街景圖像分割

Many inventories of street trees are recorded manually by an expert walking through cities and recording each tree with different attributes. This process is obviously expensive and doesn’t scale very well.

專家們在城市中穿行并記錄具有不同屬性的每棵樹，從而手動記錄了許多行道樹的清單。 這個過程顯然很昂貴，而且擴(kuò)展性很差。

To address this problem, Wegner et al. (2016) proposed a way of automating the creation of urban tree inventories by using a deep learning model that detects and assigns geo-coordinates to trees from Street View images. However, in order to perform long-term tree health studies, with time-series starting before street-level imagery was available, we still rely on these traditional inventories.

為了解決這個問題，Wegner等人。 (2016)提出了一種通過使用深度學(xué)習(xí)模型自動創(chuàng)建城市樹木清單的方法，該模型可以根據(jù)街景圖像檢測并為樹木分配地理坐標(biāo)。 但是，為了進(jìn)行長期的樹木健康研究，我們需要從獲得街道級圖像之前??的時間序列開始， 我們?nèi)匀灰蕾囘@些傳統(tǒng)清單 。

Most of the existing old databases have one crucial deficiency: the tree locations are recorded as street addresses instead of geographic coordinates.

大多數(shù)現(xiàn)有的舊數(shù)據(jù)庫都有一個關(guān)鍵缺陷： 將樹的位??置記錄為街道地址，而不是地理坐標(biāo)。

This makes them much less accurate and harder to use in practice. The goal of our project is to address this challenge: we provide an automated way to assign geo-coordinates to old tree inventories using Street View images.

這使它們的準(zhǔn)確性大大降低，在實踐中更難使用。 我們項目的目標(biāo)是解決這一挑戰(zhàn)： 我們提供了一種自動方式，可以使用街景視圖圖像將地理坐標(biāo)分配給老樹清單。

The hope is that our approach on retrofitting existing street tree inventories with geographic coordinates will enable large-scale longitudinal studies, where data of the same population of trees can be analysed over decades.

希望我們采用地理坐標(biāo)翻新現(xiàn)有街道樹木清單的方法將能夠進(jìn)行大規(guī)模的縱向研究，在該研究中，可以對數(shù)十年內(nèi)相同樹木的數(shù)據(jù)進(jìn)行分析。

為什么街上的樹木很重要？ (Why are street trees important?)

Although street trees make up a small proportion of the urban forest, they provide considerable ecosystem services to the urban landscape. These benefits include for example improvement in air quality, a reduction of the heat island effect and increased carbon capture and storage. Additionally, they can contribute to rising property values and an improvement in individual and community wellbeing.

盡管街頭樹木僅占城市森林的一小部分，但它們?yōu)槌鞘芯坝^提供了可觀的生態(tài)系統(tǒng)服務(wù)。 這些好處包括例如改善空氣質(zhì)量 ， 減少熱島效應(yīng)以及增加碳捕獲和儲存 。 此外，它們還可以促進(jìn)財產(chǎn)價值的增長以及個人和社區(qū)福祉的改善。

如何解決這樣的問題？ (How to approach such a problem?)

Our idea: We want to use the existing tool developed by Wegner et al. (2016) to automatically find trees on Street View images and then match them to the trees in the old databases based on their addresses.

我們的想法：我們想使用Wegner等人開發(fā)的現(xiàn)有工具。 (2016)自動在街景圖像上查找樹木，然后根據(jù)其地址將其與舊數(shù)據(jù)庫中的樹木匹配。

General workflow of our proposed method.我們提出的方法的一般工作流程。

Our general workflow is the following:

我們的一般工作流程如下：

The first step is to find some approximate coordinates for each tree by using a geocoding API. This means each address is converted from a textual address to a lat/long coordinate pair. We used the Geocoding API service provided by Google, which returns the geo-coordinates centered on the building of a particular parcel. This gives us a rough idea of where those trees are located.

第一步是使用地理編碼API 為每棵樹找到一些近似坐標(biāo) 。 這意味著每個地址都將從文本地址轉(zhuǎn)換為經(jīng)緯度坐標(biāo)對。 我們使用了Google提供的Geocoding API服務(wù)，該服務(wù)返回以特定宗地建筑物為中心的地理坐標(biāo)。 這使我們對這些樹的位置有了一個大概的了解。

Next, we download the Street View images in the vicinity of those coordinates.

接下來，我們下載這些坐標(biāo)附近的街景圖像 。

And detect the trees in these images using our trained deep neural network.

并使用我們訓(xùn)練有素的深度神經(jīng)網(wǎng)絡(luò)檢測這些圖像中的樹木 。

To get their coordinates, the detections are projected into geographical space.

為了獲得其坐標(biāo) ，將檢測結(jié)果投影到地理空間中。

As a final step, we need to match the trees found in the images with the existing ones corresponding to the addresses, using a global optimization approach. This assignment enhances the old databases with the coordinate information from the detected trees.

最后，我們需要使用全局優(yōu)化方法將圖像中找到的樹與對應(yīng)于地址的現(xiàn)有樹進(jìn)行匹配 。 這項任務(wù)利用來自檢測到的樹的坐標(biāo)信息增強(qiáng)了舊數(shù)據(jù)庫。

在街景圖像中查找現(xiàn)有樹木 (Finding existing trees in Street View images)

This part of the workflow follows the ideas proposed by Wegner et al. (2016) and Branson et al. (2018), with a simplified algorithm in order to make it easier to scale to huge datasets.

工作流的這一部分遵循Wegner等人提出的想法。 (2016)和Branson等人。 (2018)，采用簡化算法，以便更輕松地擴(kuò)展到龐大的數(shù)據(jù)集。

Trees in Street View panorama images are detected using Faster R-CNN (Ren et al., 2015), one of the most popular CNNs for object detection. The output of the tree detector is one bounding box per detected tree in each Street View panorama.

使用Faster R-CNN(Ren等人，2015)檢測街景全景圖像中的樹木，Raster C-NN是用于對象檢測的最受歡迎的CNN之一。 樹木檢測器的輸出是每個“街景”全景圖中每個檢測到的樹木的一個邊界框。

Automatically detected trees on a Street View panorama,在街景全景圖上自動檢測到的樹木， Imagery ?2019 Google.圖像?2019 Google。

Like all supervised learning algorithms, the network has to be trained with sufficient ground truth data. We have manually labeled 6783 individual tree instances in 718 Street View panoramas to get accurate and reliable results.

像所有監(jiān)督學(xué)習(xí)算法一樣，必須用足夠的地面真實數(shù)據(jù)來訓(xùn)練網(wǎng)絡(luò)。 我們在718 Street View全景圖中手動標(biāo)記了6783個單獨(dú)的樹實例，以獲取準(zhǔn)確可靠的結(jié)果。

After downloading all the images near an address, they are fed into the tree detector which outputs several bounding boxes per image.

在將所有圖像下載到某個地址附近后，它們被饋送到樹檢測器，該檢測器為每個圖像輸出幾個邊界框。

The metadata of the Street View images includes the camera position and heading, which lets us calculate the geographic coordinates of the detected trees by projecting the bottom center point of the bounding boxes.

街景圖像的元數(shù)據(jù)包括攝像頭的位置和航向 ，這使我們可以通過投影邊界框的底部中心點(diǎn)來計算檢測到的樹木的地理坐標(biāo) 。

使用全局優(yōu)化匹配檢測到的樹和現(xiàn)有地址 (Matching detected trees and existing addresses using global optimization)

Now we have got a database of geocoded trees. The last step of the puzzle is to match them with the geocoded addresses from the existing inventories. We cast this matching task as an optimization problem where the total distance between all pairs of trees is minimized across an entire municipality. We want to solve the following problem:

現(xiàn)在我們有了一個地理編碼樹數(shù)據(jù)庫。 難題的最后一步是將它們與現(xiàn)有庫存中的地理編碼地址進(jìn)行匹配。 我們將此匹配任務(wù)視為優(yōu)化問題，其中整個城市中所有樹木對之間的總距離最小。 我們要解決以下問題：

• x_ij is our binary decision variable, indicating if an address i is matched to a tree j (1 if i is matched to j, 0 otherwise),

  x _ ij是我們的二進(jìn)制決策變量，指示地址i是否與樹j匹配(如果i與j匹配則為1，否則為0)，

• dist_ij is simply the geographical distance between address i and tree j,

  dist _ ij只是地址i與樹j之間的地理距離，

• M is a distance threshold to avoid matching trees to addresses that are too far away (50 meters in our experiments),

  M是距離閾值，可避免將樹與太遠(yuǎn)的地址(在我們的實驗中為50米)匹配，

• We also introduce two constraints to ensure that only one street address is assigned per tree and that the number of assigned detected trees per street address does not exceed K, the total number of trees at that address given in the database.

  我們還引入了兩個約束條件，以確保每棵樹僅分配一個街道地址，并且確保每個街道地址分配的檢測到的樹木數(shù)量不超過K ，即數(shù)據(jù)庫中給定該地址的樹木總數(shù)。

By minimizing the sum over the misalignment between the geocoded trees and the geocoded addresses, we seek a globally optimal solution per city. The mathematical problem is solved using Linear Programming.

通過最小化地理編碼樹和地理編碼地址之間未對齊的總和，我們尋求每個城市的全球最佳解決方案。 使用線性編程可以解決數(shù)學(xué)問題。

加利福尼亞城市 約有 1棵Mio樹的結(jié)果 (Results for around ~1 Mio trees in Californian cities)

We ran experiments with tree inventories from five cities in California, which come with already existing coordinates such that we can validate our results. In total, our approach correctly assigned geographic coordinates to 22’363 individual street trees out of 57’938 input trees, which represents 38%. Results for the full processing pipeline are shown in the following figure:

我們對來自加利福尼亞五個城市的樹木清單進(jìn)行了實驗，這些清單具有已經(jīng)存在的坐標(biāo)，因此我們可以驗證我們的結(jié)果。 總體而言，我們的方法將正確的地理坐標(biāo)分配給了57,938棵輸入樹中的22,363棵單獨(dú)的街道樹，占38％。 下圖顯示了整個處理流程的結(jié)果：

Results of the full processing pipeline overlaid on aerial images整個處理流程的結(jié)果覆蓋在航空影像上 , Imagery ?2019 Google.，影像?2019 Google。

The green symbols denote the correctly assigned trees, the rest is categorized into the different error reasons. Those include erroneous geocoding of the addresses, missing Google Street View images, and incorrect assignments.

綠色符號表示正確分配的樹，其余的分為不同的錯誤原因。 這些包括地址的錯誤地理編碼，丟失的Google Street View圖片以及錯誤的分配。

Particularly difficult cases are street addresses of large buildings with a high number of trees such as these two examples:

尤其困難的情況是具有大量樹木的大型建筑物的街道地址，例如以下兩個示例：

Results for very large buildings overlaid on aerial images覆蓋在航空影像上的超大型建筑物的搜索結(jié)果 , Imagery ?2019 Google.，影像?2019 Google。

Our system comes at virtually no cost if we put aside costs for running computers and downloading Google images. Given appropriate hardware and a fast internet connection for downloading images, the method scales to arbitrarily large data sets.

如果我們撇開運(yùn)行計算機(jī)和下載Google圖片的費(fèi)用，我們的系統(tǒng)幾乎是免費(fèi)的。 給定適當(dāng)?shù)挠布陀糜谙螺d圖像的快速Internet連接，該方法可以擴(kuò)展到任意大的數(shù)據(jù)集。

To demonstrate this, we ran our processing pipeline on 48 municipalities containing altogether 1’100’952 input trees. Our method assigned geographic coordinates to 56% of the individual street trees.

為了證明這一點(diǎn)， 我們在48個包含1,100'952輸入樹的城市上運(yùn)行了處理管道。 我們的方法將地理坐標(biāo)分配給了56％的街道樹。

結(jié)論 (Conclusion)

We have presented a novel approach to assign geographic coordinates to street-trees given street addresses and Street View panorama images. We were able to assign correct geographic coordinates to 38% of the street trees, which is a good starting point for long-term studies on the ecosystem services value of street trees at large scale.

我們提供了一種新穎的方法，可以在給定街道地址和街景視圖全景圖像的情況下，將地理坐標(biāo)分配給街道樹。 我們能夠為38％的街道樹分配正確的地理坐標(biāo)，這是長期研究街道樹的生態(tài)系統(tǒng)服務(wù)價值的良好起點(diǎn)。

There are different reasons, which account for this percentage being relatively low. One could be the varying geocoding quality provided by the API. Also, the detection of trees can fail due to missing panoramas, occlusions or an error by the detector. The incorrect assignment is another major reason for mismatches, which is generally difficult to fix. It is due to mixing up trees close to parcel boundaries or also when several trees are on the same property, which is very often the case.

有不同的原因，導(dǎo)致該百分比相對較低。 一種可能是API提供的不同的地理編碼質(zhì)量。 此外，由于缺少全景圖，遮擋或檢測器錯誤，樹木的檢測可能會失敗。 錯誤的分配是不匹配的另一個主要原因，通常不容易解決。 這是由于混合了靠近地塊邊界的樹木，或者是由于幾棵樹木在同一屬性上，通常是這種情況。

We hope this work encourages further research to generate longer time series (over several decades) of street-tree inventories with the ultimate goal of better understanding street tree population dynamics and corresponding changes in ecosystem services at a very large scale.

我們希望這項工作能夠鼓勵進(jìn)行進(jìn)一步的研究，以產(chǎn)生更長的時間序列(超過幾十年)的行道樹清單，其最終目標(biāo)是更好地了解行道樹種群動態(tài)以及大規(guī)模的生態(tài)系統(tǒng)服務(wù)的相應(yīng)變化。

For the interested reader, we refer to our full research paper: here

對于感興趣的讀者，請參閱我們的完整研究論文： 此處

Laumer, D., Lang, N., van Doorn, N., Aodha, O.M., Perona, P., Wegner, J.D. “Geocoding of trees from street addresses and street-level images” ISPRS Journal of Photogrammetry and Remote Sensing 162(2020)

洛默(D.)，朗(N.)，朗(N.)，范多恩(N.)，密西西比州奧達(dá)(Aodha)，佩羅納(P.)，韋格納(Wegner)，京東(JD)“對街道地址和街道圖像中的樹木進(jìn)行地理編碼”， ISPRS攝影測量與遙感雜志 162 ( 2020)

Branson, S., Wegner, J., Hall, D., Lang, N., Schindler, K., Perona, P., 2018. From google maps to a fine-grained catalog of street trees. ISPRS J. Photogramm. Remote Sens. 135, 13–30.

Branson，S.，Wegner，J.，Hall，D.，Lang，N.，Schindler，K.，Perona，P.，2018年。從谷歌地圖到街樹的細(xì)粒度目錄。 ISPRS J. Photogramm。 遙感135，13-30。

Ren, S., He, K., Girshick, R., Sun, J., 2015. Faster R-CNN: Towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems (NIPS).

Ren，S.，He，K.，Girshick，R.，Sun，J.，2015。更快的R-CNN：通過區(qū)域提議網(wǎng)絡(luò)實現(xiàn)實時對象檢測。 于：神經(jīng)信息處理系統(tǒng)(NIPS)的進(jìn)展。

Wegner, J.D., Branson, S., Hall, D., Schindler, K., Perona, P., 2016. Cataloging public objects using aerial and street-level images — urban trees. In: IEEE Conference on Computer Vision and Pattern Recognition.

韋格納，法學(xué)博士，布蘭森，S。，霍爾，D。，辛德勒，K。，佩羅納，體育，2016年。使用空中和街道圖像(城市樹木)對公共對象進(jìn)行分類。 在：IEEE計算機(jī)視覺和模式識別會議上。

翻譯自: https://medium.com/ecovisioneth/towards-large-scale-tree-mortality-studies-in-cities-with-deep-learning-street-view-images-c8790c10cc7c

街景圖像分割

總結(jié)

以上是生活随笔為你收集整理的街景图像分割_借助深度学习和街景图像进行城市的大规模树木死亡率研究的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯，歡迎將生活随笔推薦給好友。