Vegetation Removal on 3D Point Cloud Reconstruction of Cut-Slopes Using U-Net

Abstract

The 3D point cloud reconstruction from photos taken by an unmanned aerial vehicle (UAV) is a promising tool for monitoring and managing risks of cut-slopes. However, surface changes on cut-slopes are likely to be hidden by seasonal vegetation variations on the cut-slopes. This paper proposes a vegetation removal method for 3D reconstructed point clouds using (1) a 2D image segmentation deep learning model and (2) projection matrices available from photogrammetry. For a given point cloud, each 3D point of it is reprojected into the image coordinates by the projection matrices to determine if it belongs to vegetation or not using the 2D image segmentation model. The 3D points belonging to vegetation in the 2D images are deleted from the point cloud. The effort to build a 2D image segmentation model was significantly reduced by using U-Net with the dataset prepared by the colour index method complemented by manual trimming. The proposed method was applied to a cut-slope in Doam Dam in South Korea, and showed that vegetation from the two point clouds of the cut-slope at winter and summer was removed successfully. The M3C2 distance between the two vegetation-removed point clouds showed a feasibility of the proposed method as a tool to reveal actual change of cut-slopes without the effect of vegetation.

1. Introduction

With the advent of unmanned aerial vehicle (UAV), the 3D reconstruction by photogrammetry has become an useful tool for numerous civil engineering applications including structural modelling [1,2], structural assessment [3], historical building protecting [4], road construction [5], road maintenance [6], construction progress measurement [7], monitoring and controlling construction site activities [8], land usage investigation [9,10], etc.

The 3D reconstruction is also useful for the cut-slope application as a drone can survey a wide range of steep cut-slopes efficiently and effectively without a risk of climbing for field investigators. Photogrammetry provides a 3D point cloud as a comprehensive and quantitative documentation of the slope at a given time, without any subjective assessment by field investigators. The 3D point cloud can provide further geometric and geological information for the slope stability analysis [11].

Ideally, engineers want to monitor long-term changes of cut-slopes. In theory, this can be done by inspecting the difference between two 3D point clouds temporarily apart, but it is often the case the difference is submerged by seasonal changes of vegetation on the cut-slopes. In order to make such technology realisable, it is important to remove vegetation from the 3D point clouds.

There are three different approaches found in the literature for vegetation removal depending on how each point in the point cloud is classified: (1) the colour index approach [12,13]; (2) the planar estimation approach [14,15,16]; or (3) the machine learning based 3D point cloud segmentation approach [17,18].

The colour index approach is simple to use and works reasonably for a specific scene with tuned parameters, but has a limited performance when the colour of a non-vegetation is close to the colour of vegetation. The planar estimation approach showed a limited performance on slopes with complex surface geometry [14]. Considering the planar estimation approach’s working principle that vegetation is separated from ground surface by distance, it is expected to work reasonably well for tall vegetation, but not for short vegetation. The comparative study [19] showed even the best performance method was not able to remove vegetation completely. The machine learning based 3D point cloud segmentation approach is relatively new, but promising with an expectation of high-performance when trained with a large dataset [17,18]. Pix4Dmapper provides a machine learning based point cloud classifier for a dense point cloud reconstructed by itself [20]. The active research for 3D segmentation in computer science, medical imaging, etc. [21,22,23,24] is likely to benefit the 3D vegetation removal approach. However, the requirement for a large 3D dataset is a practical problem in the development [17].

For a high-performance vegetation removal with less effort, this study proposes a new approach combining the deep learning-based 2D image segmentation and photogrammetry. The colour index method is employed to reduce the effort in preparing a 2D segmented image dataset.

The paper is organised as following: introduction to the cut-slope studied in this work (Section 2), methodology of the proposed 3D vegetation removal approach (Section 3), performance of the proposed approach on the cut-slope (Section 4), and the conclusion.

2. Cut-Slope of Doam Dam

The cut-slope used in this study was a rocky cut-slope located in Doam Dam at the north-east of South Korea, as shown in Figure 1a. Images taken with UAV at different times (winter and summer) were used to reconstruct 3D point clouds of the cut slope. A DJI Phantom 3 Professional UAV was used to take $4000\times 3000$ pixels images as shown in Figure 1c. The UAV was manually driven and the camera position and orientation of each image is shown as a pyramid in Figure 2. The images were processed by an open-source photogrammetry software COLMAP [25] to reconstruct a 3D point cloud of the cut-slope, as shown in Figure 1b. In the reconstruction, the “automatic reconstruction” command of COLMAP was used with the default setting.

The photogrammetry procedure consists of structure from motion (SfM) and multi-view stereo (MVS) as illustrated in Figure 3. Structure from motion estimates the camera position and orientation of each image (i.e., the structure) by triangulation of the feature points appearing on multiple images. Structure from motion also determines the 3D coordinates of the feature points, known as a sparse point cloud. As a sparse point cloud is usually not dense enough to represent the surface of a slope, multi-view stereo procedure is performed to generate a dense point cloud by triangulation on each pixel of two adjacent images to determine the corresponding 3D coordinates in the world coordinates.

As shown in Figure 1c, there was no clear distinction between the rock and vegetation, which made vegetation removal challenging on both the 2D images and 3D point cloud.

3. Methodology

3.1. Proposed Vegetation Removal Method by 2D Segmentation and Photogrammetry

There are two possible vegetation removal approaches on the 3D point cloud reconstruction using the 2D image segmentation as shown in Figure 4. Approach number (#) 1 is to perform the 2D image segmentation to generate mask files to be fed to photogrammetry procedure. A mask file of an input image is a file with the same number of pixels of the input image, but filled with 0 or 1 instructing the photogrammetry procedure whether to ignore each pixel or not. COLMAP does have such functionality for structure from motion, but not for multi-view stereo. Thus, this approach was not implementable using COLMAP.

This paper proposes an alternative, but a simpler approach as shown as approach #2 in Figure 4. The photogrammetry procedure is done without any vegetation consideration, to produce a point cloud with vegetation (Figure 5a). Each point in the point cloud is re-projected into the image coordinates (Figure 5c) using the transformation matrix identified by structure from motion. the corresponding point in the image coordinates can be determined if it belongs to vegetation or not, using the 2D vegetation segmentation by U-net on the image (Figure 5b). If it belongs to vegetation, it is deleted from the point cloud. Otherwise, it is kept. This process is repeated for every point in the point cloud to create a point cloud without vegetation (Figure 5d).

For a robust 2D segmentation, this study used a deep learning based 2D vegetation segmentation using U-Net to overcome the limitations of the colour index method, as described in Section 3.2. However, the use of the colour index method is still useful especially for green vegetation, and it is proposed to use the colour index method to minimise efforts for preparing a segmented image dataset as shown in Figure 6. Original images are segmented by the colour index method first, followed by segmentation quality check by human. Images without any apparent incorrect segmentation are fed to the segmented image dataset. Images with incorrect segmentation are manually trimmed and corrected by humans, and fed to the segmented image dataset.

3.2. Colour Index

The colour index is an efficient way for vegetation segmentation in 2D images [19,26]. The widely-used colour index is the Excessive Greenness Index ($ExG$) [27,28,29]. $ExG$ as shown in Equation (1) is a continuous index where a higher value indicates a green surface and a lower value indicates a bare ground.

$$ExG=2G-R-B$$

(1)

where R, G, B values are the normalised red, green, and blue values of each pixel in an image, $0\le R,G,B\le 1$. $ExG$ performs reasonably for vegetation segmentation from soil. However, the original $ExG$ or any widely used colour index no longer showed a good performance in this study.

A modified $ExG$ ($mExG$) was developed in this study.

$$mExG=G-1.2B$$

(2)

where the coefficients 1.2 was determined by trial and error for the best performance. Figure 7 shows a performance comparison of the original $ExG$ and $mExG$, where any vegetation pixel was displayed as it was and any non-vegetation pixel was displayed as white. The yellow traffic line, the light blue guardrail and some of the slate grey rock were classified as vegetation by $ExG$ as they also contained green. However, $mExG$ segmented the traffic line, guardrail, and rock successfully as non-vegetation.

Figure 8 shows an example that a single colour index was not sufficient for all kinds of vegetation. Two different colour indices, $mExG$ and $mExR$ ($=R-1.2B$), were used. The factor 1.2 in $mExR$ was also identified by trial and error. Figure 8b showed the green mask generated with $mExG$, Figure 8c showed the red mask generated with $mExR$, and any pixel corresponding to either $mExG$ or $mExR$ was added up to get the entire vegetation, fresh or dried, as shown in Figure 8d.

The colour index method may fail when colour of vegetation is close to the background as shown in Figure 9 and Figure 10. In the red rectangle of Figure 9, both fresh and dry leaves were segmented with the colour index method successfully (vegetation as white, and non-vegetation as black), as the colours of them were different with the rock. However, in the blue rectangle, the colour of dry bush branch (grey) was close to the rock behind, and they were segmented wrongly as rock. In Figure 10, the dry grass in the red rectangle was segmented correctly, but the colour of weathered rock surface in the blue rectangle was close to yellow leaves, so that it was segmented wrongly as vegetation.

3.3. Manual Trimming

To address the limitations of the colour index method, manual trimming by human was required. However, even with manual trimming, there were cases where segmentation was not straightforward as shown in the blue rectangle in Figure 9. Figure 11 showed how the mainstream image segmentation dataset (Cityscapes) dealt with this problem. Instead of segmenting the branch from the background, pixel-by-pixel, it gave the envelope of the branch. A similar processing was used in this study and, thus, a small part of the background was also segmented as vegetation in areas crowded with vegetation. For this reason, the area of vegetation was slightly over estimated.

Figure 12 shows examples of manual trimming with the strategy. In Figure 12a, the grey dry bush was manually segmented as vegetation. In Figure 12b, weathered rock was manually segmented as non-vegetation. Some scatters were erased since it was hard to tell from the image itself whether they were corresponding to the dry grass or not.

3.4. U-Nets

The deep learning methods used for the land usage investigation showed a good performance [9,10]. It proved that many deep learning models were suitable for vegetation recognition. In this study, the classification problem is not complex (vegetation or non-vegetation) and the aim is to find a deep learning model requiring a smaller dataset whilst keeping details of images.

U-Net (Figure 13) was firstly used in medical image segmentation, the name comes from the encoder–decoder shape. It was invented for a small dataset and was well trained with only 30 images [30]. The skip connections between the encoder and decoder keep details of the input images. In U-Net, the encoder part is similar with some image classification models and, thus, U-Net can be built based on an image classification model. The pre-trained weight of an image classification model can be used for transfer-learning reducing time-consumption for model training. Two image classification models, VGG16 and MobileNetV2, were used to build U-Nets and the results were compared. VGG16 uses a smaller convolution kernel, which helps extract complex features [31], whilst MobileNetV2 was designed for mobile applications with a smaller model size, resulting in a reduced amount of computation [32].

For training and testing, a dataset of 70 images was used: 35 images taken from summer and the other 35 images from winter. The images from winter were selected to cover the situations in Figure 9 and Figure 10 and were all manually trimmed. The images from summer were randomly selected and segmented successfully by $mExG$ only. To make full use of the 70 segmented images, each $4000\times 3000$ pixels image was cut into 25 images of $800\times 600$ pixels. In addition, horizontal-flip data augmentation was used so that the 70 images were augmented as 3500 ($=70\times 25\times 2$) images. Among them, 70% (2450 images):30% (1050 images) were used as training and test datasets respectively. The augmented images were resized to 512 × 512 pixels to be fed to the U-net. Performance metrics used in training/testing were (1) the binary accuracy on how often predictions equal to labels, and (2) the binary cross-entropy loss function between true and predicted labels. The batch size was 16, and the initial learning rate was ${10}^{-4}$ with the Adam optimiser. Figure 14 shows the training history with the two metrics.

Figure 14 shows that the accuracy on the training dataset was close to the accuracy on the test dataset. This was the same for the binary cross-entropy loss index as well. Both the accuracy and binary cross-entropy loss plots showed converging trends. U-Nets based on VGG16 and MobileNet V2 reached approximately 95% and 90% binary accuracy after 16 training epochs, respectively. The binary cross-entropy loss of VGG16 U-Net is much lower than MobileNetV2 U-Net. U-Net based on MobilenetV2 took approximately 1/3 training time of that based on VGG16, and the model size was less than half of it. Meanwhile, the binary accuracy difference between the two U-Nets was about 5%.

Figure 15 shows the vegetation segmentation results of $mExG$, and the U-Nets with VGG16 or MobileNetV2, both with 50% confidence level and above. The first row in Figure 15 shows the original images taken in different locations and seasons. The second row shows the mask obtained with the colour index method and the third row were the manually trimmed results. In the third row, the first two images were the high-quality results obtained by the colour index method only, whilst the last two images were manually trimmed based on the masks generated with the colour index method. The last two rows were the predicted masks with the two U-Nets. The predictions with the two U-Nets were close to each other and they both agreed well with the trimmed masks. VGG16 U-Net kept more details than MobileNetV2 U-Net.

4. Result of the Proposed Method on Doam Dam

This section presented (1) the vegetation removed results by the proposed method in comparison with two existing vegetation removed methods on Doam Dam’s point clouds, (2) the negative effect of vegetation on long-term cut-slope monitoring, and (3) a possibility of long-term cut-slope monitoring realised by the proposed vegetation removed method. The two existing methods used were (1) Cloth Simulation Filter [14], available in CloudCompare [33], and (2) dense point cloud classification [20], available in Pix4Dmapper [34].

Doam Dam’s point clouds used in this study were reconstructed from two sets of photos taken at two different times, one at winter (January 2018) and the other at summer (September 2017). Each image set consists of 1305 images and was processed by COLMAP. The proposed method was applied to the two point clouds and the results were shown in Figure 16, Figure 17 and Figure 18.

Figure 16 shows (a) the original 3D point cloud in winter, and (b) the point cloud classification result by Pix4Dmapper, (c) vegetation removed point cloud by the Cloth Simulation Filter, and (d) the vegetation removed point cloud by the proposed method. The point cloud classification by Pix4Dmapper showed a poor performance: many parts on the slope were classified as “building”, and the large dry vegetation patch on the left of the cut-slope was classified as a mixture of Ground and high vegetation, rather than vegetation. This may indicate that the machine learning model of Pix4D was not trained extensively for cut-slopes. The vegetation removed point cloud by the Cloth Simulation Filter (the options used were the cloth resolution of 0.1 m and the classification threshold of 0.1 m) showed the trees on the side of the road at the bottom were removed significantly. However, a large portion of the point cloud was still brown due to unremoved dry grasses and bushes on the slope. Considering the working principle of the Cloth Simulation Filter, this was an expected result. The vegetation removed result by the proposed method showed that the dry vegetation was successfully removed from the point cloud.

Figure 17 shows (a) the original 3D point cloud in summer, and (b) the point cloud classification result by Pix4Dmapper, (c) vegetation removed point cloud by the Cloth Simulation Filter, and (d) the vegetation removed point cloud by the proposed method. The point cloud classification by Pix4Dmapper showed a poor performance again: most parts on the slope were classified as “high vegetation”, and “ground” point clouds were relatively smaller than “high vegetation”. However, it was interesting to observe that “road surface” and two utility poles (“human made object”) were properly classified. Again, this may indicate that the machine learning model of Pix4D was not trained extensively for vegetation and non-vegetation on cut-slopes. The vegetation removed point cloud by the Cloth Simulation Filter showed a similar result with the winter one. It removed the trees on the side of the road at the bottom well. However, a large portion of the point cloud was still green due to unremoved fresh grasses and bushes on the slope. The vegetation removed result by the proposed method showed that the fresh vegetation was successfully removed from the point cloud.

Figure 18 shows zoom-in views of Figure 16 and Figure 17 for the same region on the cut-slope. Excellent vegetation removal performance was found for both fresh (green) and dry (brown) vegetation in comparison with the two existing methods.

For a purpose of long-term cut-slope monitoring, a geometric distance metric must be used to calculate distance between two point clouds. A promising metric is multi-scale model to model cloud comparison (M3C2) distance [35], which physical meaning is a local distance between two point clouds along the surface normal direction. The M3C2 distances between winter and summer point clouds were calculated and shown in Figure 19 and Figure 20.

Figure 19a shows orange regions with large differences. By comparing the orange regions with summer point cloud (Figure 17), it was found that the orange regions corresponded to the vegetation regions. Figure 20a confirmed this in comparison with Figure 18a,b. These demonstrated the negative effect of vegetation for long-term cut-slope monitoring using point clouds.

On the other hand, the M3C2 distance between the vegetation removed point clouds by the proposed method was shown in Figure 19a and Figure 20b. It can be found the orange regions with vegetation were removed successfully from both figures. However, the top and bottom parts of Figure 19b were shown to be green, which may indicate an alignment issue between winter and summer point clouds. This issue may be a topic of further study to improve accuracy of the M3C2 distance calculation.

5. Conclusions and Discussion

This study proposed the vegetation removal method on a 3D point cloud using the photogrammetry and the 2D segmentation by U-Net. The study demonstrated that the proposed method effectively removed seasonal change due to vegetation on the cut-slope in Doam Dam in South Korea. This showed a feasibility of the 3D point clouds to monitor long-term changes in cut-slopes. The findings in the study are summarised as following.

• The colour index was found to be successful for vegetation segmentation when the contrast between vegetation and the background was high, and the colour of vegetation was relatively simple. However, it was not successful otherwise.
• Deep learning-based 2D image segmentation using U-Net was robust to deal with such complex situations, showing the similar performance with the manually trimmed image segmentation.
• The effort to build the 2D segmentation deep learning model was significantly reduced by using U-Net with the dataset prepared by both the colour index method and manual trimming.
• Two base models, VGG16 and MobileNetV2, were used to build the U-Nets. The VGG16 U-Net showed the better performance, but the MobileNetV2 U-Net took much less time for training with a smaller model size. The segmentation accuracy of VGG16 and MobileNetV2 U-Net were approximately 95% and 90%, respectively.
• The proposed vegetation removal method on the 3D point cloud reconstruction was found to remove vegetation successfully on the two 3D point clouds constructed with images taken at two different times.
• The M3C2 distance between two vegetation-removed point clouds of the cut-slope successfully removed the negative effect due to seasonal vegetation on the cut-slope.