Augmentation-Based Methodology for Enhancement of Trees Map Detalization on a Large Scale

Abstract

Remote sensing tasks play a very important role in the domain of sensing and measuring, and can be very specific. Advances in computer vision techniques allow for the extraction of various information from remote sensing satellite imagery. This information is crucial in making quantitative and qualitative assessments for monitoring of forest clearing in protected areas for power lines, as well as for environmental analysis, in particular for making assessments of carbon footprint, which is a highly relevant task. Solving these problems requires precise segmentation of the forest mask. Although forest mask extraction from satellite data has been considered previously, no open-access applications are able to provide the high-detailed forest mask. Detailed forest masks are usually obtained using unmanned aerial vehicles (UAV) that set particular limitations such as cost and inapplicability for vast territories. In this study, we propose a novel neural network-based approach for high-detailed forest mask creation. We implement an object-based augmentation technique for a minimum amount of labeled high-detailed data. Using this augmented data we fine-tune the models, which are trained on a large forest dataset with less precise labeled masks. The provided algorithm is tested for multiple territories in Russia. The $F1$-$score$, for small details (such as individual trees) was improved to $0.929$ compared to the baseline score of $0.856$. The developed model is available in an SAAS platform. The developed model allows a detailed and precise forest mask to be easily created, which then be used for solving various applied problems.

1. Introduction

Artificial intelligence has already been successfully applied to solve various practical problems, in particular tasks related to the automatization of sensing processes and increasing their precision [1,2]. With the appearance of new technologies that allow high-quality imaging data to be obtained, the amount of collected imaging data has increased; this leads to demands for the development of effective tools for image data processing. One of the industrial and scientific domains that requires such tools is remote sensing [3,4]. Remote sensing data is widely used in various environmental studies that include measuring of the carbon footprint, for which it is crucial to obtain precise forest masks, boundaries of agriculture fields, type of crops, etc. Computer vision algorithms, in particular convolutional neural networks (CNN), can automatically process this data. The vital information such as environmental and vegetation state [5], forest inventory characteristics [6], and agriculture crop yield [7] can be effectively extracted by CNNs. Commonly, the first step in environmental studies is obtaining forest masks [8,9]. The existing satellite-based approaches for obtaining forest masks work well for vast territories where it is not important to detect and quantify small details. The usual spatial resolution for open-access landcover maps is more than 10 m [10]. Using such datasets, it is possible to create forest masks with sufficient accuracy on a large scale and make an adequate assessments of forest reserves. However, current approaches usually are not intended to detect small details such as individual trees, groups of trees, or meadows. Moreover, commonly used metrics for accuracy assessment of automatically generated forest masks do not take into account these small details in an adequate manner, for the following reason. Separate trees or groups of trees represent a tiny proportion of the target forest class, which is why the impact of detection accuracy for small objects on the overall metrics is low. Thus, a high prediction score for the entire territory does not necessarily mean high performance on small details.

For particular tasks, it is essential to obtain a detailed forest mask that approximates areas of forest. One such task is the monitoring of protected zones or natural reserves, where the territory of interest is too narrow and each small group of trees has to be taken into account [11,12]. In [13], the authors showed the importance of trees outside forests for ecosystem functions and ways to improve assessments based on aerial stereo images. In such cases, unmanned aerial vehicles (UAV) or aerial photography are usually used to obtain higher detail [14]. Obtaining the detailed forest mask on large scales is quite a challenging task, and it is common to merge data with different resolutions and from different sources [15]. The main limitation of the UAV-based approach is its cost and the difficulty of its implementation for vast territories on a country-wide scale [16]. Another datasource is satellite imagery with high spatial resolution, such as WorldView, Spot, RapidEye, and Planet. These data sources are often used to detect the crown of individual trees, which in turn can be considered in a detailed forest mask. WorldView images were used for forest cover estimation in [17,18], while RapidEye data were considered in [19,20]. However, these data are more expensive than low or medium spatial resolution satellite images.

The example of low resolution data for making large scale estimations of forest masks is described in [21], where the authors used image data collected by the MODIS mission, which has a resolution of 250 m [22] (Moderate Resolution Imaging Spectroradiometer). Using a medium resolution (10–30 m) is most frequent because of the availability of open-access data and comprehensive frameworks for data processing. For example, in [23] the authors show an approach for forest mask creation over European forests using optical Sentinel-2 data. In [24], the authors monitor forest degradation in South Asian forest ecosystems by implementing Sentinel-2 and Landsat imagery. Deforestation monitoring tasks using data from Sentinel-1, PALSAR-2 and Landsat data are discussed in [25]. The data fusion and preprocessing techniques for aerial and Sentinel-2 data are shown in [26], where the authors calculated the forest cover map for German territory and showed the accuracy of their proposed method by comparison with National forest inventory data. One major problem is deforestation connected to illegal logging. In [27,28], the authors propose and validate approaches for deforestation monitoring using Sentinel-2 data. A time series of images can be used for environmental monitoring and planning of sustainable management. In [29], the authors showed the potential of using, time series of Landsat and Sentinel-1A SAR images to identify and map mangrove forests. Time series of images can be used to detect forest degradation caused by natural reasons, anthropogenically-influenced climate change, damage by insects, etc., [30].

In this study, we propose a neural network-based approach for predicting the detailed forest mask using Basemap RGB images. We use a small dataset with detailed labelling of individual trees to fine-tune a CNN model that was initially trained on a large dataset with less accurate labels (masks) for individual trees or groups of trees. The novelty of our study includes the implementation of the object-based augmentation (OBA) technique for new training sample generation. This approach increases the amount of training data significantly and allows for the creation of physically meaningful data samples, which is important in remote sensing data analysis. The main contributions of this paper are:

• We propose and validate a pipeline for detailed forest mask segmentation using CNN
• We provide an open-access tool for detailed forest mask segmentation that can be used for environmental studies, which is available in an SAAS platform through the link provided [31].

The paper is organized as follows: Section 2 describes the characteristics of the datasets used in the present study and the methodology of the proposed solution and validation approach; Section 3 shows the obtained detailed tree maps and compares them with the baseline maps; Section 4 presents concluding remarks and plans for the possible future development of the developed methodology.

2. Materials and Methods

In this study, we considered two datasets. The first was large and lacking in precision markup for small details, while the second represented a smaller area with each individual tree presented in markup.

2.1. Large Dataset

For the large dataset, we collected data covering more than 500.000 hectares. The study area was located in the Republic of Tatarstan, Russia. There were about $45\%$ hectares of forest and $65\%$ hectares covered by other landcover types (lawns, fields, etc.) and manmade objects (roads and buildings). We used a cloud-free composite orthophotomap provided by mapbox [32] via tile-based map service. The imagery was derived from different satellite images obtained by the WorldView satellites series, consisting of three pansharpened spectral channels (RGB). The spatial resolution was about $0.5$ m per pixel, depending on the observation latitude. All images were taken during the summer period in 2018. The manual markup for this region was produced based on the aforementioned images. It was first presented in a vector GEOJSON format (as polygons coordinates), then converted into georeferenced rasters (binary masks) with spatial resolutions equal to the satellite data resolution. The study area was split into training, validation, and testing regions at a respective proportion of $70\%$, $15\%$, $15\%$.

2.2. Detailed Small Dataset

We used a high-quality small dataset with precision individual tree masks for an area in Dagestan, Russia. The environmental conditions differ from the large dataset territory in that sandy surfaces partially cover the area. The manual markup was performed for satellite images from the mapbox basemap service, with an acquisition date in the summer period of the 2020. The spatial resolution properties of satellite images were the same as for the large dataset. The entire area was $4.000$ hectares, of which approximately $40\%$ was forest cover. The final forest mask was presented in both raster (binary mask) and vector (polygon coordinates) format. The number of individual trees in the training dataset with an area smaller than 300 pixels was 6387. The test area included more than 2000 individual trees. Each subset was represented by the individual image and area.

Our solution for forest segmentation included two consecutive steps, which are shown in Figure 1. The first was model training on the large dataset in order to learn important feature representation. Then, the model was fine-tuned on the smaller and more detailed dataset that was preprocessed with an object-based augmentation technique.

2.3. Baseline Forest Segmentation

For the baseline forest segmentation, we used a large dataset. Training samples were cropped randomly from the entire study territory. Standard color and geometrical transformations (random rotation, brightness, contrast, saturation adjustment, etc.) were implemented for each sample. A neural network was trained to identify pixels belonging to the class “forest” by minimizing the binary cross-entropy loss function

$$\begin{array}{c}\hfill L(y,\widehat{y})=-\frac{1}{N}\sum _{i=1}^N{y}_{i}log\left(\widehat{y_i}\right)+(1-y_i)log(1-\widehat{y_i}),\end{array}$$

(1)

where N is the number of target mask pixels, y is the target mask, and $\widehat{y}$ is the model prediction. For the baseline forest segmentation we used the following implementations of the CNNs: UNet [33], FPN [34], and DeepLab [35]. The details of CNN training are discussed in Section 2.6. Model implementation was based on the repository in [36].

2.4. Object-Based Augmentation

After baseline training, we fine-tuned the model using a small dataset and the following augmentation approach.

The object-based augmentation approach has been previously proposed for the remote sensing domain in [37] for solving segmentation tasks. For the forest segmentation problem, we provided the following augmentation scheme, the algoritmic implementation of which can be found at the following link [38]. The initial detailed markup included both large areas and individual tree masks. In the first stage, we created a list of individual trees selected by the area according to the threshold. The threshold was established empirically and was equal to 300 pixels. Selected individual trees were ascribed IDs associated with coordinates and instance masks. During the augmentation step, the object’s ID was selected. The object (individual tree) was cropped according to its boundary. Then, shadows were added to make the generated sample more realistic. The footprint of an object was used to add a shadow. The contrast and saturation of shadows were varied in order to extend the variability of the training instances. Moreover, each individual tree could be augmented using classical color and geometrical transformations. For this task, the Albumentations package [39] was leveraged. The cropped and transformed individual trees were then merged with a new background. The background was randomly selected from the initial satellite image or from new images from another geographical location. The main requirement for the background crop was the absence of the target objects. The selected background patch was augmented using geometrical and color transformations. The final step of new training sample generation was background and target object merging. A number of objects was selected randomly for each patch from a predefined range. The maximum number was defined empirically and set to 30 according to the patch size and target object size. Intersection between the objects was restricted. It is possible that a neural network can fit exactly against generated data and lose essential properties of the original images. To avoid this, we used generated samples with a probability of $0.4$ and original samples with a probability of $0.6$. Both the original and generated samples were prepared during the training time and did not require extra memory to store patches. Examples of the generated and original samples are presented in Figure 2.

The OBA approach was compared with two alternative approaches, namely, classical augmentation (random rotation, brightness, contrast, saturation adjustment) as described in [37] (Simple_augm) and training without any image transformations (Baseline_no_augm).

Object-wise sampling was performed for all experiments with model fine-tuning on the small dataset, as this is a more powerful sampling technique for spatially distributed data in the remote sensing domain, especially in the case of target objects with coordinates [40]. In this approach, instead of cropping random patches from an image the target objects’ IDs were selected and then cropped according to their coordinates. This allowed us to form a training batch for a convolutional neural network with more valuable instances when target objects such as individual trees were rare in the study area and unevenly distributed. Object-wise sampling was alternated with a classical random cropping in order not to preserve only small objects in the training data. The probability of the object-wise sampling was set to $0.8$.

2.5. Different Dataset Size

We considered the following subsets of the training dataset in order to evaluate the effect of the dataset’s size on the prediction quality: the entire dataset size, 2/3, and 1/3 of the entire training dataset. The chosen subset was used to train the model, while the testing area was permanent and the same for all experiments. We analyzed three and two different dataset splits for the experiments with 1/3 and 2/3 of the entire dataset size, respectively. The final results were defined as an average for each training subset.

2.6. Experimental Setup

For the baseline model, we considered the following convolutional neural network architectures: U-Net [33], DeepLab [35], and FPN [34] with Inception [41] encoder. Each experiment was run with the same training parameters. The batch size was equal to 20, and the patch size was set to $256\ast 256$ pixels. There were 20 epochs with 200 steps. For each epoch, there were 4000 random patches (with size $256\ast 256$ pixels) obtained using object-wise sampling or classical random cropping from the training areas. After each epoch, the validation score was estimated. Early stopping was employed after the model reached the plateau with patience 5 epochs. According to the validation score, the best model was then considered in order to compute metrics in the test area. The RMSprop optimizer was used, with a learning rate of $0.001$. All experiments used Keras [42].

According to the previous stage, the best model among all considered architectures was employed for fine-tuning on the small dataset. The same training parameters (patch and batch sizes, training epochs number, etc.) were employed. As distinct from the first stage experiments, model weights were already pretrained on the large dataset. Therefore, the model was trained to solve individual tree segmentation and detailed forest mask prediction, which is a more complicated task.

2.7. Evaluation

To evaluate the performance of the proposed models we used the general $F1$-$score$, which is widely used in remote sensing tasks [5]. This allowed us to assess prediction quality for the entire test area. This metric is more focused on huge territories with forest cover.

$$\begin{array}{c}\hfill Precision=\frac{TP}{TP+FP{}^{\prime }}\\ \hfill Recall=\frac{TP}{TP+FN{}^{\prime }}\\ \hfill F1=\frac{TP}{TP+\frac{1}{2}(FP+FN){}^{\prime }}\end{array}$$

(2)

where $TP$ is True Positive (number of correctly classified pixels of the given class), $FP$ is False Positive (number of pixels classified as the given class while in fact being of another class, and $FN$ is False Negative (number of pixels of the given class missed by the method).

To assess the quality of our model, we estimated the average IoU between the predicted masks and the ground truth masks as follows:

$$\mathrm{IoU}=\frac{\mathrm{Area}\mathrm{of}\mathrm{Overlap}}{\mathrm{Area}\mathrm{of}\mathrm{Union}},$$

(3)

and $\mathrm{Area}\mathrm{of}\mathrm{Overlap}$ and $\mathrm{Area}\mathrm{of}\mathrm{Union}$ are computed between the ground truth and the predicted masks. In order to predict the forest mask, we used test images as an input for the trained neural network. As an output, a neural network predicts a binary mask, which we compared with the labelled binary mask and used to calculate different metrics. For example, prediction quality ($F1$-$score$), was calculated for each image in the test set. The overall prediction quality stated for each of the models is the average of the $F1$-$score$, for all images in the test set.

3. Results and Discussion

Figure 3 shows the result of the implementation of the first methodological step, namely, the performance on the test data of the model trained on the large dataset (U-Net). It should be noted that the overall model performance is appropriate in terms of metrics (see

Table 1. Forest segmentation results for baseline model on two datasets.

	Large Dataset			Small Dataset
	Precision	Recall	F1	Precision	Recall	F1
U-Net	0.965	0.963	0.964	0.862	0.851	0.856
FPN	0.961	0.958	0.959	0.856	0.849	0.852
DeepLab	0.963	0.962	0.962	0.856	0.854	0.855

) as well as by visual comparison; moreover, test images were taken from different parts of the world and represent complex environments. However, the model performance could be improved for better detection of stand-alone trees. The results of the model performance after fine-tuning on the small dataset (for which implementation of object-based augmentation was implemented) is shown in Figure 4. For the training procedure, we generated about 72,000 patches for the first dataset and about 28,000 patches for the second dataset. From Figure 4, it can be clearly seen that the predicted forest mask (see Figure 4d) is very similar to the ground truth (see Figure 4b), and the separate trees are much better detected compared to the baseline model (see Figure 4c). The $F1$ scores for the baseline and improved models are presented in

Table 2. Augmentation approaches comparison for different training set sizes on the small dataset using fine-tuned U-Net with Inception encoder (F1-scores for the test areas from small dataset and large dataset).

	Baseline_no_augm			Simple_augm			OBA
Training set size	1/3	2/3	1	1/3	2/3	1	1/3	2/3	1
	F1-score
Small dataset test	0.861	0.866	0.871	0.867	0.875	0.888	0.913	0.921	0.929
Large dataset test	0.956	0.959	0.962	0.964	0.965	0.967	0.966	0.969	0.971
	Precision
Small dataset test	0.863	0.865	0.872	0.869	0.877	0.889	0.915	0.922	0.931
Large dataset test	0.955	0.961	0.965	0.965	0.966	0.969	0.964	0.972	0.973
	Recall
Small dataset test	0.86	0.867	0.871	0.866	0.873	0.887	0.911	0.921	0.928
Large dataset test	0.957	0.958	0.959	0.963	0.964	0.965	0.968	0.967	0.97
	IoU
Small dataset test	0.754	0.761	0.768	0.774	0.783	0.799	0.851	0.856	0.867
Large dataset test	0.835	0.847	0.856	0.878	0.884	0.891	0.895	0.899	0.912

, with the best score being $F1=0.929$. The prediction quality for the initial large dataset using the fine-tuned model improved ($F1=0.971$). Moreover, it should be noticed from Table 2 that when using 1/3 of the whole dataset and implementing OBA in the training procedure the $F1=0.913$, which is higher than when using the whole dataset for training the baseline model ($F1=0.888$). This means that our proposed approach is highly relevant in view of the limited amount of high quality labelled remote sensing data.

The proposed approach allows us to obtain more precise results than the forest cover masks available through OSM serves for particular areas. Examples of generated maps are presented in Figure 5 and Figure 6. Available open-access forest masks should be updated regularly to include both newly cultivated forests and tree felling. Although trees within built-up areas can be missed in open-access maps, they are crucial for environmental analysis.

The obtained results confirmed the potential of the OBA approach for environmental studies. One of the promising direction for future study is applying precision forest mask for more accurate deforestation analysis. It can be also used for forest species classification as this task usually requires forest boundaries.

A detailed forest mask can be combined with other landcover classes and man-made objects such as the building segmentation task discussed in [37]. A promising extension of this research could be the implementation of visual transformers [43] for solving segmentation tasks using remote sensing data. The wide potential of implementing a similar augmentation approach coupled with special image collection techniques for synthetic data generation to improve neural network performance has been shown in a recent study [44]. In this study involving segmentation of damage to apples, the authors improved the F1-score by up to 4% compared with common augmentation techniques. The authors used DeepLab as the base model for comparing different augmentation techniques. Despite the demonstrated strength of the proposed method, we should take into account its limitations in processing natural scene images. We should carefully use the different types of trees in order to mix them and create the new scene, and trees and scenes should be taken from approximately one period of the year.

Basemap image use makes this approach cost-effective, and high spatial resolution provides significant features for the CNN-based model at the same time. Therefore, this data type is quite competitive with multispectral satellite images which have wider spectral range at lower spatial resolutions. The OBA approach for small precise datasets can be studied for multispectral images to solve other challenges combining RGB bands and vegetation indexes. For instance, NDVI (Normalized Difference Vegetation Index) for deforestation problems was implemented in [45].

4. Conclusions

High-resolution detailed forest masks are essential for environmental studies. However, in practice such maps are not available for large country-scale territories. Here, we have presented a novel pipeline for forest mask creation using very high spatial resolution basemap RGB images. CNN training included an object-based augmentation approach to achieve more accurate predictions of individual trees and small groups of trees. The created map showed high quality and detalization on various test territories, including in Russia and China. Model prediction showed robustness for regions with complex environmental structures. The provided approach aimed to minimize the need for labeled training data. For the test area used in this study, the F1-score for small details was $0.929$ compared with a score of $0.856$ for the baseline approach. The created forest mask is now available for large-scale and precise environmental studies as part of the open-access platform. As a possible evolution of the current study, we are planning to implement automated selection of hyperparameters and thresholds for augmentation techniques and to use our approach for solving further tree classification tasks.