Smallholder Crop Area Mapped with a Semantic Segmentation Deep Learning Method

Abstract

The growing population in China has led to an increasing importance of crop area (CA) protection. A powerful tool for acquiring accurate and up-to-date CA maps is automatic mapping using information extracted from high spatial resolution remote sensing (RS) images. RS image information extraction includes feature classification, which is a long-standing research issue in the RS community. Emerging deep learning techniques, such as the deep semantic segmentation network technique, are effective methods to automatically discover relevant contextual features and get better image classification results. In this study, we exploited deep semantic segmentation networks to classify and extract CA from high-resolution RS images. WorldView-2 (WV-2) images with only Red-Green-Blue (RGB) bands were used to confirm the effectiveness of the proposed semantic classification framework for information extraction and the CA mapping task. Specifically, we used the deep learning framework TensorFlow to construct a platform for sampling, training, testing, and classifying to extract and map CA on the basis of DeepLabv3+. By leveraging per-pixel and random sample point accuracy evaluation methods, we conclude that the proposed approach can efficiently obtain acceptable accuracy (Overall Accuracy = 95%, Kappa = 0.90) of CA classification in the study area, and the approach performs better than other deep semantic segmentation networks (U-Net/PspNet/SegNet/DeepLabv2) and traditional machine learning methods, such as Maximum Likelihood (ML), Support Vector Machine (SVM), and RF (Random Forest). Furthermore, the proposed approach is highly scalable for the variety of crop types in a crop area. Overall, the proposed approach can train a precise and effective model that is capable of adequately describing the small, irregular fields of smallholder agriculture and handling the great level of details in RGB high spatial resolution images.

1. Introduction

The demand for agricultural production is increasing all over the world as a result of the growing global population. The food consumed annually by China alone has increased significantly over the past decades [1]. China has one of the largest populations in the world, and it is currently undergoing rapid urbanization. The continually increasing population and decreasing crop area (CA) highlight the necessity of CA protection. As an effective way to protect CA and determine CA changes, classification through remote sensing (RS) has been utilized to monitor the spatial distribution of agriculture and provide basic data for crop growth monitoring and yield forecasting, particularly for smallholder family farming in China [2,3,4,5].

Smallholder family farming is characterized by family-focused motives, such as favoring the stability of the farm household system; it mainly employs family labor for production and uses part of the farm’s products for the family’s consumption [6]. CA in this system is usually characterized by small, heterogeneous, and often indistinct field patterns. Almost 70–80% of food in China is provided by this system [7]. These fields are smaller than 2 ha, which makes resolving their distribution difficult with moderate spatial resolution (30–500 m) satellite imagery. The small size and large distribution area of smallholder family farming highlight the need for a precise and automatic method to map CA via high spatial resolution remote sensing images.

However, previous studies on CA or land cover maps in China and around the world have been mostly based on medium or low spatial resolution images [8,9]. At the coarse scale (>= 500 m), using the decision tree method and Moderate Resolution Imaging Spectroradiometer (MODIS), Friedl et al. [10] and Tateishi et al. [11] finished the global land cover classification task. With the openness of Landsat series images and the development of Google Earth Engine (GEE), a number of different approaches have been applied at the moderate resolution scale (30–500 m) [12,13,14,15,16]. For medium-high resolution (10–30 m) satellites, because of its high spatial, spectral and temporal resolution, Sentinel-2 (S2) data has been used extensively in land cover mappings [17,18], crop type classifications [19,20], and monitoring of vegetation biomass [21,22]. These studies are usually based on S2’s time series information to get better results, which results in a high cost of image preprocessing work, including cloud masking, temporal gap filling and super resolution. Recent technologies have further increased the spatial resolution of available RS images (2 m spatial resolution and higher). The rich shape and context information provided by the high spatial resolution RS images allows researchers to get precise classification results with only one single-phase RS images. There have been some studies on urban land cover classification that used high spatial resolution images [23,24], but the methods have seldom aimed to map CA at a 1 m or even higher resolution. The era of using high spatial resolution RS images for mapping very small agriculture fields has only recently materialized [25].

A high spatial resolution RS image provides the details necessary to observe smallholder agriculture. However, its low spectral resolution also presents challenges in smart image interpretation for CA to the remote sensing community. Some kinds of high spatial resolution RS images have only three bands, red-green-blue (RGB), thus lacking some useful spectral information, such as the near-infrared (NIR) band, for CA classification. Although there is vast literature on the automatic mapping of CA using machine learning algorithms, such as the inverse distance weighted interpolation method [26], decision tree [6], Support Vector Machine [27], and artificial neural network [28], the approaches have usually considered the spectrum of every individual pixel and then assigned each of them to a certain class [25].

However, contextual features have proven to be very useful for classification [29], especially when it comes to small and irregularly shaped targets. Meanwhile, the most prominent advantage of high spatial resolution RS images is their rich spatial information, so it is important to take full advantage of its contextual and shape features. Although some researchers have used texture statistics [30,31], mathematical morphology [32,33], and rotation invariance [34,35] as spatial and shape features, these mid-level features cannot describe the rich contextual information offered by high spatial resolution RS images. Moreover, these methods mostly rely on hand-engineered features, and most appearance descriptors depend on a set of free parameters, which are commonly set by user experience via experimental trial-and-error or cross-validation. So, we argue that a more thorough understanding of the spatial features, such as the shape of objects, is required to aid the mapping process of small and irregularly shaped smallholder agricultural fields.

Therefore, convolutional neural networks (CNNs) [36] have attracted attention for their ability to automatically discover relevant contextual features in classification problems. CNNs, which learn the representative and discriminative features in a hierarchical manner from the data, have recently become a hotspot in the machine learning area and have been introduced to the geoscience and RS community for object detection [37,38], scene understanding [39,40], and image processing [41,42].

Recently, semantic classification tasks in remotely sensed data have also been approached by means of CNNs. In general, CNN architectures for semantic pixel-based classification use two main approaches: patch-based and pixel-to-pixel-based (end to end). At first, Patch classification was used for the task [43,44,45]. These kinds of methods commonly start with training a CNN classifier on small image patches, followed by predicting the class of the center pixel using a sliding window approach. The drawback of these approaches is that the trained network can only predict the central pixel of the input image, resulting in low classification effectiveness. Then, an end-to-end framework for pixel-based methods became more popular [46,47,48,49] for its ability to learn global context features and its high process effectiveness [50]. These frameworks are usually called semantic segmentation networks, and end-to-end usually means jointly learning a series of feature extractions from raw input data to generate a final, task-specific output. Compared with patch classification, semantic labeling-based strategies can label each pixel in the image. The network is trained to learn not only the relationships between spectral signatures and labels but also the contextual features of the whole input image.

Results from different studies have shown that both the accuracy and efficiency of end-to-end networks outperform standard patch-based strategies [51,52], suggesting that end-to-end structures are better suited for RS image classification. Recently, semantic segmentation networks, which are popular in computer vision, have been introduced to the field of RS image classification for their ability to learn both spatial and spectral information [53,54,55]. In this study, we chose the DeepLabv3+ architecture to develop a methodology because it is effective in multi-scale feature fusion and boundary description. With the proposed method, we finished the automatic mapping of CA from Satellite images with only the three RGB bands.

In the next section, we first introduce the study area and the RS data we used in this study. In Section 3, the details of our network architecture and training/classification strategies are presented. Then, we report the results of testing our semantic classification framework on WorldView-2 (WV-2) images with only RGB bands in Section 4. Different classifiers are compared with the proposed method to prove its effectiveness, and the classification results are shown in Section 5. Then, a Discussion about the strengths of the proposed method with respect to other relevant studies is given in Section 6. Finally, considerations for future work and the conclusions of the study are given in Section 7.

2. Study Area and Data

2.1. Study Area

The study area, Baodi, lies in the north of the North China Plain and south of the Yanshan Mountains, located at 117°8′–117°40′E and 39°21′–39°50′N (Figure 1). It is a part of Tianjin, China, and near Bohai Bay. Baodi occupies approximately 1450 km² and has a population of approximately 871,300. The region is dominated by plains, with elevations between 2.5 and 3 m. The dominant land cover in Baodi is small-scale agriculture fields that cover approximately 700–800 km², and it has been an important food production base for Beijing and Tianjin since the year 2000.

Baodi is characterized by a warm temperate semi-humid continental monsoon climate. The four seasons are distinct, and winter and summer are longer than spring and autumn. The annual average temperature is 11.6 °C. The annual precipitation is 612.5 mm, and the frost-free period is about 184 days [56]. There are a variety of soil types throughout Baodi. The northern high area is dominated by common tidal soils. The conditions are conducive to a high yield of various crops, such as grain, fruits, vegetables, and medicinal materials: the soil texture is loamy; the fertility is high; the water, fertilizer, gas, and heat are relatively coordinated; and the soil layer is thick. The middle part is mainly moist soil, and the texture is sticky. It is suitable for rice, sorghum, soybean, green onion, cotton, and hemp. The southern area has a salinized moist soil with a wide area and a short cultivation period. It is suitable for the development of freshwater aquaculture and plants for anti-salt and anti-humid crops. The eastern area is mostly of clay soil and is suitable for planting crops such as wheat, rice, and soybean [57].

The area is interesting from the perspective of method development for a number of reasons. First, the study area is an important grain and cotton production base in northern China. Second, the diversity of soil and crop types is abundant across Baodi county, and such variety can be used to evaluate the generalization ability of the method. Third, the climate and topographic conditions of Baodi are common in the North China Plain, so a method based on this area can be easily extended to other similar areas.

2.2. Data

As mentioned before, this study aims to map the CA of smallholder family farming systems via high spatial resolution RS images. To achieve this goal, we collected WorldView-2 (WV-2) images of the study area. DigitalGlobe’s WV-2 satellite sensor, launched on 8 October 2009, provides high optical resolution and high geometric accuracy of up to 0.46 m. The WV-2 sensor provides a high-resolution panchromatic band and eight multispectral bands: four standard bands (red, green, blue, and near-infrared 1) and four new bands (coastal, yellow, red edge, and near-infrared 2). The spectrum’s diversity provides users with the ability to perform mapping and monitoring applications, land-use planning, disaster relief, exploration, defense and intelligence, and visualization and simulation environments. The images of the study area that were selected for this paper were taken in the summer and autumn with minimum cloud cover and a 1.85-m resolution. The multispectral images were then fused with the panchromatic images and resampled to a 1-m resolution. As the RS images with the highest spatial resolution have a low spectral resolution, the WV-2 images used here are in true color fusion to prove the promotability and effectiveness of the proposed method.

3. Method

In this study, we chose DeepLabv3+ as the classification model to get the CA from the study area; the architecture of the network is presented in Section 3.1. Similar to other supervised classification methods, our approach generally has three stages (Figure 2): the training stage, the classification stage, and the accuracy evaluation stage. In the training stage, image–label pairs, with pixel-class correspondence, are input into the DeepLabv3+ network as training samples. The error between predicted class labels and ground truth (GT) labels is calculated and back-propagated through the network using the chain rule, and then the parameters of the DeepLabv3+ network are updated using the gradient descent method. In the classification stage, the trained DeepLabv3+ network is fed an input image to generate a class prediction. Then, two kinds of evaluation methods are employed in the accuracy evaluation stage to establish the effectiveness of the proposed method. The details of the training and classification stages are introduced in Section 3.2, while the two accuracy evaluation methods are detailed in Section 3.3.

3.1. Network Architecture

In the last few years, semantic segmentation has been a hot topic in computer vision. A number of network architectures have been proposed, e.g., FCN [58], U-Net [59], PspNet [60], SegNet [61], and DeepLab series [62,63,64,65]. Meanwhile, there have been some studies focusing on semantic segmentation for RS images. From the first attempt by Mnih and Hinton [66], who designed a shallow, fully connected network for road classification, different CNN architectures have been proposed for remote sensing images [67,68]. Recently, semantic segmentation networks that are popular in computer vision have been introduced to the field of RS image classification, and the results indicate that the networks are appropriate for RS images as well. In this study, we chose the DeepLabv3+ architecture (Figure 3), which has achieved state-of-the-art performance on the PASCAL VOC 2012 [69] and Cityscapes [70] datasets.

DeepLabv3+ is built on a powerful CNN backbone architecture for the most accurate results. It is an encoder–decoder architecture that employs DeepLabv3 to encode the rich contextual information and a simple yet effective decoder module to recover object boundaries. Moreover, the spatial pyramid pooling strategy is applied in the network structure, resulting in a faster and stronger encoder–decoder network for semantic segmentation. Although the DeepLabv3+ model was designed for natural image segmentation, it is compatible with multichannel inputs and is sensitive to the boundaries in the images. So, DeepLabv3+ is particularly suitable for RS images classification and CA boundary delineation.

3.2. Network Training and Classification

Compared with traditional computer vision images, such as the images on ImageNet, RS images often have more coverage and a larger size. So, it is difficult to train RS images as a whole. Therefore, before training, we split the labeled RS images into small parts. As RS images labeled with the GT are limited, we used a sliding window for overlapped sampling rather than the general sampling procedure. The sliding window can help expand the training dataset and avoid overfitting. Meanwhile, four forms of data augmentation (rotate 90°, rotate 180°, rotate 270°, and flip) were also used in this work to further enlarge the dataset. The expanded training dataset, which is organized by Image–GT label pairs, was then inputted into DeepLabv3+ as the source of training samples. For better and faster training results, the model was first trained on ImageNet and then transferred to our dataset. The Softmax function [71] was performed on the output feature map generated by the network to predict the class distribution. Then, the softmax loss was calculated and back-propagated, and finally, the network parameters were updated using Stochastic Gradient Descent (SGD) with momentum.

In the mapping stage, the trained network was used on the RS images to be classified. However, high spatial resolution RS images are often too large to be processed in only one pass through a CNN. Given current Graphic Processing Unit (GPU) memory limitations, we split our images into small patches using the same image size as that used in the training dataset. When splitting, an overlap strategy was also used. After predicting, we combined all of these small patches in order. For the overlapped part of the image predicted, we averaged the multiple predictions to obtain the final classification for overlapping pixels. This smooths the predictions along the borders of each patch and removes potential discontinuities.

3.3. Evaluation Method

3.3.1. Accuracy Evaluation Indicators

We employed the overall accuracy (OA), F1-Score, and Kappa coefficient as indicators to evaluate our approach. These indexes are calculated from the confusion matrix, where the overall accuracy is calculated as

$$OA={\displaystyle \frac{tp}{n}}\ast 100\%$$

(1)

where $tp$ represent the total number of correctly classified pixels, and n is the total number of validation pixels. Overall accuracy denotes the proportion of the pixels that are correctly classified, and the F1-Score is computed as

$$F{1}_i=2\ast {\displaystyle \frac{precisio{n}_i\ast recal{l}_i}{precisio{n}_i+recal{l}_i}}$$

(2)

$$precisio{n}_i={\displaystyle \frac{t{p}_i}{P_i}},recal{l}_i={\displaystyle \frac{t{p}_i}{C_i}}$$

(3)

where $t{p}_i$ the number of true positives for class i; $C_i$ the number of pixels belonging to class i; and $P_i$ the number of pixels attributed to class i by the model. So, $precision$ is the number of correct positive results divided by the number of all positive results returned by the classifier, $recall$ is the number of correct positive results divided by the number of all relevant samples, and F1-Score represents the harmonic average of the $precision$ and $recall$. The Kappa coefficient measures the consistency of the predicted classes with the GT classes, which is calculated as

$$KAPPA={\displaystyle \frac{OA-p_e}{1-p_e}}$$

(4)

$$p_e={\displaystyle \frac{1}{n^2}}\sum _i{n}_{i1}{n}_{i2}$$

(5)

where $n_{ik}$ is the number of times rater k predicted category i. The equations show that OA is the relative observed agreement among raters, and $p_e$ is the hypothetical probability of chance agreement. If the raters are in complete agreement, then $KAPPA=1$. If there is no agreement among the raters other than what would be expected by chance (as given by $p_e$), $KAPPA=0$.

3.3.2. Per-Pixel Accuracy Evaluation Method

We randomly chose eight slices of the whole image for the pixel-based classification evaluation (Figure 4). The size is 1024 × 1024 for all eight, and they were all labeled manually with the GT. None of the eight slices were involved in training. By obtaining the final mapping result and calculating the confusion matrix, we obtained the OA, kappa coefficient, and F1-Score for each slice. Furthermore, per-pixel accuracy was calculated for acquiring information on the spatial distribution of the classification error. Different kinds of competing methods were processed to prove the superiority of the method proposed in this paper.

3.3.3. Random Validation Point Accuracy Evaluation Method

As the RS images we used in this study have a high spatial resolution and cover the whole Baodi county at almost 1500 km², it is hard to do per-pixel accuracy evaluation all across the study area. So, the evaluation was carried out using validation points to further prove that the classification method is effective for the whole study area. One hundred validation points were collected randomly from the entire image and labeled by visual interpretation. With the classification results, the confusion matrix was also calculated, along with the OA, kappa coefficient, and F1-Score.

4. Experiments and Comparison

4.1. Experiment Setup

Our training dataset a collection from WV-2 of Baodi, Tianjin, China. The images are in true color fusion with a 1-meter resolution. As the training dataset structure has a great influence on training and the classification result, we used K-MEANS, an unsupervised clustering method, before sampling to ensure that different kinds of CAs in the study area are covered in the training samples. After clustering, we manually labeled eight slices (sizes of 2048 × 2048) of the whole image at the pixel level as GT label data. In our training dataset, using a sliding window with a stride of 32 pixels, there are a total of 120,000 pairs of samples (sizes of 128 × 128). So, each pixel in the RS images corresponds to a pixel class. We used 96,000 images (80% of the whole set) for training, and the remaining 24,000 images (20% of the whole set) were used for testing. We used ResNet-101 as the network backbone in the DeepLabv3+ model. The ResNet-101 model was pretrained on ImageNet and then adapted to our dataset with 0.00001 as the initial learning rate. The max iteration in our training step was 75,000. In the training procedure, we fed the samples into the network in batches, and each batch contained 16 images. For the classification stage, we split the whole predicted image into small patches with a sliding window whose stride was 32 pixels. Then, all the patches were combined, and an averaging method was applied for the overlapped parts. In addition, we used the deep learning framework TensorFlow, Compute Unified Device Architecture 8.0 (CUDA 8.0), and Geospatial Data Abstraction Library (GDAL) to construct a platform for all of the work steps, including sampling, training, testing, and classifying, to extract and map CA.

4.2. Competing Method

Traditional machine learning methods have been widely used in the field of RS classification in the last few years. Common methods include Maximum Likelihood (ML) [72], Support Vector Machine (SVM) [73], and Random Forest (RF) [74]. To prove that CNNs can get more thorough spatial or shape features from RGB high spatial resolution RS images, all three of these common classification methods were implemented using the same training dataset that we mentioned before. Detailed parameters are shown in

Table 1. Parameters of Maximum Likelihood (ML), Support Vector Machine (SVM), and Random Forest (RF).

Algorithm	Parameters
ML	Reject fraction (0.01)
	A priori probability weighting (Equal)
SVM	Cost or slack parameter (1.0)
	Kernel type (RBF)
	Radial basis: gamma (1/3)
RF	Number of trees (100)
	Number of variables randomly sampled as candidates at each split (3)
RBF: Radial-Based Function

.

Different kinds of CNNs have been used recently for RS image classification for their ability to learn both spatial and spectral information. In this paper, to show the superiority of DeepLabv3+ for the task of CA mapping, we chose four popular end-to-end structure networks, i.e., U-Net, PspNet, SegNet, and DeepLabv2, as competing networks. The training strategy and parameters were the same for all five networks. Similar to the above, the training dataset was the same here, as well.

5. Results

In this section, we compare our proposed method with other existing methods, including not only other CNNs but also traditional machine learning methods. These results represent an exhaustive and complete validation of our method with other popular methods for high spatial resolution RS image classification, and they show CNNs’ ability to learn more thorough spatial features from high spatial resolution RS images compared with other methods.

5.1. Results across DeepLabv3+

CA classification efforts in previous studies using different spatial resolution RS images have indicated that it is important to take full advantage of context and shape features in the mapping process, especially for smallholder agriculture. This inspired us to use deep semantic segmentation networks to get not only a more thorough spatial understanding but also more accurate CA mapping results automatically from RGB-only WV-2 images in this study. The image size was 49,895 × 56,044, the classification step length was 32 pixels, and the processing time was about 346 min. The processing burden rises with the increase in image size and the decrease in step length. The mapping result and confusion matrix for DeepLabv3+ for the whole image of Baodi are shown in Figure 5 and

Table 2. Accuracy assessment of DeepLabv3+.

		Ground Truth
		Crop Area	Non-Crop Area	Sum
Mapped	Crop Area	55	2	57
	Non-Crop Area	3	40	43
	Sum	58	42	100
	OA	95%
	KAPPA	0.90
* OA: Overall Accuracy

, respectively.

As can be seen, as a result of DeepLabv3+’s ability to learn context and shape features, along with its effectiveness in defining boundaries, the proposed method achieved high accuracy in the study area on the basis of RGB-only high spatial resolution remote sensing images. Linear objects, such as narrow roads and ridges between field blocks, were extracted from the CA mapping results. Furthermore, it is worth noticing that CAs with different crop types were extracted at the same time. This may result from the training dataset organization and deep semantic segmentation networks’ advantages of understanding spatial features.

To further justify the performance of the proposed methodology, four other deep semantic segmentation networks and three traditional machine learning methods were tested on eight valid slices of about 1 km² of Baodi county. Then, a per-pixel accuracy evaluation was processed, as shown in the next two sections. The mapping results of these competing methods for the entirety of Baodi county are not shown in this paper because the main focus is on the method itself and its comparison with other methods, such as SegNet and RF.

5.2. Comparison with Other Semantic Segmentation Networks

The previous result demonstrates that DeepLabv3+ can achieve high accuracy for CA classification. We tested four other deep learning methods, including SegNet, U-net, PspNet, and DeepLabv2, as described in this section. The OA, kappa coefficient, and F1-Score of each method for different categories are shown in

Table 3. Comparisons of U-Net, PspNet, SegNet, DeepLabv2 (DLv2), and DeepLabv3+ (DLv3+) on pixel-wise classification.

Val	OA					KAPPA
	U-Net	PspNet	SegNet	DLv2	DLv3+	U-Net	PspNet	SegNet	DLv2	DLv3+
1	95.95%	96.15%	95.61%	94.40%	96.99%	0.92	0.92	0.91	0.89	0.94
2	90.75%	88.71%	85.63%	91.38%	92.01%	0.82	0.77	0.71	0.83	0.84
3	86.65%	88.82%	88.44%	90.18%	88.87%	0.67	0.70	0.70	0.74	0.72
4	89.72%	84.24%	87.23%	89.01%	93.17%	0.79	0.68	0.74	0.78	0.86
5	90.05%	87.91%	87.24%	88.66%	91.48%	0.48	0.32	0.25	0.36	0.59
6	93.99%	85.22%	94.58%	94.15%	96.08%	0.88	0.70	0.89	0.88	0.92
7	86.40%	87.55%	83.75%	87.47%	91.70%	0.73	0.75	0.68	0.75	0.83
8	80.37%	81.09%	85.35%	81.09%	89.79%	0.45	0.51	0.58	0.55	0.73
AVG	89.24%	87.46%	88.48%	89.54%	92.51%	0.72	0.67	0.68	0.72	0.80
Val: The number of the valid slices					OA: Overall Accuracy
AVG: The average value of OA or kappa coefficient for the eight valid slices

and

Table 4. Comparisons of U-Net, PspNet, SegNet, DeepLabv2 (DLv2), and DeepLabv3+ (DLv3+) on pixel-wise classification.

Val	F1-SCORE (Crop Area)					F1-SCORE (Non-Crop Area)
	U-Net	PspNet	SegNet	DLv2	DLv3+	U-Net	PspNet	SegNet	DLv2	DLv3+
1	0.96	0.96	0.96	0.95	0.97	0.96	0.96	0.95	0.94	0.97
2	0.91	0.89	0.86	0.91	0.92	0.91	0.89	0.85	0.91	0.92
3	0.91	0.93	0.92	0.93	0.92	0.76	0.77	0.78	0.80	0.79
4	0.90	0.85	0.89	0.90	0.94	0.89	0.83	0.85	0.88	0.92
5	0.94	0.93	0.93	0.94	0.95	0.53	0.37	0.29	0.40	0.63
6	0.94	0.85	0.95	0.94	0.96	0.94	0.85	0.94	0.94	0.96
7	0.87	0.88	0.85	0.88	0.92	0.85	0.87	0.82	0.87	0.91
8	0.87	0.87	0.91	0.87	0.93	0.57	0.63	0.66	0.55	0.80
AVG	0.91	0.90	0.91	0.92	0.94	0.80	0.77	0.77	0.79	0.86
Val: The number of the valid slices
AVG: The average value of F1-Score for the eight valid slices

, along with those of DeepLabv3+. The tables show that DeepLabv3+ obtains the best performance compared with the others, and it is apparent that DeepLab series can get better CA mapping results. SegNet has a good balance between accuracy and computational cost. Although it has a simpler architecture compared with the others, SegNet still presents a similar accuracy to that of U-Net and PspNet.

Figure 6 shows the per-pixel accuracy for all five networks on eight validation slices. The first observation is that the classification maps from the five networks are similar to each other in some way. The deep semantic networks all perform well in the prediction of CA with different kinds of crops, and they are good at region extraction from original remote sensing images. Some small and scattered objects, such as country roads and brushland with small areas, are classified correctly. On the other hand, some pixels in grassland and brushland are assigned to CA. Larger area trees, shrubland, grassland, and cropland have similar spectral properties and topographic information, so it is difficult for the trained networks to distinguish between them. Meanwhile, the trained networks receive a low accuracy when it comes to mulch, which can be seen in Val.3. This may result from the lack of training samples of this type, so the networks did not learn the corresponding features in the training stage. However, there are still some differences between the classification maps from the five networks. The DeepLab series is better at classification for the pixels that are on the edge of the image. DeepLabv3+ describes the shape and the edge of the cropland more precisely and performs better on the brushes and trees located between buildings.

5.3. Comparison between CNNs and Traditional Machine Learning Methods

To verify the effectiveness of CNNs, we calculated the average accuracy of the five deep semantic segmentation networks and compared it with that of three traditional machine learning methods, i.e., ML, SVM, and RF.

Table 5. Comparisons of Maximum Likelihood (ML), Support Vector Machine (SVM), Random Forest (RF), and Convolutional Neural Networks (CNNs) on pixel-wise classification.

Val	OA				KAPPA
	ML	SVM	RF	CNNs	ML	SVM	RF	CNNs
1	80.90%	79.77%	71.49%	95.82%	0.61	0.59	0.43	0.92
2	72.11%	71.34%	69.51%	89.70%	0.44	0.43	0.39	0.79
3	85.42%	77.63%	63.02%	88.59%	0.57	0.39	0.20	0.71
4	66.70%	64.87%	62.81%	88.67%	0.31	0.28	0.25	0.77
5	81.69%	81.56%	66.00%	89.07%	0.09	0.16	0.10	0.40
6	73.06%	72.42%	65.90%	92.80%	0.46	0.44	0.32	0.85
7	67.98%	70.70%	69.51%	87.37%	0.37	0.43	0.40	0.75
8	56.58%	59.05%	56.35%	83.54%	0.08	0.13	0.13	0.56
AVG	73.06%	72.17%	65.57%	89.45%	0.37	0.36	0.28	0.72
Val: The number of the valid slices				OA: Overall Accuracy
AVG: The average value of OA or kappa coefficient for the eight valid slices

shows accuracy evaluation results from all three models and the average results from CNNs. The three traditional machine learning methods perform similarly to each other and have an average OA of 70.27%, while CNNs show a significant increase in OA (19.18%). The F1-Score shown in

Table 6. Comparisons of Maximum Likelihood (ML), Support Vector Machine (SVM), Random Forest (RF), and Convolutional Neural Networks (CNNs) on pixel-wise classification.

Val	F1-SCORE (Crop Area)				F1-SCORE (Non-Crop Area)
	ML	SVM	RF	CNNs	ML	SVM	RF	CNNs
1	0.84	0.83	0.73	0.96	0.76	0.75	0.70	0.96
2	0.77	0.76	0.72	0.90	0.65	0.65	0.67	0.90
3	0.91	0.85	0.72	0.92	0.66	0.54	0.45	0.78
4	0.73	0.71	0.66	0.90	0.57	0.56	0.59	0.87
5	0.90	0.90	0.78	0.94	0.19	0.27	0.28	0.44
6	0.75	0.75	0.66	0.93	0.71	0.69	0.65	0.93
7	0.75	0.76	0.74	0.88	0.56	0.61	0.64	0.86
8	0.66	0.68	0.64	0.89	0.40	0.42	0.44	0.64
AVG	0.79	0.78	0.71	0.92	0.56	0.56	0.55	0.80
Val: The number of the valid slices
AVG: The average value of F1-Score for the eight valid slices

also indicates that CNNs have a higher accuracy in the classification of CA. Traditional machine learning methods create a serious salt-and-pepper phenomenon in the classification maps, which can be seen in Figure 7. Using object-based classification may solve this problem, but object segmentation is a highly subjective process that needs researchers to set up the segmentation parameters on the basis of experience. However, the deep semantic segmentation networks we used can perform the segmentation automatically and classify pixels at the same time. The results prove that the deep semantic segmentation networks are effective in both segmentation and classification.

6. Discussion

Mapping CA using remotely sensed observations is important for CA protection and agricultural production. However, existing CA maps are mostly based on medium or low spatial resolution RS images that lack the essential spatial details to describe CA in a smallholder family farming system. Taking advantage of high spatial resolution RS images is an effective way to solve this problem, but it still presents several challenges when the traditional machine learning methods are used. These challenges include a thorough understanding of the rich spatial features and classification with low spectral resolution. Faced with these challenges, we developed an automatic classification framework based on deep semantic segmentation networks to map CA from WV-2 images with only three bands of RGB. Our study area, Baodi, has various types of soil and crops. It is an important grain and cotton production base in northern China. The climate and topographic conditions in the area are common in the North China Plain. All of these aspects make it representative for use in this study. Our research provides a number of key insights into how to utilize high spatial resolution RS images and deep semantic segmentation networks for mapping CA in a smallholder agricultural system.

First, our study demonstrates that, on the basis of high spatial resolution RS images and deep semantic segmentation networks, our method is suitable for the classification of CA in a smallholder family farming system. We compared our CA map with some other existing maps of Baodi. The Global Map–Global LC (GLCNMO) dataset from the International Steering Committee for Global Mapping, 2008, 500 m resolution [11] and the Finer Resolution Observation and Monitoring Global LC dataset (FROM-GLC) from China based on Landsat images, 2015, 30 m resolution [16] were used here. The same validation sample points introduced in Section 3.3.3 were utilized to prove the effectiveness of high spatial resolution RS images and the method proposed in this study.

Table 7. Accuracy assessment of the Global Map–Global LC (GLCNMO) and Finer Resolution Observation and Monitoring Global LC datasets (FROM-GLC) and our method.

	GLCNMO	FROM-GLC	Our Method
OA	58%	61%	95%
KAPPA	0.14	0.20	0.90
OA: Overall Accuracy

shows the OA and kappa coefficient of the GLCNMO and FROM-GLC datasets and our result. The FROM-GLC dataset performs better than the GLCNMO dataset in the accuracy evaluation result, but they are very similar to each other. Results from our method show a significant increase in the OA and kappa coefficient. A comparison of the detailed mapping results is shown in Figure 8. The sharpness of the CA boundaries and the details in the classification results are increased with the spatial resolution of RS images. The shape and contour of the road and river are also clearer in the results of WV-2 images. Further, the method proposed in this paper is better with the brushes and trees located between buildings and gives a more precise location of the CA in the mapping results, which is important for the small-sized fields in the smallholder family farming system.

A second major insight from our study is that deep semantic segmentation networks are effective in feature extraction from high spatial resolution RS images. Traditional machine learning methods usually rely on hand-engineered features to describe the spectral, contextual, and shape features of the input images. Most appearance descriptors depend on a set of free parameters, which are commonly set by user experience via experimental trial-and-error or cross-validation. However, as a sort of CNN, deep semantic segmentation networks are able to automatically discover relevant features in classification problems. Furthermore, the network used in this study, DeepLabv3+, can capture multi-scale information and generalize the standard convolution operation by the atrous convolution, which results in a more thorough understanding of the input information.

A third major insight is that deep semantic segmentation networks can prevent the salt-and-pepper phenomenon, which is common in pixel-based high spatial resolution RS image classification tasks. Before the appearance of deep semantic segmentation networks, the object-based classification method was usually used to solve the salt-and-pepper problem [75,76], but object segmentation also relies on researchers’ experience and knowledge to set up the segmentation parameters. However, a deep semantic segmentation network can perform segmentation and pixel classification at the same time by its encoder–decoder structure and obtain high classification accuracy and more detailed boundaries of CA. Therefore, compared with traditional machine learning methods, deep semantic segmentation can get better CA classification results, especially when it comes to high spatial resolution RS images with low spectral resolution.

A final key insight from our study is that it is possible to apply deep learning methods to the larger-scale task of RS image classification. Deep learning techniques were originally rooted in the computer vision fields for classification and recognition tasks, and they have only recently been introduced to the RS community. As a new research branch in RS image analysis, most of the recent studies have focused on the optimization of models and algorithms using a small-scale study area [77,78]. Studies using deep learning models for larger-scale RS classification are lacking. However, classification over large areas is one of the fundamental topics in the RS application field. In this study, we applied the proposed method to classify the CA of Baodi, which occupies 1500 km². With the development of hardware and the improvement of model training strategies, problems such as low efficiency and large numbers of training samples may be solved, and then deep learning models can be applied to the classification (or other analysis) of larger study areas, such as those at the regional or even national level.

Our method yielded automatic and precise smallholder agricultural CA maps, but a few uncertainties remain. First, our classification framework performed well with only one single-phase RS image and three RGB bands in this study. However, multi-temporal and multi-spectral features may help optimize the CA mapping results. Furthermore, although traditional machine learning methods had a lower classification accuracy, they were easier to train on a smaller dataset with point labels, and they had a higher efficiency in both the training and classification stages. The great performance of deep semantic segmentation models is often due to the availability of massive datasets. However, recent studies on semi-supervised [79] or even unsupervised [80] deep learning methods, as well as the transfer learning strategy [81], indicate that this problem can be solved.

7. Conclusions

Prior studies have documented the effectiveness of RS image classification for the purpose of CA protection. However, these studies have been mostly based on RS images with medium or low spatial resolution, neglecting high spatial resolution RS images’ advantages for precise smallholder agriculture observation. Meanwhile, the methods used in these studies have often lacked a more thorough understanding of the context, such as the shape of objects. In this paper, using CNNs and WV-2 images, we developed a methodology to get better shapes and deeper contextual features of cropland, and we accomplished the automatic mapping of the CA of satellite images with only the three RGB bands.

We found that deep semantic segmentation networks, as a sort of CNN, are able to automatically extract the deep features from the input images and prevent salt-and-pepper problems by their encoder–decoder structure. Therefore, methods based on deep semantic segmentation models can get a higher classification accuracy and more detailed boundaries of CA than traditional machine learning methods, such as ML, SVM, and RF. These findings indicate that the deep semantic segmentation networks are effective for both the segmentation and classification of high spatial resolution RS images. Furthermore, we used the proposed method to classify the CA of the whole study area, which occupies 1500 km². This study, therefore, provides insight into introducing deep learning methods to the larger-scale task of RS image classification.

Most notably, to our knowledge, this is the first study to use CNNs to extract the CA of a whole county area. Our results provide compelling evidence for CNNs’ ability to learn shape and contextual features and show that this approach appears to be effective for smallholder agriculture CA mapping. Owing to the representativeness of the study area and the generalization of the proposed framework, the methodology can be applied to other similar areas in the North China Plain or extended to mapping more refined cropland attributes such as crop types. However, there are still some limitations worth noting. Although our method achieved high accuracy in the study with only one single-phase RS image and three RGB bands, multi-temporal RS images and the near-infrared band have been proved to be important for CA classification. Future work should, therefore, focus on multi-source RS image fusion and multi-time sequence data processing with recurrent neural networks (RNNs).