Deep Neural Networks to Detect Weeds from Crops in Agricultural Environments in Real-Time: A Review

Abstract

Automation, including machine learning technologies, are becoming increasingly crucial in agriculture to increase productivity. Machine vision is one of the most popular parts of machine learning and has been widely used where advanced automation and control have been required. The trend has shifted from classical image processing and machine learning techniques to modern artificial intelligence (AI) and deep learning (DL) methods. Based on large training datasets and pre-trained models, DL-based methods have proven to be more accurate than previous traditional techniques. Machine vision has wide applications in agriculture, including the detection of weeds and pests in crops. Variation in lighting conditions, failures to transfer learning, and object occlusion constitute key challenges in this domain. Recently, DL has gained much attention due to its advantages in object detection, classification, and feature extraction. DL algorithms can automatically extract information from large amounts of data used to model complex problems and is, therefore, suitable for detecting and classifying weeds and crops. We present a systematic review of AI-based systems to detect weeds, emphasizing recent trends in DL. Various DL methods are discussed to clarify their overall potential, usefulness, and performance. This study indicates that several limitations obstruct the widespread adoption of AI/DL in commercial applications. Recommendations for overcoming these challenges are summarized.

1. Introduction

Weeds constitute one of the most devastating constraints for crop production, and efficient weed control is a prerequisite for increasing crop yield and food production for a growing world population [1]. However, weed control may negatively affect the environment [2]. The application of herbicides may result in pollution of the environment because, in most cases, only a tiny proportion of the applied chemicals hits the targets while most herbicides hit the ground, and a part of them may drift away [2,3]. Mechanical weed control may result in erosion and harm beneficial organisms such as earthworms in the soil and spiders on the soil surface [4,5]. Other weed control methods have other disadvantages and often affect the environment negatively. Sustainable weed control methods need to be designed only to affect the weed plants and interfere as little as possible with the surroundings. Weed control could be improved and be more sustainable if weeds were identified and located in real-time before applying any control methods.

1.1. Motivation

Rehman et al. [6] considered how machine vision could automate the weed detection problem using field or airborne cameras. Machine vision was used to detect and kill weeds with powerful lasers. A cascade classifier was trained using Haar-like features to detect weeds on images.

However, the Haar features strongly depend on the orientation of the object being monitored, especially on the angle of rotation [7,8,9]. Histogram of Oriented Gradients (HOG) has similar problems. HOG are descriptors of special points that are used in computer vision and image processing for the purpose of object recognition. Abouzahir et al. [10] used HOG as an auxiliary tool to generate visual words and a backpropagation neural network for weed detection and plant classification. Che’Ya et al. [11] used a hyperspectral reflectance method for the assessment of weed classification for Amaranthus macrocarpus, Urochoa panicoides, and Malva sp. The images were from a real field but they did not consider gaps with dense scenes, which greatly simplifies the weedy identification process.

For classification tasks, Bayesian classification, discriminant analysis, and the nearest neighbor method have been widely used. Rainville et al. [12] used Bayesian classification and unsupervised learning for the isolation of weeds in row crops. Finally, the authors correctly classified an average of 94% of corn and soybean plants and 85% of weeds (multiple species). Islam et al. [13] considered several machine learning algorithms, random forest (RF), support vector machine (SVM), and k-nearest neighbors (KNN), to detect weeds using UAV images. The authors concluded that RF performed better than other classifiers. For the analyses, the authors only used images from one field. However, under other conditions, another classifier may be preferable. Hung et al. [14] presented an overview of machine learning methods in weed classification tasks.

Weed detection is an applied task and is not conducted for statistics, but for the subsequent control of weeds. Therefore, it is important to know the position of the weed and, at the same time, identify it quickly, because the cameras are installed on moving objects such as tractors, autonomous vehicles, and drones. The methods described above perfectly cope with the task of classification in conditions close to those of a laboratory, where they have certain criteria and requirements for images, such as background, light, angle, etc., but in field conditions, these factors vary all the time. Accurately determining the position of the weed in natural conditions remains a difficult task because of the high variability in weed size, color, occlusion, high density of weed and crop plants, and the overlapping of plant parts. For this reason, this study focuses only on papers that have successfully used machine vision for weed detection in images with high accuracy, emphasizing the deep learning (DL) technique.

1.2. Research Methodology and Criteria for Comparison

The fields of probabilistic modelling, AI, and neural networks are broad, covering many fields. In this section, we describe the process of selecting and analyzing articles included in this review. The search for articles was performed mainly via keywords, carried out directly for the following publishers through their websites and search features/menus:

• Elsevier;
• Taylor & Francis;
• Springer;
• Wiley;
• IEEE;
• Informa;
• MDPI;
• Hindawi.

The keyword-based search was also performed on the following academic search engines:

• Google Scholar (http://www.scholar.google.com, accessed on 5 November 2021);
• ResearchGate (https://www.researchgate.net/, accessed on5 November 2021);
• Academia (https://www.academia.edu/, accessed on 5 November 2021);
• Publons (https://publons.com/, accessed on 5 November 2021).

We focused on papers published over the last ten years. Keywords were used in various combinations (e.g., “weed corn detection”, “weed corn classification”, and “object detection in agriculture”), including different names of crops (e.g., tomato, apple, and corn) as well as modern, specific, widely used DL architectures (e.g., EfficientNet, EfficientDet, SpineNet, CenterNet, ThunderNet, CSPNet, DenseNet, SAUNet, DetNASNet, SM-NAS, AmoebaNet, Graph Neural Network, and Growing Neural Cellular Automata). The following criteria were considered when analyzing the retrieved papers:

• The DL model/architecture used because this directly affects the requirements for the hardware part, as shown by Pourghassemi et al. [15]. In particular, the possibility of using neural networks on families of single-board computers was included in the review process.
• The number of images (i.e., dataset size) used for training the neural network.
• The types of platforms used to collect the images, with an option of whether these were mobile.
• The time of day.
• The training and inference time, overall speed of the DL model, and memory requirements. We examined whether authors used low-cost tools for training models, such as Google Collab (https://colab.research.google.com/, accessed on 5 November 2021). Expensive GPU or GPU farms significantly complicated the process of verifying the results presented by the authors.
• The availability of open-source code for the DL model and dataset used for training/testing. The type of license was not considered.
• The dataset type and quality.
• The camera used and the distance from the points of interest (i.e., weeds), and the number/volume of weeds captured on images. The dimensions of the camera, and whether it was installed on a vehicle, were also considered.

We did not include research where experimental studies were not reported (e.g., Kulkarni et al. [16]) or papers where only artificial conditions were presented (e.g., [17]).

For completeness, we also mention some commercial efforts for addressing the weed control problem. The company Tertill (https://tertill.com/, accessed on 5 November 2021) sells a small compact robot for the elimination of small weeds. Ecorobotix (http://www.ecorobotix.com, accessed on 5 November 2021) offers a robot that uses a DL neural network to detect weeds and destroy them using herbicides. These companies provide solutions that have been tested in operational environments (TRL7+), and they deliver an entirely commercial product, which covers user manual and product specifications but not the underlying algorithms and methods. Therefore, such products are difficult to assess because of a lack of technical information. These commercial efforts were not included in the review that follows.

1.3. Contribution and Previous Reviews

This study contributes to the application of machine vision technologies for weed detection by identifying and studying relevant works that have used neural networks (especially DL-based approaches) in real-world agricultural datasets. Several key points that determine the success of weed classification and weed control technologies, including the distribution and proximity of weeds around the crop plants, various types of occlusion, different illumination/lighting conditions, and the color/texture similarities of weeds and plants, are considered during the review.

Several similar reviews exist in this field. Most of them provide only general information, while others focus mainly on analyzing the neural network methods used rather than their application, performance, and effectiveness. For example, Gikunda et al. [18] and Jouandeau et al. [19] analyzed modern neural networks that were used for agricultural tasks. However, their review of neural networks was only partly tied to agriculture. Gikunda [20] provided an overview of DL in crop production. Kamilaris [21] highlighted the problems of detecting small objects in the scene, but possible solutions to this problem were not provided or discussed.

In contrast to the existing ones mentioned above, our review has a specific task (machine vision for weed control). Therefore, it focuses less on general agricultural problems and processes—such as general automatization, yield estimation, crops’ collection, pest management, etc. This research is intended for researchers who investigate weed classification and automatic control of weeds. Appendix A shows a block diagram on the various stages of the weed detection process that were considered in this paper.

2. A Brief Overview of DL

2.1. The History: Birth, Decline and Prosperity

Deep neural networks became popular after 2012 when the neural network AlexNet [22] won the ImageNet competition (https://image-net.org/update-mar-11-2021.php access on 5 November 2021). After that, it became customary to evaluate the accuracy of a neural network on massive public datasets (ImageNet, MS COCO dataset, CIFAR, etc.). Accordingly, various neural networks were published by their authors/creators with weights obtained while training their models based on these datasets. Most of these datasets focus on general tasks such as tracking a person, animals, transportation, etc.

DL techniques benefit from a large amount of training data, which has sufficiently been captured at all possible variations that can exist in the targeted environment (e.g., a natural environment with variation in lighting conditions), adequately covering the distribution of data. Transfer learning techniques have been widely used to train new models on several other applications where the training data were limited. Such methods can save time and labor that would otherwise be used for capturing images and manually labelling classes (annotation). However, transfer learning and fine-tuning are efficient only if the new detected objects’ classes being modelled are similar to some of the detected objects’ classes that contributed to the training process.

The need for weed recognition datasets, which could aid in accurately controlling weeds using automated means, has been in high demand. Since automatic weed detection and control can reduce time and labor while increasing productivity, there has been a significant interest in an efficient solution. Some works performed before the appearance of DL embrace various methods for increasing productivity, without any of them prevailing over the rest. For example, Liu et al. [23] proposed the combination of HSI (hue, saturation, lightness) and RGB (red, green, blue) color for weed detection. Watchareeruetai et al. [24] proposed two methods for detecting weeds on lawns using computer vision technology. The first exploited statistical differences between weed and grass areas in the edge images and used a Bayesian classifier to distinguish them. Padmapriya et al. [25], Olsen et al. [26], and Downey et al. [27] demonstrated an approach that included various steps (i.e., preliminary processing, feature extraction, and a classification stage with a pronounced background) to detect weeds with high accuracy. The information presented in these works may still be helpful for image pre-processing and feature extraction before training DL models, which can be especially important when only a limited number of images is available in the dataset.

2.2. Architecture and Advantages of CNN

Convolutional neural networks (CNNs) constitute a special architecture of DL networks proposed by Cun in 1988 [28]. They target effective image recognition based on the use of convolutions. CNNs are a set of machine learning (ML) methods based on general feature representations rather than specialized algorithms for specific tasks. While several CNNs architectures have been proposed, and their accuracy has been benchmarked on various standard public datasets, it is still difficult to determine the best performer for some specific applications such as weed detection. Wen et al. [29], Gothai et al. [30], and Su [31] presented reviews of numerous neural networks carried out in the last years for automated weed control. The authors provided general information about the state of the field of neural networks of weeds to a wide range of readers, but did not discuss how the accuracy of object detection could be approved. Li et al. [32] and Kattenborn et al. [33] explained the benefits of deep neural networks over other methods. They can recognize deeper and unexpected patterns in the data with improved performance over previous techniques used. However, the authors only gave a few examples. Mahony et al. [34] presented various benefits of employing DL for machine vision problems and concluded that, for the detection of objects in real-life, DL is the best tool. However, a disadvantage of DL is the limited ability of algorithms to learn visual relations. Wang et al. [35] published a review paper summarizing advances in weed detection using ground-based vision and imaging technologies. An important point was that the authors, besides the standard use of DL, presented methods that utilized color indices and threshold- and learning-based functions. They used four categories of biological feature morphology, spectral features, visual textures, and spatial contexts. Dhillon et al. [36], Ren et al. [37], Gorach [38], Naranjo-Torres et al. [39], and Jiao et al. [40] presented overviews of various deep architectures and models. The operation of various CNN architectures and their components were described in detail in these overviews, including popular architectures and models such as LeNet, AlexNet, ZFNet, GoogleNet, VGGNet, ResNet, ResNeXt, SENet, DenseNet, Xception, and PNAS/ENAS.

Hundreds of varieties of neural networks have been presented. In recent years, computer science researchers have tended to publish the results as preprints because the relevance of their studies quickly becomes obsolete. Consequently, the information presented remains unreviewed. Generally, neural networks should be tested on published datasets and the results, such as training time and FPS, should be reported. In most cases, the efficiencies of neural networks are tested on unpublished datasets, which makes it extremely difficult to evaluate the capabilities of the neural network in question.

The object detection efficiency largely depends on the choice of training parameters for a convolutional neural network. One of the most important points is hyperparameter optimization. Hyperparameter optimization is a machine learning task that involves choosing a set of optimal hyperparameters for a training algorithm. The settings of hyperparameters were discussed by He et al. [41], Ma et al. [42], and Aydoğan et al. [43].

2.3. DL and CNN in Generic Object Detection in Agriculture

CNN has already been employed in research in the agricultural domain. Agarwal et al. [44] developed a custom CNN model with only eight layers for the identification of tomato crop diseases. The authors proposed the use of image pre-processing by changing the image brightness after image augmentation. If the weed has a color contrast with the crop, this method can be used even on low-power processors, such as single-board computers including the Raspberry PI. The topic of the identification of crop diseases through neural networks was also considered by Li et al. [32]. The authors concluded, in line with the findings of Agarwal et al. [44], that for some tasks involving the identification of leaves, shallow CNNs are useful. Combining shallow CNN and classical ML classification algorithms is a promising and simple way to deal with the identification of plant diseases. Boulent et al. [45] reviewed papers where neural networks were used for the detection of diseases on plant leaves. The authors analyzed 19 studies that used CNNs to identify crop diseases automatically and provided recommendations to maximize the potential of CNNs.

Jiang et al. [46] performed a large survey of studies in which various CNN architectures had been employed for plant stress assessment, plant development, and post-harvest quality assessment. The authors organized the studies in their review based on technical developments resulting from image classification, object detection, and image segmentation. Noon et al. [47] reviewed different deep learning techniques for the identification of plant leaf stresses. The authors paid special attention to one of the most popular frameworks created in python, the Keras DL framework (https://keras.io/, accessed on 5 November 2021), and its ability to increase the speed of weed recognition.

Moreover, Mishra et al. [48] presented a pretrained deep convolutional neural networks (DCNN) model deployed on a RaspberryPi 3 (https://www.raspberrypi.com/products/raspberry-pi-3-model-b/ accessed on 5 November 2021) using an Intel Movidius Neural Compute Stick consisting of dedicated CNN hardware units. The DL model achieved an accuracy of 88.46% in recognizing corn leaf diseases. The authors used a single-board computer with power limitations, and therefore, they proposed that their method should be employed on more powerful machines with GPUs. Many current applications of machine vision in agriculture consider the use of stereovision and time-of-flight (TOF) cameras (e.g., Kinect v2), as described by Badhan et al. [49] and Gai et al. [50], to determine the camera-to-object distance. It is necessary to understand that the quality of the image strongly depends on the choice of the camera. A comparison of the most popular technologies for cameras (CMOS and CCD) was made by Gottardi [51], Helmers [52], and Silfhout [53]. Krishna et al. [54] presented an overview of camera frameworks in their review paper. In all the above cases, the application falls into the machine vision pipeline but is too broad to be considered in this paper.

Generally, simple/shallow neural networks and classical ML techniques were mainly used during the previous decade to model problems requiring simple classification. However, in recent years, DL-based algorithms and architectures have proven to be superior in difficult classification tasks, as illustrated by Trung et al. [55], Wang et al. [56], and Bui et al. [57]. DL methods trained on synthetic data have achieved satisfactory results on real data as well [58]. Barth et al. [59] reported better classification results in the modelling of artificial conditions (training/testing on synthetic images or controlled lighting conditions) to improve the segmentation of the details of yellow pepper. However, such methods can fail when exposed to unstructured natural environments with several variations in lighting conditions.

3. Datasets and Image Pre-Processing

3.1. Datasets for Training Neural Networks

A dataset is needed to train neural networks, and image annotation of datasets is one of the main tasks for developing a computer vision system. Neural networks can be trained via multi-dimensional data and have the potential to model and extract meaning from a wide range of input information to address complex and challenging image classification and object detection problems. However, image datasets targeted for training neural networks should contain a significant number of images and enough variation in the object classes. When tracking an object with an accuracy of a centimeter, the dataset must be marked up appropriately, which is a highly labor-intensive task in terms of the time and effort required. At the same time, whichever neural network architecture is used, it is advisable to initialize the neurons using weights from a trained model that was trained using similar/related objects/images (i.e., the transfer learning technique). Transfer learning is a way to transfer the understanding (trained weights) of a model trained especially on the large dataset for initialization on a new dataset with similar objects, or on a smaller training set. Zichao [60] used 3500 images in 12 types of weeds and trained the first 14 layers of VGG16 with the Keras framework. Chen et al. [61] collected 5187 color images with 15 types of weeds and used 27 types of deep learning models through transfer learning (Figure 1). The paper considered, in detail, many specific moments as unweighted cross-entropy (CE) loss functions. The authors conducted an impressive amount of work, and, at the same time, provided links to both the software on GitHub and the dataset on Kaggle.

Espejo-Garcia et al. [62] used five pre-trained convolutional networks (Xception, Inception-Resnet, VGNets, Mobilenet and Densenet) for detecting two weed species (black nightshade (Solanum nigrum L.) and velvetleaf (Abutilon theophrasti Medik.)). They attained a weed detection accuracy of 99.29%. The advantage of transfer learning is that we can consider other objects than weeds. In transfer learning, a model trained in plant identification can also be used to find weeds with pretrained weights (after adding the last layers, it can be trained for weed objects) Al-Qurran et al. [63] discussed the possibility of applying transfer learning to the identification of plants in images from the environment with natural backgrounds. Data augmentation is a preprocessing technique, which is becoming increasingly popular in ML for the training of neural networks that works by creating large amounts of training data with a significant variance based on smaller datasets [64]. This technique helps to create more efficient and robust models [65]. Zheng et al. [66] proposed a dataset consisting of 31,147 images with over 49,000 annotated instances from 31 different types of crops. In contrast to existing vision datasets, the images were collected from a variety of cameras and equipment installed in greenhouses (Figure 2).

Sudars et al. [67] presented a smaller dataset that consists of 1118 images, identifying six food crops and eight weed species. Pictures were taken of food crops and weeds grown in a controlled environment and the field at various growth stages. Cap et al. [68] took a completely different path and created a dataset to identify the disease on the leaves of a cultivated culture with Leaf Artifact-Suppression Super Resolution (LASSR). According to the results, the preselected novel artefact-suppression super-resolution method had higher accuracy than the GAN and the CylceGAN models [69] (Figure 3).

In general, the selection of datasets for DL model training plays a crucial role in the model’s accuracy and prediction capacity. At the same time, it is extremely difficult to predict which results a different dataset will provide. Thus, considerations such as which dataset we need and how to properly prepare and pre-process the images are quite difficult. Huang et al. [70] and He et al. [71] discussed various recommendations for creating training datasets. From their recommendations, the resolution of the images and markups in the dataset, the format of the data shapes, the size of the objects, and their relative sizes, rotation, tilt, and lighting seem to affect the accuracy of neural networks trained with these datasets. Equally important are the scales of the input images, turns, lighting from different sides, and different backgrounds. Information about popular datasets are presented in

Table 1. Popular open weed image datasets.

	Name	Size, Pixels	Plant/Object	Amount	References
1	CropDeep	1000 × 1000	31 different types of crops	31,147	[67]
2	Food crops and weed images	720 × 1280	6 food crops and 8 weed species	1118	[68]
3	DeepWeeds	256 × 256	8 different weed species and various off-target (or negative) plants native to Australia.	17,509	[26]
4	Crop and weed	1200 x 2048	Maize, weeds	2489	[72]
5	Dataset with RGB images taken under variable light conditions	3264 × 2448	Carrot and weed	39	[73]
6	Crop and weed	1200 × 2048	6 food crops and 8 weed species	1176	[63]
7	V2 Plant seedlings Dataset	10 pixels per mm.	960 unique plants	5539	[74]
8	Early crop weed	6000 × 4000	tomato, cotton, velvetleaf and black nightshade	508	[62]
9	Weed detection dataset with RGB images taken under variable light conditions	3200 × 2400	carrot seedlings with weeds	39	[73]
10	Datasets for sugar beet crop/weed detection	1200 × 2048	Capsella bursa pastoris	8518	[75]

.

Lu et al. [76] devoted one chapter to the topic of weed datasets in their review article on datasets in the agricultural sector.

3.2. Image Preprocessing

Image preprocessing (e.g., color space transform, rescaling, flipping, rotation, and translation) is a stage of image preparation that focuses on creating accurate data in the format needed for the efficient use of image processing algorithms. The advantages are that these methods do not require high computing power and these processes can be implemented in real-time on single-board computers. Faisal et al. [77] presented a good example of a typical preprocessing process, exploring the use of a support vector machine (SVM) and a Bayesian classifier as an ML algorithm for the classifying of pigweed (Amaranthus retroflexus L.) in image classification processes. With sufficiently bright color contrast, this method accurately recognizes the position of the object (Figure 4).

Image preprocessing has a broad scope with many different possible filters and different options for transforming an image. Dyrmann [78] presented a review of various filters for preprocessing but did not give enough examples with images. Preprocessing can also be used in real-time. Pajares et al. [64] provided guidance on choosing a vision system for optimal performance given adverse outdoor conditions with large light variability, uneven terrain, and different plant growth conditions. Calibration was used for color balance, especially when the illumination changed (Figure 5). This kind of system can be used in conjunction with DL. Since DL can be trained to show only one illumination, this will reduce the number of images required to train the model.

With a sufficiently large image dataset and the use of data augmentation (e.g., brightness, hue, and saturation alteration), the color balance may not be required since the neural network will already be trained for different illuminations. However, in addition to adjusting the camera settings, for example, when working through the digital camera interface (DMI) protocol in real-time, the presence of color balance can significantly enhance the results. For machine vision, tasks use simple technology, as a rule, and various filters for preprocessing with the subsequent definition of an object against a pronounced background. For example, Chang et al. [79] used image processing methods such as HSV (hue (H), saturation (S), value (V)) color conversion for weed detection. Slaughter et al. [80] used the OpenCV library for weed control with autonomous robotic systems, and Abhisesh [81] used it for robotic apple harvesting. The OpenCV library is well suited for objects with a pronounced contrast.

Despite the high efficiency of preprocessing methods in some works, these methods are not suitable for real-time use. The parameters of the preprocessing methods can be effectively adjusted to specific ones by human observation, and after that, these settings are no longer possible without human intervention to effect change in real-time. At the slightest change in the color contrast of the image, the effectiveness of the method decreases. Therefore, these methods may be suitable for analyzing yields on farms where artificial conditions for growing corn are created. Consequently, the robot can only identify weeds under these specific conditions.

3.3. Available Weed Detection Systems

The real application of machine vision systems is closely related to robotics. Controlling weeds with robotic systems is a practice that has recently gained increasing interest. In robotic systems, mechanical aspects usually require higher precision; hence, detecting weeds from machine vision becomes even more challenging. Qiu et al. [82] and Ren et al. [83] used popular models of neural networks, such as R-CNN and VGG16, for weed detection with a robotics system. Asha et al. [84] created a robot for weed control, but realized the challenge of accurate weed detection after deployment in the field due to the minor differences in size between the weed and the crop (Figure 3). When the weed was close to a cultivated crop and its leaves (e.g., when they covered each other), the robot had a problem identifying the weeds. After color segmentation, the weed was perceived as a single object together with the crop.

Chang et al. [79] and Shinde et al. [85] used neural networks for image segmentation and the subsequent identification of weeds by finding the shapes that represent the weed’s structure (Figure 6).

Raja et al. [86] proposed the marking of the crops, making them machine-readable for robotic systems. After such annotation, a robot could be trained to cut a stem with high accuracy. A drawback is that this work requires a lot of time. Additionally, several works have focused on laser technology for weed control in ideal imaging conditions, without much emphasis on the use of machine vision in real-world fields [87,88,89,90,91,92,93,94].

Unmanned aerial vehicles (UAV) have been a promising application for data collection and weed control in the field of agriculture (e.g., broad-acre farming), as demonstrated by Librán-Embid et al. [91] and Boursianis et al. [92], and, in the future machine vision in agricultural farms, may mainly be used via the UAV. The use of UAVs for creating an accurate map of weeds with DL has been presented by Huang [93], Hunter et al. [94], and Cerro et al. [95]. UAV images are not only used to obtain general information about the crop. Drones are used for patch spraying of herbicide; they are also used when precise detection of the target weeds in images is required, either in real-time or based on weed maps [96]. Rijk et al. [97] used a drone with the OpenCV library for image processing in real-time. Liang et al. [98] presented an automated image classification system based on a CNN model that relies on simple imaging tools to spray herbicide on patches of weeds in fields.

The development of robots to control weeds is a complicated and time-consuming task that includes mechanics, weed identification, and weed control systems working together. Selecting a neural network, collecting datasets, training the neural network, and setting up hyper-parameters takes a lot of time. Therefore, authors often prefer already existing proven and previously used tools that are available on the market.

4. DL for Weed Detection

This section analyzes the research papers identified through the criteria set out in Section 1.3. First, relevant applications of machine vision for weed detection in agronomic automation are broadly mentioned. Then, Section 4.1 presents studies that have used neural networks and DL for object detection in relevant agricultural applications.

4.1. The Curse of Dense Scenes in Agricultural Fields

Dense scenes are the ones where there is dense vegetation, including both crops and weeds, and large occlusion between weeds and crops. Zhang et al. [99] presented a review of DL for dense scenes in agriculture. The purpose of this study is illustrated in Figure 7a, presenting a cluster of fruits on a dense background (including trees from other rows). As a result, the authors give several general recommendations such as increasing the dataset, generating synthetic data, and setting hyper-parameters. They concluded that it is advisable to continue increasing the depth of neural networks to solve problems with dense scenes. Assad et al. [100] focused on weeds (Figure 7b,c). The authors used semantic segmentation when conducting DL with a ResNet-50-based SegNet model. As a result, they attained a Frequency-Weighted Intersection Over Union (FWIOU) value of 0.9869, but the authors did not consider situations where the weed and the crop came into contact; in such cases, this method will not be efficient.

Proper annotation of the training dataset was crucial to detect the objects precisely. Bounding box-based detection methods used several IoU (intersection over union) and NMS (non-maximum suppression) values for model training and inference. Some segmentation methods can be used as well, with blob detection and thresholding techniques for the identification of individual objects in clusters. Identifying individual objects in clusters is a problem in various fields of activity, and much research has been devoted to solving it. Shawky et al. [101] and Zhuoyao et al. [102] considered the problem of dense scenes and concluded that each situation is unique, and it is advisable to combine various convolutional neural networks with each other. However, it is very difficult to predict the result in advance because such a method may take a lot of time. Dyrmann et al. [78] considered an object as a combination of different objects, depending on their position, which made the model a little heavier but, at the same time, increased the tracking accuracy. This method is one of the most promising.

4.2. State-of-the-Art Methods in Weed Detection

This section presents studies that used deep neural networks for object detection in agricultural applications focusing on weed detection. Works that explored practical application of neural network models for real-time agricultural automation, focusing on the precise detection of weeds in natural environments, are considered. Recommendations are made on the type of DL networks and their suitability for practical farm applications.

Liakos et al. [103] provided a comprehensive review of research on the application of ML in agricultural production systems in all its aspects, including both vehicle management and machine vision systems. However, the authors focused more on static data on the use of neural networks than on the technical information needed to implement a neural network. Similarly, Hasan et al. [104] reported on existing methods of weed detection and classification based on DL. The authors considered only 70 papers, but all papers were considered using the same criteria. In the conclusion, they explained common ideas related to the use of DL in the field of agriculture.

Many scientists have used DL to detect a single feature on a specific background. Rehman et al. [105] considered the role of CV only in fruits and vegetables among various horticultural products in agricultural areas with statistical machine learning technology. Similarly, Osorio et al. [106] considered a DL approach for weed detection in lettuce crops, while Ferreira et al. [107] considered weed detection in soybean crops. It is critical that the authors only used a limited dataset from one field. Therefore, the results of the neural network presented in the work have high accuracy, but at the slightest change in the detection condition (e.g., a new field or a change of season), the accuracy may change. As a rule, the results of such works can be used only under consistent conditions, which very rarely happens.

Santos et al. [108] presented a review of several DL methods applied to various agricultural problems such as disease detection, the classification of fruits and plants, and weed classification. However, the review was very short and limited to a small number of studies.

Dokic et al. [109] concluded that DL neural methods were better than classical ML methods for plant classification in images. The analysis of deep neural networks in the context of agriculture is not fully disclosed because problems with dense scenes have not been solved. A review by Tian et al. [110] considered practical applications of machine vision in agriculture. Their analysis showed that computer vision could help in developing agricultural automation for small fields in order to achieve benefits such as low cost, high efficiency, and accuracy. However, more emphasis was placed on the automation process than on machine vision. Khaki et al. [111] proposed the use of a CNN to detect and count corn kernels in images. The authors trained different models for the detection of objects at high speed under various conditions, lighting, and angles. The authors applied the standard sliding window approach to detecting the kernel, obtaining high accuracy due to the correct annotation of the dataset (Figure 8).

Osorio et al. [106] classified weeds on images by employing architectures such as SVM, YOLOv3, and Mask R-CNN, and F1 scores of 88%, 94%, and 94%, respectively, were achieved for weed detection. F1 is a harmonic mean between precision and recall values for a model. Dunn’s test was introduced to obtain statistical measurements between each assessment (man vs. machine). They showed that the DL models could improve the accuracy of weed coverage estimates and minimize human bias. Yu et al. [112] used several models of DCNN for the detection of Bermuda herbs. The VGGNet model that was used achieved the highest F1 scores (> 0.95) and outperformed the GoogLeNet model in detecting weeds. The authors provided various tips to improve the detection accuracy for each DL model used in the study.

Asad et al. [100] used DL meta-architectures such as SegNet and UNET, and encoder blocks such as VGG16 and ResNet-50, to detect weed plants in canola fields. The ResNet-50-based SegNet model performed best, with a mean crossover at the pooled value of 0.8288 and a frequency-weighted crossover over the pooled value of 0.9869. Yu et al. [113] analyzed several deep convolutional neural networks (DCNNs) to detect dandelion (Taraxacum officinale Web.), Ground ivy (Glechoma hederacea L.), and euphorbia (Euphorbia maculata L.). They considered GoogleNet, DetectNet, and AlexNet. From those, DetectNet had the highest F (≥0.9843) in test datasets for weed detection.

Gao et al. [114] significantly improved the accuracy of the neural network by developing their neural network based on YOLOv3 and Tiny YOLO. Specifically, the average accuracy for the detection of hedge bindweed (Calystegia sepium (L.) R.Br.) and sugar beet were 0.761 and 0.897, respectively, for images of 800 × 1200 pixels. Such a redesign of existing standard object detection frameworks for specific applications has good potential for improved accuracy and speed.

Scott et al. [115] analyzed two models, the Faster R-CNN and the Single Shot Detector (SSD), for weed detection using mean intersection over the pool (IoU) and inference speed. The authors showed that the Faster RCNN with a 200-box proposal had the same weed detection performance as the SSD model in terms of accuracy, recall, F1, and IoU score, as well as similar hatch times. They concluded that Single-stage (one-shot) object detectors (e.g., SSD and YOLO) provided better detection speed (real-time) but were considered less accurate as compared to double-stage detectors such as Faster R-CNN

Narvekar et al. [116] developed a prototype of the CNN model architecture for the problem of classification of flower species with a dense overlap between flowers and weeds. The authors compared the result of transfer learning across the VGG16, MobileNet2, and Resnet50 CNN architectures.

Sharma et al. [117] used image segmentation to train CNN models. As a result, the S-CNN model was trained using segmented images and showed an accuracy of 98.6% when classifying ten classes of plant diseases. Object/instance segmentation was used to draw a free shape (polygon) around the target objects. Such segmentation methods are useful in cases where there is a high degree of object overlapping, which causes the common method of drawing rectangular bounding boxes to be inappropriate.

An overview of the methods and DL techniques used to identify weeds and crops is shown in Appendix B.

5. Technical Aspects

This section discusses various technical aspects considered in the state-of-the-art work presented in this review. Such elements include models and architectures that have been employed, methods used to improve performance, methods used to detect small objects, and complexity vs. processing.

5.1. Models and Architectures

The number of neural network models and architectures proposed for solving the task under study is increasing quickly due to the large interest in employing DL among scientists in this field. Therefore, we only present an analysis of the latest, most popular, and successfully used DL neural networks, which have shown promising, good results for datasets used by the authors for training and/or testing.

Du et al. [118] described a SpineNet model introducing multi-scale features via convolutional layers of mixed sizes. The model was originally developed to classify medical images, but is also used in the agricultural field due to its excellent efficiency. Koh et al. [119] successfully used this model for high-throughput image-based plant phenotyping. Shah et al. [120] described AmoebaNetalso, which refers to algorithms used for the automatic creation of neural networks. AmoebaNet uses evolutionary algorithms instead of reinforcement learning algorithms to automatically find optimal neural network architectures.

Yao et al. [121] used SM-NAS neural networks for high accuracy object detection, which are models that offer a two-step coarse search strategy called Structural-to-Modular NAS (SM-NAS). The first stage of the search, at the structural level, was aimed at finding an effective combination of different modules. The second stage of the search, at the modular level, developed each specific module. Jia et al. [122] described the CenterNet and CornerNet-Lite lightweight real-time object detection systems. Zhao et al. [123] used this system for fruit detection from digital images. As a result, this system showed the best results compared with ResNet-18, DLA-34, and HourglassNet. CenterNet models the object as a single point in the center of the bounding box. The size of the object is retrieved at a second phase through the image features. An input image is fed into the neural network, and the neural network generates a heatmap. The peaks in this heatmap correspond to the centers of the objects. The image characteristics at each peak in the heatmap predict the size of the bounding box around the object.

Xu et al. [124] considered SegNet, FCN, and U-Net for weed image segmentation at the seedling stage in paddy fields. Kong et al. [125] successfully used MCF-Net for crop species recognition in precision agriculture. Wosner et al. [126] used the EfficientDet neural network for object detection in agriculture. The general architecture of EfficientDet represents the one-stage detector paradigm. EfficientNet, pre-trained on ImageNet, was taken as a basis.

Some of the most popular neural networks for tracking objects in real-time are YoloV3, YoloV4, and YoloV5. Most articles concerning the detection of weeds or crop classification in agriculture used these models. For example, Wu et al. [127] used YoloV4 to accurately detect apple flowers in real-time. Kuznetsova et al. [128] considered YoloV5 in comparison with YoloV3 for apple detection. General information about YOLOV3 was presented by Tian et al. [129] and Wu et al. [130]. The open-source code of YOLO is regularly maintained, with various features continuously added. YOLO models have been widely used in real-time applications because of their well-maintained repository and documentation, and the availability of light, medium, and heavy models for speed–accuracy trade-offs.

5.2. Future Directions

Popular DL models have been designed for a wide range of tasks. Therefore, they can detect and classify many different objects. Summarizing previously reviewed papers, we can conclude that deep neural networks do not currently perform well on challenging tasks, such as finding weeds in dense scenes and accurately detecting their position. Using popular DL models without any modifications/adaptations makes detecting the position of a weed plant challenging.

Waheed et al. [131] and Atila [132] gave general advice, such as using transfer learning when all layers of the models were trained, to improve the accuracy of CNN, and used corn leaf diseases as an example. In addition, these recommendations were implemented by Pang et al. [133], who determined early-season corn stands using geometric descriptor information and deep neural networks.

Liang et al. [134] proposed the use of linear-phase point-convolution kernels (LPPC kernels) to reduce the computational complexity and storage costs of convolutional neural networks. This method was partially implemented by Taravat et al. [135] for agricultural field boundary detection. Isufi et al. [136] developed a custom DL model to study joint convolutional representations from the nearest neighbor and the graph of the farthest neighbor. Wei et al. [137] used this method to increase accuracy in the slice positioning method for the laser cutting of corn seeds.

Another option for increasing the accuracy of a deep neural network is to combine several types of networks (ensemble techniques). For example, Koo et al. [138] proposed a hierarchical classification model for CNN fusion to extract hierarchical representations of images. The authors applied residual learning to the RNN part to make it easier to train the composite model, and finally improved the model’s generalization. The experimental results showed that hierarchical networks perform better than modern CNNs. Agarap [139] combined neural networks and ML by combining CNN and SVM for images. This combination achieved a test classification accuracy of ≈99.04% using the MNIST dataset. Khaki et al. [140] used these techniques to create a DL model, which demonstrated the ability to generalize yield forecasts from untested media without significantly reducing the forecast accuracy.

Finally, proper annotation of the training dataset was crucial to detect the objects precisely. Bounding box-based detection methods used several IoU (intersection over union) and NMS (non-maximum suppression) values for model training and inference. Some segmentation methods can be used with blob detection and thresholding techniques for the identification of individual objects in clusters. Identifying individual objects in clusters is a problem in various fields of activity, and much research has been devoted to solving it. Dyrmann et al. [141] considered an object as a combination of different objects, depending on their position, which made the model a little heavier, but at the same time increased tracking accuracy.

5.3. Detection of Small Objects

Identifying and separating weeds from crops are processes that require high accuracy in object detection. Here, we highlight papers where the task was to detect small objects. Barth et al. [142] showed that with a large dataset with high image resolution (Nikon D7200 with a resolution of 4000 × 6000 pixels), it is possible to achieve high accuracy for real-time sugar beet and weed counting using the proposed deep neural network. Nguyen et al. [143] considered the possibility of tracking small objects in a filtered dataset from PASCAL VOC 2007, using the Fast RCNN, Faster RCNN, RetinaNet, and YOLOv3 DL models. They concluded that the deeper the architecture was, the higher the accuracy of detection achieved. Chen et al. [144] compiled a reference dataset adapted for detecting small objects to better assess the performance in the detection of small objects. The authors supplemented the state-of-the-art R-CNN algorithm with a contextual model and small area suggestion generator to improve the performance of small object detection.

Yu et al. [145] presented a mask-region convolutional neural network (Mask-RCNN) to detect strawberries using a robot. Resnet50 was adopted as the underlying network, combined with the architecture’s Pyramid Network (PN) feature for feature extraction. It was concluded that segmentation is especially useful for identifying objects in dense clusters and correctly calculating the gripping position. Finally, Boukhris et al. [146] trained Mask-RCNN to automatically detect small lesions on leaves and fruits, locate them, classify their severity, and visualize them.

5.4. Complexity vs. Processing Capacity

As the depth of the network increases, the number of layers increases, and the network requires more parameters for training; thus, longer training times and more processing capacity are needed. Initially, up to VGG Net, it was found that the accuracy increased with depth. However, the vanish gradient increased with the increasing of the depth, and then the model was not efficiently trained [147]. Afterwards, ResNet was produced using residual networks, and GoogleNet was developed using a similar technique (inception blocks) to flow information up to the end layers for very deep neural networks. Later, more accurate models were created, increasing width and not just depth. A logical solution for improving the accuracy of the neural network is to use more layers. However, heavy networks can no longer run on low-cost single-board computers (RaspberryPI, OrangePI, etc.), which can be conveniently installed directly on a vehicle. For example, Jetson Nano on YoloV3 can only detect a few frames per second [148,149]. The following possibilities are available for using deep neural networks on single board computers:

• DeepStream SDK—software that allows the use of multiple neural networks to process each video stream, making it possible to apply different deep ML techniques;
• The AWS IoT Greengrass Platform, which extends AWS Edge Web Services by enabling them to work locally with data;
• The RAPIDS suite of software libraries, based on the CUDA-X AI, makes it possible to work continuously, complete data processing, and analyze pipelines entirely on GPUs;
• Google Colab is a similar service to Jupyter-Notebook that has been offering free access to GPU Instances for a long time. Colab GPUs have been updated to the new NVIDIA T4 GPUs. This update opens up new software packages, allowing experimentation with RAPIDS at Colab;
• NVIDIA TensorRT is an SDK for high performance DL output. It includes a DL inference optimizer and runtime that provides low latency and high throughput for DL inference applications. TensorRT is a very promising direction for single-board computers because we can obtain 39 FPS by using tkDNN + TensorRT [150] with Jetson Nano. tkDNN is a deep neural network library built with cuDNN and tensorRT primitives specifically designed to run on NVIDIA Jetson boards. This requires a conversion of Darknet weights to TensorRT weights using the TensorFlow version of YOLOv4 (https://github.com/hunglc007/tensorflow-yolov4-tflite#convert-to-tensorrt, accessed on 5 November 2021) or the Pytorch version of YOLOv4 (https://github.com/Tianxiaomo/pytorch-YOLOv4#5-onnx2tensorrt, accessed on 5 November 2021).

5.5. Limitations

In this survey, it is worth noting that all the criteria set for comparing works were impossible to fulfill, because some authors explained their design decisions only partially, or employed a wide variety of performance metrics on their own datasets, which are not publicly available for fair comparisons. More fairness should be achieved when comparing work in the field of machine vision and agriculture by considering popular, open, and public datasets based on well-accepted and well-understood metrics for assessment. Another critical problem in the surveyed papers was the lack of links to/information on initial sources: neural network codes, trained weights, and datasets. More than 95% of the papers did not provide any link to the initial source. In the absence of source code, it is not possible to practically test and apply the results presented in the manuscripts. Direct comparison of papers is complicated because the assessment of the effectiveness of neural network models is vague. A neural network can be evaluated for the accuracy of detection (classification), for the FPS rate, for the volume of weights (relevant for single-board computers), the required number of images for training, and others. Furthermore, many authors agree that the main indicator is the accuracy of determining the position of the detected object and often cite this parameter to prove the effectiveness of the model they have proposed. It is worth noting that the accuracy depends on many factors such as the setting of the model parameters, the quality of the labeling, the number of images, and the background on which the plant is located. The comparison of models requires that the models are checked on the same or similar datasets, and it is essential to have full access to the original materials. In the reviewed papers, the characteristics of the camera were not fully disclosed. For example, the OV7725 camera, which is popular in machine vision, has about a hundred registers, which can be changed in real-time.

6. Conclusions

This paper performed a survey on the state-of-the-art methods used for weed identification in agricultural fields, which robots can use for effective weed control. Various methods and techniques were reviewed, focusing on shallow neural networks and DL, and also mentioning different approaches from machine vision that have been applied in the field. Our findings indicate that DL performs better than traditional machine vision-based approaches. Authors of related work in agriculture, especially with respect to weed detection and elimination, are recommended to embrace modern approaches based on neural networks and DL because of the overwhelming number of papers reviewed that show the superiority of deep neural networks in such machine vision problems.

From our observation in this review, it seems logical to consider multifunctional adaptive unions to improve the performance of a neural network. The structural synthesis of multilayer neural networks will help in the efficient use of spatial information by giving different weights to different layers of objects. CNN architectures are combined well with other DL algorithms such as RNN, showing better results. Regarding DL architectures, the most popular neural network seems to be Yolo, but for practical use, the position of the weed must still be precisely determined. As we have seen, due to various frameworks and libraries, the line between single-board computers and computers with powerful graphical interfaces gradually decreases. Therefore, deep neural networks are expected to be increasingly used in real-time weed recognition in the future.

We conclude that currently, there is no neural network that is ideal for determining the exact position of the weeds for real-time weed detection. This is an ongoing challenge, with more solutions expected to come in the near future. We encourage scientists to continue their efforts in the field, aiming to solve the problem efficiently, accurately and without expensive resources.

We recognize that more advanced techniques and promising architectures might appear soon, making the proposed methodology obsolete. Finally, we intend to experimentally investigate the proposed recommendations for offering a complete weed detection and control system, identifying the exact position of weeds via DL and then killing them with lasers or robotic arms under realistic, real-world conditions.