Automatic Detection and Monitoring of Insect Pests—A Review

Abstract

Many species of insect pests can be detected and monitored automatically. Several systems have been designed in order to improve integrated pest management (IPM) in the context of precision agriculture. Automatic detection traps have been developed for many important pests. These techniques and new technologies are very promising for the early detection and monitoring of aggressive and quarantine pests. The aim of the present paper is to review the techniques and scientific state of the art of the use of sensors for automatic detection and monitoring of insect pests. The paper focuses on the methods for identification of pests based in infrared sensors, audio sensors and image-based classification, presenting the different systems available, examples of applications and recent developments, including machine learning and Internet of Things. Future trends of automatic traps and decision support systems are also discussed.

1. Introduction

Environmental concerns about the use of pesticides, overexploitation of natural resources, expansion of global trade, increasing human population, changes in consumption patterns, and advances in technology are thriving, leading to a new revolution in agriculture. This revolution consists of the use of digital tools to increase productivity, optimizing the management of natural resources and agricultural inputs concurrently [1]. Precision agriculture tools enable farmers to analyze the spatial-temporal variability of several key factors that affect plant health and productivity. Obtained through sensors, these data are stored and combined in digital platforms in order to guide the decision-making process [2]. Integrated pest management (IPM) systems are being developed in order to improve the management of insect pests, reducing the overall use of pesticides and focusing on more precise applications. However, the efficiency of these systems depends on the accuracy of the chosen pest population monitoring method. In addition, it is essential to gather information of population dynamics and their associated ecological factors in order to develop an appropriate pest control strategy [3]. Recently, modern technologies started to be applied in field surveys of several pests, such as radar technologies monitoring pest migration, video equipment to observe flying insects, thermal infrared imaging and chemiluminescent tags for tracking insect movement in the darkness, Global Navigation Satellite System(GNSS )for wildlife telemetry, habitat mapping and echo-sounding detection of larvae movement [3]. Nevertheless, these techniques are usually high-cost and not affordable for farmers. More recently, advances in miniaturized sensors, microprocessors, telecommunications engineering and digital processing techniques allowed the reduction in costs in novel insect automatic detection and monitoring systems. These new devices can be easily connected to the internet, allowing real-time surveillance on the field level. Some of these devices can be connected to wireless sensors networks (Internet of Things) for monitoring field areas and/or use cloud-computing services to help in the decision-making process. This paper covers an in-depth review on recent publications in the field of insect automatic detection and monitoring. Furthermore, this review intends to highlight the methods used for different orders of insects, including Lepidoptera, Diptera (fruit flies), Coleoptera (weevil in palm trees) and sucking insects belonging to different orders of insects, considering that the majority of relevant publications were focused on them.

2. Automatic Monitoring of Lepidoptera Pest Species

A wide range of moth and butterfly species, such as gypsy moth Lymantria dispar, codling moth Cydia pomonella, diamondback moth Plutella xylostella, are known to cause significant yield losses in many crops worldwide [4,5]. These insects are able to oviposit a large amount of eggs, and the larval stages feed voraciously, causing direct defoliation, thus leading to huge losses when populations are well developed. Currently, the usual surveillance method is based on delta traps with pheromone lures. However, due to the multiple poses that these insects can display when attached to sticky traps, the development of automatic detection and identification models is quite challenging [6,7].

Silveira and Monteiro [8] developed a tool to automatically identify eyespot patterns of the nymphalid butterfly Bicyclus anynana, using a machine learning algorithm with features based on circularity and symmetry to detect eyespots on the images. The software was also able to successfully recognize patterns of other butterfly species.

For the identification of species based on images, Wen et al. [6] used a suitable combination of shape, color, texture and numerical features extracted for moth description. Later, a pyramidal stacked de-noising auto-encoder (IpSDAE) was proposed to generate a deep neural network for moth identification regardless of the pose of insects. This model reached a level of 98.13% of moth identification at genus level without classifying the species. Guarnieri et al. [9] created an automatic model for codling moth Cydia pomonella monitoring. Using a modified trap at field level using mobile phone cameras of different resolutions for remote visual inspection, they reached up to 100% efficacy when compared to local visual inspection.

Many models were developed using artificial neural networks (ANNs) for species identification. ANNs are computational models inspired in biological neural networks that can be trained to perform different tasks, such as identifying patterns in images. Kaya et al. [10] combined ANNs with binary patterns to identify five butterfly species of the family Papilinidae, whilst Wang et al. [11] combined ANNs with a series of morphological features and a support vector machine (SVM) to develop a high efficacy system (93%) for the identification of over 200 species from 64 families of different insect orders including Coleoptera, Hemiptera, Hymenoptera, Lepidoptera, Megaloptera, Neuroptera, Odonata and Orthoptera. Kang et al. [7] developed a model for butterfly identification based on their shapes from different angles, having efficacy ranging between 89% and 100% for 15 species of different families, and Kaya and Kayci [12] proposed a method using color and texture features.

The use of texture descriptors, especially the gray level co-occurrence matrix (GLCM) in the context of machine learning showed to be useful in the identification and monitoring of Lepidoptera species. Kayci and Kaya [12] reached 96.3% accuracy in the identification of 19 species belonging to the family Pieridae.

Kaya et al. [13] applied a texture gabor filter-based and extreme machine learning model for butterfly identification of five species with an accuracy of 97%.

Applying deep convolutional neural networks and deep learning techniques on three publicly available insect datasets, Thenmozhi and Reddy [14] were able to identify several species of Lepidoptera, as well many Coleoptera and Orthoptera with accuracy varying from 95% to 97%.

Using histograms of multi-scale curvature (HoMSC), gray-level co-occurrence matrix of image blocks (GLCMoIB) and weight-based k-nearest neighbor classifier, Li and Yin [15] reached 98% efficacy in the identification of 50 lepidopteran species.

Different approaches were conducted by two studies [16,17] that developed a mobile robot car with camera for real-time identification of Pyralidae species on field level. Applying the Gaussian Mixture Model (GMM), Aggregation Dispersion Variance (ADV) and Distance Regularization Level Set Evolution (DRLSE), Zhao et al. [17] were able to identify the target species with an accuracy of 95%. Liu and colleagues [16] designed a two-step recognition, using first a colour space (HSV) in which candidates were evaluated by applying Otsu segmentation thresholds, and secondly an object contour recognition procedure was performed based on Hu moments; the model was able to detect Pyralidae species with 94.3% accuracy, superior to the support vector machine method.

Commercial solutions are produced by EFOS (Figure 1) based on cloud computing image processing. The model integrating bucket/funnel type of trap is especially suitable for larger moth species that come in high numbers. It can remotely identify Helicoverpa armigera, Autographa gamma and Spodoptera spp. (Figure 1a).

3. Automatic Monitoring of Sucking Insects

Sucking pests are among the critical factors causing losses in greenhouse environments. Thrips, aphids, and whiteflies are recognized as some of the most problematic pests when crops are cultivated in congested conditions in closed areas. Usually in greenhouses, sticky traps are placed in order to monitor the populations of these pests. This method can be considered a difficult task for not specialized professionals regarding their size, the complex morphology and the low efficiency score of fatigued or unskilled human observers [18].

Several studies were conducted for automatic identification of greenhouse sucking pests. For the identification of the greenhouse whitefly Trialeurodes vaporariorum at the mature stage, models applying computer vision techniques were proposed by a few authors [19,20]. Using threshold algorithms, Bodhe and Mukherji [21] developed a system to detect and count whiteflies using image analysis. Applying texture and shape analysis, Ghods and Shojaeddini [22] created an algorithm that could identify whiteflies in plant leaves with 85% accuracy. Blasco et al. [23] developed a prototype to monitor insect traps placed in the field by capturing and sending images of the trapped insects to a remote server. The device was created on the basis of a Raspberry Pi platform and incorporates a camera to capture the images, a control board to program the image capture intervals and a modem to send the images and additional information to a remote media server.

Applying noise removal, contrast enhancement techniques and k-means, Dey et al. [24] used statistical feature extraction methods such as GLCM and gray level run length matrix (GLRLM) before the use of classifiers like support vector machine, Bayesian classifier, artificial neural network, binary decision tree classifier and k-nearest neighbor to distinguish white fly pest infested from healthy leaf images. All these classifiers present a high accuracy (90–98%) and high sensitivity (93.9–98.8%).

Using a support vector machine, Ebrahimi et al. [25] developed a model to detect and classify multiple stages of whitefly. Automatic models based on color transformations were proposed for counting and measuring whiteflies in soybean leaves [26], in different leaves [27], and in yellow sticky traps [28,29].

With a more practical IPM-based approach, multiple techniques were developed. Using samples of yellow sticky traps from greenhouses, Qiao et al. [30] proposed a model for density estimation of the silverleaf whitefly Bemisia tabaci based on image processing system. In this work, the system proved to be more efficient with medium and high densities of the pest.

For score and identification of Bemisia tabaci in yellow sticky traps and the western flower thrips Frankinella occidentalis on blue sticky traps, Sun et al. [31] applied a novel smart vision algorithm using two-dimensional Fourier transformed spectra, finding high correlation with human vision accuracy. Similar results were obtained by Solis-Sanchez [32].

Combining image processing (segmentation, morphological and color property estimation) and ANN, Espinoza et al. [33] proposed a model for the identification and monitoring of Frankinella occidentalis and Bemisia tabaci with high precision (96%). Bauch and Rath [34] used shape and colour properties for the identification of Bemisia tabaci and Trialeurodes vaporarium, achieving 85% efficacy.

Furthermore, Lu et al. [35] proposed a convolutional network (CNN) classifier model in combination with a generative adversarial network (GAN) image augmentation. For this method, a Raspberry Pi v2 camera was used, both whiteflies and thrips were analyzed, and synthetic images were created through the GAN-based data augmentation method, in order to enhance CNN classifier with limited image data. This method resulted in a precision of a range between 85% and 95%.

Seeking out a way to process images with a low computational cost, a model fitting embedded system for the detection of aphids, whiteflies and thrips was proposed by Xia et al. [18], whilst Xuesong et al. [36] developed a model for the counting and identification of aphids based on machine learning and an adapted smartphone. In yellow sticky traps placed in greenhouse conditions, this method showed 95% accuracy and 92.5% when placed outside. Liu et al. [37] developed a model that uses a maximally stable extremal region descriptor, and then used histograms of oriented gradient features and a support vector machine for identification of wheat aphids in field conditions. This new method provides an 86.81% identification rate. In addition, Li et al. [38] developed a convolutional neural network (CNN) of Zeiler and Fergus model and a region proposal network (RPN) with non-maximum suppression (NMS), achieving a precision of over 88.5%.

Using leaf samples, Maharlooei et al. [39] designed a model for identification of soybean aphid (Aphis glycines) based on image processing techniques with different types of cameras and two illumination conditions and compared results with human counting. The best results were obtained with low illumination and Sony camera with 96% accuracy. Also using leaf samples (of pakchoi), Chen et al. [40] proposed a model for the segmentation and counting of aphid nymphs. This system showed a high accuracy (99%) when compared to human counting, but it is not selective and can be used for other pests. As a limitation, this and some of the systems described appear just to count and not distinguish between the whiteflies or aphid species.

Using sticky traps, a camera, a temperature sensor and an ambient light sensor, Rustia et al. [41] developed a RGB to LUV color model conversion in order to extract the V-channel color component, achieving an accuracy of between 90% and 96%.

Furthermore, Gutierrez et al. [42] conducted a study for monitoring and identifying whiteflies. For this purpose, two cameras, a dataset generator and two microcontrollers were used in combination with a K-nearest neighbor (KNN) and multilayer perceptron (MLP), resulting in an accuracy of between 66% and 81%.

4. Automatic Identification and Monitoring of Fruit Flies

One of the biggest challenges of horticultural and fruit production in the Mediterranean, tropical, and subtropical areas of the world is the frugivorous fruit flies (Diptera: Tephritidae) [43]. This group of pest causes crop losses amounting to billions of dollars each year worldwide, totaling USD 242 million/year in Brazil alone [44]. Depending on the crop and on the lack of control methods, fruit fly damages can lead to from 80% to 100% of crop losses [45]. The fruit fly belongs to the tribe Dacini and have 932 recognized species, in which about 10% are currently recognized as pests of commercial fruit and vegetable production, causing quarantine issues and trade embargos [46]. Females of Ceratitis capitata (Widemann), Bactrocera Dorsalis (Hendel) and Bactrocera oleae (Gmelin), among other species, exhibit high reproductive rates. These insects lay eggs below the fruit surface and when they hatch, the larvae feed inside the fruit. For this reason, the fruit drops or loses quality [45].

The global fruit fly issue is intensified by a small group of highly polyphagous and highly invasive pest species that competitively dominate the local fauna if they enter and establish in a region [47,48,49]. These pests can be controlled with pesticide sprays, mass trapping containing pheromone lures [50], and by the release of sterile males [51]. The efficiency of the control method depends on the time of application, being more effective when the pest is detected early. Due to a low threshold of control of these pests, an automatic detection system will be useful to prevent and monitoring the infestation of these quarantine pests in a faster way.

In order to improve the operation of a low-cost McPhail trap, Potamitis et al. [47] inserted optoelectronics sensors to monitor the entrance of the pests and identify the species of incoming insects from the optoacoustic spectrum analysis of their wingbeat. With this system, it was possible to distinguish fruit flies from other insects with 91% accuracy, but not between fruit fly species. Improving this system, Potamitis et al. [52] presented a novel bimodal optoelectronic sensor based on a stereo-recording device that records the wingbeat of an insect in flight and Fresnel lens. This system was able to distinguish between Ceratitis capitata and Bactrocera oleae with 98.99% accuracy (Figure 2 and Figure 3).

Another system was developed to identify different species of the Anastrepha group based on image datasets of wings and aculeus, a specialized ovipositor structure. Faria et al. [53] used a multimodal fusion classifier approach to distinguish images of three species: Anastrepha fraterculus, Anastrepha obliqua, Anastrepha sororcula. In these experiments, the fuzzy support vector machines (FSVM) multimodal approach could account for 98.8% of classification accuracy in the laboratory conditions. In a more applied study, Doitsidis et al. [54] modified a McPhail trap with a camera connected to a Web system that automatically counts the Bactrocera oleae and provides the images to expert entomologists that can remotely assess the potential threat at any time and rate, reducing the need for visiting and collect data on site. Okuyama et al. [55] applied an automatic count system to monitor the dynamics of Bactrocera dorsalis at the field level using an automatic fly census developed by Jiang et al. [3]. This automatic fly census system reports the environmental conditions and the pests in real time. It is composed of an infrared device counting the insects that enter the trap and recording environmental data (remote monitoring platform—RMP) (Figure 2). The device also has an external host control platform that receives data from the RMP through a short cell phone message (GSM).

In order to improve this system and develop a more accurate and precise device, Liao et al. [56] designed a monitoring system built on two different wireless protocols: GSM and ZigBee, with three major components: remote sensing information gateway (RSIG), a host control platform (HCP) and wireless monitoring nodes (WMNs). The WMNs transmit the collected data (relative humidity, illumination, temperature and the number of Oriental fruit flies captured) to the RSIG, and the RSIG delivers the data to the database server (HCP) for storage and analysis. The server can process the data and classify the information after analysis in three event types: a normal status event, a pest outbreak event and a sensor fault event, which can be accessed through an online platform (Figure 4).

Another electronic trap for the monitoring of adult fruit flies was developed by Shaked et al. [57] and extensively tested at the field level. One of the traps was based on specific volatiles for male and female adult Ceratitis capitata, and another on the attraction of the adults to the color yellow in order to capture Bactrocera oleae, Dacus ciliates and Rhagoletis cerasi. These traps were image based. Real-time images of the surface of the traps were taken automatically and sent to a server. From the office, the entomologists could classify fruit flies from images with an accuracy superior to 88%. These traps showed good specificity among the different fruit fly species, and do not differ from the results of conventional traps that were operating simultaneously.

A different approach was proposed by Haff et al. [58] using hyperspectral images. It was possible to identify spots caused by fruit fly larvae in mangoes fruits in post-harvest on line with 87.7% accuracy. This approach used image processing parameters such as gaussian blur radius, ball radius, threshold for binary conversion, and minimum particle size.

A commercial automatic trap with a high-resolution camera is manufactured by TrapView, EFOS, Slovenia, for the monitoring of Mediterranean fruit fly (Ceratitis capitata) in citrus and peaches and spotted wing drosophila (Drosophila suzukii) in fruits and grapes (Figure 1b).

5. Automatic Monitoring of Weevil in Palm Trees and Other Borer Insects

The Rhynchophorus ferrugineus or red palm weevil (RPW) is a key pest of palm species in the Mediterranean region, Middle East, Asia and North Africa. It is one of the most important invasive pests worldwide, being considered the single most destructive pest in 40 species of palm trees. RPW has an important socio-economic impact; present in more than 60 countries, this pest is reducing date production and destroying ornamental palms, causing economic losses of millions of dollars each year [59]. The RPW larvae are located deep inside of the palm crowns, offshoots and trunks, causing considerable damage without being visually detected [60]. The intensive trade of palms without visual symptoms was one of the reasons for spreading the pest worldwide. The early detection of the RPW is the key of success to control and eradicate this pest. The difficulty in controlling with insecticides requires an early detection method for minimizing the losses.

Currently, the use of pheromone traps and human inspection are the most widely and effectively used techniques. Recently, the use of entomopathogenic nematodes in green areas inside Spain have presented successful results in controlling the pest. In addition, the overall efficiency and speed of detection can be increased by using promising technologies in automatic detection such as thermal cameras and acoustic detectors, in order to develop an easy-to-handle, cost-effective, quick and reliable device for early detection of RPW [59].

The acoustic activity of Rhynchophorus ferrugineus was studied in detail by several researchers [52,61,62,63] that concluded that the sound produced by the pest can be isolated and differentiated from environmental sounds and other insects. Dosunmu et al. [64] collected RPW sounds inside the palm trees in different conditions and create a model based in the pattern and frequency spectra of the emitted sounds. In their study, the RPW could be differentiated from Chamaemyia elegans based on the burst duration, and the detection of the larvae in the offshoots was easier than inside the trunk. Studying the activities of RPW in the larvae stage, Martin et al. [61] concluded that the sound spectrum is constant during chewing and biting activities and differs during the insect movement. Using a speech recognition technique, the presence of the RPW larvae was identified and validated by tree dissections. For the identification of hidden RPW inside coconut trunks, Martin and Juliet [65] used the MfCC (Mel Frequency Cepstral Coefficient) algorithm and vector quantization. Pinhas et al. [61] developed a mathematical model based on speech recognition (Gaussian Mixture Model and Vector Quantization) with a detection rate of 98%. Using Gaussian Mixture Models, Potamitis et al. [52] also had detection rates above 94%. A bioacoustic sensor was developed to detect the weevil in the early infestation stages with 90% success, despite the environmental noises. This prototype also was solar energy based and was able to send information to a control station in schedule defined by the user via wireless [66].

Multiple studies were conducted in order to identify the RPW using acoustic sensors [67,68,69,70,71,72,73]. A wireless network connected with acoustic sensors was proposed by Srinivas et al. [71] in order to precisely monitor palm fields. Hetrzoni et al. [72] developed a ‘learning data set’ based on the multivariate distribution of nine pre-selected frequencies to detect RPW larvae activity, using a piezoelectric sensor. The sensitivity was low (around 30%) in early larval stages but had a significant improvement (up to 95%) as larvae developed. Herrick and Mankin [74] developed a custom-written insect signal analysis program: “Digitize, Analyze, View, Insect Sounds” (DAVIS), and were able to detect larval burst in 80% of palms inoculated with neonates in the previous day.

Makin et al. [75], using microphones and amplifiers had applied the DAVIS (digitize, analyze, view, insect sounds), achieved 90% accuracy.

Soroker et al. [76] developed a method for identifying digital signature of larvae on trunks, resulting in improvements in detection efficiency of weevils in palms offshoots.

Using a titanium drill bit inserted into the palm tree trunk and a sensor-preamplifier module, Fiaboe et al. [77] could detect the activity of RPW larvae through oscillogram and spectrogram analysis.

A portable, user-friendly acoustic sensor system enabled the identification of larvae in individual infested trees through the use of signal processing analyses that screened out bird and wind noise.

Nowadays, automatic acoustic sensor (seismic) for early detection of RPW in ornamental and date palm trees using wireless sensor networks is commercialized by Agrint, Rockville, USA and for IOTrees, Spain. The platform is connected to a network of palm tree sensor devices and gives the user treat alerts in real-time (Figure 4).

6. Conclusions

The identification and monitoring of insect pests using automatic traps brings a novel approach to the integrated pest management. Systems that use image recognition techniques and neural networks are the most studied ones, being reliable for the fully automatized identification of orders and counting of insects; however, not so many proposed models are able to identify the species level. Other promising image-based systems developed are the ones that aim to send the insect image to a specialist and then the insects could be identified and counted remotely in real time. The infrared sensor traps were shown to be useful for counting insects, but are limited because they cannot identify the species, which can result to misleading data in the survey. Audio traps are another deeply studied approach for monitoring pests. The research presented in the RPW resulted in the creation of commercial solutions. Image-based commercial solutions can be also be found nowadays in the market.

These new systems promise to facilitate the implementation of IPM systems soon. Appendix A summarizes the automatic detection techniques and sensors used for the different insect groups and Appendix B brings a list of commercial providers previously discussed in this review. The record of data will be useful for population dynamics studies and, if related with climate data, can be used in decision support systems and provide real-time information about pest infestation risk.