A Hybrid Approach for the Detection and Classification of Tomato Leaf Diseases

Featured Application

This system will help the farmer to detect plant any disease at very early stage to take timely pre-emptive action.

Abstract

In this paper, we proposed a hybrid deep learning approach for detecting and classifying tomato plant leaf diseases early. This hybrid system is a combination of a convolutional neural network (CNN), convolutional attention module (CBAM), and support vector machines (SVM). Initially, the proposed model can detect nine different tomato diseases but is not limited to this. The proposed system is tested using a database containing images of tomato leaves. The obtained results were very encouraging, giving us accuracy up to 97.2%, which can be improved with the improvement of learning processes. The proposed system is very efficient and lightweight, so the farmer can install it on any smart device having a digital camera and processing capabilities. With a bit of training, a farmer can detect any disease immediately, which will help him take timely pre-emptive action.

1. Introduction

The tomato is a universal crop with high nutritional values and is considered one of the primary consumable vegetables in the world. According to [1], worldwide production of tomatoes was more than 180 million tons in 2018, which is much higher now, where Asia is the largest market and producer. However, this crop is often affected by many diseases and insects, which leads to financial losses and poses a fundamental challenge for food production. Timely detection, correct classification, and early prevention by disease intervention are vital and can save production costs and output yield. Automatic and autonomous tools and technology are quickly emerging to detect these issues. In this regard, several research studies have been conducted to develop computer vision and artificial-intelligence-based algorithms for the early detection and classification of plant disease to minimize the adverse effects. Machine learning and deep learning approaches have become the foremost research direction to diagnose crop disease at an early stage. It will enhance the survival rate of crops, including fruits, vegetables, flowers, and grass, through disease management at the right time.

The traditional disease diagnosis systems for tomato leaves involve high costs and risk of misjudgment and are now supplanted with computer technology, including computer vision, deep learning, and machine learning methods. However, these computer technologies provide better and easier disease detection and control mechanisms. They are still not fully efficient and error-free as they need tailored feature extraction and segmentation methods. Thus, considering the importance of developing an efficient disease classification model to correctly identify and classify the type of disease at an early stage, we gained the motivation to write this paper.

This paper proposes an efficient hybrid classification approach for identifying nine diseases in tomato plants. The developed system consists of three different technologies, the convolutional neural network (CNN), support vector machine (SVM), and convolutional block attention module (CBAM), and The ResNet50 model is adopted to implement CNN. Rather than fully connected network layers, SVM is used to connect to the CNN model, which will enhance the classification accuracy. Finally, the CBAM is employed to process the feature extraction function. The proposed model achieves an efficient classification accuracy in comparison to the recently developed approaches. Therefore, the main contribution of this paper lies in the following aspects:

• Studying and evaluating the recently developed tomato leaf disease classification and identification approaches.
• Developing an efficient tomato leaf disease identification and classification system based on a customized CNN model and SVM approach.
• Validating the efficiency of the proposed hybrid approach using a dataset of tomato leaves.

The rest of this paper is organized as follows: Section 2 discusses the recently developed disease classification systems for tomato crops. Section 3 discusses the proposed system’s design, whereas Section 4 presents an analysis of the results. Section 5 discusses the results and compares the obtained classification accuracy with the existing classification models. Finally, Section 6 concludes the work presented in this paper and draws a future outcome.

2. Related Work

The most vital issue for farmers is to manage plant-related diseases. The plant leaves are the part that is affected first of all and show a noticeable change in case any attack is approaching. However, the segmentation of a sick leaf image is prone to produce uneven illumination and a cluttered composite environment. Therefore, this section discusses the recently developed classification models capable of identifying tomato leaf diseases and analyzing their efficiency.

In a study [2], the authors developed a deep CNN model named efficient net CNN. The developed model was fine-tuned and trained to detect healthy and unhealthy tomato leaf images. The authors tested the performance of two segmentation models, the U-net and the modified U-net model, for segmenting the leaves. They also conducted experiments for binary classification models, six-class classification, and 10-class classification. They found that only the U-net model revealed an accuracy of 98.66% for segmenting the leaf images. On the other hand, the EfficientNet-B7 showed a reliable performance when employed with binary and six-class classification. The average accuracy obtained using segmented pictures is 99.95% for binary classification and 99.12% for six-class classification.

Authors of [3] proposed an approach that can automatically detect and classify tomato disease by employing super-pixel optimized segmentation for natural images. The preprocessing phase used a color-balancing algorithm that can aid in dropping the unwarranted influence of illumination and assist in finding the local optimum threshold that different types of image datasets can adapt. In addition, a novel method based on a histogram of gradients and color variations has been employed to separate the leaves from the background in an effective manner. The feature extraction that is based on the pyramid of the histogram of gradients, shape descriptor, and gray level co-occurrence matrix (GLCM) is proved to be reliable in providing the uniqueness of specific disease patterns. In this work, several classifiers have been implemented. Still, the random forest classifier, which used a hundred trees dataset, showed the best efficiency in obtaining the performance of the proposed framework. The results obtained in this research are compared with several similar works in studies [4,5,6,7] where the comparative analysis using the estimation parameters proved the best.

Authors of [8] proposed a multi-class feature extraction system for classifying the tomato leaf disease. The system is based on a deep CNN model incorporating the attention method and the residual block. The outcomes demonstrated that the model could successfully extract the intricate characteristics of various diseases. Moreover, it outperforms several recently developed deep learning studies by adopting the most popular PlantVillage dataset. This work obtained an overall identification accuracy of 99.24%.

The work presented in [9] involves analyzing the performance of classifying different types of tomato disease. A lightweight CNN approach has been designed and illustrated by incorporating several attention methods to enhance the performance of the models. The network architecture, performance, and computational complexity were studied in this work. The employed dataset consists of nine diseases and a single healthy case. The obtained results showed an improvement in the classification accuracy through designing a tiny and computationally effective model. The work presented in [10] involves creating different types of CNN architecture (LeNet, VGGNet, ResNet50, and Xception) for detecting tomato leaf disease. Authors trained a deep CNN using the PlantVillage dataset with 14,903 images of healthy and diseased tomato plant leaves. The obtained results showed that the fine-tuned VGGNet architecture offers the best classification, with an accuracy of 99.25%, and achieves the minimum loss among all the evaluated architectures at the cost of a long training time and needs high hardware configurations.

In [11], the authors investigated grape plant leaf diseases by designing an efficient deep transfer learning-based model. The authors introduced a fully connected layer to extract the most significant features. Then, they employed the variance technique to remove extraneous features from the feature extractor vector. The PlantVillage dataset was used in this work, where the photos were fed to retrain the EfficientNet B7 deep architecture. Then, the collected features were minimized using the logistic regression technique. The obtained classification accuracy was equal to 98.7%.

Authors of [12] designed a CNN model to effectively identify and classify the tomato diseases using the google collab framework to conduct the complete experiment by adopting a dataset containing 3000 different tomato leaf images that are affected by nine various diseases and a single healthy leaf class. The developed system consists of three main phases: the images are preprocessed, and the targeted area in the pictures are segmented. Then, the images are further processed through varying hyper-parameters of the CNN model. Finally, the CNN extracts the characteristics from the image, including colors, edges, texture, etc. The developed classification model achieves a prediction accuracy of 98.49%.

The work presented in [13] developed a CNN model consisting of three convolutional and max-pooling layers with a different number of filters in each layer. The authors’ first time revealed that the PlantVillage dataset is unbalanced, and augmentation techniques are used to balance the images inside each class. The obtained average accuracy for this model is 91.2%, whereas the needed storage space is almost 1.5 MB, and the pretrained model needs a space of 100 MB.

In [14], the authors employed the transfer learning model to minimize the training data size, computational cost, and training time when building a deep learning model. The authors used five deep network structures, including MobileNet, Resnet50, Xception, Densenet121 Xception, and ShuffleNet, to perform the feature extraction. The authors developed several experiments using different learning rates and different training parameters. The best classification accuracy was obtained with the Densenet Xception, with 97.10% classification accuracy.

The work presented in [15] involves designing a dense residual network for classifying tomato leaf diseases. The authors adjusted the residual dense network architecture. The model has been transformed into a classification model that offers higher accuracy of 95%.

In [16], the authors presented an automatic leaf disease identification and classification model. In the first stage, the input images are resized to minimize the computational complexity, and then segmented by blacking out the background region. In this project, the authors employed a feature extraction method using two deep learning models, VGG19 and AlexNet, that are then classified using the ECOC-based support vector machine classifier. The results show that VGG19 and AlexNet achieved 98.9% and 98.8% classification accuracy, respectively.

Authors of [17] focus on the soybean plant disease classification using the multilayer perceptron deep learning and support vector machine algorithms. The support vector machine was able to classify 19 classes of diseases. The employed dataset consists of 683 instances, where 643 instances were correctly classified, and 40 instances were incorrectly classified, with a classification accuracy of 94.14%.

As presented above, several deep learning and machine learning approaches for classifying tomato leaf diseases have been proposed and discussed with various classification accuracies.

Table 1. A comparison between the recently developed crop classification systems.

Ref.	Algorithm	Disease Classification	Accuracy
[2]	EfficientNet-B7 EfficientNet-B4	Binary (healthy/unhealthy) 6 disease classes 10 disease classes	Binary classification 99.95% Six-class classification 99.12% 10-class classification 99.89%
[3]	Random Forest	4 disease classes	RF: 97%
[8]	ResNet-50 + SeNet	10 disease classes	99.24%
[9]	Light weight CNN	10 disease classes	99.25%
[10]	LeNet VGGNet ResNet50 Xception	10 disease classes	VGGNet: 99.25%
[11]	Hybrid CNN (Hy-CNN)	10 disease classes	97%
[12]	Segmentation-based CNN	10 disease classes	98.49%
[13]	CNN model	10 disease classes	91.2%
[14]	MobileNet, Resnet50 Xception, Densenet121 Xception, ShuffleNet	10 disease classes	Densenet121 Xception: 97.10%
[15]	Restructured residual dense network	10 disease classes	95%
[16]	VGG19, AlexNet		VGG19: 98.9% AlexNet: 98.8%
[17]	Deep learning, SVM	19 classes of diseases	94.14%

further discusses the aforementioned classification systems and analyzes them in terms of the employed algorithm, disease classification capability, and the obtained accuracy.

3. Proposed System

We proposed an effective and state-of-the-art hybrid approach for classifying tomato diseases based on a CNN model trained on infected tomato leaves images.

3.1. Data Collection and Preprocessing

Our data consist of ten distinct classes, of which nine represent different tomato diseases, while one is for healthy tomatoes. The data are a collection of RGB images of tomato leaves of dimensions 256 × 256 pixels. Distention between infected and healthy leaves is based on their image classes. While taking the images, it has been ensured that each image should have a single centroid leaf. Furthermore, lighting and photography conditions are also kept identical. The 10 categories of images described above are shown in Figure 1.

The images are a part of the PlantVillage dataset [18], which consists of many plants, but we chose tomato as the base of our research. The images were collected from the fields and each classified by their disease.

3.1.1. Preprocessing

The preprocessing method downsizes input images to 256 × 256 × 3. The images are then converted from RGB to BGR; then, each color channel is turned to zero-centered concerning the PlantVillage [18] dataset without scaling.

3.1.2. Data Augmentation

Data augmentation is crucial to data preparation to increase the number of images and reduce overfitting. We used data augmentation to produce more images in each class and managed to multiply the data by 30 folds by applying the following data augmentation techniques:

• Random image rotation: It makes a new, slightly rotated image with a random angle ranging up to 90 degrees each time.
• Random horizontal shift: The random shift pushes the image from the center horizontally, making a new image where the leaf is not in the center, helping the model to be more robust and less prone to overfitting.
• Random vertical shift: Like the horizontal shift, the vertical shift pushes the image from the center vertically. Both filters produce random shifts in the image size range of (1,20) percent.
• Horizontal and vertical flip: This filter produces a flipped image identical to the original, either vertically or horizontally.
• Zoom: This involves a random zoom at every image by up to 20 percent.

3.1.3. Pipeline Preparation

Preprocessed and augmented images are compiled into batches of 64 during training and one during evaluation, so the final input shape is a four-dimensional matrix of 64 × 256 × 256 × 3.

Images are categorized using one-hot encoding, and each training batch is paired with a 64 × 10 matrix representing the targets for the current batch. The preprocessing data stages are depicted in Figure 2.

3.2. Architecture

Image classification has made significant progress with the help of deep learning. Currently, CNN and many other architectures have been tried and tested and achieved various results, as discussed in the related work section. We are proposing here a new hybrid architecture that combines different techniques. Our approach consists of three main parts: a CNN model, CBAM attention mechanism, and SVM classifier. In this new proposed model, layer activation functions and hyper-parameters are also modified and customized accordingly.

3.2.1. Feature Extraction Stage

In this proposed system, ResNet50 [19] is used as a backbone of the model. The Resnet-50 contains four stages, each containing multiple residual blocks to increase the network’s performance. Using the identity connection also shields the network from vanishing gradient problems. Furthermore, it prevents overfitting by utilizing batch normalization (BN) layers. Each residual block contains two sets of weight layers, followed by a batch normalization layer separated by the “ReLU” activation layer. After that, we added the input $x$ of the residual block and obtained the output $f\left(x\right)$. Once again, this $f\left(x\right)$ passes through the “ReLU” layer to obtain the final output of all the blocks, which can be expressed as $g\left(x\right)=f\left(x\right)+x$. The architecture of the residual block is shown in Figure 3.

3.2.2. Convolutional Block Attention Module

In general, the convolutional block attention module (CBAM) employs two attention modules: the channel attention module, as in Figure 4, and the spatial attention module shown in Figure 5. The channel attention module comprises two feature maps, each consisting of two intermediate layers: average and maximum pooling. Both feature maps are joined by a shared multilayer perceptron layer (MLP), and then the output of the feature maps is added using the sigmoid activation function. Finally, the multiplied features between the convolutional layer and the channel attention module were applied to the spatial attention module to identify the location of the most significant features in the given image. Attention modules make CNN learn and focus more on the critical information rather than on learning non-useful background information. We added CBAM [19] block to our base model, giving the model as shown in Figure 6.

3.2.3. SVM

A support vector machine (SVM) is a supervised machine learning algorithm for regression, classification, and outlier detection. SVM also includes support vectors to find the co-ordinates of discrete observations. SVM is applied when the total number of dimensions exceeds the total number of samples, generating efficient results in high-dimensional spaces. SVM works by constructing a hyperplane in multidimensional space to distinguish between different classes. In addition, SVM has been intensively employed in image classification scenarios and achieved efficient classification accuracy. In the proposed model, we replaced the multilayer perceptron output of the CNN model with an SVM classifier, as shown in Figure 7.

4. Results

We started the training process by creating an image data generator that produces a batch of the image in every training iteration. The made batch is a set of normalized, augmented, and shuffled images, each paired with the desired output in the form of a one-hot encoding vector. The image data generator parameters are shown in

Table 2. Image data generator hyper-parameters.

Augmentation Algorithm	Description
Image zooming	0.2
Horizontal flip	True
Vertical flip	True
Image rotation	0.3
Width shift	0.2
Height shift	0.2

, and a random set of augmented images are shown as an example in Figure 8.

We divided the training process into two stages. The first stage is training the ResNet-50 with an attention module with a fully connected output using a categorical cross-entropy loss function. We utilized two dropout layers, each 50 percent, to decrease overfitting, and the final output layer had a Softmax activation function.

Table 3. Output layers of the first stage of training.

Layer	Activation Function	Trainable Parameters	Output Shape
Global average pooling	-	0	2048
Dropout 50%	-	0	-
Dense Layer	ReLU	4,196,352	2048
Dropout 50%	-	0	-
Dense Layer	Softmax	20,490	10

expresses the shape of the output, the number of trainable parameters, and the activation function for each layer, while

Table 4. First-stage training parameters.

Particular	Description
Original training data size	10,000 images
Original validation data size	1000 images
Training batch size	64
Validation batch size	1
Steps per epoch (train)	156
Steps per epoch (evaluation)	2
Optimizer	Adam
Loss Function	Categorical Cross entropy
Learning rate	0.001 (default)
Epochs	50

shows the training hyper-parameters and the training and data validation size.

In Table 4, different training hyper-parameters exist where the Adam optimization algorithm is used to categorize the cross-entropy with a fixed learning rate of 0.001.

We trained the model for 50 epochs, reaching an accuracy of 97.78%, and the final loss value was 0.076, as shown in Figure 9.

In the second stage, we froze the weights of the ResNet-50 layers paired with the attention module. We replaced the multilayer perceptron output with an SVM classifier using the Gaussian RBF kernel function. Furthermore, we squared hinge loss function and trained SVM only. Training parameters of this stage are shown in

Table 5. Second-stage training parameters.

Particular	Description
Original training data size	10,000 images
Original validation data size	1000 images
Training batch size	64
Validation batch size	1
Steps per epoch (train)	156
Steps per epoch (evaluation)	2
Optimizer	AdaDelta
Loss Function	Squared Hinge loss
Learning rate	0.001 (default)
Epochs	60

.

We trained the SVM model for 75 epochs and reached an accuracy of 97.2% and a final loss value of 0.089, and the output results are shown in Figure 10.

Table 6. Training and validation of our two different models with obtained accuracy and loss.

Model	Training		Validation
	Loss	Accuracy	Loss	Accuracy
ResNet-50-CBAM (MLP)	0.108	0.9664	0.076	0.9778
ResNet-50-CBAM (SVM)	0.087	0.9632	0.891	0.9723

compares system accuracy while training a CNN model in the presence of MLP versus replacing the MLP with the SVM classifier.

5. Discussion

The proposed hybrid model was able to separate the data with high accuracy, although different diseases cause similar symptoms, resulting in low-class variance. Furthermore, it avoids overfitting, and resultant validation accuracy is conditional on training accuracy, as shown in Table 6. The biggest challenge related to this hybrid system was to conduct separate training at different stages, as every stage may have different loss functions. We tried different CNN architectures, including VGG16 and ResNet-50, but, finally, ResNet-50 is selected as a backbone for our proposed model due to better accuracy. We also tried different attention techniques and achieved different results but, ultimately, CBAM suited well to our model. It is also worth mentioning that training the entire model from scratch by introducing CNN layers and squared hinge loss function gave us inferior results and needed more epochs to reach acceptable results.

Although our two-stage hybrid approach is more complex, it improved the learning speed dramatically because we froze the base model layers during the second stage of training, which decreased massively the number of trainable parameters. Most importantly, compared to other techniques, we were able to reach comparable results in a considerably lower number of epochs, which helped us in the early detection of existing diseases. The main improvement in our approach was the speed at which the model was learning, especially during the second stage. However, there is always room for improvement, for example, more than one CBAM attention module can be utilized in the same model. Furthermore, different attention techniques can be combined to achieve better preferences. Additionally, we do not recommend using a deeper network, such as ResNet-101 or ResNet-152, for this under-studied problem.

Initially, the proposed model is enough to separate the 10 different classes and deepen the network. The performance of the proposed model is compared with some existing research studies, as shown in

Table 7. Performance comparison of the proposed model with related work.

Refs.	Algorithm	Accuracy
[2]	EfficientNet-B4	99.89%
[8]	ResNet-50 + SeNet	96.81%
[9]	Light weight CNN	99.69%
[11]	Hybrid CNN (Hy-CNN)	98.7%
[12]	Segmentation-based CNN	98.49%
[13]	CNN model	91.2%
Our Model	ResNet50-CBAM + SVM	97.2%

. Table 7 compares the performance of existing semantic models that are similar to our system and used the same tomato disease datasets, named PlantVillage, and all of them studied the same number of disease classes (10 diseases). It was found that the results of some studies have better accuracy as compared to ours, but the results of our model could be improved by training more epochs, changing the value of the learning rate and the optimizer, and reducing the dropout ratio. Overall, our model has good overall performance and acceptable diagnostic accuracy, adding the benefit of early detection of tomato leaf diseases.

6. Conclusions

This paper proposed a hybrid classification model based on three main subsystems: CNN, CBAM, and SVM, with two stages implementing transfer learning. It can classify nine different crop diseases in tomato leaves and the intact case. With the implementation of transfer learning, we could reduce the number of training epochs from 1000 to 3000 in numbers, which was 175 epochs in total previously. Moreover, the second training stage was much faster and utilized much fewer hardware resources, since the base model was frozen. The initial results showed an improvement over the existing crop disease classification systems, with 97.2% classification accuracy.

Although our approach of using the hybrid model and transfer learning might seem complex and not as accurate as other existing systems. The accuracy of the proposed system can improve with more training data and a number of model epochs. Additional benefits of this system are that it has a fast processing speed, fewer epochs, fewer parameters, and early detection. These advantages lead this system to work in a real-time environment. Nowadays, drones are being used to take images of underlying crops or forests that can be used to make an accurate and least dataset. Due to the faster processing of the proposed system, it can be an excellent candidate to install in drones to detect crop diseases in real time.