Potato Surface Defect Detection Based on Deep Transfer Learning

Abstract

Food defect detection is crucial for the automation of food production and processing. Potato surface defect detection remains challenging due to the irregular shape of potato individuals and various types of defects. This paper employs deep convolutional neural network (DCNN) models for potato surface defect detection. In particular, we applied transfer learning by fine-tuning a base model through three DCNN models—SSD Inception V2, RFCN ResNet101, and Faster RCNN ResNet101—on a self-developed dataset, and achieved an accuracy of 92.5%, 95.6%, and 98.7%, respectively. RFCN ResNet101 presented the best overall performance in detection speed and accuracy. It was selected as the final model for out-of-sample testing, further demonstrating the model’s ability to generalize.

1. Introduction

As one of the most significant crops, potatoes are planted on more than 18.9 million hectares of farms in the world [1]. In 2015, potatoes were promoted as a staple food in China, with a target of 30% of the total potatoes consumed as a staple food by 2020 [2], which drives potatoes to become the World’s fourth-largest food product after wheat, rice, and corn. After harvest, the potato surface is often subject to sprouting and mechanical damage due to human or other external factors [3]. Therefore, grading based on surface quality is crucial to classify products into different categories, which can greatly improve packing, storage, transportation, and other post-harvest operations. Traditional defect detection methods rely on manual inspection [4], which is highly subjective, slow, error prone, and difficult to achieve uniform standards [5].

As a non-intrusive technique, computer vision systems have been adopted to build automated grading systems during post-harvesting [6,7,8,9]. A variety of devices, such as CCD camera, ultraviolet camera, hyperspectral camera, and X-ray CT, have been utilized and demonstrated their efficacy in capturing essential features of potato surface quality [10]. Sensory images generated by these devices can be used to build estimators to detect a potato’s physical size and internal and external defects [3,11]. However, early automated grading systems have extensively utilized image processing algorithms and relied on manually defined image features to build classifiers [12,13,14,15], limiting the robustness and generalization [16] of detection performance due to the variance of potato types, appearances, and damage defects [17]. These machine vision systems form the basis of optical sorters used for potato grading and quality inspection [18,19,20]. For example, Noordam et al. develop a high-speed sorter to grade potatoes on on size, shape, and color based on several image processing and classification algorithms: Linear Discriminant Analysis combined with a Mahalanobis distance classifier for color segmentation; a Fourier-based estimator for shape classification; and features such as central moments, area, and eccentricity are used to discriminate between similar colored defects [20]. To build such an optical sorter, extensive domain knowledge is required to identify a set of features to represent a potato sample in an image. Compared to the feature-based conventional learning algorithms, deep learning can learn features automatically during training and demonstrate superior performance in a wide spectrum of computer vision tasks, including image classification and object detection [21,22]. Recent years have witnessed the development of deep learning-based optical sorters [23,24], which have been proven to be more efficient and effective than traditional machine vision-based systems [25].

The rise of deep learning has created explosive impact in every industry [26]. Deep convolutional neural networks (DCNNs), a revolutionary neural architecture, can eliminate the tedious feature engineering step and learn features automatically during training, pushing numerous computer vision tasks to the state of the art [22]. Specifically, a DCNN consists of multiple convolutional layers that can extract and represent features at different levels. The network parameters, or weights, are optimized during training in a ways that the critical patterns can be captured, leading to activated signals on the output feature maps, which jointly determine the prediction result.

DCNN-based models have also been developed for surface quality assessment of agricultural products [27,28,29]. We investigate a collection of relevant studies published in the last five years and present a subset of them in

Table 1. Literature in DCNN-based surface defect detection for agricultural products. Abbreviation: T.L. = Transfer Learning; OOS T. = Out-of-sample testing; Cls. = Classification; O.D. = Object Detection.

Ref	Product	# Classes	Task	T.L.	OOS T.
Fan et al. (2020) [30]	Apple	Two	Cls.	✗	✗
Valdez et al. (2020) [31]	Apple	Two	O.D.	✓	✗
Xie et al. (2021) [32]	Carrot	Six	Cls.	✓	✗
Deng et al. (2021) [33]	Carrot	Six	Cls.	✓	✗
Azizah et al. (2017) [34]	Mangosteen	Two	Cls.	✗	✗
Shi et al. (2019) [35]	Tomato	Two	O.D.	✓	✗
da Costa et al. (2020) [36]	Tomato	Two	Cls.	✓	✗
Turaev et al. (2020) [37]	12 types	Five	Cls.	✓	✗
Casano et al. (2020) [38]	Potato	Two	Cls.	✗	✗
Su et al. (2020) [3]	Potato	Six	Cls.	✗	✗
Hasan et al. (2021) [39]	Potato	Twenty	Cls.	✓	✗
Ours	Potato	Three	O.D.	✓	✓

, from which we identify the commonly studied products, e.g., apples, carrots, mangosteens, tomatoes, and potatoes. In addition, we compare these efforts in terms of the number of classes, the learning task, the use of transfer learning and out-of-sample (OOS) testing. We highlight the differences between our work and the ones in the literature as follows.

• # classes. Half of the listed studies classify the target samples into two categories: healthy and defective. One study, by Hasan et al. [39], considered a fine-grained classification problem with twenty categories, which describe the different types of potato diseases. Our study considers three potato classes: normal, scratch, and sprout. Based on an experiment conducted by Al-Dosary et al. [40], 2–3% of potato tubers are superficially or deeply scratched, accounting for over 70% of the total damaged potatoes during harvesting. Our review reveals that the scratched type is rarely considered in surface defect detection models by prior efforts. Furthermore, sprouted potatoes can be toxic to human being due to the higher level of glycoalkaloids [41]. The two defective types we consider are crucial and meaningful, as they represent the major defects caused during and post-harvesting.
• Learning task. We have seen both classification [42] and object detection [43] used in the literature to build a detection model. The former takes an input image and outputs a predicted class, meaning that there is only one object in the image. On the other hand, the latter allows an input image to contain objects of different classes and outputs bounding boxes and classes of the predicted objects. Apparently, object detection is a more powerful model with more practical value, suitable for large-scale and real-time detection systems [44]. Our investigation shows that classification is mainly adopted by most existing studies, and we develop detection models based on object detection algorithms in this study.
• Transfer learning. Training a robust DCNN model requires a large amount of data, which is usually not available for the surface defect detection task. Transfer learning [45] addresses the low-resource issue by transferring knowledge from a source domain, where a base model can be trained with sufficient data, to a target domain. We find that most existing studies have adopted transfer learning, i.e., a pretrained DCNN model is only fine tuned on the target dataset for surface defect detection.
• OOS testing. OOS testing is essential to evaluate how robust a model is by testing it on a different batch of sample and potentially in a different environment [46]. A grading system can be installed and utilized in various scenarios. It is thus crucial to simulate the image variance caused by environment change via an OOS test set gathered in a different scenario than the one where the original dataset is developed. We did not find another related study using an OOS test set.

The contributions of this paper are as follows.

• We develop a dataset for potato surface defect detection with three categories and a total of 2770 images. Compared to the existing studies, we treat the potato surface defect task as an object detection problem and consider the scratched and sprouted potatoes, which are rarely seen in prior efforts.
• Three pretrained DCNN models—SSD Inception V2, Faster RCNN ResNet101, and RFCN ResNet101—have been fine-tuned on our dataset and achieved an accruacy of 92.5%, 95.6%, and 98.7%, respectively. In addition, we develop an OOS test set to evaluate the best model, namely, RFCN ResNet101, in three scenarios. Results show that the RFCN ResNet101 model demonstrates robust performance with moderate inference speed. To our best knowledge, this is the first time DCNN-based transfer learning is employed for potato surface detect detection, with three object detection algorithms evaluated on both original and OOS test sets. Our work can serve as a credible baseline for future research.

2. Material and Methods

2.1. Potato Surface Defect

Surface defect refers to the external characteristics of potatoes, including sprouting, greening, insect eyes, rotting, disease spots, damage, etc. In this paper, we focus on three specific categories: normal, scratch, and sprout. Figure 1 shows a sample image for each category.

• Normal: Potatoes that are yellow without sprouts, damage, scratch, rot, etc. and are ready for everyday human consumption.
• Scratch: The potatoes were artificially scratched to simulate damage at the time of harvesting. In our experiment, we keep the number of scratches to one to three.
• Sprout: Potatoes germinate in the temperature range of 15 to 20 °C. Below 10 °C, germination is slow; above 25 °C, germination is rapid, but the shoot roots are small. Therefore, potatoes are placed in a room with a room temperature of 18 to 25 °C and high humidity.

2.2. Dataset

2.2.1. Potato Samples Acquisition

A total of 642 normal and edible potatoes were taken from a local vegetable market in Huizhou, a southern city in China. The acquired potatoes belong to the Favorita variety [47]. We took photos on the same day of purchase for these normal potatoes to create an image set for the normal category. After that, the 642 normal potatoes were divided into two groups in a ratio of 1:2. The 214 potatoes in the first group were manually and intentionally scratched. The 428 potato samples in the second group were then placed in a relatively humid environment at 18–25 °C and left to germinate. We eventually collected a total of 428 sprouted potatoes. The sample size of sprouted potatoes is twice as large as that of scratched potatoes for two reasons: first, there are more uncontrollable factors when making samples of sprouted potatoes, which may cause undesirable results such as rotting [48], and second, the sprouted potatoes have relatively small bud-eye characteristics [49], and more samples are beneficial to training a better DCNN model.

2.2.2. Image Data Collection

To increase the quality and diversity of potato images, we utilized an industrial camera (model Basler Scout scA1390-17fc made by Basler AG at Ahrensburg, Germany). The camera has a SONY ICX285 CCD sensor and operates at 17 frames per second (FPS) with global shutter, generating JPEG images with a resolution of 1392 × 1040. Images were taken at different sites with different angles. Figure 2 displays some images from the dataset. It is noted that the images have different backgrounds, which increases the diversity of the dataset and allows deep learning models to learn more patterns to locate and classify potatoes. Photos of the normal potatoes were taken on the same day of purchase. We then created a group of scratched potatoes and took photos on the second day of purchase. Last, the photos for the sprouted potatoes were taken after 20 days since purchase.

Training a DCNN model from scratch requires a large amount of annotated data. We took advantage of transfer learning, which takes a pretrained model on the COCO object detection dataset [50] and fine-tunes the model on the target dataset. As the pretrained model can capture numerous generic object features such as shape, color, and texture, we can further train it on our dataset to feed it with more domain-specific features. Transfer learning performs exceptionally well on a small and medium-sized dataset. To this end, we created a small dataset and trained a well-performing DCNN model. In all, we gathered a total of 2770 images, which were divided into training and test sets in the ratio of 3:1.

Table 2. Basic information of the potato surface defect dataset.

Category	Samples	Images	Percentage	Training	Test
Normal	428	813	29.4%	610	203
Scratch	214	841	30.4%	631	210
Sprout	428	1116	40.2%	837	279
Total	1070	2770	100%	2078	693

displays the basic information of the dataset.

2.2.3. Out-of-Sample Test Set

To further validate the robustness and generalization ability of the DCNN models, we created an OOS test set using a total of 642 potato samples purchased from a different local food market. We followed the steps similar to the ones used for the first dataset and eventually obtained 214 samples per category. We set up a real test environment to validate the models in three scenarios. For each scenario, we used an industrial camera to take images that were sent to the model for real-time detection.

• Scenario one. We took one image per sample at different sites within the campus, with the camera directly facing down to the potato sample, creating 214 sample images per category.
• Scenario two. This experiment was conducted indoor. We set up a clean desktop. The camera was placed 30 cm above the desktop. Taking the point where the camera was facing the desktop as the center position, we divided the shooting range equally into four areas A, B, C, and D, as shown in Figure 3a. The potatoes were put into A, B, C, and D areas in turn for testing. Scenario two also created 214 × 4 sample images per category.
• Scenario three. Our last experiment aims to study the impact of different light intensities on model performance. We moved the test platform into a closed dark box, within which the light intensity can be changed by a ring-shaped adjustable light source, as shown in Figure 3b. The adjustable light source has four levels of brightness. At each brightness level, we placed a potato sample on the platform with the feature parts (if any) of the potato facing up to the camera. We obtained 214 × 4 images for each category.

2.2.4. Data Labeling

To train a supervised DCNN model, we need to first manually label each image in the dataset to indicate its category. In particular, we employed a tool called LabelImg, which allows one to draw a bounding box around a potato object in an image and mark its category. As shown in Figure 4, the blue rectangle is the bounding box enclosing a normal potato object. After the labeling, each image comes with a file that stores the coordinates of the bounding box and the potato’s category.

2.3. DCNN Models for Object Detection

2.3.1. Model 1: SSD Inception V2

Figure 5 shows a simplified neural architecture of SSD Inception V2. The SSD algorithm is a single-stage detection model that allows object localization and classification to be done in a single forward pass of the neural network. The vanilla architecture of SSD uses VGG-16 [51] as the backbone network to extract features, with a set of convolutional layers added after the backbone to perform feature extraction at multiple scales, which enables the model to discover more distinguishable patterns at different granularities. Moreover, to ensure the speed and accuracy of target detection, the output space of the bounding box is discretized into a series of default boxes with different aspect ratios, and the boxes can be adjusted during training to better match the size and shape of the ground truth object, while the prediction results on multiple feature maps of different resolutions are combined to extract feature maps of different scales for better detection.

Our study utilized the SSD Inception V2 model, which replaces VGG16 with the Inception V2 neural structure [52] that splits the large convolutional kernel into several small convolutional kernels, and introduces asymmetric convolution to reduce the number of parameters and mitigate overfitting. As shown in Figure 6, an inception module performs multiple convolutional or pooling operations on the input image in parallel and stitches all the results into a deep feature map. It uses different filters to perform convolution operations on the input to obtain different information about the input image. Processing these operations in parallel and combining all the feature maps would result in a better image representation. Inception V2 also uses batch normalization to speed up training while removing dropout and mitigating L2 regularization.

2.3.2. Model 2: Faster RCNN ResNet101

Regions with CNN features (RCNN), proposed by Girshick et al. [53], work by performing a selective search that extracts 2000 regions, i.e., region proposals (RP), from an image. Each RP is then warped and passed through a CNN to produce a feature vector, which acts as input of a support vector machine (SVM) for the final classification. Despite its inspiring idea, RCNN is not practical for real-time systems due to its low detection speed, mainly caused by the CNN operations individually done on the 2000 image patches. To speed up RCNN, Fast RCNN, proposed by the same group [54], works by passing an entire image through a CNN to generate a feature map, from which RPs are located and warped to make a feature vector. Fast RCNN greatly reduces the time consumed by the CNN as only one image passes through it, rather than 2000 patches.

Despite the improvement brought by Fast RCNN, it still adopts a selective search process to identify RPs, which is inefficient. Ren et al. [55] develop Faster RCNN that adopts a Region Proposal Network (RPN), which can learn where and how to place RPs through training. Specifically, the input of RPN is an image, and the output is a collection of anchor boxes, also known as object proposals, each assigned a an objectness score. The top-N scored anchor boxes are sent to a region of interests (ROI) pooling layer to produce a feature map with proposals, which is fed into the detection head for bounding box regression and object classification.

Our study employed the Faster RCNN ResNet101 model, as shown in Figure 7, which adopts ResNet101 as the backbone network for feature extraction. In deep learning, as the network deepens, accuracy drops due to gradient dispersion. This issue is well addressed by a residual network (ResNet) [56], which allows the network to deepen as much as possible without causing gradient dispersion. A ResNet adds an identity mapping (constant mapping) that transfers the current output directly to the next layer of the network, taking a shortcut and skipping the current layer of operations. During backward propagation, the gradients of a layer are passed directly to the upper layer of the network. ResNet101 is a variant of ResNet that is 101 layers deep.

2.3.3. Model 3: RFCN ResNet101

Another notable target detection model is the RFCN algorithm, which adopts the same neural structure as Faster RCNN. RFCN generates proposals through an RPN that shares previous feature maps with the detection layer. Our work utilized RFCN ResNet101 with a convolutional layer of ResNet101 as the base network structure, immediately followed by a convolutional layer to downscale and a position-sensitive convolutional layer capable of generating score maps. The following is a position-sensitive ROI pooling layer with a softmax layer to determine the type of ROI. The RFCN model framework is shown in Figure 8.

The ResNet101 used in Faster RCNN is shared in the first 91 layers, then inserted into ROI pooling, and the last ten layers are computed separately for each ROI. In contrast, RFCN ResNet101 puts all 101 layers in the front shared subnetwork, and the final convolution used for prediction is only one layer, and then ROI pooling is performed. This dramatically reduces the amount of computation. The location information is lost in ROI pooling, so the location information is added before pooling based on the concept of position-sensitive score maps, i.e., different score maps are specified to be responsible for detecting different locations of the target. After pooling, the score maps obtained from different locations can be combined to reproduce the original location information, which further improves the accuracy of RFCN in detecting targets.

2.4. Transfer Learning for Potato Surface Defect Detection

Transfer learning refers to a machine learning method where a model trained on a source dataset is reused as the starting point to train another model on a target dataset [21]. For our task, we adopt the COCO object detection dataset [50], which consists of 330 K images, 1.5 million object instances, and 80 object categories, as the source dataset. As shown in Figure 9, the trained DCNN backbone network, namely, the parameters or weights, are directly utilized for our task. As the backbone network is thoroughly trained on a large dataset, it has learned extract general object features, such as color, shape, texture, and edge [45]. To fit our objective, we choose to freeze the first T layers of the pretrained backbone, and only fine-tune the remaining layers on our dataset. This strategy is fast and effective, as demonstrated in Section 3.

2.5. Performance Metrics

To gain a comprehensive understanding of the model performance, we employed five metrics: accuracy (Acc), precision (Pre), Recall (Rec), Specificity (Spe), and the F1 score. Intuitively, Acc measures the ratio of correct predictions over the total number of predictions (see Equation (1)), Pre is the fraction of true positives (TP) among the detected positives (see Equation (2)), Rec is the fraction of TP among the correct predictions (see Equation (3)), Spe measures the fraction of negatives that are misclassified as positives (see Equation (4)), and F1 is the harmonic mean of precision and recall (see Equation (5)). As our task is a multi-class classification problem with three categories, we can use the following equations to calculate the five metrics per category and perform a micro average to obtain the overall performance.

$$Acc=\frac{TP+FN}{TP+FP+TN+FN}$$

(1)

$$Pre=\frac{TP}{TP+FP}$$

(2)

$$Rec=\frac{TP}{TP+FN}$$

(3)

$$Spe=\frac{TN}{TN+FP}$$

(4)

$$F1=2\times \frac{Pre\times Rec}{Pre+Rec}$$

(5)

3. Results

3.1. Experimental Setting

We conducted the experiments on a workstation equipped with an Intel Core i7-8700 CPU, a 16 GB RAM, and an Nvidia GeForce RTX 2070 GPU. To implement and train the DCNN models, we adopted Python 3.6.7 and TensorFlow r2.1.

3.2. Implementation Details

The model implementation details are provided as follows.

• SSD Inception V2. The loss function of SSD is a weighted combination of localization and confidence loss. The former is defined as the smooth L1 loss with the offset from the predicted bounding box to the ground truth bounding box, and the latter is calculated as the softmax over the confidences of multiple classes. We adopt a learning rate of 0.001, 0.9 momentum, 0.0005 weight decay, and batch size 32. The model is trained using a stochastic gradient decent (SGD) optimizer. The used model in this study is offered by supervise.ly at https://supervise.ly/explore/models/ssd-inception-v-2-coco-1861/overview (accessed on 6 February 2021).
• RFCN ResNet101. The loss function of RFCN on each RoI is the sum of cross-entropy loss and the box regression loss, which correspond to the confidence and localization loss defined in SSD, respectively. We adopt a learning rate of 0.001, a weight decay of 0.0005, and a momentum of 0.9, with an Adam optimizer, also used by the original authors of RFCN in [57]. In addition, the batch size is 32. The released model used in this study is at https://supervise.ly/explore/models/rfcn-res-net-101-coco-1862/overview (accessed on 6 February 2021).
• Faster RCNN ResNet101. Faster RCNN uses the same loss function as SSD and RFCN. For training, we adopt similar settings as [55], with a learning rate of 0.003, a batch size of 16, a momentum of 0.9, a weight decay of 0.0005, and an optimizer of SGD. We used the released model at https://supervise.ly/explore/models/faster-r-cnn-res-net-101-coco-1866/overview (accessed on 6 February 2021) for our experiment.

As the models have been pretrained, fine-tuning on our dataset is fast. For each model, we only fine-tuned with five epochs, and the models were sufficiently trained and converged. In addition, all images in the dataset are rescaled with a shorter side of 600 px, which is a common setting [55,57].

3.3. Model Evaluation and Selection

The primary performance metric used in this study is Acc, which is adequate because the three categories in our dataset are relatively balanced. In addition, we used the model loss to monitor the training progress as it serves an effective indicator of model convergence.

Figure 10 shows the results that plot accuracy vs. training steps and loss vs. training steps. In Figure 10b, we observe that the loss gradually drops and eventually converges as the step number increases and reaches 15,000. All three models present a quick convergence with a low loss, indicating the models’ ability to learn rich features to distinguish potatoes of categories. In particular, both RFCN ResNet101 and Faster RCNN ResNet101 outperform SSD Inception V2 with a loss close to zero. Performance-wise, as the number of steps increased, the test accuracy of all three models reached more than 90%, and the final accuracy of SSD Inception V2, RFCN ResNet101, and Faster RCNN ResNet101 network models was 92.5%, 95.6%, and 98.7%, respectively.

We plot the confusion matrices of the three models in Figure 11, from which we obtain a better understanding about the correct predictions and misclassifications. Taking SSD Inception V2 (the left matrix) as an example, 188 out of 203 normal samples were correctly predicted, nine were misclassified as scratch, and six were misclassified as sprout. After checking, we found that these errors are caused by the mud, spot, or sprout eyes on the potato surface, which confused the model. Furthermore, 195 out of 210 scratch samples were correctly predicted, nine were predicted as normal, and six were predicted as sprout; an error analysis reveals that the potatoes with superficial scratches might be predicted as normal, and the ones with deep and large scratches could be predicted as sprout. As for the sprouted samples, 257 were correctly predicted, eight were misclassified as normal, and fourteen were predicted as scratch; our error analysis shows that the model tends to predict the potatoes with small sprouts or the ones with large but hidden sprouts due to camera shooting anger. RFCN ResNet101 presented the best performance, also reflected in the confusion matrix, in which only nine out of 693 potatoes were misclassified.

In addition, we report the Acc, Pre, Rec, Spe, and F1 in

Table 3. Model performance on the test set. Abbreviations: S.I.V2 = SSD Inception V2; F.R.R.N.101 = Faster RCNN ResNet101; R.R.N.101 = RFCN ResNet101.

Model	Category	Acc	Pre	Rec	Spe	F1
S.I.V2	Normal	92.5%	91.7%	92.6%	92.4%	92.2%
	Scratch	92.5%	89.4%	92.9%	92.3%	91.1%
	Sprout	92.5%	95.5%	92.1%	92.7%	93.8%
	Macro avg.	92.5%	92.2%	92.5%	92.5%	92.4%
F.R.R.N.101	Normal	95.7%	95.1%	96.1%	95.5%	95.6%
	Scratch	95.7%	93.9%	94.8%	96.1%	94.3%
	Sprout	95.7%	97.5%	96.1%	95.4%	96.8%
	Macro avg.	95.7%	95.5%	95.6%	95.7%	95.6%
R.R.N.101	Normal	98.7%	99.0%	98.5%	98.8%	98.8%
	Scratch	98.7%	98.1%	99.0%	98.5%	98.6%
	Sprout	98.7%	98.9%	98.6%	98.8%	98.7%
	Macro avg.	98.7%	98.7%	98.7%	98.7%	98.7%

. It is observed that the variance across metrics is small for all three models, meaning that (1) classes in the dataset are relatively balanced so that models can gain about equal amount of knowledge from each category during training, and (2) the hard cases were not many. Furthermore, the inter-class variance is slightly larger. Taking SSD Inception V2 as an example, the Pre for the three categories were 91.7%, 89.4%, and 95.5%, respectively. As a higher Pre indicates less false alarms, and vice versa, we find that the false alarm rate for the scratch samples is the highest among the three classes, and that the false alarm rate for the sprouted samples is the lowest.

Figure 12 shows the average inference speed of the three models. SSD Inception V2 is the fastest (51.3 FPS) because it is a single-stage object detector. Both Faster RCNN (21.2 FPS) and RFCN (28.5 FPS) are two-stage detectors. All three models can meet the requirement of real-time inference in practice with an FPS obove 20 [58,59,60]. Furthermore, as RFCN ResNet101 achieves a good balance between accuracy and speed, we select it as the best one among the three and further evaluate it on the OOS test set.

3.4. Out-of-Sample Testing

3.4.1. Scenario One: Effect of Different Batches

The results are shown in Figure 13a. The model performed exceptionally well with an Acc of 100% for all three categories. After checking, we find that scenario one represent an ideal situation where the camera was facing down pointing to the potato sample so that the image can capture the most informative features, which can be easily distinguished by our model.

3.4.2. Scenario Two: Effect of Different Detection Regions

Figure 13b and

Table 4. Effect of different detection regions on model performance for the OOS test set.

Category	Acc	Pre	Rec	Spe	F1
Normal	95.60%	96.00%	100.00%	93.50%	97.90%
Scratch	95.60%	92.70%	95.30%	95.80%	94.00%
Sprout	95.60%	98.50%	91.60%	97.70%	94.90%
Macro avg.	95.60%	95.70%	95.60%	95.60%	95.70%

show the model performance in scenario two. It is observed that a number of scratched and sprouting potatoes were predicted as normal. After error analysis, we found that for those misclassified samples, the characteristic parts of the surface, i.e., scratches and sprouts, happened to locate in the blind area of the camera shooting, leading to misclassification due to the lack of enough predictive features. This finding shows that it is crucial to capture sample images from multiple angles to prevent key surface features from being missed.

3.4.3. Scenario Three: Effect of Different Light Intensities

In scenario three, four brightness levels were offered during test. We find that for light levels two and three, the RFCN ResNet101 presented an Acc of 100% for all categories, which is reasonable as with moderate brightness and in a closed and controlled space (even better than scenario one), the model should behave consistently well. As shown in Figure 14 and

Table 5. Effect of light intensities on model performance for the OOS test set.

Light Level	Category	Acc	Pre	Rec	Spe	F1
One	Normal	97.50%	97.30%	100.00%	96.30%	98.60%
	Scratch	97.50%	96.70%	96.30%	98.10%	96.50%
	Sprout	97.50%	98.60%	96.30%	98.10%	97.40%
	Macro avg.	97.50%	97.50%	97.50%	97.50%	97.50%
Four	Normal	93.30%	93.00%	100.00%	90.00%	96.40%
	Scratch	93.30%	91.30%	88.80%	95.60%	90.00%
	Sprout	93.30%	95.60%	91.10%	94.40%	93.30%
	Macro avg.	93.30%	93.30%	93.30%	93.30%	93.30%

, for level one, which is the case with the lowest light, RFCN ResNet100 showed an Acc of 97.5%, with a total of sixteen errors out of 642 samples; for level four, the case with the highest light, RFCN ResNet100 showed a decreased Acc of 93.3%. An error analysis on level one tells us that a darker image did affect the model’s ability to discover patterns, which were of low quality given insufficient light. Furthermore, for level four, stronger light resulted in light reflection on the potato surface, which also lowered the image quality since certain informative areas were covered by reflection.

Despite the performance degradation, RFCN ResNet101 can still achieve satisfying performance in this OOS testing. Results demonstrated that this model is in line with the requirements for industrial-level applications.

4. Discussion

Detecting defective surface for potatoes is a crucial task in post-harvest quality control during potato production. In this study, we consider two typical defective types, including sprouting and scratches caused by mechanical damage in harvest. Sprouted potatoes are toxic to human beings owing to the high level of glycoalkaloids, while scratched potatoes account for over 70% of damaged potatoes during harvesting. The surface patterns of these two damages carry rich imagery information that can be exploited to build automated and intelligent grading systems for potatoes. Such a system can tremendously improve the efficiency and efficacy of the quality control process and reduce the cost of post-harvest operations.

In the past two decades, computer vision-based grading systems have been developed and applied to sort potatoes and other food products by size, shape, and external defects. The early sorting systems have extensively applied image processing algorithms to extract a set of hand-crafted image features from the potato images to build a classifier. As the features, such as central moments, area, and eccentricity, are manually defined, extensive domain knowledge is required. In addition, these feature-based learning models do not generalize well, leading to performance degradation in OOS testing. The recent advances of deep learning is proven to be effective to address the weaknesses of traditional feature-based models. In computer vision, DCNN models have refreshed numerous performance records in a wide spectrum of tasks, such as image classification, object detection, and semantic segmentation. In the food industry, DCNN models have been adopted to build intelligent quality control systems. However, the use of DCNN models for potato surface defect detection has not been sufficiently explored. To our best knowledge, this study is the first to consider both sprouted and scratched potato damages and treat the detection problem as a transfer learning-based object detection task.

To validate the DCNN models, we developed an annotated dataset with a total of 2770 images, consisting of 813 normal potato images, 841 scratched ones, and 1116 sprouted ones. To further evaluate a model’s robustness, a total of 642 potatoes were used to build an OOS test set with three test scenarios. We investigated the effectiveness of three DCNN models, including SSD Inception V2, RFCN ResNet101, and Faster RCNN ResNet101, using the self-developed dataset. After a thorough comparison, the RFCN ResNet101 model outperformed the other two and was selected as the final model for out-of-sample testing. We conducted an error analysis on the misclassified samples, which were mainly caused by avoidable factors like hidden areas of scratches or sprouts and light reflection. Our findings lay the foundation of automated systems for potato surface defect inspection.

This work is subject to the following limitations that can be addressed in future work. First, this study on considered the three most important categories, while there are other categories and/or potato properties, such as broken, shape, size, and color that are also desired in a grading system. Second, gathering samples is time-consuming and costly, we plan to investigate data augmentation techniques to enhance the diversity and the size of the training set, which has been validated by other applications as an effective performance booster [61]. Third, recent advances in attention mechanisms have offered us another direction. Attention in computer vision tasks allow a DCNN to learn what and where to focus on in the input image, which would also benefit the surface defect detection task.