Gastrointestinal Disease Classification in Endoscopic Images Using Attention-Guided Convolutional Neural Networks

Abstract

Gastrointestinal (GI) diseases constitute a leading problem in the human digestive system. Consequently, several studies have explored automatic classification of GI diseases as a means of minimizing the burden on clinicians and improving patient outcomes, for both diagnostic and treatment purposes. The challenge in using deep learning-based (DL) approaches, specifically a convolutional neural network (CNN), is that spatial information is not fully utilized due to the inherent mechanism of CNNs. This paper proposes the application of spatial factors in improving classification performance. Specifically, we propose a deep CNN-based spatial attention mechanism for the classification of GI diseases, implemented with encoder–decoder layers. To overcome the data imbalance problem, we adapt data-augmentation techniques. A total of 12,147 multi-sited, multi-diseased GI images, drawn from publicly available and private sources, were used to validate the proposed approach. Furthermore, a five-fold cross-validation approach was adopted to minimize inconsistencies in intra- and inter-class variability and to ensure that results were robustly assessed. Our results, compared with other state-of-the-art models in terms of mean accuracy (ResNet50 = 90.28, GoogLeNet = 91.38, DenseNets = 91.60, and baseline = 92.84), demonstrated better outcomes (Precision = 92.8, Recall = 92.7, F1-score = 92.8, and Accuracy = 93.19). We also implemented t-distributed stochastic neighbor embedding (t–SNE) and confusion matrix analysis techniques for better visualization and performance validation. Overall, the results showed that the attention mechanism improved the automatic classification of multi-sited GI disease images. We validated clinical tests based on the proposed method by overcoming previous limitations, with the goal of improving automatic classification accuracy in future work.

1. Introduction

Gastrointestinal (GI) diseases are common in the human digestive system. Stomach cancer, esophageal cancer, and colorectal cancer are most common in terms of incidence and fatality [1,2]. Endoscopic examinations are thus vital to detect diseases and form the critical initial step for diagnosing GI tract diseases generally [3]. Such examinations also enhance the assessment of the clinical features of lesions to determine their severity and type and to arrive at proper diagnoses. Variations in the expertise of different clinicians could introduce errors in some cases, especially with respect to controversial aspects of diagnostic images and videos from endoscopic examinations. Such inconsistency may lead to misdiagnoses and negative impact on patient care.

Automatic disease classification potentially addresses this problem by providing clinicians with objective and reliable identification of several GI endoscopic images, thereby minimizing the misdiagnosis rate, improving prognosis, and economizing clinicians’ valuable time. Automatic GI disease classification thus remains an open area of research for attaining better lesion detection and classification accuracy [4].

Recently, artificial intelligence based on deep learning (DL) has demonstrated remarkable progress in classification, detection, fault diagnosis, and segmentation tasks [5,6,7,8,9]. However, a drawback to the potential of DL is that large amounts of data are required for optimum performance. In addition, difficulty in obtaining GI images is increased by patient privacy concerns and annotation costs [2,10,11]. In short, optimal use of the DL approach to automatic GI disease classification is constrained by data scarcity. Unlike classic machine learning-based classifiers such as support vector machines (SVMs) that are used for feature extraction [12], convolutional neural networks (CNNs) have shown better performance in feature extraction, making them state-of-the-art for DL applications [8]. Effective use of CNNs has improved image recognition- and classification-related tasks [8].

The foundation of CNN is the convolution operation that fuses spatial and channel-wise information within local receptive fields to extract information features, a process known as spatial encoding [13]. However, the vanilla CNN mechanism would allow the CNN operation to pay more attention to the pixel regions that play a decisive role in classifying input images while ignoring irrelevant information [14]. To improve the suggestive power of a CNN network, several recent studies have shown the benefit of enhancing the spatial encoding ability of CNN via spatial attention modules. Other studies have indicated promising CNN-based approaches for disease classifications [9,12,15,16,17,18,19,20]. In addition, spatial attention mechanisms have been studied and applied to related tasks [16,17,21,22,23].

To the best of our knowledge, GI-tract disease classification of multi-sited, multi-class diseases and artifacts, from GI endoscopic images using attention-guided CNNs, has not previously been applied. Multi-class disease and artifact classification and generalization are essential not only for diagnosis but also to avoid training biases. Our goal in this study, therefore, is to design, test, and validate an efficient spatial attention-guided CNN-based disease classification model that can be used in a computer-aided diagnosis (CAD) system for automatic GI disease classification. We examine a total of 10 classes: esophagitis, polyps, ulcerative colitis, early esophagus cancer, normal cecum, normal Z-line, normal pylorus, dyed-lifted polyps, dyed-resection-margins, and artifacts. Moreover, we adapt a t-distributed stochastic neighborhood embedding (t–SNE) visualization technique to better understand the data. [14].

Automatic classification of GI diseases using CAD can assist in the diagnosis of GI disease and in the efficient, effective, and safe removal of lesions [24,25,26]. Dimensionality reduction (DR) is a key step in feature extraction in images that often contain irrelevant information or labeling. When applied in this work, t–SNE was effective in enhancing data pattern visualization and feature extraction by preserving high intrinsic dimensionality [27,28]. Accordingly, this study proposes an efficient DL-based CAD system for the classification of multi-class diseases and artifacts in GI endoscopic images.

The main contributions of this paper can be summarized as follows:

(1)

We propose an efficient method that incorporates spatial attention CNN for classifying multi-class diseases and artifacts in GI endoscopic images.

(2)

We performed extensive experiments to validate the effectiveness of the proposed model. Moreover, we compared our results with recent related models and demonstrated better outcomes.

(3)

The proposed method demonstrated significant performance accuracy in GI disease classification by using spatial attention mechanisms and t–SNE.

(4)

The proposed GI disease classification method was validated for clinical applications and has great potential for medical communities.

In the next section, we describe the materials and method of our proposed approach. In Section 3, we address the experiments and results. In Section 4, we provide a detailed discussion of the results before concluding the paper in Section 5.

2. Materials and Methods

2.1. Materials

A total of 12,147 GI endoscopic images were obtained from hospital and public data sources and used to validate our proposed approach. The details of each dataset are described below:

2.1.1. Kvasir Multi-Class Dataset

Eight thousand multi-class GI endoscopy images (8 classes with 1000 images in each class) were collected from the Kvasir dataset v2 [2] and verified by experienced medical specialists. The datasets were collected using endoscopic equipment at Vestre Viken Health Trust in Norway. The pathological results included esophagitis (ESO), polyps (POL), ulcerative colitis (ULC), etc., while the anatomical landmarks included Z-line (ZLI), pylorus (PYL), cecum (CEC), etc. In addition to images related to removing lesions, such as a dyed and lifted polyp (DLP), the dyed resection margins (DRM) were used for the experiments. The dataset consisted of images with resolutions ranging from 720 × 576 up to 1920 × 1072 pixels. Some of the included GI images have artifacts indicating the configuration of the position of the endoscope inside the bowel, using an electromagnetic imaging system [2].

2.1.2. Endoscopy Artifact Detection Challenge Dataset

The endoscopy artifact detection (EAD2019) challenge dataset [4] provided 2147 images collected from six different centers. The images varied in resolution from 295 × 299 to 1920 × 1080 pixels in JPG format. Most of the images were used for endoscopic artifact detection (EAD) classification.

2.1.3. Gastrointestinal Endoscopy Dataset

We used 2000 early esophagus cancer (EEC) GI endoscopy images collected from 389 patients and verified by physicians from the Digestive Endoscopy Center (Lab) of the West China Hospital in Sichuan, China. The images were saved in the RGB channel in JPG format, with resolutions ranging from 512 × 512 to 764 × 572 pixels. Permission was obtained from the medical ethical review committee of the Electronic University of Science and Technology of China (UESTC) and West China Hospital, and the informed consent of patients was received. Samples of raw GI endoscopic images are depicted in Figure 1.

2.2. Methods

This subsection describes the DL-based method for using the attention mechanism in the GI disease classification of multi-class and multi-sited endoscopic images. In step 1, unnecessary background features, text, and other artifacts were removed from each sample dataset during pre-processing. Images were resized to 224 × 224 pixels for the experiments. As shown in Figure 1, the raw images that contained unnecessary background and text were removed during pre-processing, and the class balancing technique was applied with ECA-DDCNN [15]. Eighty percent of the data were used for training and 20% for testing. To avoid overfitting and imbalance, data augmentation techniques were used, such as transposing, flipping, rotating, random brightness, blurring, distortion, contrasting, and limited adaptive histogram equalization. After data augmentation, the total training data were comprised of forty thousand images. In step 2, the pre-processed data were utilized for the whole model’s experiments. In step 3, disease detection and classification were implemented with attention mechanisms. In step 4, disease classification or lesion categorization of the proposed method was validated using 20% of the test data. Finally, in step 5, the output was evaluated quantitatively and qualitatively using appropriate evaluation metrics and visualization techniques as shown in Figure 2.

In this work, we propose an attention network, a CNN, that adopts a mixed attention mechanism, as shown in Figure 3 (main network) and Figure 4 (sub-network). It comprises multiple attention layers/modules, which generate attention features by stacking layers to optimize the network [18]. The proposed model explores the performance of a robust classifier on CNN architecture. The spatial attention module compresses the channel and performs average and maximum pooling in the channel dimensions. The fully connected (FC) layers are then used to capture the non-linear cross-channel interaction, during which the complexity of the model is minimized by reducing dimensionality. The performance of the proposed approach was tested on the data used in training and validation. Attention features generated at stage 2 are shown in Figure 5.

3. Experimental Setup

The proposed method explores the effect of a robust classifier on deep CNN architecture. Most common CNN architectures, such as GoogLeNet and ResNet, are used as a backbone [29]. The important features of each architecture were initially learned, followed by training with the three sets of GI endoscopic datasets. In the feature extraction procedure, the original architectures were not changed. The primary attention-based model was used for baseline network training. Stochastic gradient descent (SGD) was implemented to optimize the network with a 0.001 initial training rate. Total training epochs and batch size were 120 and 32, respectively. Input was resized to 128 × 128 pixels to crop the ROI. The programming language implemented in python 3.6.12 and Pytorch 1.7.1 DL library (https://github.com/pytorch/pytorch (accessed on 11 February 2021)) was used for the experiments. Experiments were implemented on a server-based Ubuntu 16.04.6 LTS. The system was equipped with four graphic processing units (GPUs) of NVIDIA GeForce RTX 2080Ti with 11 GB memory each.

We used several evaluation metrics to validate the classification performance of the proposed method, including accuracy (Acc), precision (Pre), recall (Rec), sensitivity (Sen), specificity (Spe), and F1-score (F1). F1 measures a test accuracy by calculating the harmonic mean of Pre and Rec [2]. Acc specifies the ratio of GI-disease patients to non-diseased patients.

$$Acc=\frac{TP+TN}{TP+TN+FP+FN}$$

(1)

$$Pre=\frac{TP}{TP+FP}$$

(2)

$$Rec=\frac{TP}{TP+FP}$$

(3)

$$F1=2\times \frac{Pre\times Rec}{Pre+Rec}$$

(4)

4. Results and Discussion

This section presents results and a discussion of the experimental results based on the DL approach. The DL approach has been shown to enhance the performance of GI disease classification tasks significantly.

Table 1. Model performance complexity comparisons.

Models	Evaluation Metrics
	Mean Accuracy (%)	Parameters (Million)
ResNet50 [30]	90.28	21.71
GoogLeNet [31]	91.38	5.61
DenseNets [32]	91.60	25.6
Baseline (Ours)	92.84	19.92

depicts the model’s performance and its parameters. The most common architectures [30,31,32] and baseline (ResNet50 = 90.28%, GoogLeNet = 91.38%, DenseNets = 91.60%, and baseline = 92.84%) demonstrated classification accuracy.

Table 2. Cross-validation statistical comparisons (mean values) on test datasets.

Folds	Evaluation Metrics
	Precision	Recall	F1-Score	Accuracy
Fold1	91.8	91.8	91.7	92.16
Fold2	92.5	92.4	92.4	92.88
Fold3	92.4	92.5	92.6	92.91
Fold4	92.8	92.7	92.8	93.19
Fold5	92.8	92.6	92.7	93.12

reveals cross-validation results in different five-folds, with reference to the size of the test dataset and other parameters. In

Table 3. Statistical comparisons of related works (mean values).

Methods	Evaluation Metrics
	Precision	Recall	F1-Score	Accuracy
ECA–Net [23]	92.4	92.4	92.2	92.81
DL–OCT [10]	90.1	90.05	90.1	90.60
LSMT–CNN [29]	92.8	92.8	92.6	93.10
Liu et al. [5]	92.7	92.6	92.7	93
Ours	92.8	92.7	92.8	93.19

, we compared the proposed method with other related methods [10,23,29]. The mean accuracy for ECA–Net equaled 92.81; for DL–OCT, it was 90.60; for LSTM-CNN, it was 93.10; for Liu et al. it was 93; for our method, it was 93.19.

Table 3 demonstrates a statistical comparison with other related methods. The comparison between the proposed method and other state-of-the-art methods was evaluated based on the same data and on standard evaluation metrics.

Figure 6 demonstrates statistical comparisons of the DL method’s performance in the confusion matrix. The confusion matrix calculates the actual and predicted rates for each disease category. The results of all four methods presented on the confusion matrix indicate disease classification performance. The test data ranges from 300 to 400 in each class. Overall performance accuracies of the respective methods were ECA–Net = 92.81; DL–OCT = 90.60; LSTM + CNN = 93.10; and Ours = 93.19. The ECA–Net method [23] revealed the GI disease or artifacts classification performance, with the pylorus classification of 100% and the lowest classification in esophagitis (82%). The DL–OCT [10] method also demonstrated competitive classification accuracy, classifying pylorus at 98% and polyps at 81%. The LSMT–CNN [29] method exhibited competitive classification performance in pylorus (100%) and showed the lowest performance in the esophagitis class (82%). The proposed method classified pylorus at 100% and showed the lowest performance in the esophagitis disease classification (84%). Nearly all the methods classified pylorus best and misclassified the Z-line. Only a few data items were misclassified. Overall classification accuracy of all related methods was promising. The proposed method outperformed other methods significantly on GI disease or lesion classification by achieving a mean accuracy of 94.33%. Overall, the proposed method validated actual and predicted classifications better than other methods.

It is essential to determine the location (anatomical and physiological) of the digestive system. Figure 7 confirms disease classification results using t-distributed stochastic neighbor embedding (t–SNE) visualization techniques, which show the transformation of high-dimensional features into low dimensions by preserving key features of endoscopic images. This technique provides better identification of each class in terms of depth and color. The proposed method, including other related works [10,23,29] demonstrated t–SNE-based classification performance. Our method achieved better qualitative results when compared with other works.

In this study, t–SNE was applied to preserve the local structure of the data by a non-linear dimension reduction (DR) approach. A performance analysis was implemented to demonstrate the performance difference in the methods [33]. The t–SNE visualization showed significant improvement in terms of feature extraction and DR for GI disease classification. It preserved complex non-linear data structures by maintaining the local similarity structure of the data [27]. The t–SNE technique could efficiently project complex 2D endoscopic datasets. High-dimensional space was preserved as much as possible. Our t–SNE-based disease classification exhibited better feature extraction performance than other methods.

The proposed method achieved better results in classifying GI diseases when compared with other recent related methods. However, the results of the proposed method come with some limitations. A fully supervised learning method needs a massive GI endoscopic image dataset, which is the method’s main drawback. Another limitation of the proposed approach may be its lack of focus on pixel-wise classification and detection.

5. Conclusions

This work presents a deep CNN-based GI disease classification method by employing an efficient spatial attention mechanism. Three GI endoscopic image datasets with multi-classes were used to validate the proposed method. We validated model performance complexity and conducted five-fold cross-validation to confirm the proposed method. The experimental results show that our approach is more efficient when compared with other related methods. Accordingly, our proposed method has the potential to aid in the clinical diagnosis of various GI diseases.

In future works, we propose that special attention should be given to improving classification accuracy by combining DL techniques with clinical features and by exploring the stability of the proposed method. In addition, we intend to perform clinical testing of the proposed model in determining its value in assisting the diagnoses of GI diseases.