Accelerating Detection of Lung Pathologies with Explainable Ultrasound Image Analysis

Abstract

Care during the COVID-19 pandemic hinges upon the existence of fast, safe, and highly sensitive diagnostic tools. Considering significant practical advantages of lung ultrasound (LUS) over other imaging techniques, but difficulties for doctors in pattern recognition, we aim to leverage machine learning toward guiding diagnosis from LUS. We release the largest publicly available LUS dataset for COVID-19 consisting of 202 videos from four classes (COVID-19, bacterial pneumonia, non-COVID-19 viral pneumonia and healthy controls). On this dataset, we perform an in-depth study of the value of deep learning methods for the differential diagnosis of lung pathologies. We propose a frame-based model that correctly distinguishes COVID-19 LUS videos from healthy and bacterial pneumonia data with a sensitivity of $0.90\pm 0.08$ and a specificity of $0.96\pm 0.04$. To investigate the utility of the proposed method, we employ interpretability methods for the spatio-temporal localization of pulmonary biomarkers, which are deemed useful for human-in-the-loop scenarios in a blinded study with medical experts. Aiming for robustness, we perform uncertainty estimation and demonstrate the model to recognize low-confidence situations which also improves performance. Lastly, we validated our model on an independent test dataset and report promising performance (sensitivity $0.806$, specificity $0.962$). The provided dataset facilitates the validation of related methodology in the community and the proposed framework might aid the development of a fast, accessible screening method for pulmonary diseases. Dataset and all code are publicly available at: https://github.com/BorgwardtLab/covid19_ultrasound.

1. Introduction

To date, SARS-CoV-2 has infected more than 90 million and killed more than 1.9 million patients around the globe (https://0-coronavirus-jhu-edu.brum.beds.ac.uk/map.html (accessed on 11 January 2020)). Its long and dispersive incubation time calls for fast, accurate, and reliable techniques for early disease diagnosis to successfully fight the spread [1]. The standard genetic test (RT-PCR ) has a processing time of up to 2 days [2]. Several publications have reported sensitivity as low as 70% [3,4] and a meta-analysis estimated the false negative rate to be at least 20% over the course of the infection [5]. Medical imaging complements the diagnostic process that can guide further PCR-testing, especially in triage situations [6]. CT (Computed Tomography) scanning is the imaging gold standard for pulmonary diseases [7] and is considered reliable for COVID-19 diagnosis in some countries [4,8,9], although a significant amount of patients exhibit normal CT scans [10]. However, CT scanning is expensive and highly irradiating, carries significant risk of cross infection to healthcare workers and requires extensive, time-consuming sterilization [11]. It is furthermore reserved for developed countries; there are only ∼30 k CT scanners globally [12]. A chest X-ray (CXR) is the most common first line procedure in diagnostic imaging, despite reports of low specificity and sensitivity for COVID-19 (for example, Reference [13] found 89% normal CXR in 493 COVID-19 patients). Ultrasound (US), by contrast, is a cheap, safe, non-invasive and repeatable technique that can be performed with portable devices at patient bedside and is ubiquitously available around the globe [14]. Lung ultrasound (LUS) developed into an established tool to diagnose pulmonary diseases [15,16,17], as it is superior to CXR for detecting pulmonary conditions [18,19,20,21].

During the COVID-19 pandemic, the growing body of evidence for disease-specific patterns in lung US, mostly B-lines and pleural line irregularities, has led to the advocacy for an amplified role of LUS [22,23,24,25]. It was argued that LUS could be performed routinely in COVID-19 suspects and become part of the diagnostic toolkit for COVID-19 differential diagnosis [23,24,26,27]. Notably, in COVID-19 patients, LUS has a higher diagnostic sensitivity than CXR [28]. Moreover, radiologists reported inter and intra-observer agreement between US and CT findings [29,30]. Some studies found the diagnostic accuracy of LUS for COVID-19 to be on par with CT [31,32], and even more sensitive in detecting pulmonary imaging biomarkers [10]. Hence, in triage or resource limited settings, LUS can be a valuable technique [33,34] and serve as a globally available first-line examination method to guide downstream testing [35]. Despite this encouraging evidence, US is not (yet) widely adopted in clinical practice for cardiac or lung imaging and its wider deployment is hampered for a multitude of reasons such as operator-dependent acquisition [36]. Notably, a recent “perspective” paper in The Lancet has pointed out the adoption of deep learning (DL) technologies for the guidance and interpretation of US images as a major challenge for the digitation of medicine [26] Citing: “Theoretically, if these (POCUS) devices were widely used, rapid, point-of-care imaging could become routine and reduce the need for formal studies in a dedicated radiology suite. But that shift of adoption is predicated on the ability of clinicians to become facile in obtaining high-quality ultrasound scans. To date, that has not occurred for various reasons including cost of the devices, issues for reimbursement, and difficulty with performing image capture, which has historically been a specialised task for sonographers”. Moreover, the relevant LUS pattern can be hard to discern, requiring time and trained personal [37], This calls into play medical image analysis systems based on machine learning (ML) which aim to be utilized as clinical support tools for physicians that aid data acquisition, patient diagnostics or monitoring. Here, we release a novel LUS dataset with diverse pulmonary manifestations and perform a first study on the automatic detection of lung pathologies for differential diagnosis.

1.1. Related Work

Literature on exploiting medical image analysis and deep learning (DL) to classify or segment CT or CXR data of COVID-19 patients recently exploded (for reviews, see Ulhaq et al. [38], Shi et al. [39]). More than 300 publications on this topic appeared only in 2020 [40]. For instance, Mei et al. [2] achieved equal sensitivity (but lower specificity) compared to senior radiologists in detecting COVID-19 from CT and clinical information. A meta-analysis on COVID-19 revealed a mismatch between clinical and ML communities and found that US is significantly under-explored by ML researchers [40].

With the rise of deep learning in computer vision, learning-based approaches to medical ultrasound analysis became increasingly popular over the last years [41]. While more than 50 papers were published on deep learning on US in 2017 [41], sparse work has been done on lung US, with B-line detection/quantification being the most common task [42,43,44]. Others focus on pleural line extraction [45], subpleural pulomnary lesions [46] or lung cancer detection [47].

In our initial preprint dubbed POCOVID-Net [48], we were first to regard the problem of automatic differential diagnosis of COVID-19 from LUS data. The released dataset has been incorporated by several authors into their work [49,50,51,52]. Others developed a quality assessment module for COVID-19 pattern detection [53] or a classifier for typical LUS patterns of COVID-19 patients [54,55]. Especially related to our work is the preprint by Arntfield et al. [56] which proposed a model to differentiate types of B-lines according to the clinical diagnosis (COVID-19, non-COVID-19-ARDS, hydrostatic pulmonary edema). However, none of the above works have released their utilized datasets to the public.

An exception to the neglect of US during the first months of COVID-19 a deep learning approach for a severity assessment of COVID-19 from US data [57]. The work convincingly predicts disease severity and segments COVID-19 specific patterns building up on their previous work on localizing B-lines [42]. The paper claims to release a segmentation dataset of COVID-19 cases, but to date, only a small fraction of class-labelled data and no segmentation annotations are available. While their effort (and follow-ups [45,58,59]) on severity assessment are highly relevant for disease monitoring, they are not applicable for diagnosis, where the main problem lies in distinguishing COVID-19 from other lung pathologies (i.e., clinical diagnoses). We aim to close this gap with our approach to classify COVID-19, healthy, and bacterial pneumonia from LUS data.

1.2. Our Contributions

Figure 1 depicts a graphical overview of our contributions.

First, we release the largest to-date publicly-available dataset of lung US recordings, consisting of 202 videos and 59 images. Since the initial release of our database in a preprint [48], further research has used our dataset [52,60] or followed up on our approach with studies of different neural network architectures [55] or even leveraging robustness [49]. Our dataset is heterogeneous and includes clinical data from one hospital as well as data from public sources, which were curated manually and approved by two medical experts.

Second, we take a first step towards a tool for differential diagnosis of pulmonary diseases, here especially focused on bacterial and viral pneumonia, exemplified with COVID-19 as viral pneumonia. Specifically, we demonstrate that competitive performance can be achieved from raw US recordings, thereby challenging the current focus on irradiating imaging techniques. Moreover, we employ explainability techniques such as class activation maps or uncertainty estimates and present a roadmap towards an automatic detection system that can highlight relevant spatio-temporal patterns. Our proposed system presents a step towards a tool that aids medical care which, in the midterm, could potentially help to reduce time and cost in the clinical workflow, as shown for CT [2].

2. A Lung Ultrasound Dataset for COVID-19 Detection

2.1. Dataset Description

We release the largest publicly-available LUS dataset (https://github.com/BorgwardtLab/covid19_ultrasound), comprising samples of COVID-19 patients, patients with bacterial pneumonia, (non-COVID-19) viral pneumonia and healthy controls. As shown in

Table 1. Dataset size. Number of videos and images in our dataset, per class and probe.

	Convex		Linear
	Vid.	Img.	Vid.	Img.	Sum
COVID-19	64	18	6	4	92
Bacterial Pneu.	49	20	2	2	73
Viral Pneu.	3	–	3	–	6
Healthy	66	15	9	–	90
Sum	182	53	20	6	261

, we collected and gathered 261 recordings (202 videos + 59 images) recorded with either convex or linear probes from a total of 216 patients. Linear probes are higher frequency, yielding a greater resolution that allows better to study abnormalities around the pleural line [61]. However, the linear probe penetrates the tissue less than the convex probe which can hamper the differentiation of B-lines [62] and does not allow the assessment of deeper lung tissue. In Section 5.2 we discuss limitations of our dataset regarding the heterogeneity of US probes in the data, as well as shortcomings with respect to the distinction of COVID-19 from other viral pneumonias. Note that due to the low number of samples (3 convex videos), we did not use the non-COVID-19 viral pneumonia data in the analysis, but instead only distinguish the remaining three classes.

Our dataset comprises clinical data donated from hospitals or academic ultrasound course instructors, as well as LUS recordings published in other scientific literature, community platforms, open medical repositories and health-tech companies. An overview about the sources and a full list of utilized publications is listed in the supplementary material (

Table A1. Data sources. Overview of all data sources, including a comprehensive list of publications and educational websites that provide data included in our analysis.

Data Source	Data Selected	Description
Northumbria (NH NHS-FT)	Convex: 47 videos and 23 images (31 healthy and 39 bacterial pneumonia infected patients)	The Northumbria Healthcase NHS Foundation Trust (NH NHS-FT) contributed patient data (images and videos) to our dataset
Neuruppin (MHB)	Convex: 28 videos Linear: 3 videos (all healthy)	Ultrasoud course instructors from Medizinische Hochschule Brandenburg Theodor Fontane (MHB) recorded volunteers that did not show any symptoms of COVID-19 infections and were not tested positively
Publications	Convex: 15 images and 30 videos from all classes Linear: 4 images, 4 videos	Miscallaneous LUS videos and images were fetched from publications [19,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98]
GrepMed	Convex: 9 COVID-19, 9 pneumonia and 2 healthy Linear: 3 COVID-19, 1 healthy (all videos)	GrepMed is a community-sourced, medical image repository for referencing clinically relevant medical images
Butterfly	Convex: 18 COVID-19 and 2 healthy videos	Butterfly is a vendor of a portable US device needing only a single probe usable on the whole body that connects to a smartphone
ThePocusAtlas	Convex: 8 COVID-19, 2 pneumonia and 3 healthy videos Linear: 2 COVID-19 videos	ThePocusAtlas is a Collaborative Ultrasound Education Platform
LITFL	Convex: 5 bacterial and 2 viral pneumonia (H1N1), 2 healthy Linear: 1 H1N1 (all videos)	Australasian critical care physicians maintain an educational platform and provide an ultrasound library with case studies
Web	Convex: 15 images, 15 videos Linear: 2 images, 6 videos from all classes	Remaining online sources were: https://www.stemlynsblog.org/, https://clarius.com/, https://everydayultrasound.com/, https://radiopaedia.org, https://www.acutemedicine.org, https://www.bcpocus.ca, https://www.youtube.com, www.sonographiebilder.de/
Bolzano AG	45 linear videos of probably COVID-19 infected patients
(Data not used for any analysis presented herein)	Videos were recorded in spring 2020 in Piacenza (Italy) from patients suspected of COVID-19. Diagnosis was not confirmed via PCR or thorax imaging. Bolzano AG donated this data to our dataset.

). The COVID-19 diagnosis was normally obtained via RT-PCR, but for details on individual recordings, we refer to the respective sources. We consider it a major contribution to assemble this dataset from 41 distinct sources, which included web scraping, labeling, pre-processing (cropping, artifact removal etc.), commenting and approving by medical professionals and US operators. The dataset is accompanied by extensive metadata table, listing the source URL, source ID, an anonymized patient ID and, when available, age, gender, disease symptoms and pathological patterns. Further technical details comprise image resolution, frame rate and the number of frames for each video after pre-processing. Importantly, all samples of our database were reviewed and approved by two medical experts (a paediatric physician with $10+$ years of clinical LUS experience and an academic US course instructor) and annotated with visible LUS patterns.

2.2. Data Collection

2.2.1. Northumbria Data

Patient data was made available by the Northumbria Healthcare NHS Foundation Trust. The Trust serves a population of approximately 600,000 over a large geographical area in the North East of the United Kingdom. During the COVID-19 pandemic, inpatient respiratory services were centralised onto the acute care centre. Thoracic ultrasound was conducted with a convex probe Venue™ ultrasound machine by GE Healthcare (2–5 MHz). Patients were scanned according to the established BLUE protocol [62] which has high diagnostic sensitivity, specificity and accuracy for pleural effusions, alveolar consolidation and interstitial syndromes [63]. RT-PCR was done to confirm/reject COVID-19 diagnosis and standard care (including thoracic X-Ray and CT) was done to identify bacterial pneumonia. Local Caldicott approval was sought and granted. Patient were consented appropriately [64,65]. Recordings with appropriate pathology were stored, anonymized and electronically shared. A total of 70 videos and images from 44 male and 26 female patients were provided.

2.2.2. Neuruppin Data

As an independent control group, healthy volunteers were recruited at Brandenburg Medical School Theodor Fontane in Neuruppin, Germany. Data was acquired at a time of low prevalence. Similarly to the clinical data, LUS was conducted with a GE Healthcare US device and executed according to the BLUE protocol [62]. 31 videos (28 convex, 3 linear) from 6 patients (3M/3F) were provided.

2.3. Dataset Analysis

In images and videos from 41 different sources, we count 216 distinct patients. Unfortunately, in many online sources or publications no patient metadata was given, but we were able to collect age and gender for 42% of the data. Out of those, 57% were male, and the average age is $41.3$ years (median: 35, standard deviation: $24.7$). Additionally, descriptions of symptoms were available for 30% of our LUS recordings and are distributed as shown in Figure 2a.

Last, the pathological patterns as annotated by the physicians are shown in Figure 2b. The results are in accordance with prior knowledge and corroborate the non-specificity of COVID-19 patterns compared to other viral pneumonia, a detriment shared across all imaging techniques [66,67].

3. Classification of Lung Ultrasound Data

3.1. Methods

3.1.1. Data Processing

Due to data availability, all experiments are conducted on the convex ultrasound probes. For the same reason we exclude the three non-COVID-19 viral pneumonia videos (Table 1) and focus on the distinction of healthy lungs, bacterial pneumonia and COVID-19 viral pneumonia. We manually processed all convex ultrasound recordings (179 videos and 53 images) and split the videos into images at a frame rate of 3 Hz (with maximal 30 frames per video, leading to a database of 1204 COVID-19, 704 bacterial pneumonia, and 1326 healthy images. The videos are diverse in length and kind ($160\pm 144$ frames) with a frame rate of $25\pm 10$ Hz. All images were cropped to a quadratic window excluding measure bars, texts and artifacts on the borders before they were resized to $224\times 224$ pixels (for examples see Figure 1A). Apart from the independent test data, all reported results were obtained in a 5-fold cross validation stratified by the number of samples per class. Data was split on a patient-level, hence it was ensured that the frames of a single video are present within a single fold only, and that the number of videos per class is similar in all folds. All models were trained to classify images as COVID-19, pneumonia, healthy, or uninformative. The latter consists of  ImageNet [68] pictures as well as neck US data [69]; added for the purpose of detecting out-of-distribution samples. This is particularly relevant for public web-based inference services. In this paper, we present all results omitting the uninformative class, as it is not relevant for the analysis of differential diagnosis performance and would bias the results, that is, lead to a higher classification accuracy due to the recall and precision of almost 100% for the uninformative class (please refer to Appendix C.1 for results including uninformative data). Furthermore, we use data augmentation, specifically flips and rotations (up to 10${}^{\circ }$) and translations (up to 10%) to diversify the dataset and prevent overfitting.

3.1.2. Frame-Based Models

Our backbone neural architecture is a VGG-16 [70] that is compared to NasNET Mobile, a light-weight alternative [71] that uses less than $1/3$ of the parameters of VGG-16 and was optimized for applications on portable devices. Both models are pre-trained on ImageNet and fine-tuned on the frames sampled from the videos. Specifically, we use two variants of VGG-16 that we name VGG and VGG-CAM. VGG-CAM has a single dense layer following the convolutions, thus enabling the usage of plain class activation maps (CAMs [72]), whereas VGG has an additional dense layer with ReLU activation and batch normalization. Considering the recent work of [57] on lung US segmentation and severity prediction for COVID-19, we investigated whether a segmentation-targeted network can also add value to the prediction in differential diagnosis. We implemented two approaches building upon the pre-trained model of [57], an ensemble of three separate U-Net-based models (U-Net, U-Net++, and DeepLabv3+, with a total of ∼19.5 M parameters). First, VGG-Segment is identical to VGG, however instead of training on the raw US data, we train on the segmented images from the ensemble (see example in Appendix B.1). Although it might seem unconventional, we hypothesized that the colouring entails additional information that might simplify classification. Secondly, in Segment-Enc the bottleneck layer of each of the three models is used as a feature encoding of the images, resulting in 560 filter maps that are fed through a two layer MLP with hidden layer sizes 512 and 256 respectively. The encoding weights are fixed during training. Both settings are compared to the other models that directly utilize the raw images. For more details on the architectures and the training procedure, please refer to Appendix B.

3.1.3. Video-Based Model

In comparison to a naïve, frame-based video classifier (obtained either by averaging scores or by related aggregation/selection schemes of all frames), we also investigate Models Genesis, a generic model for 3D medical image analysis pretrained on lung CT scans [73]. For Models Genesis, the videos are split into chunks of 5 frames each, sampled at a frame rate of 5Hz (input size: $224\times 224\times 5$). Stratified 5-fold cross validation is performed using the same split as for frame-based classifiers. Individual images were excluded, leaving aside 178 videos which were split into 770 video chunks.

3.2. Results

3.2.1. Frame-Based Experiments

Table 2. Comparison of the tested classification models on 5-fold cross validation for each class. Accuracy abbreviates accuracy, Bal. balanced accuracy, Coefficient and Par. the number of parameters (in millions). For each class and each column the best model is highlighted in bold. The accuracy of $88\%$ of VGG and VGG-CAM cannot be outperformed by pre-trained lung US segmentation models from [57], and unfortunately it cannot be matched by NASNetMobile, a model that is significantly smaller.

	Class	Recall (Sens.)	Precision	F1-Score	Specificity
VGG
Accuracy: 87.8%	COVID-19	$\mathbf{0.88}\pm 0.07$	$\mathbf{0.90}\pm 0.07$	$\mathbf{0.89}\pm 0.06$	$\mathbf{0.94}\pm 0.05$
Balanced: 87.1%	Pneumonia	$\mathbf{0.90}\pm 0.11$	$0.81\pm 0.08$	$0.85\pm 0.08$	$0.94\pm 0.04$
#Param: 14.7 M	Healthy	$0.83\pm 0.11$	$0.90\pm 0.06$	$0.86\pm 0.08$	$0.94\pm 0.03$
VGG-CAM
Accuracy: 87.4%	COVID-19	$0.86\pm 0.11$	$0.86\pm 0.07$	$0.86\pm 0.08$	$0.91\pm 0.04$
Balanced: 86.1%	Pneumonia	$0.87\pm 0.15$	$\mathbf{0.87}\pm 0.06$	$\mathbf{0.86}\pm 0.10$	$0.96\pm 0.03$
#Param: 14.7 M	Healthy	$\mathbf{0.86}\pm 0.11$	$0.88\pm 0.90$	$\mathbf{0.87}\pm 0.90$	$0.93\pm 0.04$
NASNetMobile
Accuracy: 62.5%	COVID-19	$0.63\pm 0.22$	$0.67\pm 0.14$	$0.63\pm 0.15$	$0.79\pm 0.12$
Balanced: 55.2%	Pneumonia	$0.22\pm 0.27$	$0.46\pm 0.42$	$0.28\pm 0.30$	$\mathbf{0.98}\pm 0.03$
#Param: 4.8 M	Healthy	$0.80\pm 0.11$	$0.58\pm 0.05$	$0.67\pm 0.06$	$0.60\pm 0.13$
VGG-Segment
Accuracy: 85.1%	COVID-19	$0.81\pm 0.20$	$0.86\pm 0.07$	$0.82\pm 0.13$	$0.91\pm 0.06$
Balanced: 83.9%	Pneumonia	$0.86\pm 0.08$	$0.84\pm 0.07$	$0.85\pm 0.03$	$0.95\pm 0.04$
#Param: 34.0 M	Healthy	$0.85\pm 0.12$	$0.86\pm 0.06$	$0.85\pm 0.08$	$0.90\pm 0.08$
Segment-Enc
Accuracy: 85.7%	COVID-19	$0.84\pm 0.11$	$0.89\pm 0.07$	$0.86\pm 0.07$	$0.92\pm 0.07$
Balanced: 84.4%	Pneumonia	$0.89\pm 0.04$	$0.75\pm 0.10$	$0.81\pm 0.06$	$0.92\pm 0.03$
#Param: 20.0 M	Healthy	$0.82\pm 0.18$	$\mathbf{0.90}\pm 0.05$	$0.85\pm 0.12$	$\mathbf{0.94}\pm 0.02$

shows a detailed comparison of all trained models in terms of recall, precision, specificity and F1-scores, computed in a one-vs.-all fashion for all three classes. Overall, both VGG and VGG-CAM achieve promising performance with an accuracy of $88\pm 5\%$ on a 5-fold CV of 3234 frames. The results are sufficiently balanced across classes, with the best F1-score achieved for COVID-19 in VGG ($0.89$) and for healthy data ($0.87$) in VGG-CAM. Figure 3a visualizes the results of the best model, VGG, for each binary detection task as a ROC curve, showing ROC-AUC scores of $0.94$ and above for all classes, while depicting the point where the accuracy is maximal for each class.

The false positive rate at the maximal-accuracy point is slightly larger for COVID-19 than for pneumonia and healthy patient. In a clinical setting, where false positives are less problematic than false negatives, this property is desirable. Since the data is imbalanced, we also plot the precision–recall curve in Figure 3b, which confirms that the imbalance is not of concern. In addition, the confusion matrices in Figure 3c,d further detail the predictions of VGG; we observe that the sensitivity of bacterial pneumonia is highest, but confusions occur to similar extent for each pair of classes. This complies with findings from Section 2 indicating that patients infected with COVID-19 might not show any abnormalities in LUS (e.g., for asymptomatic patients) or present similar patterns as bacterial pneumonia such as consolidated areas and effusions. For further results including the ROC- and precision–recall curves of all three models see Appendix C.

Ablation Study with Segmentation Models

Lung US recordings are noisy and operator-dependent, posing difficulties for the classification of raw data. Hence, we compare VGG and VGG-CAM to approaches from related work where all frames are segmented (i.e., classified on a pixel level into pathological patterns) with the model from [57]; see Appendix B for an example input image. The relevant rows in Table 2 exhibit mixed results, with Segment-Enc marginally outperforming VGG-Segment, but both obtaining substantially lower scores than VGG. The results are thus not encouraging to explore VGG-Segment further, since it comes at the cost of a much larger model, with an ensemble of three models (U-Net, U-Net++, and DeepLabv3+) segmenting the images prior to classification (restricting segmentation to a single model resulted in inferior performance). Possible reasons for the inferior performance of this approach are limited accuracy of the underlying segmentation model (Roy et al. [57] reported a Dice coefficient of $0.75$) and the algorithm’s pathology-to-color mapping that may dispel a notion of similarity between types of patterns. Future work might explore other architecture along the lines of Segment-Enc, that is, a dense model classifying the encoding produced by a segmentation model.

Ablation Study on Other Architectures

Last, we tested several smaller networks such as MobileNet [74] as an additional ablation study, with NASNetMobile [71] performing best, obtaining $62.5$% accuracy, but still suffering from low precision and recall on data of bacterial pneumonia. As most ultrasound devices are portable and real-time inference on the devices is technically feasible, resource-efficient networks are highly relevant and could supersede web-based inference. In an attempt torward real-time on-device inference, our fine-tuned NASNetMobile requires less than a third of the parameters of VGG but does not yet yield satisfactory results ($63\%$ accuracy).

3.2.2. Video-Based Experiments

To investigate the need for a model with the ability to detect spatiotemporal patterns in lung US, we explored Models Genesis, a pretrained 3D-CNN designed for 3D medical image analysis [73].

Table 3. Video classification results. The frame-based model VGG-CAM outperforms the 3D CNN Models Genesis in all shown metrics, displaying high accuracy (94%), recall, precision for COVID-19 and pneumonia detection.

	Class	Recall	Precision	F1-Score	Specificity
VGG
Accuracy: 90%	COVID-19	$0.90\pm 0.08$	$0.92\pm 0.07$	$0.91\pm 0.06$	$0.96\pm 0.04$
Balanced: 90%	Pneumonia	$0.93\pm 0.10$	$0.88\pm 0.08$	$0.90\pm 0.08$	$0.95\pm 0.04$
#Param.: 14.7 M	Healthy	$0.88\pm 0.06$	$0.91\pm 0.08$	$0.89\pm 0.06$	$0.95\pm 0.05$
Models Genesis
Accuracy: 78%	COVID-19	$0.74\pm 0.17$	$0.77\pm 0.15$	$0.75\pm 0.15$	$0.87\pm 0.11$
Balanced: 77%	Pneumonia	$0.79\pm 0.14$	$0.80\pm 0.15$	$0.78\pm 0.11$	$0.91\pm 0.07$
#Param.: 7.6 M	Healthy	$0.78\pm 0.28$	$0.79\pm 0.12$	$0.77\pm 0.22$	$0.88\pm 0.08$

contrasts the frame-based performance of VGG model to Model Genesis. The video classifier is outperformed by VGG, with a video accuracy of 90% compared to 78%. Note that by summarizing frames of one video, the accuracy of VGG was improved, indicating that in some cases only a minority of the frames are misclassified. In particular, the sensitivity and precision of COVID-19 increases. In contrast, Model Genesis struggles with the classification of COVID-19 patients, obtaining an F1-score of only $0.75$. Notably, the varying frame rate across the videos was accounted for during data preparation and cannot serve as a reason to explain the performance gap. One caveat in Models Genesis that may hinder better generalization is that it was pretrained on 3D volumes rather than 2D time series. But considering that only 770 video-chunks were available for training Model Genesis, while 3234 images are used to train VGG-CAM, even extended through data augmentation techniques, it is plausible that video-based classification may improve with increasing data availability.

3.2.3. Evaluation on Independent Test Data

To the best of our knowledge, the only other publicly-available lung ultrasound database was released by the ICLUS initiative [57], comprising 60 lung US recordings from Italian patients suspected for COVID-19 (39 convex + 21 linear probes). The data was annotated by medical experts by observation, with severity scores from 0 to 3 (Roy et al. [57] “Score 0 indicates the presence of a continuous pleural-line accompanied by horizontal artifacts called A-lines [33], which characterize a healthy lung surface. In contrast, score 1 indicates the first) signs of abnormality, that is, the appearance of alterations in the pleural-line in conjunction with vertical artifacts. Scores 2 and 3 are representative of a more advanced pathological state, with the presence of small or large consolidations, respectively. Finally score 3 is associated with the presence of a wider hyperechogenic area below the pleural surface, which can be referred to as white lung”. We evaluated the performance of the VGG model on all 39 convex probes, predicting each frame as the average of the five VGG models trained in cross-validation. Since many frames in a raw lung US video are not informative, Roy et al. [57] train an aggregation layer to combine frame-based scores into a video prediction. Instead, we propose to use confidence estimates as explained in Section 4.2 and discard frames with low confidence. Discarding the 5% frames with lowest certainty, and assuming that a severity score of 0 corresponds to a healthy lung and severity of 1, 2 or 3 are patients infected with COVID-19, the ensemble achieves a frame-prediction accuracy of $72.9\%$, and a video classification accuracy of $82.1$%. The sensitivity for COVID-19 is $0.806$ and the specificity $0.962$.

We also compare the predicted probability of COVID-19 to the severity scores in Figure 4. Although, in contrast to their model, our model was not trained to assess severity, there is a clear correlation between the severity and the probability of our model (Pearson’s $r=0.36$). At this point, we can safely conclude that test data performance is encouraging, but requires further validation with labeled data including healthy patients and patients infected with other pneumonias.

4. Model Explainability

4.1. Class Activation Maps

Class activation mapping (CAM) is a popular technique for model explainability that exploits global average pooling and allows to compute class-specific heatmaps that indicate the discriminative regions of the image that caused the particular class activity of interest [72]. For healthcare applications, CAMs, or their generalization Grad-CAMs [75], can provide valuable decision support by unravelling whether a model’s prediction was based on visible pathological patterns. Moreover, CAMs can guide doctors and point to informative patterns, especially relevant in time-sensitive (triage) or knowledge-sensitive (third-world countries) situations. Here, the Grad-CAMs yielded with the VGG model were compared to CAMs of VGG-CAM, and the latter were found of better quality by observation. Since the accuracy of VGG-CAM is very similar to the one of VGG, we analyze only the outputs of VGG-CAM in this section as the most performant interpretable model.

4.1.1. Results

Figure 5 shows representative CAMs in the three rightmost panels. They highlight the most frequent US pattern for the three classes, COVID-19 (vertical B-lines), bacterial pneumonia (consolidations), and healthy (horizontal A-line). For a more quantitative estimate, we computed the points of maximal activation of the CAMs for each class (abbreviated as $\mathbf{C}$, $\mathbf{P}$, and $\mathbf{H}$) and all samples of the dataset (see Figure 5 left). While, in general, the heatmaps are fairly distributed across the probe, pneumonia related features were rather found in the center and bottom part, especially compared to COVID-19 and healthy patterns. Please refer to Appendix D for a density plot. The interactive HTML and a few exemplary CAM videos are available as Supplementary Material: https://bit.ly/2HH4sUt To assess to what extent the differences between the individual distributions are significant, we employed maximum mean discrepancy (MMD), a metric between statistical distributions [76] that enables the comparison of distributions via kernels, that is, generic similarity functions. Given two coordinates $x,y\in {\mathbb{R}}^2$ and a smoothing parameter $\sigma \in \mathbb{R}$, we use a Gaussian kernel $k(x,y):=exp(-{\parallel x-y\parallel }^2/{\sigma }^2)$ to assess the dissimilarity between x and y. Following [76], we set $\sigma$ to the median distance in the aggregated samples (i.e., all samples, without considering labels). We then calculate MMD values for the distance between the three classes, that is, $MMD(\mathbf{C},\mathbf{P})\approx 0.0051$, $MMD(\mathbf{C},\mathbf{H})\approx 0.0061$, and $MMD(\mathbf{P},\mathbf{H})\approx 0.0065$. Repeating this calculation for 5000 bootstrap samples per class (see Figure A5 for the resulting histograms), we find that the observe achieved significance levels of the intra-class MMD values of well below an $\alpha =0.05$ significance level.

4.1.2. Expert Validation of CAMs for Human-in-the-Loop Settings

A potential application of our framework is a human-in-the-loop (HITL) setting with CAMs as a core component of the decision support tool that highlights pulmonary biomarkers and guides the decision makers. Since the performance of qualitative methods like CAMs can only be validated with the help of doctors, we conducted a blinded study with two medical experts experienced in the diagnostic process with ultrasound recordings (one physician with >10 years of clinical experience with LUS and one academic US course instructor). The experts were shown a set of 50 videos comprising non-proprietary video data which was correctly classified by the model. The class activation map for the respective class was computed two times, first with one of the four models that were trained on this video, and secondly only with the model that was not exposed to that video during training (called train- and test-CAMs in the following). Both experts were asked to compare both activation maps for all 50 videos, and to score them on a scale of $-3$ (“the heatmap is only distracting”) to 3 (“the heatmap is very helpful for diagnosis”).

First, the CAMs were overall perceived useful and the test CAMs were assigned a higher average score of $0.81$ than the train CAMs ($0.45$). Secondly, disagreeing in only $8\%$ of the cases, both experts independently decided for the test-CAM with a probability of $56\%$. Hence, the test-CAMs are superior to the train-CAMs, however non-significant in a Wilcoxon signed-rank test.

However, train- and test-CAM both scored best for videos of bacterial pneumonia, lacking performance for videos of healthy and COVID-19 patients. Specifically, test-CAM received an average score of $0.81$, divided into $-0.25$ for COVID-19, $2.05$ for pneumonia, and 0 for healthy patients. Thirdly, the experts were asked to name the pathological patterns visible in general, as well as the patterns that were highlighted by the heatmap. Figure 6 shows the average ratio of pattern that were correctly highlighted by the CAM model, where the patterns listed by the more senior expert are taken as the ground truth for each video. Interestingly, the high performance of our model in classifying videos of bacterial pneumonia is probably explained by the model’s ability to detect consolidated areas, where 17 out of 18 are correctly classified. Moreover, A-lines are highlighted in ∼60% of the normal lung recordings. Problematically, in 13 videos mostly fat, muscles or skin is highlighted, which has to be studied and improved in future work.

4.2. Confidence Estimates

The ability to quantify states of high uncertainty is of crucial importance for medical image analysis and computer vision applications in healthcare. We assessed this via independent measures of epistemic (model) uncertainty (by drawing Monte Carlo samples from the approximate predictive posterior [77]) and aleatoric (data) uncertainty (by means of test time data augmentation [78]). The sample standard deviation of 10 forward passes is interpreted as inverse, empirical confidence score $\in [0,1]$. In detail, for both aleatoric and epistemic uncertainty, the confidence estimate $c_i$ of sample i is computed by scaling the sample’s standard deviation to $\in [0,1]$ and interpreting it as an inverse precision:

$$c_i=-\left(\frac{{\sigma }_{i,j}-{\sigma }_{min}}{{\sigma }_{max}-{\sigma }_{min}}\right)+1,$$

(1)

where ${\sigma }_{i,j}$ is the sample standard deviation of the ten class probabilities of the winning class j, ${\sigma }_{min}$ is the minimal standard deviation (0, that is, all probabilities for the winning class are identical) and ${\sigma }_{max}$ is the maximal standard deviation, that is, $0.5$. Practically, for epistemic uncertainty, dropout was set to $0.5$ across the VGG model and for aleatoric uncertainty the same transformations as during training are employed. The epistemic confidence estimate was found to be highly correlated with the correctness of the predictions ($\rho =0.41,p<4\times {10}^{-73}$, mean confidence of $0.75$ and $0.27$ for correct and wrong predictions), while the aleatoric confidence was found correlated to a lesser extent ($\rho =0.29,p<6\times {10}^{-35}$, mean confidence of $0.88$ and $0.73$, respectively). Across the entire dataset, both scores are highly correlated ($\rho =0.52,p<4\times {10}^{-124}$), suggesting to exploit them jointly to detect and remove predictions of low confidence in a possible application. In the evaluation of our model on independent test data from the ICLUS initiative (see above), we found that the frame-wise classification accuracy improved monotonically from $72.2$% to $77.4\%$ with increasing exclusion of frames with insufficient confidence.

5. Discussion

5.1. Prediction Performance Evaluation

We provide strong evidence that automatic detection of COVID-19 is a promising future endeavour and competitive compared to CT and CXR based models, with a sensitivity of $90\%$ and a specificity of $96\%$ for COVID-19, achieved on our dataset of 202 lung US videos. In comparison, sensitivity up to $98\%$ and specificity up to $92\%$ was reported for CT [2,79]. We verified our results with independent test data, compared to other architectures and models of related work [57], studied model uncertainty and concluded a significant ability of our model to recognize low-confidence situations. If implemented and validated as a decision support tool, it might bring value to the diagnostic process of symptomatic patients, but could also pose risks as we expand on in Appendix E.

Certainly, there are many approaches yet to be explored in order to improve on the results presented here, including further work on video classification, improving prediction robustness [49], incorporating a LUS probe quality assessment module [53] or a pulmonary symptom classifier [54] which could be coupled with our disease classifier. Furthermore, differentiating COVID-19 from other viral pneumonias remains a key challenge for diagnostic imaging [67] and is beyond the scope of this work. Further work could explore the possibility to use interpretable methods to determine differences of COVID-19 to other viral pneumonia which could be exploited in (automatic) differential diagnosis. Here, we investigated the value of interpretable methods in a quantitative manner with the implementation and validation of class activation mapping in a study involving medical experts. While the analysis provides excellent evidence for the successful detection of pathological patterns like consolidations, A-lines and effusion, it reveals problems in the model’s “focal point” (e.g., missing B-lines and sometimes highlighting muscles instead of the lung) which should be further addressed using ultrasound segmentation techniques [42].

5.2. Dataset Limitations

Our dataset certainly suffers from certain limitations, as gathering data in an organized manner during a pandemic is a logistic and time-sensitive challenge. We ensured that the Northumbria and Neuruppin cohort used US devices from the same vendor, but this was not possible for external data from online sources and publications. Processing exemplary videos of publications carries the risk of population bias and may overestimate the frequency of stereotypical symptoms and pathologies. In general, the data inevitably remains heterogeneous and information is at least partially unknown, including patient metadata (age, gender, symptoms etc.), technical details (recording frequency, imaging depths) and disease progression (duration of symptoms at day of examination). While this heterogeneity may induce biases, it can also aid in developing robust methods, as models based on single-center data often fail to generalize [80]. Moreover, most types of patterns in LUS are, just like for CT or CXR, not disease-specific [66,67] (see also Figure 2b) and we by no means claim to develop a diagnostic tool in here. For the future, a more homogeneous dataset of LUS, associated with demographics and anamnesis of patients should be collected in a controlled manner from different hospitals. Conclusively however, given the non-availability of other public databases for LUS data in the context of COVID-19, this does not decrease the benefit of our dataset; but we do advise to be careful in drawing far-ranging conclusions from simulations on our dataset.

6. Conclusions

Lung ultrasound as an established diagnosis tool that is both safe and highly available constitutes a method with potentially huge impact that has nevertheless been neglected by the machine learning community. This work presents a novel LUS dataset for COVID-19 alongside new methods and analyses that pave the way towards computer vision-assisted differential diagnosis of COVID-19 from US. We provide an extensive analysis of interpretable methods that are relevant not only in the context of COVID-19, but in general for the diagnosis of viral and bacterial pneumonias.

Our published database is constantly updated and verified by medical experts and researchers are invited to contribute to our initiative. We strongly believe in the value of such open-source databases not only for follow-up work, but also for related fields. Our data in particular can be utilized for pre-training (transfer learning) in other US applications [41], as done in Model Genesis [73]. Even research on noise and artifact removal from US [81], or US image simulation [82] might benefit from the availability of US data. The available metadata also opens up opportunities for further analysis with respect to pre-conditions, symptoms and patterns, as we touched upon in Figure 2a,b.

We envision the proposed tool as a step toward a decision support system to aid diagnosis by providing a “second opinion” to increase reliability. The promising results of our model are to be validated in a controlled clinical study that investigates the predictive power of US for automatic detection of COVID-19, especially in comparison to CT and CXR.