Human-to-AI Interrater Agreement for Lung Ultrasound Scoring in COVID-19 Patients.

OBJECTIVES: Lung ultrasound (LUS) has sparked significant interest during COVID-19. LUS is based on the detection and analysis of imaging patterns. Vertical artifacts and consolidations are some of the recognized patterns in COVID-19. However, the interrater reliability (IRR) of these findings has not been yet thoroughly investigated. The goal of this study is to assess IRR in LUS COVID-19 data and determine how many LUS videos and operators are required to obtain a reliable result. METHODS: A total of 1035 LUS videos from 59 COVID-19 patients were included. Videos were randomly selected from a dataset of 1807 videos and scored by six human operators (HOs). The videos were also analyzed by artificial intelligence (AI) algorithms. Fleiss' kappa coefficient results are presented, evaluated at both the video and prognostic levels. RESULTS: Findings show a stable agreement when evaluating a minimum of 500 videos. The statistical analysis illustrates that, at a video level, a Fleiss' kappa coefficient of 0.464 (95% confidence interval [CI] = 0.455-0.473) and 0.404 (95% CI = 0.396-0.412) is obtained for pairs of HOs and for AI versus HOs, respectively. At prognostic level, a Fleiss' kappa coefficient of 0.505 (95% CI = 0.448-0.562) and 0.506 (95% CI = 0.458-0.555) is obtained for pairs of HOs and for AI versus HOs, respectively. CONCLUSIONS: To examine IRR and obtain a reliable evaluation, a minimum of 500 videos are recommended. Moreover, the employed AI algorithms achieve results that are comparable with HOs. This research further provides a methodology that can be useful to benchmark future LUS studies.

Multi-objective automatic analysis of lung ultrasound data from COVID-19 patients by means of deep learning and decision trees.

COVID-19 raised the need for automatic medical diagnosis, to increase the physicians' efficiency in managing the pandemic. Among all the techniques for evaluating the status of the lungs of a patient with COVID-19, lung ultrasound (LUS) offers several advantages: portability, cost-effectiveness, safety. Several works approached the automatic detection of LUS imaging patterns related COVID-19 by using deep neural networks (DNNs). However, the decision processes based on DNNs are not fully explainable, which generally results in a lack of trust from physicians. This, in turn, slows down the adoption of such systems. In this work, we use two previously built DNNs as feature extractors at the frame level, and automatically synthesize, by means of an evolutionary algorithm, a decision tree (DT) that aggregates in an interpretable way the predictions made by the DNNs, returning the severity of the patients' conditions according to a LUS score of prognostic value. Our results show that our approach performs comparably or better than previously reported aggregation techniques based on an empiric combination of frame-level predictions made by DNNs. Furthermore, when we analyze the evolved DTs, we discover properties about the DNNs used as feature extractors. We make our data publicly available for further development and reproducibility.

Lung Ultrasound in COVID-19 and Post-COVID-19 Patients, an Evidence-Based Approach.

OBJECTIVES: Worldwide, lung ultrasound (LUS) was utilized to assess coronavirus disease 2019 (COVID-19) patients. Often, imaging protocols were however defined arbitrarily and not following an evidence-based approach. Moreover, extensive studies on LUS in post-COVID-19 patients are currently lacking. This study analyses the impact of different LUS imaging protocols on the evaluation of COVID-19 and post-COVID-19 LUS data. METHODS: LUS data from 220 patients were collected, 100 COVID-19 positive and 120 post-COVID-19. A validated and standardized imaging protocol based on 14 scanning areas and a 4-level scoring system was implemented. We utilized this dataset to compare the capability of 5 imaging protocols, respectively based on 4, 8, 10, 12, and 14 scanning areas, to intercept the most important LUS findings. This to evaluate the optimal trade-off between a time-efficient imaging protocol and an accurate LUS examination. We also performed a longitudinal study, aimed at investigating how to eventually simplify the protocol during follow-up. Additionally, we present results on the agreement between AI models and LUS experts with respect to LUS data evaluation. RESULTS: A 12-areas protocol emerges as the optimal trade-off, for both COVID-19 and post-COVID-19 patients. For what concerns follow-up studies, it appears not to be possible to reduce the number of scanning areas. Finally, COVID-19 and post-COVID-19 LUS data seem to show differences capable to confuse AI models that were not trained on post-COVID-19 data, supporting the hypothesis of the existence of LUS patterns specific to post-COVID-19 patients. CONCLUSIONS: A 12-areas acquisition protocol is recommended for both COVID-19 and post-COVID-19 patients, also during follow-up.

On the Impact of Different Lung Ultrasound Imaging Protocols in the Evaluation of Patients Affected by Coronavirus Disease 2019: How Many Acquisitions Are Needed?

Lung ultrasound (LUS) is currently being extensively used for the evaluation of patients affected by coronavirus disease 2019. In the past months, several imaging protocols have been proposed in the literature. However, how the different protocols would compare when applied to the same patients had not been investigated yet. To this end, in this multicenter study, we analyzed the outcomes of 4 different LUS imaging protocols, respectively based on 4, 8, 12, and 14 LUS acquisitions, on data from 88 patients. Results show how a 12-area acquisition system seems to be a good tradeoff between the acquisition time and accuracy.

Lung Ultrasound for COVID-19 Patchy Pneumonia: Extended or Limited Evaluations?

OBJECTIVES: The 2019 novel coronavirus (severe acute respiratory syndrome coronavirus 2) is causing cases of severe pneumonia. Lung ultrasound (LUS) could be a useful tool for physicians detecting a bilateral heterogeneous patchy distribution of pathologic findings in a symptomatic suggestive context. The aim of this study was to focus on the implications of limiting LUS examinations to specific regions of the chest. METHODS: Patients were evaluated with a standard sequence of LUS scans in 14 anatomic areas. A scoring system of LUS findings was reported, ranging from 0 to 3 (worst score, 3). The scores reported on anterior, lateral, and posterior landmarks were analyzed separately and compared with each other and with the global findings. RESULTS: Thirty-eight patients were enrolled. A higher prevalence of score 0 was observed in the anterior region (44.08%). On the contrary, 21.05% of posterior regions and 13.62% of lateral regions were evaluated as score 3, whereas only 5.92% of anterior regions were classified as score 3. Findings from chest computed tomography performed in 16 patients with coronavirus disease 2019 correlated with and matched the distribution of findings from LUS. CONCLUSIONS: To assess the quantity and severity of lung disease, a comprehensive LUS examination is recommended. Omitting areas of the chest misses involved lung.

Dependence of lung ultrasound vertical artifacts on frequency, bandwidth, focus and angle of incidence: An in vitro study.

Lung ultrasound (LUS) is nowadays widely adopted by clinicians to evaluate the state of the lung surface. However, being mainly based on the evaluation of vertical artifacts, whose genesis is still unclear, LUS is affected by qualitative and subjective analyses. Even though semi-quantitative approaches supported by computer aided methods can reduce subjectivity, they do not consider the dependence of vertical artifacts on imaging parameters, and could not be classified as fully quantitative. They are indeed mainly based on scoring LUS images, reconstructed with standard clinical scanners, through the sole evaluation of visual patterns, whose visualization depends on imaging parameters. To develop quantitative techniques is therefore fundamental to understand which parameters influence the vertical artifacts' intensity. In this study, we quantitatively analyzed the dependence of nine vertical artifacts observed in a thorax phantom on four parameters, i.e., center frequency, focal point, bandwidth, and angle of incidence. The results showed how the vertical artifacts are significantly affected by these four parameters, and confirm that the center frequency is the most impactful parameter in artifacts' characterization. These parameters should hence be carefully considered when developing a LUS quantitative approach.

Numerical study on lung ultrasound B-line formation as a function of imaging frequency and alveolar geometries.

Lung ultrasound (LUS) has become a widely adopted diagnostic method for several lung diseases. However, the presence of air inside the lung does not allow the anatomical investigation of the organ. Therefore, LUS is mainly based on the interpretation of vertical imaging artifacts, called B-lines. These artifacts correlate with several pathologies, but their genesis is still partly unknown. Within this framework, this study focuses on the factors affecting the artifacts' formation by numerically simulating the ultrasound propagation within the lungs through the toolbox k-Wave. Since the main hypothesis behind the generation of B-lines relies on multiple scattering phenomena occurring once acoustic channels open at the lung surface, the impact of changing alveolar size and spacing is of interest. The tested domain is of size 4 cm × 1.6 cm, the investigated frequencies vary from 1 to 5 MHz, and the explored alveolar diameters and spacing range from 100 to 400 µm and from 20 to 395 µm, respectively. Results show the strong and entangled relation among the wavelength, the domain geometries, and the artifact visualization, allowing for better understanding of propagation in such a complex medium and opening several possibilities for future studies.

Deep learning applied to lung ultrasound videos for scoring COVID-19 patients: A multicenter study.

In the current pandemic, lung ultrasound (LUS) played a useful role in evaluating patients affected by COVID-19. However, LUS remains limited to the visual inspection of ultrasound data, thus negatively affecting the reliability and reproducibility of the findings. Moreover, many different imaging protocols have been proposed, most of which lacked proper clinical validation. To address these problems, we were the first to propose a standardized imaging protocol and scoring system. Next, we developed the first deep learning (DL) algorithms capable of evaluating LUS videos providing, for each video-frame, the score as well as semantic segmentation. Moreover, we have analyzed the impact of different imaging protocols and demonstrated the prognostic value of our approach. In this work, we report on the level of agreement between the DL and LUS experts, when evaluating LUS data. The results show a percentage of agreement between DL and LUS experts of 85.96% in the stratification between patients at high risk of clinical worsening and patients at low risk. These encouraging results demonstrate the potential of DL models for the automatic scoring of LUS data, when applied to high quality data acquired accordingly to a standardized imaging protocol.

Investigating training-test data splitting strategies for automated segmentation and scoring of COVID-19 lung ultrasound images.

Ultrasound in point-of-care lung assessment is becoming increasingly relevant. This is further reinforced in the context of the COVID-19 pandemic, where rapid decisions on the lung state must be made for staging and monitoring purposes. The lung structural changes due to severe COVID-19 modify the way ultrasound propagates in the parenchyma. This is reflected by changes in the appearance of the lung ultrasound images. In abnormal lungs, vertical artifacts known as B-lines appear and can evolve into white lung patterns in the more severe cases. Currently, these artifacts are assessed by trained physicians, and the diagnosis is qualitative and operator dependent. In this article, an automatic segmentation method using a convolutional neural network is proposed to automatically stage the progression of the disease. 1863 B-mode images from 203 videos obtained from 14 asymptomatic individual,14 confirmed COVID-19 cases, and 4 suspected COVID-19 cases were used. Signs of lung damage, such as the presence and extent of B-lines and white lung areas, are manually segmented and scored from zero to three (most severe). These manually scored images are considered as ground truth. Different test-training strategies are evaluated in this study. The results shed light on the efficient approaches and common challenges associated with automatic segmentation methods.

Proposal for International Standardization of the Use of Lung Ultrasound for Patients With COVID-19: A Simple, Quantitative, Reproducible Method.

Growing evidence is showing the usefulness of lung ultrasound in patients with the 2019 new coronavirus disease (COVID-19). Severe acute respiratory syndrome coronavirus 2 has now spread in almost every country in the world. In this study, we share our experience and propose a standardized approach to optimize the use of lung ultrasound in patients with COVID-19. We focus on equipment, procedure, classification, and data sharing.

On the influence of imaging parameters on lung ultrasound B-line artifacts, in vitro study.

The clinical relevance of lung ultrasonography (LUS) has been rapidly growing since the 1990s. However, LUS is mainly based on the evaluation of visual artifacts (also called B-lines), leading to subjective and qualitative diagnoses. The formation of B-lines remains unknown and, hence, researchers need to study their origin to allow clinicians to quantitatively evaluate the state of lungs. This paper investigates an ambiguity about the formation of B-lines, leading to the formulation of two main hypotheses. The first hypothesis states that the visualization of these artifacts is linked only to the dimension of the emitted beam, whereas the second associates their appearance to specific resonance phenomena. To verify these hypotheses, the frequency spectrum of B-lines was studied by using dedicated lung-phantoms. A research programmable platform connected to an LA533 linear array probe was exploited both to implement a multifrequency approach and to acquire raw radio frequency data. The strength of each artifact was measured as a function of frequency, focal point, and transmitting aperture by means of the artifact total intensity. The results show that the main parameter that influences the visualization of B-lines is the frequency rather than the focal point or the number of transmitting elements.

Low-complexity lung ultrasound video scoring by means of intensity projection-based video compression.

Since the outbreak of COVID-19, efforts have been made towards semi-quantitative analysis of lung ultrasound (LUS) data to assess the patient's condition. Several methods have been proposed in this regard, with a focus on frame-level analysis, which was then used to assess the condition at the video and prognostic levels. However, no extensive work has been done to analyze lung conditions directly at the video level. This study proposes a novel method for video-level scoring based on compression of LUS video data into a single image and automatic classification to assess patient's condition. The method utilizes maximum, mean, and minimum intensity projection-based compression of LUS video data over time. This enables to preserve hyper- and hypo-echoic data regions, while compressing the video down to a maximum of three images. The resulting images are then classified using a convolutional neural network (CNN). Finally, the worst predicted score given among the images is assigned to the corresponding video. The results show that this compression technique can achieve a promising agreement at the prognostic level (81.62%), while the video-level agreement remains comparable with the state-of-the-art (46.19%). Conclusively, the suggested method lays down the foundation for LUS video compression, shifting from frame-level to direct video-level analysis of LUS data.

Ultrasound multifrequency strategy to estimate the lung surface roughness, in silico and in vitro results.

Lung ultrasound (LUS) is an important imaging modality to assess the state of the lung surface. Nevertheless, LUS is limited to the visual evaluation of imaging artifacts, especially the vertical ones. These artifacts are observed in pathologies characterized by a reduction of dimensions of air-spaces (alveoli). In contrast, there exist pathologies, such as chronic obstructive pulmonary disease (COPD), in which an enlargement of air-spaces can occur, which causes the lung surface to behave essentially as a perfect reflector, thus not allowing ultrasound penetration. This characteristic high reflectivity could be exploited to characterize the lung surface. Specifically, air-spaces of different sizes could cause the lung surface to have a different roughness, whose estimation could provide a way to assess the state of the lung surface. In this study, we present a quantitative multifrequency approach aiming at estimating the lung surface's roughness by measuring image intensity variations along the lung surface as a function of frequency. This approach was tested both in silico and in vitro, and it showed promising results. For the in vitro experiments, radiofrequency (RF) data were acquired from a novel experimental model. The results showed consistency between in silico and in vitro experiments.

Benchmark methodological approach for the application of artificial intelligence to lung ultrasound data from COVID-19 patients: From frame to prognostic-level.

Automated ultrasound imaging assessment of the effect of CoronaVirus disease 2019 (COVID-19) on lungs has been investigated in various studies using artificial intelligence-based (AI) methods. However, an extensive analysis of state-of-the-art Convolutional Neural Network-based (CNN) models for frame-level scoring, a comparative analysis of aggregation techniques for video-level scoring, together with a thorough evaluation of the capability of these methodologies to provide a clinically valuable prognostic-level score is yet missing within the literature. In addition to that, the impact on the analysis of the posterior probability assigned by the network to the predicted frames as well as the impact of temporal downsampling of LUS data are topics not yet extensively investigated. This paper takes on these challenges by providing a benchmark analysis of methods from frame to prognostic level. For frame-level scoring, state-of-the-art deep learning models are evaluated with additional analysis of best performing model in transfer-learning settings. A novel cross-correlation based aggregation technique is proposed for video and exam-level scoring. Results showed that ResNet-18, when trained from scratch, outperformed the existing methods with an F1-Score of 0.659. The proposed aggregation method resulted in 59.51%, 63.29%, and 84.90% agreement with clinicians at the video, exam, and prognostic levels, respectively; thus, demonstrating improved performances over the state of the art. It was also found that filtering frames based on the posterior probability shows higher impact on the LUS analysis in comparison to temporal downsampling. All of these analysis were conducted over the largest standardized and clinically validated LUS dataset from COVID-19 patients.

State of the Art in Lung Ultrasound, Shifting from Qualitative to Quantitative Analyses.

Lung ultrasound (LUS) has been increasingly expanding since the 1990s, when the clinical relevance of vertical artifacts was first reported. However, the massive spread of LUS is only recent and is associated with the coronavirus disease 2019 (COVID-19) pandemic, during which semi-quantitative computer-aided techniques were proposed to automatically classify LUS data. In this review, we discuss the state of the art in LUS, from semi-quantitative image analysis approaches to quantitative techniques involving the analysis of radiofrequency data. We also discuss recent in vitro and in silico studies, as well as research on LUS safety. Finally, conclusions are drawn highlighting the potential future of LUS.