Checklist for Reproducibility of Deep Learning in Medical Imaging.

The application of deep learning (DL) in medicine introduces transformative tools with the potential to enhance prognosis, diagnosis, and treatment planning. However, ensuring transparent documentation is essential for researchers to enhance reproducibility and refine techniques. Our study addresses the unique challenges presented by DL in medical imaging by developing a comprehensive checklist using the Delphi method to enhance reproducibility and reliability in this dynamic field. We compiled a preliminary checklist based on a comprehensive review of existing checklists and relevant literature. A panel of 11 experts in medical imaging and DL assessed these items using Likert scales, with two survey rounds to refine responses and gauge consensus. We also employed the content validity ratio with a cutoff of 0.59 to determine item face and content validity. Round 1 included a 27-item questionnaire, with 12 items demonstrating high consensus for face and content validity that were then left out of round 2. Round 2 involved refining the checklist, resulting in an additional 17 items. In the last round, 3 items were deemed non-essential or infeasible, while 2 newly suggested items received unanimous agreement for inclusion, resulting in a final 26-item DL model reporting checklist derived from the Delphi process. The 26-item checklist facilitates the reproducible reporting of DL tools and enables scientists to replicate the study's results.

The 2021 SIIM-FISABIO-RSNA Machine Learning COVID-19 Challenge: Annotation and Standard Exam Classification of COVID-19 Chest Radiographs.

We describe the curation, annotation methodology, and characteristics of the dataset used in an artificial intelligence challenge for detection and localization of COVID-19 on chest radiographs. The chest radiographs were annotated by an international group of radiologists into four mutually exclusive categories, including "typical," "indeterminate," and "atypical appearance" for COVID-19, or "negative for pneumonia," adapted from previously published guidelines, and bounding boxes were placed on airspace opacities. This dataset and respective annotations are available to researchers for academic and noncommercial use.

Frequency of Statin Prescription Among Individuals with Coronary Artery Calcifications Detected Through Lung Cancer Screening.

Individuals eligible for lung cancer screening (LCS) are at risk for atherosclerotic cardiovascular disease (ASCVD) due to smoking history. Coronary artery calcifications (CAC), a common incidental finding on low-dose CT (LDCT) for LCS, is a predictor of cardiovascular events. Despite findings of high ASCVD risk and CAC, a substantial proportion of LCS patients are not prescribed primary preventive statin therapy for ASCVD. We assessed the frequency of statin prescription in LCS patients with moderate levels of CAC. Among 259 individuals with moderate CAC, 95% had ASCVD risk ≥ 7.5%. Despite this, 27% of patients were statin-free prior to LDCT and 21.2% remained statin-free after LDCT showing moderate CAC. Illustratively, while a substantial proportion of LCS patients are statin-eligible, many lack a statin prescription, even after findings of CAC burden. CAC reporting should be standardized, and interdisciplinary communication should be optimized to ensure that LCS patients are placed on appropriate preventive therapy.

Using Deep Learning Segmentation for Endotracheal Tube Position Assessment.

PURPOSE: The purpose of this study was to determine the efficacy of using deep learning segmentation for endotracheal tube (ETT) position on frontal chest x-rays (CXRs). MATERIALS AND METHODS: This was a retrospective trial involving 936 deidentified frontal CXRs divided into sets for training (676), validation (50), and 2 for testing (210). This included an "internal test" set of 100 CXRs from the same institution, and an "external test" set of 110 CXRs from a different institution. Each image was labeled by 2 radiologists with the ETT-carina distance. On the training images, 1 radiologist manually segmented the ETT tip and inferior wall of the carina. A U-NET architecture was constructed to label each pixel of the CXR as belonging to either the ETT, carina, or neither. This labeling allowed the distance between the ETT and carina to be compared with the average of 2 radiologists. The interclass correlation coefficients, mean, and SDs of the absolute differences between the U-NET and radiologists were calculated. RESULTS: The mean absolute differences between the U-NET and average of radiologist measurements were 0.60±0.61 and 0.48±0.47 cm on the internal and external datasets, respectively. The interclass correlation coefficients were 0.87 (0.82, 0.91) and 0.92 (0.88, 0.94) on the internal and external datasets, respectively. CONCLUSION: The U-NET model had excellent reliability and performance similar to radiologists in assessing ETT-carina distance.

Endotracheal Tube Position Assessment on Chest Radiographs Using Deep Learning.

PURPOSE: To determine the efficacy of deep learning in assessing endotracheal tube (ETT) position on radiographs. MATERIALS AND METHODS: In this retrospective study, 22 960 de-identified frontal chest radiographs from 11 153 patients (average age, 60.2 years ± 19.9 [standard deviation], 55.6% men) between 2010 and 2018 containing an ETT were placed into 12 categories, including bronchial insertion and distance from the carina at 1.0-cm intervals (0.0-0.9 cm, 1.0-1.9 cm, etc), and greater than 10 cm. Images were split into training (80%, 18 368 images), validation (10%, 2296 images), and internal test (10%, 2296 images), derived from the same institution as the training data. One hundred external test radiographs were also obtained from a different hospital. The Inception V3 deep neural network was used to predict ETT-carina distance. ETT-carina distances and intraclass correlation coefficients (ICCs) for the radiologists and artificial intelligence (AI) system were calculated on a subset of 100 random internal and 100 external test images. Sensitivity and specificity were calculated for low and high ETT position thresholds. RESULTS: On the internal and external test images, respectively, the ICCs of AI and radiologists were 0.84 (95% CI: 0.78, 0.92) and 0.89 (95% CI: 0.77, 0.94); the ICCs of the radiologists were 0.93 (95% CI: 0.90, 0.95) and 0.84 (95% CI: 0.71, 0.90). The AI model was 93.9% sensitive (95% CI: 90.0, 96.7) and 97.7% specific (95% CI: 96.9, 98.3) for detecting ETT-carina distance less than 1 cm. CONCLUSION: Deep learning predicted ETT-carina distance within 1 cm in most cases and showed excellent interrater agreement compared with radiologists. The model was sensitive and specific in detecting low ETT positions.© RSNA, 2020.

Interstitial lung abnormalities and pulmonary fibrosis in COVID-19 patients: a short-term follow-up case series.

Fibrotic lung changes are well-known complications of SARS, MERS, and ARDS from other causes and are anticipated in recovered COVID patients. However, there is limited data so far showing a temporal relationship between lung changes on imaging in the acute phase and follow-up imaging after recovery from the infection. We present 12 patients who demonstrate the development of interstitial lung changes and pulmonary fibrosis in the same distribution and pattern as the acute phase findings, up to 6 months after the acute infection, demonstrating a direct relationship between these changes and COVID-19 pneumonia.

Performance of a Severity Score on Admission Chest Radiography in Predicting Clinical Outcomes in Hospitalized Patients With Coronavirus Disease (COVID-19).

BACKGROUND. Chest radiographs (CXRs) are typically obtained early in patients admitted with coronavirus disease (COVID-19) and may help guide prognosis and initial management decisions. OBJECTIVE. The purpose of this study was to assess the performance of an admission CXR severity scoring system in predicting hospital outcomes in patients admitted with COVID-19. METHODS. This retrospective study included 240 patients (142 men, 98 women; median age, 65 [range, 50-80] years) admitted to the hospital from March 16 to April 13, 2020, with COVID-19 confirmed by real-time reverse-transcriptase polymerase chain reaction who underwent chest radiography within 24 hours of admission. Three attending chest radiologists and three radiology residents independently scored patients' admission CXRs using a 0- to 24-point composite scale (sum of scores that range from 0 to 3 for extent and severity of disease in upper and lower zones of left and right lungs). Interrater reliability of the score was assessed using the Kendall W coefficient. The mean score was obtained from the six readers' scores for further analyses. Demographic variables, clinical characteristics, and admission laboratory values were collected from electronic medical records. ROC analysis was performed to assess the association between CXR severity and mortality. Additional univariable and multivariable logistic regression models incorporating patient characteristics and laboratory values were tested for associations between CXR severity and clinical outcomes. RESULTS. Interrater reliability of CXR scores ranged from 0.687 to 0.737 for attending radiologists, from 0.653 to 0.762 for residents, and from 0.575 to 0.666 for all readers. A composite CXR score of 10 or higher on admission achieved 53.0% (35/66) sensitivity and 75.3% (131/174) specificity for predicting hospital mortality. Hospital mortality occurred in 44.9% (35/78) of patients with a high-risk admission CXR score (≥ 10) versus 19.1% (31/162) of patients with a low-risk CXR score (< 10) (p < .001). Admission composite CXR score was an independent predictor of death (odds ratio [OR], 1.17; 95% CI, 1.10-1.24; p < .001). composite CXR score was a univariable predictor of intubation (OR, 1.23; 95% CI, 1.12-1.34; p < .001) and continuous renal replacement therapy (CRRT) (OR, 1.15; 95% CI, 1.04-1.27; p = .007) but was not associated with these in multivariable models (p > .05). CONCLUSION. For patients admitted with COVID-19, an admission CXR severity score may help predict hospital mortality, intubation, and CRRT. CLINICAL IMPACT. CXR may assist risk assessment and clinical decision-making early in the course of COVID-19.

Crowdsourcing pneumothorax annotations using machine learning annotations on the NIH chest X-ray dataset.

Pneumothorax is a potentially life-threatening condition that requires prompt recognition and often urgent intervention. In the ICU setting, large numbers of chest radiographs are performed and must be interpreted on a daily basis which may delay diagnosis of this entity. Development of artificial intelligence (AI) techniques to detect pneumothorax could help expedite detection as well as localize and potentially quantify pneumothorax. Open image analysis competitions are useful in advancing state-of-the art AI algorithms but generally require large expert annotated datasets. We have annotated and adjudicated a large dataset of chest radiographs to be made public with the goal of sparking innovation in this space. Because of the cumbersome and time-consuming nature of image labeling, we explored the value of using AI models to generate annotations for review. Utilization of this machine learning annotation (MLA) technique appeared to expedite our annotation process with relatively high sensitivity at the expense of specificity. Further research is required to confirm and better characterize the value of MLAs. Our adjudicated dataset is now available for public consumption in the form of a challenge.

Assessment of Critical Feeding Tube Malpositions on Radiographs Using Deep Learning.

Assess the efficacy of deep convolutional neural networks (DCNNs) in detection of critical enteric feeding tube malpositions on radiographs. 5475 de-identified HIPAA compliant frontal view chest and abdominal radiographs were obtained, consisting of 174 x-rays of bronchial insertions and 5301 non-critical radiographs, including normal course, normal chest, and normal abdominal x-rays. The ground-truth classification for enteric feeding tube placement was performed by two board-certified radiologists. Untrained and pretrained deep convolutional neural network models for Inception V3, ResNet50, and DenseNet 121 were each employed. The radiographs were fed into each deep convolutional neural network, which included untrained and pretrained models. The Tensorflow framework was used for Inception V3, ResNet50, and DenseNet. Images were split into training (4745), validation (630), and test (100). Both real-time and preprocessing image augmentation strategies were performed. Receiver operating characteristic (ROC) and area under the curve (AUC) on the test data were used to assess the models. Statistical differences among the AUCs were obtained. p < 0.05 was considered statistically significant. The pretrained Inception V3, which had an AUC of 0.87 (95 CI; 0.80-0.94), performed statistically significantly better (p < .001) than the untrained Inception V3, with an AUC of 0.60 (95 CI; 0.52-0.68). The pretrained Inception V3 also had the highest AUC overall, as compared with ResNet50 and DenseNet121, with AUC values ranging from 0.82 to 0.85. Each pretrained network outperformed its untrained counterpart. (p < 0.05). Deep learning demonstrates promise in differentiating critical vs. non-critical placement with an AUC of 0.87. Pretrained networks outperformed untrained ones in all cases. DCNNs may allow for more rapid identification and communication of critical feeding tube malpositions.

Lung Cancer Screening Guidelines: How Readable Are Internet-Based Patient Education Resources?

OBJECTIVE: Following the findings of the National Lung Screening Trial, several national societies from multiple disciplines have endorsed the use of low-dose chest CT to screen for lung cancer. Online patient education materials are an important tool to disseminate information to the general public regarding the proven health benefits of lung cancer screening. This study aims to evaluate the reading level at which these materials related to lung cancer screening are written. MATERIALS AND METHODS: The four terms "pulmonary nodule," "radiation," "low-dose CT," and "lung cancer screening" were searched on Google, and the first 20 online resources for each term were downloaded, converted into plain text, and analyzed using 10 well-established readability scales. If the websites were not written specifically for patients, they were excluded. RESULTS: The 80 articles were written at a 12.6 ± 2.7 (mean ± SD) grade level, with grade levels ranging from 4.0 to 19.0. Of the 80 articles, 62.5% required a high school education to comprehend, and 22.6% required a college degree or higher (≥ 16th grade) to comprehend. Only 2.5% of the analyzed articles adhered to the recommendations of the National Institutes of Health and American Medical Association that patient education materials be written at a 3rd- to 7th-grade reading level. CONCLUSION: Commonly visited online lung cancer screening-related patient education materials are written at a level beyond the general patient population's ability to comprehend and may be contributing to a knowledge gap that is inhibiting patients from improving their health literacy.

Hello World Deep Learning in Medical Imaging.

There is recent popularity in applying machine learning to medical imaging, notably deep learning, which has achieved state-of-the-art performance in image analysis and processing. The rapid adoption of deep learning may be attributed to the availability of machine learning frameworks and libraries to simplify their use. In this tutorial, we provide a high-level overview of how to build a deep neural network for medical image classification, and provide code that can help those new to the field begin their informatics projects.

Machine Learning in Radiology: Applications Beyond Image Interpretation.

Much attention has been given to machine learning and its perceived impact in radiology, particularly in light of recent success with image classification in international competitions. However, machine learning is likely to impact radiology outside of image interpretation long before a fully functional "machine radiologist" is implemented in practice. Here, we describe an overview of machine learning, its application to radiology and other domains, and many cases of use that do not involve image interpretation. We hope that better understanding of these potential applications will help radiology practices prepare for the future and realize performance improvement and efficiency gains.

Deep Convolutional Neural Networks for Endotracheal Tube Position and X-ray Image Classification: Challenges and Opportunities.

The goal of this study is to evaluate the efficacy of deep convolutional neural networks (DCNNs) in differentiating subtle, intermediate, and more obvious image differences in radiography. Three different datasets were created, which included presence/absence of the endotracheal (ET) tube (n = 300), low/normal position of the ET tube (n = 300), and chest/abdominal radiographs (n = 120). The datasets were split into training, validation, and test. Both untrained and pre-trained deep neural networks were employed, including AlexNet and GoogLeNet classifiers, using the Caffe framework. Data augmentation was performed for the presence/absence and low/normal ET tube datasets. Receiver operating characteristic (ROC), area under the curves (AUC), and 95% confidence intervals were calculated. Statistical differences of the AUCs were determined using a non-parametric approach. The pre-trained AlexNet and GoogLeNet classifiers had perfect accuracy (AUC 1.00) in differentiating chest vs. abdominal radiographs, using only 45 training cases. For more difficult datasets, including the presence/absence and low/normal position endotracheal tubes, more training cases, pre-trained networks, and data-augmentation approaches were helpful to increase accuracy. The best-performing network for classifying presence vs. absence of an ET tube was still very accurate with an AUC of 0.99. However, for the most difficult dataset, such as low vs. normal position of the endotracheal tube, DCNNs did not perform as well, but achieved a reasonable AUC of 0.81.

Diagnostic Performance of the Visual Reading of 123I-Ioflupane SPECT Images With or Without Quantification in Patients With Movement Disorders or Dementia.

Visual interpretation of 123I-ioflupane SPECT images has high diagnostic accuracy for differentiating parkinsonian syndromes (PS), from essential tremor and probable dementia with Lewy bodies (DLB) from Alzheimer disease. In this study, we investigated the impact on accuracy and reader confidence offered by the addition of image quantification in comparison with visual interpretation alone. Methods: We collected 304 123I-ioflupane images from 3 trials that included subjects with a clinical diagnosis of PS, non-PS (mainly essential tremor), probable DLB, and non-DLB (mainly Alzheimer disease). Images were reconstructed with standardized parameters before striatal binding ratios were quantified against a normal database. Images were assessed by 5 nuclear medicine physicians who had limited prior experience with 123I-ioflupane interpretation. In 2 readings at least 1 mo apart, readers performed either a visual interpretation alone or a combined reading (i.e., visual plus quantitative data were available). Readers were asked to rate their confidence of image interpretation and judge scans as easy or difficult to read. Diagnostic accuracy was assessed by comparing image results with the standard of truth (i.e., diagnosis at follow-up) by measuring the positive percentage of agreement (equivalent to sensitivity) and the negative percentage of agreement (equivalent to specificity). The hypothesis that the results of the combined reading were not inferior to the results of the visual reading analysis was tested. Results: A comparison of the combined reading and the visual reading revealed a small, insignificant increase in the mean negative percentage of agreement (89.9% vs. 87.9%) and equivalent positive percentages of agreement (80.2% vs. 80.1%). Readers who initially performed a combined analysis had significantly greater accuracy (85.8% vs. 79.2%; P = 0.018), and their accuracy was close to that of the expert readers in the original studies (range, 83.3%-87.2%). Mean reader confidence in the interpretation of images showed a significant improvement when combined analysis was used (P < 0.0001). Conclusion: The addition of quantification allowed readers with limited experience in the interpretation of 123I-ioflupane SPECT scans to have diagnostic accuracy equivalent to that of the experienced readers in the initial studies. Also, the results of the combined reading were not inferior to the results of the visual reading analysis and offered an increase in reader confidence.

Deep Learning at Chest Radiography: Automated Classification of Pulmonary Tuberculosis by Using Convolutional Neural Networks.

Purpose To evaluate the efficacy of deep convolutional neural networks (DCNNs) for detecting tuberculosis (TB) on chest radiographs. Materials and Methods Four deidentified HIPAA-compliant datasets were used in this study that were exempted from review by the institutional review board, which consisted of 1007 posteroanterior chest radiographs. The datasets were split into training (68.0%), validation (17.1%), and test (14.9%). Two different DCNNs, AlexNet and GoogLeNet, were used to classify the images as having manifestations of pulmonary TB or as healthy. Both untrained and pretrained networks on ImageNet were used, and augmentation with multiple preprocessing techniques. Ensembles were performed on the best-performing algorithms. For cases where the classifiers were in disagreement, an independent board-certified cardiothoracic radiologist blindly interpreted the images to evaluate a potential radiologist-augmented workflow. Receiver operating characteristic curves and areas under the curve (AUCs) were used to assess model performance by using the DeLong method for statistical comparison of receiver operating characteristic curves. Results The best-performing classifier had an AUC of 0.99, which was an ensemble of the AlexNet and GoogLeNet DCNNs. The AUCs of the pretrained models were greater than that of the untrained models (P < .001). Augmenting the dataset further increased accuracy (P values for AlexNet and GoogLeNet were .03 and .02, respectively). The DCNNs had disagreement in 13 of the 150 test cases, which were blindly reviewed by a cardiothoracic radiologist, who correctly interpreted all 13 cases (100%). This radiologist-augmented approach resulted in a sensitivity of 97.3% and specificity 100%. Conclusion Deep learning with DCNNs can accurately classify TB at chest radiography with an AUC of 0.99. A radiologist-augmented approach for cases where there was disagreement among the classifiers further improved accuracy. © RSNA, 2017.

Automated extraction of critical test values and communications from unstructured radiology reports: an analysis of 9.3 million reports from 1990 to 2011.

PURPOSE: To determine the frequency of critical radiology results in 9.3 million radiology reports from our health system, to identify those containing documentation of communication by using automated text-classification algorithms, and to assess the impact of a policy requiring documentation of critical results communication. MATERIALS AND METHODS: This HIPAA-compliant retrospective study received institutional review board approval. Text-mining algorithms that were previously validated to have mean accuracies of more than 90% for identifying certain critical results and documentation of communications were applied to a database of 9.3 million radiology reports. The frequency of critical results and documentation of communication were then determined from 1990 to 2011. RESULTS: There was an increase in documentation of communication for all critical results from 1990 to 2011. In 1990, 19.0% of reports with critical values had evidence of documentation of communication compared with 72.4% of reports in 2010. The linear trend for increasing documentation of communications began in 1997 and continued until 2011 (P < .001). From 1990 to 2011, documentation of communication was highest in acute scrotal torsion (70.6%) and ectopic pregnancy (65.4%) and lowest in unexplained free-intraperitoneal air (29.5%) and malpositioned tubes (30.4%). In 2010-2011, radiologists were least likely to document communication of results for malpositioned endotracheal and enteric tubes (2010, 58.56%; 2011, 57.50%) and unexplained free-intraperitoneal air (2010, 59.57%; 2011, 75.51%). They were most likely to document communication of results for ectopic pregnancy (2010, 94.12%; 2011, 93.48%) and acute appendicitis (2010, 86.87%; 2011, 84.31%). CONCLUSION: There was an increase in documentation of communication of critical results, which demonstrated a rising linear trend that began in 1997 and continued until 2011. The increasing trend began well before policy implementation, indicating that other factors such as heightened awareness among radiologists likely had a role.