Chest Radiograph Interpretation with Deep Learning Models: Assessment with Radiologist-adjudicated Reference Standards and Population-adjusted Evaluation.

BackgroundDeep learning has the potential to augment the use of chest radiography in clinical radiology, but challenges include poor generalizability, spectrum bias, and difficulty comparing across studies.PurposeTo develop and evaluate deep learning models for chest radiograph interpretation by using radiologist-adjudicated reference standards.Materials and MethodsDeep learning models were developed to detect four findings (pneumothorax, opacity, nodule or mass, and fracture) on frontal chest radiographs. This retrospective study used two data sets. Data set 1 (DS1) consisted of 759 611 images from a multicity hospital network and ChestX-ray14 is a publicly available data set with 112 120 images. Natural language processing and expert review of a subset of images provided labels for 657 954 training images. Test sets consisted of 1818 and 1962 images from DS1 and ChestX-ray14, respectively. Reference standards were defined by radiologist-adjudicated image review. Performance was evaluated by area under the receiver operating characteristic curve analysis, sensitivity, specificity, and positive predictive value. Four radiologists reviewed test set images for performance comparison. Inverse probability weighting was applied to DS1 to account for positive radiograph enrichment and estimate population-level performance.ResultsIn DS1, population-adjusted areas under the receiver operating characteristic curve for pneumothorax, nodule or mass, airspace opacity, and fracture were, respectively, 0.95 (95% confidence interval [CI]: 0.91, 0.99), 0.72 (95% CI: 0.66, 0.77), 0.91 (95% CI: 0.88, 0.93), and 0.86 (95% CI: 0.79, 0.92). With ChestX-ray14, areas under the receiver operating characteristic curve were 0.94 (95% CI: 0.93, 0.96), 0.91 (95% CI: 0.89, 0.93), 0.94 (95% CI: 0.93, 0.95), and 0.81 (95% CI: 0.75, 0.86), respectively.ConclusionExpert-level models for detecting clinically relevant chest radiograph findings were developed for this study by using adjudicated reference standards and with population-level performance estimation. Radiologist-adjudicated labels for 2412 ChestX-ray14 validation set images and 1962 test set images are provided.© RSNA, 2019Online supplemental material is available for this article.See also the editorial by Chang in this issue.

A prospective clinical study to evaluate the safety and performance of wireless localization of nonpalpable breast lesions using radiofrequency identification technology.

OBJECTIVE: The purpose of this study was to evaluate the safety and performance of localizing nonpalpable breast lesions using radiofrequency identification technology. SUBJECTS AND METHODS: Twenty consecutive women requiring preoperative localization of a breast lesion were recruited. Subjects underwent placement of both a hook wire and a radiofrequency identification tag immediately before surgery. The radiofrequency identification tag was the primary method used by the operating surgeon to localize each lesion during excision, with the hook wire serving as backup in case of tag migration or failed localization. Successful localization with removal of the intended lesion was the primary outcome measured. Tag migration and postoperative infection were also noted to assess safety. RESULTS: Twenty patients underwent placement of a radiofrequency identification tag, 12 under ultrasound guidance and eight with stereotactic guidance. In all cases, the radiofrequency identification tag was successfully localized by the reader at the level of the skin before incision, and the intended lesion was removed along with the radiofrequency identification tag. There were no localization failures and no postoperative infections. Tag migration did not occur before incision, but in three cases, occurred as the lesion was being retracted with fingers to make the final cut along the deep surface of the specimen. CONCLUSION: In this initial clinical study, radiofrequency tags were safe and able to successfully localize nonpalpable breast lesions. Radiofrequency identification technology may represent an alternative method to hook wire localization.

Evaluation of lossy data compression in primary interpretation for full-field digital mammography.

OBJECTIVE. For full-field digital mammography (FFDM), federal regulations prohibit lossy data compression for primary reading and archiving, unlike all other medical images, where reading physicians can apply their professional judgment in implementing lossy compression. Faster image transfer, lower costs, and greater access to expert mammographers would result from development of a safe standard for primary interpretation and archive of lossy-compressed FFDM images. This investigation explores whether JPEG 2000 80:1 lossy data compression affects clinical accuracy in digital mammography. MATERIALS AND METHODS. Randomized FFDM cases (n = 194) were interpreted by six experienced mammographers with and without JPEG 2000 80:1 lossy compression applied. A cancer-enriched population was used, with just less than half of the cases (42%) containing subtle (< 1 cm) biopsy-proven cancerous lesions, and the remaining cases were negative as proven by 2-year follow-up. Data were analyzed using the jackknife alternative free-response ROC (JAFROC) method. RESULTS. The differences in reader performance between lossy-compressed and non-lossy-compressed images using lesion localization (0.660 vs 0.671), true-positive fraction (0.879 vs 0.879), and false-positive fraction (0.283 vs 0.271) were not statistically significant. There was no difference in the JAFROC figure of merit between lossy-compressed and non-lossy-compressed images, with a mean difference of -0.01 (95% CI, -0.03 to 0.01; F1,5 = 2.30; p = 0.189). CONCLUSION. These results suggest that primary interpretation of JPEG 2000 80:1 lossy-compressed FFDM images may be viable without degradation of clinical quality. Benefits would include lower storage costs, faster telemammography, and enhanced access to expert mammographers.

Computed tomography machine learning classifier correlates with mortality in interstitial lung disease.

BACKGROUND: A machine learning classifier system, Fibresolve, was designed and validated as an adjunct to non-invasive diagnosis in idiopathic pulmonary fibrosis (IPF). The system uses a deep learning algorithm to analyze chest computed tomography (CT) imaging. We hypothesized that Fibresolve is a useful predictor of mortality in interstitial lung diseases (ILD). METHODS: Fibresolve was previously validated in a multi-site >500-patient dataset. In this analysis, we assessed the usefulness of Fibresolve to predict mortality in a subset of 228 patients with IPF and other ILDs in whom follow up data was available. We applied Cox regression analysis adjusting for the Gender, Age, and Physiology (GAP) score and for other known predictors of mortality in IPF. We also analyzed the role of Fibresolve as tertiles adjusting for GAP stages. RESULTS: During a median follow-up of 2.8 years (range 5 to 3434 days), 89 patients died. After adjusting for GAP score and other mortality risk factors, the Fibresolve score significantly predicted the risk of death (HR: 7.14; 95% CI: 1.31-38.85; p = 0.02) during the follow-up period, as did forced vital capacity and history of lung cancer. After adjusting for GAP stages and other variables, Fibresolve score split into tertiles significantly predicted the risk of death (p = 0.027 for the model; HR 1.37 for 2nd tertile; 95% CI: 0.77-2.42. HR 2.19 for 3rd tertile; 95% CI: 1.22-3.93). CONCLUSIONS: The machine learning classifier Fibresolve demonstrated to be an independent predictor of mortality in ILDs, with prognostic performance equivalent to GAP based solely on CT images.

Analysis of Validation Performance of a Machine Learning Classifier in Interstitial Lung Disease Cases Without Definite or Probable Usual Interstitial Pneumonia Pattern on CT Using Clinical and Pathology-Supported Diagnostic Labels.

We previously validated Fibresolve, a machine learning classifier system that non-invasively predicts idiopathic pulmonary fibrosis (IPF) diagnosis. The system incorporates an automated deep learning algorithm that analyzes chest computed tomography (CT) imaging to assess for features associated with idiopathic pulmonary fibrosis. Here, we assess performance in assessment of patterns beyond those that are characteristic features of usual interstitial pneumonia (UIP) pattern. The machine learning classifier was previously developed and validated using standard training, validation, and test sets, with clinical plus pathologically determined ground truth. The multi-site 295-patient validation dataset was used for focused subgroup analysis in this investigation to evaluate the classifier's performance range in cases with and without radiologic UIP and probable UIP designations. Radiologic assessment of specific features for UIP including the presence and distribution of reticulation, ground glass, bronchiectasis, and honeycombing was used for assignment of radiologic pattern. Output from the classifier was assessed within various UIP subgroups. The machine learning classifier was able to classify cases not meeting the criteria for UIP or probable UIP as IPF with estimated sensitivity of 56-65% and estimated specificity of 92-94%. Example cases demonstrated non-basilar-predominant as well as ground glass patterns that were indeterminate for UIP by subjective imaging criteria but for which the classifier system was able to correctly identify the case as IPF as confirmed by multidisciplinary discussion generally inclusive of histopathology. The machine learning classifier Fibresolve may be helpful in the diagnosis of IPF in cases without radiological UIP and probable UIP patterns.

Development and validation of a CT-based deep learning algorithm to augment non-invasive diagnosis of idiopathic pulmonary fibrosis.

RATIONALE: Non-invasive diagnosis of idiopathic pulmonary fibrosis (IPF) involves identification of usual interstitial pneumonia (UIP) pattern by computed tomography (CT) and exclusion of other known etiologies of interstitial lung disease (ILD). However, uncertainty in identification of radiologic UIP pattern leads to the continued need for invasive surgical biopsy. We thus developed and validated a machine learning algorithm using CT scans alone to augment non-invasive diagnosis of IPF. METHODS: The primary algorithm was a deep learning convolutional neural network (CNN) with model inputs of CT images only. The algorithm was trained to predict IPF among cases of ILD, with reference standard of multidisciplinary discussion (MDD) consensus diagnosis. The algorithm was trained using a multi-center dataset of more than 2000 cases of ILD. A US-based multi-site cohort (n = 295) was used for algorithm tuning, and external validation was performed with a separate dataset (n = 295) from European and South American sources. RESULTS: In the tuning set, the model achieved an area under the receiver operating characteristic curve (AUC) of 0.87 (CI: 0.83-0.92) in differentiating IPF from other ILDs. Sensitivity and specificity were 0.67 (0.57-0.76) and 0.90 (0.83-0.95), respectively. By contrast, pre-recorded assessment prior to MDD diagnosis had sensitivity of 0.31 (0.23-0.42) and specificity of 0.92 (0.87-0.95). In the external test set, c-statistic was also 0.87 (0.83-0.91). Model performance was consistent across a variety of CT scanner manufacturers and slice thickness. CONCLUSION: The presented deep learning algorithm demonstrated consistent performance in identifying IPF among cases of ILD using CT images alone and suggests generalization across CT manufacturers.

Improving reference standards for validation of AI-based radiography.

OBJECTIVE: Demonstrate the importance of combining multiple readers' opinions, in a context-aware manner, when establishing the reference standard for validation of artificial intelligence (AI) applications for, e.g. chest radiographs. By comparing individual readers, majority vote of a panel, and panel-based discussion, we identify methods which maximize interobserver agreement and label reproducibility. METHODS: 1100 frontal chest radiographs were evaluated for 6 findings: airspace opacity, cardiomegaly, pulmonary edema, fracture, nodules, and pneumothorax. Each image was reviewed by six radiologists, first individually and then via asynchronous adjudication (web-based discussion) in two panels of three readers to resolve disagreements within each panel. We quantified the reproducibility of each method by measuring interreader agreement. RESULTS: Panel-based majority vote improved agreement relative to individual readers for all findings. Most disagreements were resolved with two rounds of adjudication, which further improved reproducibility for some findings, particularly reducing misses. Improvements varied across finding categories, with adjudication improving agreement for cardiomegaly, fractures, and pneumothorax. CONCLUSION: The likelihood of interreader agreement, even within panels of US board-certified radiologists, must be considered before reads can be used as a reference standard for validation of proposed AI tools. Agreement and, by extension, reproducibility can be improved by applying majority vote, maximum sensitivity, or asynchronous adjudication for different findings, which supports the development of higher quality clinical research. ADVANCES IN KNOWLEDGE: A panel of three experts is a common technique for establishing reference standards when ground truth is not available for use in AI validation. The manner in which differing opinions are resolved is shown to be important, and has not been previously explored.

Author Correction: End-to-end lung cancer screening with three-dimensional deep learning on low-dose chest computed tomography.

An amendment to this paper has been published and can be accessed via a link at the top of the paper.

End-to-end lung cancer screening with three-dimensional deep learning on low-dose chest computed tomography.

With an estimated 160,000 deaths in 2018, lung cancer is the most common cause of cancer death in the United States1. Lung cancer screening using low-dose computed tomography has been shown to reduce mortality by 20-43% and is now included in US screening guidelines1-6. Existing challenges include inter-grader variability and high false-positive and false-negative rates7-10. We propose a deep learning algorithm that uses a patient's current and prior computed tomography volumes to predict the risk of lung cancer. Our model achieves a state-of-the-art performance (94.4% area under the curve) on 6,716 National Lung Cancer Screening Trial cases, and performs similarly on an independent clinical validation set of 1,139 cases. We conducted two reader studies. When prior computed tomography imaging was not available, our model outperformed all six radiologists with absolute reductions of 11% in false positives and 5% in false negatives. Where prior computed tomography imaging was available, the model performance was on-par with the same radiologists. This creates an opportunity to optimize the screening process via computer assistance and automation. While the vast majority of patients remain unscreened, we show the potential for deep learning models to increase the accuracy, consistency and adoption of lung cancer screening worldwide.

Radiofrequency identification tags for preoperative tumor localization: proof of concept.

OBJECTIVE: The objective of our study was to experimentally explore the potential for tumor localization using radiofrequency identification (RFID) tags and a newly developed handheld RFID detector. MATERIALS AND METHODS: A unique RFID detector that combines the use of multiple interchangeable detector probes with both audio and LCD display signals was invented, allowing precise localization and identification of RFID tags. Accurate localization and identification were validated using this handheld RFID detector (TagFinder) and RFID tags of 2-mm diameter and 8- or 12-mm lengths. Experiments included the following: validation in various breast phantoms; differentiation of 4- to 6-cm-diameter tissue specimens with and without tags; determination of the nearest differentiable distance between two tags; proof of visualization of tags on sonography, radiography, and MRI; and experimental localization and resection of RFID-labeled tissue specimens. RESULTS: Both 8- and 12-mm-length RFID tags implanted < 6 cm deep were accurately localized and uniquely identified. Chicken breast specimens of 4- to 6-cm diameter implanted with RFID tags were accurately differentiated from specimens without tags. Tags in proximity could be reliably differentiated and uniquely identified when placed as close as 0-2 cm apart, depending on the tags' precise orientations. RFID tags were easily visualized with sonography, mammography, and MRI, with artifacts present only on MRI. Localization and resection of RFID tags in the labeled tissue region were successful in grocery store-bought chicken breasts. CONCLUSION: The combination of RFID tags and a new handheld RFID detector shows promise for preoperative imaging-guided tumor localization.

Increased prevalence of tracheal diverticula in cystic fibrosis patients.

OBJECTIVE: This investigation sought to evaluate the prevalence and imaging characteristics of tracheal diverticula (TD) among patients with cystic fibrosis (CF). METHODS: A total of 113 patients with CF at our institution, with a median age of 29 years, had chest CT examinations between 2002 and 2014. These imaging studies were retrospectively reviewed to assess for the presence and characteristics of TD, including quantity, size and location. The severity of the CF disease was assessed using the Bhalla CT scoring system and pulmonary function tests. RESULTS: Of the 113 cases reviewed, 20 (17.7%) patients were found to have 1 or more TD. The presence of TD was associated with more severe disease by imaging criteria, with a Bhalla CT score of 13.9 ± 4.3 in patients with TD and 11.5 ± 4.3 in patients without TD. For the pulmonary function tests, forced expiratory volume in 1 s (FEV1) and FEV1 percent predicted demonstrated a trend towards worsening function in patients with TD, although the difference was not statistically significant. CONCLUSION: The prevalence of TD in our patient population with CF based on chest CT exams was 17.7%. In addition, the presence of TD was associated with more severe CF disease by imaging criteria. ADVANCES IN KNOWLEDGE: TD appear to have a higher prevalence in patients with CF than in the general population, are associated with more severe CF pulmonary disease by CT criteria and are frequently underreported by radiologists.

Case report: benign gastric ulcer erosion leading to a gastropericardial fistula in a patient with no known risk factors.

A 53-year-old homeless male presented to the emergency department with sudden onset chest pain and was found to have a large pneumopericardium on chest X-ray. The patient had no history of surgery, hiatal hernia, or ulcer disease. A contrast-enhanced computed tomography scan demonstrated the pneumopericardium and raised concern for possible gastropericardial fistula from a benign gastric ulcer. An esophagogastroduodenoscopy confirmed the fistula, as did surgery, and intraoperatively vegetable particular matter was removed from the anatomic space continuous with the pericardium.