Performance of AI to exclude normal chest radiographs to reduce radiologists' workload.

INTRODUCTION: This study investigates the performance of a commercially available artificial intelligence (AI) system to identify normal chest radiographs and its potential to reduce radiologist workload. METHODS: Retrospective analysis included consecutive chest radiographs from two medical centers between Oct 1, 2016 and Oct 14, 2016. Exclusions comprised follow-up exams within the inclusion period, bedside radiographs, incomplete images, imported radiographs, and pediatric radiographs. Three chest radiologists categorized findings into normal, clinically irrelevant, clinically relevant, urgent, and critical. A commercial AI system processed all radiographs, scoring 10 chest abnormalities on a 0-100 confidence scale. AI system performance was evaluated using the area under the ROC curve (AUC), assessing the detection of normal radiographs. Sensitivity was calculated for the default and a conservative operating point. the detection of negative predictive value (NPV) for urgent and critical findings, as well as the potential workload reduction, was calculated. RESULTS: A total of 2603 radiographs were acquired in 2141 unique patients. Post-exclusion, 1670 radiographs were analyzed. Categories included 479 normal, 332 clinically irrelevant, 339 clinically relevant, 501 urgent, and 19 critical findings. The AI system achieved an AUC of 0.92. Sensitivity for normal radiographs was 92% at default and 53% at the conservative operating point. At the conservative operating point, NPV was 98% for urgent and critical findings, and could result in a 15% workload reduction. CONCLUSION: A commercially available AI system effectively identifies normal chest radiographs and holds the potential to lessen radiologists' workload by omitting half of the normal exams from reporting. CLINICAL RELEVANCE STATEMENT: The AI system is able to detect half of all normal chest radiographs at a clinically acceptable operating point, thereby potentially reducing the workload for the radiologists by 15%. KEY POINTS: The AI system reached an AUC of 0.92 for the detection of normal chest radiographs. Fifty-three percent of normal chest radiographs were identified with a NPV of 98% for urgent findings. AI can reduce the workload of chest radiography reporting by 15%.

Nodule Detection and Generation on Chest X-Rays: NODE21 Challenge.

Pulmonary nodules may be an early manifestation of lung cancer, the leading cause of cancer-related deaths among both men and women. Numerous studies have established that deep learning methods can yield high-performance levels in the detection of lung nodules in chest X-rays. However, the lack of gold-standard public datasets slows down the progression of the research and prevents benchmarking of methods for this task. To address this, we organized a public research challenge, NODE21, aimed at the detection and generation of lung nodules in chest X-rays. While the detection track assesses state-of-the-art nodule detection systems, the generation track determines the utility of nodule generation algorithms to augment training data and hence improve the performance of the detection systems. This paper summarizes the results of the NODE21 challenge and performs extensive additional experiments to examine the impact of the synthetically generated nodule training images on the detection algorithm performance.

How AI should be used in radiology: assessing ambiguity and completeness of intended use statements of commercial AI products.

BACKGROUND: Intended use statements (IUSs) are mandatory to obtain regulatory clearance for artificial intelligence (AI)-based medical devices in the European Union. In order to guide the safe use of AI-based medical devices, IUSs need to contain comprehensive and understandable information. This study analyzes the IUSs of CE-marked AI products listed on AIforRadiology.com for ambiguity and completeness. METHODS: We retrieved 157 IUSs of CE-marked AI products listed on AIforRadiology.com in September 2022. Duplicate products (n = 1), discontinued products (n = 3), and duplicate statements (n = 14) were excluded. The resulting IUSs were assessed for the presence of 6 items: medical indication, part of the body, patient population, user profile, use environment, and operating principle. Disclaimers, defined as contra-indications or warnings in the IUS, were identified and compared with claims. RESULTS: Of 139 AI products, the majority (n = 78) of IUSs mentioned 3 or less items. IUSs of only 7 products mentioned all 6 items. The intended body part (n = 115) and the operating principle (n = 116) were the most frequently mentioned components, while the intended use environment (n = 24) and intended patient population (n = 29) were mentioned less frequently. Fifty-six statements contained disclaimers that conflicted with the claims in 13 cases. CONCLUSION: The majority of IUSs of CE-marked AI-based medical devices lack substantial information and, in few cases, contradict the claims of the product. CRITICAL RELEVANCE STATEMENT: To ensure correct usage and to avoid off-label use or foreseeable misuse of AI-based medical devices in radiology, manufacturers are encouraged to provide more comprehensive and less ambiguous intended use statements. KEY POINTS: • Radiologists must know AI products' intended use to avoid off-label use or misuse. • Ninety-five percent (n = 132/139) of the intended use statements analyzed were incomplete. • Nine percent (n = 13) of the intended use statements held disclaimers contradicting the claim of the AI product. • Manufacturers and regulatory bodies must ensure that intended use statements are comprehensive.

Comparison of Commercial AI Software Performance for Radiograph Lung Nodule Detection and Bone Age Prediction.

Background Multiple commercial artificial intelligence (AI) products exist for assessing radiographs; however, comparable performance data for these algorithms are limited. Purpose To perform an independent, stand-alone validation of commercially available AI products for bone age prediction based on hand radiographs and lung nodule detection on chest radiographs. Materials and Methods This retrospective study was carried out as part of Project AIR. Nine of 17 eligible AI products were validated on data from seven Dutch hospitals. For bone age prediction, the root mean square error (RMSE) and Pearson correlation coefficient were computed. The reference standard was set by three to five expert readers. For lung nodule detection, the area under the receiver operating characteristic curve (AUC) was computed. The reference standard was set by a chest radiologist based on CT. Randomized subsets of hand (n = 95) and chest (n = 140) radiographs were read by 14 and 17 human readers, respectively, with varying experience. Results Two bone age prediction algorithms were tested on hand radiographs (from January 2017 to January 2022) in 326 patients (mean age, 10 years ± 4 [SD]; 173 female patients) and correlated strongly with the reference standard (r = 0.99; P < .001 for both). No difference in RMSE was observed between algorithms (0.63 years [95% CI: 0.58, 0.69] and 0.57 years [95% CI: 0.52, 0.61]) and readers (0.68 years [95% CI: 0.64, 0.73]). Seven lung nodule detection algorithms were validated on chest radiographs (from January 2012 to May 2022) in 386 patients (mean age, 64 years ± 11; 223 male patients). Compared with readers (mean AUC, 0.81 [95% CI: 0.77, 0.85]), four algorithms performed better (AUC range, 0.86-0.93; P value range, <.001 to .04). Conclusions Compared with human readers, four AI algorithms for detecting lung nodules on chest radiographs showed improved performance, whereas the remaining algorithms tested showed no evidence of a difference in performance. © RSNA, 2024 Supplemental material is available for this article. See also the editorial by Omoumi and Richiardi in this issue.