Interobserver Agreement and Performance of Concurrent AI Assistance for Radiographic Evaluation of Knee Osteoarthritis.

Background Due to conflicting findings in the literature, there are concerns about a lack of objectivity in grading knee osteoarthritis (KOA) on radiographs. Purpose To examine how artificial intelligence (AI) assistance affects the performance and interobserver agreement of radiologists and orthopedists of various experience levels when evaluating KOA on radiographs according to the established Kellgren-Lawrence (KL) grading system. Materials and Methods In this retrospective observer performance study, consecutive standing knee radiographs from patients with suspected KOA were collected from three participating European centers between April 2019 and May 2022. Each center recruited four readers across radiology and orthopedic surgery at in-training and board-certified experience levels. KL grading (KL-0 = no KOA, KL-4 = severe KOA) on the frontal view was assessed by readers with and without assistance from a commercial AI tool. The majority vote of three musculoskeletal radiology consultants established the reference standard. The ordinal receiver operating characteristic method was used to estimate grading performance. Light kappa was used to estimate interrater agreement, and bootstrapped t statistics were used to compare groups. Results Seventy-five studies were included from each center, totaling 225 studies (mean patient age, 55 years ± 15 [SD]; 113 female patients). The KL grades were KL-0, 24.0% (n = 54); KL-1, 28.0% (n = 63); KL-2, 21.8% (n = 49); KL-3, 18.7% (n = 42); and KL-4, 7.6% (n = 17). Eleven readers completed their readings. Three of the six junior readers showed higher KL grading performance with versus without AI assistance (area under the receiver operating characteristic curve, 0.81 ± 0.017 [SEM] vs 0.88 ± 0.011 [P < .001]; 0.76 ± 0.018 vs 0.86 ± 0.013 [P < .001]; and 0.89 ± 0.011 vs 0.91 ± 0.009 [P = .008]). Interobserver agreement for KL grading among all readers was higher with versus without AI assistance (κ = 0.77 ± 0.018 [SEM] vs 0.85 ± 0.013; P < .001). Board-certified radiologists achieved almost perfect agreement for KL grading when assisted by AI (κ = 0.90 ± 0.01), which was higher than that achieved by the reference readers independently (κ = 0.84 ± 0.017; P = .01). Conclusion AI assistance increased junior readers' radiographic KOA grading performance and increased interobserver agreement for osteoarthritis grading across all readers and experience levels. Published under a CC BY 4.0 license. Supplemental material is available for this article.

Artificial intelligence-assisted double reading of chest radiographs to detect clinically relevant missed findings: a two-centre evaluation.

OBJECTIVES: To evaluate an artificial intelligence (AI)-assisted double reading system for detecting clinically relevant missed findings on routinely reported chest radiographs. METHODS: A retrospective study was performed in two institutions, a secondary care hospital and tertiary referral oncology centre. Commercially available AI software performed a comparative analysis of chest radiographs and radiologists' authorised reports using a deep learning and natural language processing algorithm, respectively. The AI-detected discrepant findings between images and reports were assessed for clinical relevance by an external radiologist, as part of the commercial service provided by the AI vendor. The selected missed findings were subsequently returned to the institution's radiologist for final review. RESULTS: In total, 25,104 chest radiographs of 21,039 patients (mean age 61.1 years ± 16.2 [SD]; 10,436 men) were included. The AI software detected discrepancies between imaging and reports in 21.1% (5289 of 25,104). After review by the external radiologist, 0.9% (47 of 5289) of cases were deemed to contain clinically relevant missed findings. The institution's radiologists confirmed 35 of 47 missed findings (74.5%) as clinically relevant (0.1% of all cases). Missed findings consisted of lung nodules (71.4%, 25 of 35), pneumothoraces (17.1%, 6 of 35) and consolidations (11.4%, 4 of 35). CONCLUSION: The AI-assisted double reading system was able to identify missed findings on chest radiographs after report authorisation. The approach required an external radiologist to review the AI-detected discrepancies. The number of clinically relevant missed findings by radiologists was very low. CLINICAL RELEVANCE STATEMENT: The AI-assisted double reader workflow was shown to detect diagnostic errors and could be applied as a quality assurance tool. Although clinically relevant missed findings were rare, there is potential impact given the common use of chest radiography. KEY POINTS: • A commercially available double reading system supported by artificial intelligence was evaluated to detect reporting errors in chest radiographs (n=25,104) from two institutions. • Clinically relevant missed findings were found in 0.1% of chest radiographs and consisted of unreported lung nodules, pneumothoraces and consolidations. • Applying AI software as a secondary reader after report authorisation can assist in reducing diagnostic errors without interrupting the radiologist's reading workflow. However, the number of AI-detected discrepancies was considerable and required review by a radiologist to assess their relevance.

ESR Essentials: radiomics-practice recommendations by the European Society of Medical Imaging Informatics.

Radiomics is a method to extract detailed information from diagnostic images that cannot be perceived by the naked eye. Although radiomics research carries great potential to improve clinical decision-making, its inherent methodological complexities make it difficult to comprehend every step of the analysis, often causing reproducibility and generalizability issues that hinder clinical adoption. Critical steps in the radiomics analysis and model development pipeline-such as image, application of image filters, and selection of feature extraction parameters-can greatly affect the values of radiomic features. Moreover, common errors in data partitioning, model comparison, fine-tuning, assessment, and calibration can reduce reproducibility and impede clinical translation. Clinical adoption of radiomics also requires a deep understanding of model explainability and the development of intuitive interpretations of radiomic features. To address these challenges, it is essential for radiomics model developers and clinicians to be well-versed in current best practices. Proper knowledge and application of these practices is crucial for accurate radiomics feature extraction, robust model development, and thorough assessment, ultimately increasing reproducibility, generalizability, and the likelihood of successful clinical translation. In this article, we have provided researchers with our recommendations along with practical examples to facilitate good research practices in radiomics. KEY POINTS: Radiomics' inherent methodological complexity should be understood to ensure rigorous radiomic model development to improve clinical decision-making. Adherence to radiomics-specific checklists and quality assessment tools ensures methodological rigor. Use of standardized radiomics tools and best practices enhances clinical translation of radiomics models.

Validation of a commercially available CAD-system for lung nodule detection and characterization using CT-scans.

OBJECTIVES: This study aims to externally validate a commercially available Computer-Aided Detection (CAD)-system for the automatic detection and characterization of solid, part-solid, and ground-glass lung nodules (LN) on CT scans. METHODS: This retrospective study encompasses 263 chest CT scans performed between January 2020 and December 2021 at a Dutch university hospital. All scans were read by a radiologist (R1) and compared with the initial radiology report. Conflicting scans were assessed by an adjudicating radiologist (R2). All scans were also processed by CAD. The standalone performance of CAD in terms of sensitivity and false-positive (FP)-rate for detection was calculated together with the sensitivity for characterization, including texture, calcification, speculation, and location. The R1's detection sensitivity was also assessed. RESULTS: A total of 183 true nodules were identified in 121 nodule-containing scans (142 non-nodule-containing scans), of which R1 identified 165/183 (90.2%). CAD detected 149 nodules, of which 12 were not identified by R1, achieving a sensitivity of 149/183 (81.4%) with an FP-rate of 49/121 (0.405). CAD's detection sensitivity for solid, part-solid, and ground-glass LNs was 82/94 (87.2%), 42/47 (89.4%), and 25/42 (59.5%), respectively. The classification accuracy for solid, part-solid, and ground-glass LNs was 81/82 (98.8%), 16/42 (38.1%), and 18/25 (72.0%), respectively. Additionally, CAD demonstrated overall classification accuracies of 137/149 (91.9%), 123/149 (82.6%), and 141/149 (94.6%) for calcification, spiculation, and location, respectively. CONCLUSIONS: Although the overall detection rate of this system slightly lags behind that of a radiologist, CAD is capable of detecting different LNs and thereby has the potential to enhance a reader's detection rate. While promising characterization performances are obtained, the tool's performance in terms of texture classification remains a subject of concern. CLINICAL RELEVANCE STATEMENT: Numerous lung nodule computer-aided detection-systems are commercially available, with some of them solely being externally validated based on their detection performance on solid nodules. We encourage researchers to assess performances by incorporating all relevant characteristics, including part-solid and ground-glass nodules. KEY POINTS: Few computer-aided detection (CAD) systems are externally validated for automatic detection and characterization of lung nodules. A detection sensitivity of 81.4% and an overall texture classification sensitivity of 77.2% were measured utilizing CAD. CAD has the potential to increase single reader detection rate, however, improvement in texture classification is required.

Artificial intelligence in musculoskeletal imaging: realistic clinical applications in the next decade.

This article will provide a perspective review of the most extensively investigated deep learning (DL) applications for musculoskeletal disease detection that have the best potential to translate into routine clinical practice over the next decade. Deep learning methods for detecting fractures, estimating pediatric bone age, calculating bone measurements such as lower extremity alignment and Cobb angle, and grading osteoarthritis on radiographs have been shown to have high diagnostic performance with many of these applications now commercially available for use in clinical practice. Many studies have also documented the feasibility of using DL methods for detecting joint pathology and characterizing bone tumors on magnetic resonance imaging (MRI). However, musculoskeletal disease detection on MRI is difficult as it requires multi-task, multi-class detection of complex abnormalities on multiple image slices with different tissue contrasts. The generalizability of DL methods for musculoskeletal disease detection on MRI is also challenging due to fluctuations in image quality caused by the wide variety of scanners and pulse sequences used in routine MRI protocols. The diagnostic performance of current DL methods for musculoskeletal disease detection must be further evaluated in well-designed prospective studies using large image datasets acquired at different institutions with different imaging parameters and imaging hardware before they can be fully implemented in clinical practice. Future studies must also investigate the true clinical benefits of current DL methods and determine whether they could enhance quality, reduce error rates, improve workflow, and decrease radiologist fatigue and burnout with all of this weighed against the costs.

Is the generalizability of a developed artificial intelligence algorithm for COVID-19 on chest CT sufficient for clinical use? Results from the International Consortium for COVID-19 Imaging AI (ICOVAI).

OBJECTIVES: Only few published artificial intelligence (AI) studies for COVID-19 imaging have been externally validated. Assessing the generalizability of developed models is essential, especially when considering clinical implementation. We report the development of the International Consortium for COVID-19 Imaging AI (ICOVAI) model and perform independent external validation. METHODS: The ICOVAI model was developed using multicenter data (n = 1286 CT scans) to quantify disease extent and assess COVID-19 likelihood using the COVID-19 Reporting and Data System (CO-RADS). A ResUNet model was modified to automatically delineate lung contours and infectious lung opacities on CT scans, after which a random forest predicted the CO-RADS score. After internal testing, the model was externally validated on a multicenter dataset (n = 400) by independent researchers. CO-RADS classification performance was calculated using linearly weighted Cohen's kappa and segmentation performance using Dice Similarity Coefficient (DSC). RESULTS: Regarding internal versus external testing, segmentation performance of lung contours was equally excellent (DSC = 0.97 vs. DSC = 0.97, p = 0.97). Lung opacities segmentation performance was adequate internally (DSC = 0.76), but significantly worse on external validation (DSC = 0.59, p < 0.0001). For CO-RADS classification, agreement with radiologists on the internal set was substantial (kappa = 0.78), but significantly lower on the external set (kappa = 0.62, p < 0.0001). CONCLUSION: In this multicenter study, a model developed for CO-RADS score prediction and quantification of COVID-19 disease extent was found to have a significant reduction in performance on independent external validation versus internal testing. The limited reproducibility of the model restricted its potential for clinical use. The study demonstrates the importance of independent external validation of AI models. KEY POINTS: • The ICOVAI model for prediction of CO-RADS and quantification of disease extent on chest CT of COVID-19 patients was developed using a large sample of multicenter data. • There was substantial performance on internal testing; however, performance was significantly reduced on external validation, performed by independent researchers. The limited generalizability of the model restricts its potential for clinical use. • Results of AI models for COVID-19 imaging on internal tests may not generalize well to external data, demonstrating the importance of independent external validation.

Automatic detection of actionable findings and communication mentions in radiology reports using natural language processing.

OBJECTIVES: To develop and validate classifiers for automatic detection of actionable findings and documentation of nonroutine communication in routinely delivered radiology reports. METHODS: Two radiologists annotated all actionable findings and communication mentions in a training set of 1,306 radiology reports and a test set of 1,000 reports randomly selected from the electronic health record system of a large tertiary hospital. Various feature sets were constructed based on the impression section of the reports using different preprocessing steps (stemming, removal of stop words, negations, and previously known or stable findings) and n-grams. Random forest classifiers were trained to detect actionable findings, and a decision-rule classifier was trained to find communication mentions. Classifier performance was evaluated by the area under the receiver operating characteristic curve (AUC), sensitivity, and specificity. RESULTS: On the training set, the actionable finding classifier with the highest cross-validated performance was obtained for a feature set of unigrams, after stemming and removal of negated, known, and stable findings. On the test set, this classifier achieved an AUC of 0.876 (95% CI 0.854-0.898). The classifier for communication detection was trained after negation removal, using unigrams as features. The resultant decision rule had a sensitivity of 0.841 (95% CI 0.706-0.921) and specificity of 0.990 (95% CI 0.981-0.994) on the test set. CONCLUSIONS: Automatic detection of actionable findings and subsequent communication in routinely delivered radiology reports is possible. This can serve quality control purposes and may alert radiologists to the presence of actionable findings during reporting. KEY POINTS: • Classifiers were developed for automatic detection of the broad spectrum of actionable findings and subsequent communication mentions in routinely delivered radiology reports. • Straightforward report preprocessing and simple feature sets can produce well-performing classifiers. • The resultant classifiers show good performance for detection of actionable findings and excellent performance for detection of communication mentions.

Comparing two artificial intelligence software packages for normative brain volumetry in memory clinic imaging.

PURPOSE: To compare two artificial intelligence software packages performing normative brain volumetry and explore whether they could differently impact dementia diagnostics in a clinical context. METHODS: Sixty patients (20 Alzheimer's disease, 20 frontotemporal dementia, 20 mild cognitive impairment) and 20 controls were included retrospectively. One MRI per subject was processed by software packages from two proprietary manufacturers, producing two quantitative reports per subject. Two neuroradiologists assigned forced-choice diagnoses using only the normative volumetry data in these reports. They classified the volumetric profile as "normal," or "abnormal", and if "abnormal," they specified the most likely dementia subtype. Differences between the packages' clinical impact were assessed by comparing (1) agreement between diagnoses based on software output; (2) diagnostic accuracy, sensitivity, and specificity; and (3) diagnostic confidence. Quantitative outputs were also compared to provide context to any diagnostic differences. RESULTS: Diagnostic agreement between packages was moderate, for distinguishing normal and abnormal volumetry (K = .41-.43) and for specific diagnoses (K = .36-.38). However, each package yielded high inter-observer agreement when distinguishing normal and abnormal profiles (K = .73-.82). Accuracy, sensitivity, and specificity were not different between packages. Diagnostic confidence was different between packages for one rater. Whole brain intracranial volume output differed between software packages (10.73%, p < .001), and normative regional data interpreted for diagnosis correlated weakly to moderately (rs = .12-.80). CONCLUSION: Different artificial intelligence software packages for quantitative normative assessment of brain MRI can produce distinct effects at the level of clinical interpretation. Clinics should not assume that different packages are interchangeable, thus recommending internal evaluation of packages before adoption.

Differential Diagnosis and Molecular Stratification of Gastrointestinal Stromal Tumors on CT Images Using a Radiomics Approach.

Treatment planning of gastrointestinal stromal tumors (GISTs) includes distinguishing GISTs from other intra-abdominal tumors and GISTs' molecular analysis. The aim of this study was to evaluate radiomics for distinguishing GISTs from other intra-abdominal tumors, and in GISTs, predict the c-KIT, PDGFRA, BRAF mutational status, and mitotic index (MI). Patients diagnosed at the Erasmus MC between 2004 and 2017, with GIST or non-GIST intra-abdominal tumors and a contrast-enhanced venous-phase CT, were retrospectively included. Tumors were segmented, from which 564 image features were extracted. Prediction models were constructed using a combination of machine learning approaches. The evaluation was performed in a 100 × random-split cross-validation. Model performance was compared to that of three radiologists. One hundred twenty-five GISTs and 122 non-GISTs were included. The GIST vs. non-GIST radiomics model had a mean area under the curve (AUC) of 0.77. Three radiologists had an AUC of 0.69, 0.76, and 0.84, respectively. The radiomics model had an AUC of 0.52 for c-KIT, 0.56 for c-KIT exon 11, and 0.52 for the MI. The numbers of PDGFRA, BRAF, and other c-KIT mutations were too low for analysis. Our radiomics model was able to distinguish GISTs from non-GISTs with a performance similar to three radiologists, but less observer dependent. Therefore, it may aid in the early diagnosis of GIST, facilitating rapid referral to specialized treatment centers. As the model was not able to predict any genetic or molecular features, it cannot aid in treatment planning yet.

Radiology in the Era of Value-based Healthcare: A Multi-Society Expert Statement from the ACR, CAR, ESR, IS3R, RANZCR, and RSNA.

Background The Value-Based Healthcare (VBH) concept is designed to improve individual healthcare outcomes without increasing expenditure, and is increasingly being used to determine resourcing of and reimbursement for medical services. Radiology is a major contributor to patient and societal healthcare at many levels. Despite this, some VBH models do not acknowledge radiology's central role; this may have future negative consequences for resource allocation. Methods, findings and interpretation This multi-society paper, representing the views of Radiology Societies in Europe, the USA, Canada, Australia, and New Zealand, describes the place of radiology in VBH models and the health-care value contributions of radiology. Potential steps to objectify and quantify the value contributed by radiology to healthcare are outlined. Published under a CC BY 4.0 license.

Radiology in the Era of Value-Based Healthcare: A Multi-Society Expert Statement From the ACR, CAR, ESR, IS3R, RANZCR, and RSNA.

BACKGROUND: The Value-Based Healthcare (VBH) concept is designed to improve individual healthcare outcomes without increasing expenditure, and is increasingly being used to determine resourcing of and reimbursement for medical services. Radiology is a major contributor to patient and societal healthcare at many levels. Despite this, some VBH models do not acknowledge radiology's central role; this may have future negative consequences for resource allocation. METHODS, FINDINGS AND INTERPRETATION: This multi-society paper, representing the views of Radiology Societies in Europe, the USA, Canada, Australia, and New Zealand, describes the place of radiology in VBH models and the health-care value contributions of radiology. Potential steps to objectify and quantify the value contributed by radiology to healthcare are outlined.

The Value of Quantitative Musculoskeletal Imaging.

Musculoskeletal imaging is mainly based on the subjective and qualitative analysis of imaging examinations. However, integration of quantitative assessment of imaging data could increase the value of imaging in both research and clinical practice. Some imaging modalities, such as perfusion magnetic resonance imaging (MRI), diffusion MRI, or T2 mapping, are intrinsically quantitative. But conventional morphological imaging can also be analyzed through the quantification of various parameters. The quantitative data retrieved from imaging examinations can serve as biomarkers and be used to support diagnosis, determine patient prognosis, or monitor therapy.We focus on the value, or clinical utility, of quantitative imaging in the musculoskeletal field. There is currently a trend to move from volume- to value-based payments. This review contains definitions and examines the role that quantitative imaging may play in the implementation of value-based health care. The influence of artificial intelligence on the value of quantitative musculoskeletal imaging is also discussed.

The Need for Sustainable Teleconsultation Systems in the Aftermath of the First COVID-19 Wave.

The physical and social distancing measures that have been adopted worldwide because of COVID-19 will probably remain in place for a long time, especially for senior adults, people with chronic conditions, and other at-risk populations. Teleconsultations can be useful in ensuring that patients continue to receive clinical care while reducing physical crowding and avoiding unnecessary exposure of health care staff. Implementation processes that typically take months of planning, budgeting, pilot testing, and education were compressed into days. However, in the urgency to deal with the present crisis, we may be forgetting that the introduction of digital health is not exclusively a technological issue, but part of a complex organizational change problem. This viewpoint offers insight regarding issues that rapidly adopted teleconsultation systems may face in a post-COVID-19 world.

Using Cost-Effectiveness Analysis to Measure Value in Musculoskeletal Imaging.

In the era of value-based health care, adding value is a key element in providing care. The choice of appropriate imaging modality and protocol should be based on consideration of patients' values, health care outcomes, and cost-effectiveness, taking into account the perspective of the decision maker, the health care system, and society at large. This article provides an overview of the available tools to measure value, outcomes, and cost-effectiveness in musculoskeletal radiology, illustrated with relevant examples.

Talus-derived reference coordinate system for 3D calcaneal assessment: A novel approach to improve morphological measurements.

In 3D-analysis of the calcaneus, a consistent coordinate system aligned with the original anatomical directions is crucial for pre- and postoperative analysis. This importance stems from the calcaneus's key role in weight-bearing and biomechanical alignment. However, defining a reliable coordinate system based solely on fractured or surgically reconstructed calcanei presents significant challenges. Given its anatomical prominence and consistent orientation, the talus offers a potential solution to this challenge. Our work explores the feasibility of talus-derived coordinate systems for 3D-modeling of the calcaneus across its various conditions. Four methods were tested on nonfractured, fractured and surgically reconstructed calcanei, utilizing Principal Component Analysis, anatomical landmarks, bounding box, and an atlas-based approach. The methods were compared with a self-defined calcaneus reference coordinate system. Additionally, the impact of deviation of the coordinate system on morphological measurements was investigated. Among methods for constructing nonfractured calcanei coordinate systems, the atlas-based method displayed the lowest Root Mean Square value in comparison with the reference coordinate system. For morphological measures like Böhler's Angle and the Critical angle of Gissane, the atlas talus-based system closely aligned with ground truth, yielding differences of 0.6° and 1.2°, respectively, compared to larger deviations seen in other talus-based coordinate systems. In conclusion, all tested methods were feasible for creating a talus derived coordinate system. A talus derived coordinate system showed potential, offering benefits for morphological measurements and clinical scenarios involving fractured and surgically reconstructed calcanei. Further research is recommended to assess the impact of these coordinate systems on surgical planning and outcomes.

Impact of AI on radiology: a EuroAIM/EuSoMII 2024 survey among members of the European Society of Radiology.

In order to assess the perceptions and expectations of the radiology staff about artificial intelligence (AI), we conducted an online survey among ESR members (January-March 2024). It was designed considering that conducted in 2018, updated according to recent advancements and emerging topics, consisting of seven questions regarding demographics and professional background and 28 AI questions. Of 28,000 members contacted, 572 (2%) completed the survey. AI impact was predominantly expected on breast and oncologic imaging, primarily involving CT, mammography, and MRI, and in the detection of abnormalities in asymptomatic subjects. About half of responders did not foresee an impact of AI on job opportunities. For 273/572 respondents (48%), AI-only reports would not be accepted by patients; and 242/572 respondents (42%) think that the use of AI systems will not change the relationship between the radiological team and the patient. According to 255/572 respondents (45%), radiologists will take responsibility for any AI output that may influence clinical decision-making. Of 572 respondents, 274 (48%) are currently using AI, 153 (27%) are not, and 145 (25%) are planning to do so. In conclusion, ESR members declare familiarity with AI technologies, as well as recognition of their potential benefits and challenges. Compared to the 2018 survey, the perception of AI's impact on job opportunities is in general slightly less optimistic (more positive from AI users/researchers), while the radiologist's responsibility for AI outputs is confirmed. The use of large language models is declared not only limited to research, highlighting the need for education in AI and its regulations. CRITICAL RELEVANCE STATEMENT: This study critically evaluates the current impact of AI on radiology, revealing significant usage patterns and clinical implications, thereby guiding future integration strategies to enhance efficiency and patient care in clinical radiology. KEY POINTS: The survey examines ESR member's views about the impact of AI on radiology practice. AI use is relevant in CT and MRI, with varying impacts on job roles. AI tools enhance clinical efficiency but require radiologist oversight for patient acceptance.

Towards clinical implementation of an AI-algorithm for detection of cervical spine fractures on computed tomography.

BACKGROUND: Artificial intelligence (AI) applications can facilitate detection of cervical spine fractures on CT and reduce time to diagnosis by prioritizing suspected cases. PURPOSE: To assess the effect on time to diagnose cervical spine fractures on CT and diagnostic accuracy of a commercially available AI application. MATERIALS AND METHODS: In this study (June 2020 - March 2022) with historic controls and prospective evaluation, we evaluated regulatory-cleared AI-software to prioritize cervical spine fractures on CT. All patients underwent non-contrast CT of the cervical spine. The time between CT acquisition and the moment the scan was first opened (DNT) was compared between the retrospective and prospective cohorts. The reference standard for determining diagnostic accuracy was the radiology report created in routine clinical workflow and adjusted by a senior radiologist. Discrepant cases were reviewed and clinical relevance of missed fractures was determined. RESULTS: 2973 (mean age, 55.4 ± 19.7 [standard deviation]; 1857 men) patients were analyzed by AI, including 2036 retrospective and 938 prospective cases. Overall prevalence of cervical spine fractures was 7.6 %. The DNT was 18 % (5 min) shorter in the prospective cohort. In scans positive for cervical spine fracture according to the reference standard, DNT was 46 % (16 min) shorter in the prospective cohort. Overall sensitivity of the AI application was 89.8 % (95 % CI: 84.2-94.0 %), specificity was 95.3 % (95 % CI: 94.2-96.2 %), and diagnostic accuracy was 94.8 % (95 % CI: 93.8-95.8 %). Negative predictive value was 99.1 % (95 % CI: 98.5-99.4 %) and positive predictive value was 63.0 % (95 % CI: 58.0-67.8 %). 22 fractures were missed by AI of which 5 required stabilizing therapy. CONCLUSION: A time gain of 16 min to diagnosis for fractured cases was observed after introducing AI. Although AI-assisted workflow prioritization of cervical spine fractures on CT shows high diagnostic accuracy, clinically relevant cases were missed.

Radiology AI Deployment and Assessment Rubric (RADAR) to bring value-based AI into radiological practice.

OBJECTIVE: To provide a comprehensive framework for value assessment of artificial intelligence (AI) in radiology. METHODS: This paper presents the RADAR framework, which has been adapted from Fryback and Thornbury's imaging efficacy framework to facilitate the valuation of radiology AI from conception to local implementation. Local efficacy has been newly introduced to underscore the importance of appraising an AI technology within its local environment. Furthermore, the RADAR framework is illustrated through a myriad of study designs that help assess value. RESULTS: RADAR presents a seven-level hierarchy, providing radiologists, researchers, and policymakers with a structured approach to the comprehensive assessment of value in radiology AI. RADAR is designed to be dynamic and meet the different valuation needs throughout the AI's lifecycle. Initial phases like technical and diagnostic efficacy (RADAR-1 and RADAR-2) are assessed pre-clinical deployment via in silico clinical trials and cross-sectional studies. Subsequent stages, spanning from diagnostic thinking to patient outcome efficacy (RADAR-3 to RADAR-5), require clinical integration and are explored via randomized controlled trials and cohort studies. Cost-effectiveness efficacy (RADAR-6) takes a societal perspective on financial feasibility, addressed via health-economic evaluations. The final level, RADAR-7, determines how prior valuations translate locally, evaluated through budget impact analysis, multi-criteria decision analyses, and prospective monitoring. CONCLUSION: The RADAR framework offers a comprehensive framework for valuing radiology AI. Its layered, hierarchical structure, combined with a focus on local relevance, aligns RADAR seamlessly with the principles of value-based radiology. CRITICAL RELEVANCE STATEMENT: The RADAR framework advances artificial intelligence in radiology by delineating a much-needed framework for comprehensive valuation. KEYPOINTS: • Radiology artificial intelligence lacks a comprehensive approach to value assessment. • The RADAR framework provides a dynamic, hierarchical method for thorough valuation of radiology AI. • RADAR advances clinical radiology by bridging the artificial intelligence implementation gap.