Accuracy of an artificial intelligence-enabled diagnostic assistance device in recognizing normal chest radiographs: a service evaluation.

Objectives: Artificial intelligence (AI) enabled devices may be able to optimize radiologists' productivity by identifying normal and abnormal chest X-rays (CXRs) for triaging. In this service evaluation, we investigated the accuracy of one such AI device (qXR). Methods: A randomly sampled subset of general practice and outpatient-referred frontal CXRs from a National Health Service Trust was collected retrospectively from examinations conducted during November 2022 to January 2023. Ground truth was established by consensus between 2 radiologists. The main objective was to estimate negative predictive value (NPV) of AI. Results: A total of 522 CXRs (458 [87.74%] normal CXRs) from 522 patients (median age, 64 years [IQR, 49-77]; 305 [58.43%] female) were analysed. AI predicted 348 CXRs as normal, of which 346 were truly normal (NPV: 99.43% [95% CI, 97.94-99.93]). The sensitivity, specificity, positive predictive value, and area under the ROC curve of AI were found to be 96.88% (95% CI, 89.16-99.62), 75.55% (95% CI, 71.34-79.42), 35.63% (95% CI, 28.53-43.23), and 91.92% (95% CI, 89.38-94.45), respectively. A sensitivity analysis was conducted to estimate NPV by varying assumptions of the prevalence of normal CXRs. The NPV ranged from 88.96% to 99.54% as prevalence increased. Conclusions: The AI device recognized normal CXRs with high NPV and has the potential to increase radiologists' productivity. Advances in knowledge: There is a need for more evidence on the utility of AI-enabled devices in identifying normal CXRs. This work adds to such limited evidence and enables researchers to plan studies to further evaluate the impact of such devices.

Assessing the effectiveness of artificial intelligence (AI) in prioritising CT head interpretation: study protocol for a stepped-wedge cluster randomised trial (ACCEPT-AI).

INTRODUCTION: Diagnostic imaging is vital in emergency departments (EDs). Accessibility and reporting impacts ED workflow and patient care. With radiology workforce shortages, reporting capacity is limited, leading to image interpretation delays. Turnaround times for image reporting are an ED bottleneck. Artificial intelligence (AI) algorithms can improve productivity, efficiency and accuracy in diagnostic radiology, contingent on their clinical efficacy. This includes positively impacting patient care and improving clinical workflow. The ACCEPT-AI study will evaluate Qure.ai's qER software in identifying and prioritising patients with critical findings from AI analysis of non-contrast head CT (NCCT) scans. METHODS AND ANALYSIS: This is a multicentre trial, spanning four diverse sites, over 13 months. It will include all individuals above the age of 18 years who present to the ED, referred for an NCCT. The project will be divided into three consecutive phases (pre-implementation, implementation and post-implementation of the qER solution) in a stepped-wedge design to control for adoption bias and adjust for time-based changes in the background patient characteristics. Pre-implementation involves baseline data for standard care to support the primary and secondary outcomes. The implementation phase includes staff training and qER solution threshold adjustments in detecting target abnormalities adjusted, if necessary. The post-implementation phase will introduce a notification (prioritised flag) in the radiology information system. The radiologist can choose to agree with the qER findings or ignore it according to their clinical judgement before writing and signing off the report. Non-qER processed scans will be handled as per standard care. ETHICS AND DISSEMINATION: The study will be conducted in accordance with the principles of Good Clinical Practice. The protocol was approved by the Research Ethics Committee of East Midlands (Leicester Central), in May 2023 (REC (Research Ethics Committee) 23/EM/0108). Results will be published in peer-reviewed journals and disseminated in scientific findings (ClinicalTrials.gov: NCT06027411) TRIAL REGISTRATION NUMBER: NCT06027411.

Radiograph accelerated detection and identification of cancer in the lung (RADICAL): a mixed methods study to assess the clinical effectiveness and acceptability of Qure.ai artificial intelligence software to prioritise chest X-ray (CXR) interpretation.

INTRODUCTION: Diagnosing and treating lung cancer in early stages is essential for survival outcomes. The chest X-ray (CXR) remains the primary screening tool to identify lung cancers in the UK; however, there is a shortfall of radiologists, while demand continues to increase. Image analysis by machine-learning software has the potential to support radiology workflows with a focus on immediate triage of suspicious X-rays. The RADICAL study will evaluate Qure.ai's 'qXR' software in reducing reporting time for suspicious X-rays in NHS Greater Glasgow & Clyde. METHODS AND ANALYSIS: This is a stepped-wedge cluster-randomised study consisting of a retrospective technical evaluation and prospective clinical effectiveness study alongside the assessment of acceptability via qualitative work and evaluation of cost-effectiveness via a cost utility analysis. The primary objective is to assess the clinical effectiveness of qXR to prioritise patients suspected with lung cancer on CXR for follow-up CT. Secondary objectives will look at the utility, safety, technical performance, health economics and acceptability of the intervention. The study period is 24 months, consisting of an initial 12 month data collection period and a 12 month follow-up period. All the standard care CXRs from outpatient and primary care requests will be securely transmitted to Qure.ai software 'qXR' for interpretation. Images with features of cancer will be flagged as 'Urgent Suspicion of Cancer' and be prioritised for radiologist review within the existing reporting workflow. ETHICS AND DISSEMINATION: The study will follow the principles of Good Clinical Practice. The protocol was granted REC approval in August 2023 from North West-Greater Manchester West Research Ethics Committee (REC 23/NW/0211). This study was registered on clinicaltrials.gov (NCT06044454). An interim report will be produced for use by the Scottish Government. The results from this study will be presented at artificial intelligence, radiology and respiratory meetings and published in peer-reviewed journals. TRIAL REGISTRATION NUMBER: NCT06044454.

AI assisted reader evaluation in acute CT head interpretation (AI-REACT): protocol for a multireader multicase study.

INTRODUCTION: A non-contrast CT head scan (NCCTH) is the most common cross-sectional imaging investigation requested in the emergency department. Advances in computer vision have led to development of several artificial intelligence (AI) tools to detect abnormalities on NCCTH. These tools are intended to provide clinical decision support for clinicians, rather than stand-alone diagnostic devices. However, validation studies mostly compare AI performance against radiologists, and there is relative paucity of evidence on the impact of AI assistance on other healthcare staff who review NCCTH in their daily clinical practice. METHODS AND ANALYSIS: A retrospective data set of 150 NCCTH will be compiled, to include 60 control cases and 90 cases with intracranial haemorrhage, hypodensities suggestive of infarct, midline shift, mass effect or skull fracture. The intracranial haemorrhage cases will be subclassified into extradural, subdural, subarachnoid, intraparenchymal and intraventricular. 30 readers will be recruited across four National Health Service (NHS) trusts including 10 general radiologists, 15 emergency medicine clinicians and 5 CT radiographers of varying experience. Readers will interpret each scan first without, then with, the assistance of the qER EU 2.0 AI tool, with an intervening 2-week washout period. Using a panel of neuroradiologists as ground truth, the stand-alone performance of qER will be assessed, and its impact on the readers' performance will be analysed as change in accuracy (area under the curve), median review time per scan and self-reported diagnostic confidence. Subgroup analyses will be performed by reader professional group, reader seniority, pathological finding, and neuroradiologist-rated difficulty. ETHICS AND DISSEMINATION: The study has been approved by the UK Healthcare Research Authority (IRAS 310995, approved 13 December 2022). The use of anonymised retrospective NCCTH has been authorised by Oxford University Hospitals. The results will be presented at relevant conferences and published in a peer-reviewed journal. TRIAL REGISTRATION NUMBER: NCT06018545.