Effects of a comprehensive brain computed tomography deep learning model on radiologist detection accuracy.

OBJECTIVES: Non-contrast computed tomography of the brain (NCCTB) is commonly used to detect intracranial pathology but is subject to interpretation errors. Machine learning can augment clinical decision-making and improve NCCTB scan interpretation. This retrospective detection accuracy study assessed the performance of radiologists assisted by a deep learning model and compared the standalone performance of the model with that of unassisted radiologists. METHODS: A deep learning model was trained on 212,484 NCCTB scans drawn from a private radiology group in Australia. Scans from inpatient, outpatient, and emergency settings were included. Scan inclusion criteria were age ≥ 18 years and series slice thickness ≤ 1.5 mm. Thirty-two radiologists reviewed 2848 scans with and without the assistance of the deep learning system and rated their confidence in the presence of each finding using a 7-point scale. Differences in AUC and Matthews correlation coefficient (MCC) were calculated using a ground-truth gold standard. RESULTS: The model demonstrated an average area under the receiver operating characteristic curve (AUC) of 0.93 across 144 NCCTB findings and significantly improved radiologist interpretation performance. Assisted and unassisted radiologists demonstrated an average AUC of 0.79 and 0.73 across 22 grouped parent findings and 0.72 and 0.68 across 189 child findings, respectively. When assisted by the model, radiologist AUC was significantly improved for 91 findings (158 findings were non-inferior), and reading time was significantly reduced. CONCLUSIONS: The assistance of a comprehensive deep learning model significantly improved radiologist detection accuracy across a wide range of clinical findings and demonstrated the potential to improve NCCTB interpretation. CLINICAL RELEVANCE STATEMENT: This study evaluated a comprehensive CT brain deep learning model, which performed strongly, improved the performance of radiologists, and reduced interpretation time. The model may reduce errors, improve efficiency, facilitate triage, and better enable the delivery of timely patient care. KEY POINTS: • This study demonstrated that the use of a comprehensive deep learning system assisted radiologists in the detection of a wide range of abnormalities on non-contrast brain computed tomography scans. • The deep learning model demonstrated an average area under the receiver operating characteristic curve of 0.93 across 144 findings and significantly improved radiologist interpretation performance. • The assistance of the comprehensive deep learning model significantly reduced the time required for radiologists to interpret computed tomography scans of the brain.

Pathophysiological Mechanisms of Gastro-esophageal Reflux After Sleeve Gastrectomy.

OBJECTIVE: To evaluate the mechanisms associated with reflux events after sleeve gastrectomy (SG). SUMMARY BACKGROUND DATA: Gastro-esophageal reflux (GERD) post-SG is a critical issue due to symptom severity, impact on quality of life, requirement for reoperation, and potential for Barrett esophagus. The pathophysiology is incompletely delineated. METHODS: Post-SG patients, stratified into asymptomatic and symptomatic, underwent protocolized nuclear scintigraphy (n = 83), 24-hour esophageal pH monitoring, and stationary manometry (n = 143) to characterize reflux patterns. Ten patients underwent fasting and postprandial concurrent manometry and pH for detailed analysis of reflux events. RESULTS: Baseline demographics between cohorts were similar: Age 47.2 ± 11.6 versus 44.1 ± 11.3 years ( P = 0.121); females 73.2% versus 90.8% ( P = 0.005); excess weight loss 53.8 ± 28.1% versus 57.4 ± 25.5% ( P = 0.422), follow-up duration 12.3 versus 7.4 months ( P = 0.503). Nuclear scintigraphy delineated bolus-induced deglutitive reflux events (29.6% vs 62.5%, P = 0.005) and postprandial reflux events [4 (IQR2) versus 4 (IQR 3) events, P = 0.356]. Total acid exposure was significantly elevated in the symptomatic population (7.7% vs 3.6%, P < 0.001), especially fasting acid exposure (6.0% vs 1.3%, P < 0.001). pH/manometry analysis demonstrated acute elevations of the gastro-esophageal pressure gradient (>10 mm Hg) underpinned most reflux events. Swallow-induced intragastric hyper-pressur-ization was associated with individual reflux events in most patients (90% in fasting state and 40% postprandial). CONCLUSIONS: We found reflux to be strongly associated with SG and identified 3 unique categories. Bolus-induced deglutitive and postprandial reflux occurred in most patients. Elevated fasting esophageal acid exposure mediated symptoms. Frequent, significant elevation in the gastro-esophageal pressure gradient was the mechanism of reflux and seemed to relate to the noncompliant proximal stomach.

Machine learning approaches for imaging-based prognostication of the outcome of surgery for mesial temporal lobe epilepsy.

OBJECTIVES: Around 30% of patients undergoing surgical resection for drug-resistant mesial temporal lobe epilepsy (MTLE) do not obtain seizure freedom. Success of anterior temporal lobe resection (ATLR) critically depends on the careful selection of surgical candidates, aiming at optimizing seizure freedom while minimizing postoperative morbidity. Structural MRI and FDG-PET neuroimaging are routinely used in presurgical assessment and guide the decision to proceed to surgery. In this study, we evaluate the potential of machine learning techniques applied to standard presurgical MRI and PET imaging features to provide enhanced prognostic value relative to current practice. METHODS: Eighty two patients with drug resistant MTLE were scanned with FDG-PET pre-surgery and T1-weighted MRI pre- and postsurgery. From these images the following features of interest were derived: volume of temporal lobe (TL) hypometabolism, % of extratemporal hypometabolism, presence of contralateral TL hypometabolism, presence of hippocampal sclerosis, laterality of seizure onset volume of tissue resected and % of temporal lobe hypometabolism resected. These measures were used as predictor variables in logistic regression, support vector machines, random forests and artificial neural networks. RESULTS: In the study cohort, 24 of 82 (28.3%) who underwent an ATLR for drug-resistant MTLE did not achieve Engel Class I (i.e., free of disabling seizures) outcome at a minimum of 2 years of postoperative follow-up. We found that machine learning approaches were able to predict up to 73% of the 24 ATLR surgical patients who did not achieve a Class I outcome, at the expense of incorrect prediction for up to 31% of patients who did achieve a Class I outcome. Overall accuracies ranged from 70% to 80%, with an area under the receiver operating characteristic curve (AUC) of .75-.81. We additionally found that information regarding overall extent of both total and significantly hypometabolic tissue resected was crucial to predictive performance, with AUC dropping to .59-.62 using presurgical information alone. Incorporating the laterality of seizure onset and the choice of machine learning algorithm did not significantly change predictive performance. SIGNIFICANCE: Collectively, these results indicate that "acceptable" to "good" patient-specific prognostication for drug-resistant MTLE surgery is feasible with machine learning approaches utilizing commonly collected imaging modalities, but that information on the surgical resection region is critical for optimal prognostication.

Delayed intracranial hemorrhage after trauma.

Introduction: Delayed Intracranial Hemorrhage (D-ICH), defined as finding of ICH on subsequent imaging after a normal computed tomography of the brain (CTB), is a feared complication after head trauma. The aim of this study was to determine the incidence and severity of D-ICH.Methods: This retrospective cohort study included patients that presented directly from the scene of injury to an adult major trauma center from Jan 2013 to Dec 2018.Results: There were 6536 patients who had an initial normal CTB and 23 (0.3%; 95%CI: 0.20-0.47) had D-ICH. There were 653 patients who had a repeat CTB (incidence of D-ICH 3.5%; 95%CI: 2.2-5.2). There was no significant association of D-ICH with age>65 years (OR 1.33; 95%CI: 0.54-3.29), presenting GCS <15 (OR 1.21; 95% CI: 0.52-2.80) and anti-platelet medications (OR 0.68; 95%CI: 0.26-1.74). Exposure to anti-coagulant medications was associated with lower odds of D-ICH (OR 0.23; 95%CI: 0.05-0.99). All cases of D-ICH were diffuse injury type II lesions on the Marshall classification. There were no cases that underwent neurosurgical intervention and no deaths were attributed to D-ICH.Conclusions: These results question observation of patients with head injury in hospital after a normal CTB for the sole purpose of excluding D-ICH.

Missed opportunities for HIV testing persist despite a single educational intervention: how can we close this evidence-practice gap?

BACKGROUND: In Australia, one-third of human immunodeficiency virus (HIV) diagnoses occur late, with an estimated 11% of people with HIV unaware of their diagnosis. Undiagnosed and untreated HIV infection increases morbidity in the HIV positive person and allows onward transmission of HIV. AIM: To determine the rate of HIV testing in acute general medicine patients with HIV indicator conditions (IC) and evaluate the effectiveness of an educational intervention in improving testing rates. METHODS: Single-centre, tertiary hospital, before-after study of general medicine inpatients with IC for 12 weeks prior and 10 weeks post an educational intervention focusing on recommendations for HIV testing including IC. The REASON Cohort Discovery Tool was used to search for the IC using ICD-10 codes and laboratory data. The presence of IC was estimated, and HIV testing rates before and after the intervention were compared. Regression analysis was utilised to identify characteristics associated with HIV testing. RESULTS: Of 1414 admissions in the baseline period and 946 in the post-period, 161 (11.4%) and 132 (14.0%) had at least one IC present respectively. There were 18 (11.2%) HIV tests performed for admissions with IC in the pre-period which increased to 27 (20.5%) (P = 0.028) in the post-period. Younger patients were more likely to be tested and regression analysis identified the educational intervention (adjusted odds ratio) 2.2 (1.1, 4.4) to be significantly associated with testing. CONCLUSIONS: Although HIV testing rates for IC doubled following the intervention, they remained unacceptably low. The recently introduced electronic medical record presents opportunities to prompt HIV testing.

Chest Radiographs in Congestive Heart Failure: Visualizing Neural Network Learning.

Purpose To examine Generative Visual Rationales (GVRs) as a tool for visualizing neural network learning of chest radiograph features in congestive heart failure (CHF). Materials and Methods A total of 103 489 frontal chest radiographs in 46 712 patients acquired from January 1, 2007, to December 31, 2016, were divided into a labeled data set (with B-type natriuretic peptide [BNP] result as a marker of CHF) and unlabeled data set (without BNP result). A generative model was trained on the unlabeled data set, and a neural network was trained on the encoded representations of the labeled data set to estimate BNP. The model was used to visualize how a radiograph with high estimated BNP would look without disease (a "healthy" radiograph). An overfitted model was developed for comparison, and 100 GVRs were blindly assessed by two experts for features of CHF. Area under the receiver operating characteristic curve (AUC), κ coefficient, and mixed-effects logistic regression were used for statistical analyses. Results At a cutoff BNP of 100 ng/L as a marker of CHF, the correctly trained model achieved an AUC of 0.82. Assessment of GVRs revealed that the correctly trained model highlighted conventional radiographic features of CHF as reasons for an elevated BNP prediction more frequently than the overfitted model, including cardiomegaly (153 [76.5%] of 200 vs 64 [32%] of 200, respectively; P < .001) and pleural effusions (47 [23.5%] of 200 vs 16 [8%] of 200, respectively; P = .003). Conclusion Features of congestive heart failure on chest radiographs learned by neural networks can be identified using Generative Visual Rationales, enabling detection of bias and overfitted models. © RSNA, 2018 See also the editorial by Ngo in this issue.

Generative Large Language Models for Detection of Speech Recognition Errors in Radiology Reports.

This study evaluated the ability of generative large language models (LLMs) to detect speech recognition errors in radiology reports. A dataset of 3233 CT and MRI reports was assessed by radiologists for speech recognition errors. Errors were categorized as clinically significant or not clinically significant. Performances of five generative LLMs-GPT-3.5-turbo, GPT-4, text-davinci-003, Llama-v2-70B-chat, and Bard-were compared in detecting these errors, using manual error detection as the reference standard. Prompt engineering was used to optimize model performance. GPT-4 demonstrated high accuracy in detecting clinically significant errors (precision, 76.9%; recall, 100%; F1 score, 86.9%) and not clinically significant errors (precision, 93.9%; recall, 94.7%; F1 score, 94.3%). Text-davinci-003 achieved F1 scores of 72% and 46.6% for clinically significant and not clinically significant errors, respectively. GPT-3.5-turbo obtained 59.1% and 32.2% F1 scores, while Llama-v2-70B-chat scored 72.8% and 47.7%. Bard showed the lowest accuracy, with F1 scores of 47.5% and 20.9%. GPT-4 effectively identified challenging errors of nonsense phrases and internally inconsistent statements. Longer reports, resident dictation, and overnight shifts were associated with higher error rates. In conclusion, advanced generative LLMs show potential for automatic detection of speech recognition errors in radiology reports. Keywords: CT, Large Language Model, Machine Learning, MRI, Natural Language Processing, Radiology Reports, Speech, Unsupervised Learning Supplemental material is available for this article.

Machine Learning Augmented Interpretation of Chest X-rays: A Systematic Review.

Limitations of the chest X-ray (CXR) have resulted in attempts to create machine learning systems to assist clinicians and improve interpretation accuracy. An understanding of the capabilities and limitations of modern machine learning systems is necessary for clinicians as these tools begin to permeate practice. This systematic review aimed to provide an overview of machine learning applications designed to facilitate CXR interpretation. A systematic search strategy was executed to identify research into machine learning algorithms capable of detecting >2 radiographic findings on CXRs published between January 2020 and September 2022. Model details and study characteristics, including risk of bias and quality, were summarized. Initially, 2248 articles were retrieved, with 46 included in the final review. Published models demonstrated strong standalone performance and were typically as accurate, or more accurate, than radiologists or non-radiologist clinicians. Multiple studies demonstrated an improvement in the clinical finding classification performance of clinicians when models acted as a diagnostic assistance device. Device performance was compared with that of clinicians in 30% of studies, while effects on clinical perception and diagnosis were evaluated in 19%. Only one study was prospectively run. On average, 128,662 images were used to train and validate models. Most classified less than eight clinical findings, while the three most comprehensive models classified 54, 72, and 124 findings. This review suggests that machine learning devices designed to facilitate CXR interpretation perform strongly, improve the detection performance of clinicians, and improve the efficiency of radiology workflow. Several limitations were identified, and clinician involvement and expertise will be key to driving the safe implementation of quality CXR machine learning systems.

Analysis of Line and Tube Detection Performance of a Chest X-ray Deep Learning Model to Evaluate Hidden Stratification.

This retrospective case-control study evaluated the diagnostic performance of a commercially available chest radiography deep convolutional neural network (DCNN) in identifying the presence and position of central venous catheters, enteric tubes, and endotracheal tubes, in addition to a subgroup analysis of different types of lines/tubes. A held-out test dataset of 2568 studies was sourced from community radiology clinics and hospitals in Australia and the USA, and was then ground-truth labelled for the presence, position, and type of line or tube from the consensus of a thoracic specialist radiologist and an intensive care clinician. DCNN model performance for identifying and assessing the positioning of central venous catheters, enteric tubes, and endotracheal tubes over the entire dataset, as well as within each subgroup, was evaluated. The area under the receiver operating characteristic curve (AUC) was assessed. The DCNN algorithm displayed high performance in detecting the presence of lines and tubes in the test dataset with AUCs > 0.99, and good position classification performance over a subpopulation of ground truth positive cases with AUCs of 0.86-0.91. The subgroup analysis showed that model performance was robust across the various subtypes of lines or tubes, although position classification performance of peripherally inserted central catheters was relatively lower. Our findings indicated that the DCNN algorithm performed well in the detection and position classification of lines and tubes, supporting its use as an assistant for clinicians. Further work is required to evaluate performance in rarer scenarios, as well as in less common subgroups.

Prime Time for Artificial Intelligence in Interventional Radiology.

Machine learning techniques, also known as artificial intelligence (AI), is about to dramatically change workflow and diagnostic capabilities in diagnostic radiology. The interest in AI in Interventional Radiology is rapidly gathering pace. With this early interest in AI in procedural medicine, IR could lead the way to AI research and clinical applications for all interventional medical fields. This review will address an overview of machine learning, radiomics and AI in the field of interventional radiology, enumerating the possible applications of such techniques, while also describing techniques to overcome the challenge of limited data when applying these techniques in interventional radiology. Lastly, this review will address common errors in research in this field and suggest pathways for those interested in learning and becoming involved about AI.

Diagnostic accuracy of a commercially available deep-learning algorithm in supine chest radiographs following trauma.

OBJECTIVES: Trauma chest radiographs may contain subtle and time-critical pathology. Artificial intelligence (AI) may aid in accurate reporting, timely identification and worklist prioritisation. However, few AI programs have been externally validated. This study aimed to evaluate the performance of a commercially available deep convolutional neural network - Annalise CXR V1.2 (Annalise.ai) - for detection of traumatic injuries on supine chest radiographs. METHODS: Chest radiographs with a CT performed within 24 h in the setting of trauma were retrospectively identified at a level one adult trauma centre between January 2009 and June 2019. Annalise.ai assessment of the chest radiograph was compared to the radiologist report of the chest radiograph. Contemporaneous CT report was taken as the ground truth. Agreement with CT was measured using Cohen's κ and sensitivity/specificity for both AI and radiologists were calculated. RESULTS: There were 1404 cases identified with a median age of 52 (IQR 33-69) years, 949 males. AI demonstrated superior performance compared to radiologists in identifying pneumothorax (p = 0.007) and segmental collapse (p = 0.012) on chest radiograph. Radiologists performed better than AI for clavicle fracture (p = 0.002), humerus fracture (p < 0.0015) and scapula fracture (p = 0.014). No statistical difference was found for identification of rib fractures and pneumomediastinum. CONCLUSION: The evaluated AI performed comparably to radiologists in interpreting chest radiographs. Further evaluation of this AI program has the potential to enable it to be safely incorporated in clinical processes. ADVANCES IN KNOWLEDGE: Clinically useful AI programs represent promising decision support tools.

Retrospective analysis and prospective validation of an AI-based software for intracranial haemorrhage detection at a high-volume trauma centre.

Rapid detection of intracranial haemorrhage (ICH) is crucial for assessing patients with neurological symptoms. Prioritising these urgent scans for reporting presents a challenge for radiologists. Artificial intelligence (AI) offers a solution to enable radiologists to triage urgent scans and reduce reporting errors. This study aims to evaluate the accuracy of an ICH-detection AI software and whether it benefits a high-volume trauma centre in terms of triage and reducing diagnostic errors. A peer review of head CT scans performed prior to the implementation of the AI was conducted to identify the department's current miss-rate. Once implemented, the AI software was validated using CT scans performed over one month, and was reviewed by a neuroradiologist. The turn-around-time was calculated as the time taken from scan completion to report finalisation. 2916 head CT scans and reports were reviewed as part of the audit. The AI software flagged 20 cases that were negative-by-report. Two of these were true-misses that had no follow-up imaging. Both patients were followed up and exhibited no long-term neurological sequelae. For ICH-positive scans, there was an increase in TAT in the total sample (35.6%), and a statistically insignificant decrease in TAT in the emergency (- 5.1%) and outpatient (- 14.2%) cohorts. The AI software was tested on a sample of real-world data from a high-volume Australian centre. The diagnostic accuracy was comparable to that reported in literature. The study demonstrated the institution's low miss-rate and short reporting time, therefore any improvements from the use of AI would be marginal and challenging to measure.

Charting the potential of brain computed tomography deep learning systems.

Brain computed tomography (CTB) scans are widely used to evaluate intracranial pathology. The implementation and adoption of CTB has led to clinical improvements. However, interpretation errors occur and may have substantial morbidity and mortality implications for patients. Deep learning has shown promise for facilitating improved diagnostic accuracy and triage. This research charts the potential of deep learning applied to the analysis of CTB scans. It draws on the experience of practicing clinicians and technologists involved in development and implementation of deep learning-based clinical decision support systems. We consider the past, present and future of the CTB, along with limitations of existing systems as well as untapped beneficial use cases. Implementing deep learning CTB interpretation systems and effectively navigating development and implementation risks can deliver many benefits to clinicians and patients, ultimately improving efficiency and safety in healthcare.

Evaluation of an Artificial Intelligence Model for Detection of Pneumothorax and Tension Pneumothorax in Chest Radiographs.

Importance: Early detection of pneumothorax, most often via chest radiography, can help determine need for emergent clinical intervention. The ability to accurately detect and rapidly triage pneumothorax with an artificial intelligence (AI) model could assist with earlier identification and improve care. Objective: To compare the accuracy of an AI model vs consensus thoracic radiologist interpretations in detecting any pneumothorax (incorporating both nontension and tension pneumothorax) and tension pneumothorax. Design, Setting, and Participants: This diagnostic study was a retrospective standalone performance assessment using a data set of 1000 chest radiographs captured between June 1, 2015, and May 31, 2021. The radiographs were obtained from patients aged at least 18 years at 4 hospitals in the Mass General Brigham hospital network in the United States. Included radiographs were selected using 2 strategies from all chest radiography performed at the hospitals, including inpatient and outpatient. The first strategy identified consecutive radiographs with pneumothorax through a manual review of radiology reports, and the second strategy identified consecutive radiographs with tension pneumothorax using natural language processing. For both strategies, negative radiographs were selected by taking the next negative radiograph acquired from the same radiography machine as each positive radiograph. The final data set was an amalgamation of these processes. Each radiograph was interpreted independently by up to 3 radiologists to establish consensus ground-truth interpretations. Each radiograph was then interpreted by the AI model for the presence of pneumothorax and tension pneumothorax. This study was conducted between July and October 2021, with the primary analysis performed between October and November 2021. Main Outcomes and Measures: The primary end points were the areas under the receiver operating characteristic curves (AUCs) for the detection of pneumothorax and tension pneumothorax. The secondary end points were the sensitivities and specificities for the detection of pneumothorax and tension pneumothorax. Results: The final analysis included radiographs from 985 patients (mean [SD] age, 60.8 [19.0] years; 436 [44.3%] female patients), including 307 patients with nontension pneumothorax, 128 patients with tension pneumothorax, and 550 patients without pneumothorax. The AI model detected any pneumothorax with an AUC of 0.979 (95% CI, 0.970-0.987), sensitivity of 94.3% (95% CI, 92.0%-96.3%), and specificity of 92.0% (95% CI, 89.6%-94.2%) and tension pneumothorax with an AUC of 0.987 (95% CI, 0.980-0.992), sensitivity of 94.5% (95% CI, 90.6%-97.7%), and specificity of 95.3% (95% CI, 93.9%-96.6%). Conclusions and Relevance: These findings suggest that the assessed AI model accurately detected pneumothorax and tension pneumothorax in this chest radiograph data set. The model's use in the clinical workflow could lead to earlier identification and improved care for patients with pneumothorax.

Artificial intelligence in medical imaging: implications for patient radiation safety.

Artificial intelligence, including deep learning, is currently revolutionising the field of medical imaging, with far reaching implications for almost every facet of diagnostic imaging, including patient radiation safety. This paper introduces basic concepts in deep learning and provides an overview of its recent history and its application in tomographic reconstruction as well as other applications in medical imaging to reduce patient radiation dose, as well as a brief description of previous tomographic reconstruction techniques. This review also describes the commonly used deep learning techniques as applied to tomographic reconstruction and draws parallels to current reconstruction techniques. Finally, this paper reviews some of the estimated dose reductions in CT and positron emission tomography in the recent literature enabled by deep learning, as well as some of the potential problems that may be encountered such as the obscuration of pathology, and highlights the need for additional clinical reader studies from the imaging community.

Chest radiographs and machine learning - Past, present and future.

Despite its simple acquisition technique, the chest X-ray remains the most common first-line imaging tool for chest assessment globally. Recent evidence for image analysis using modern machine learning points to possible improvements in both the efficiency and the accuracy of chest X-ray interpretation. While promising, these machine learning algorithms have not provided comprehensive assessment of findings in an image and do not account for clinical history or other relevant clinical information. However, the rapid evolution in technology and evidence base for its use suggests that the next generation of comprehensive, well-tested machine learning algorithms will be a revolution akin to early advances in X-ray technology. Current use cases, strengths, limitations and applications of chest X-ray machine learning systems are discussed.

Assessment of the effect of a comprehensive chest radiograph deep learning model on radiologist reports and patient outcomes: a real-world observational study.

OBJECTIVES: Artificial intelligence (AI) algorithms have been developed to detect imaging features on chest X-ray (CXR) with a comprehensive AI model capable of detecting 124 CXR findings being recently developed. The aim of this study was to evaluate the real-world usefulness of the model as a diagnostic assistance device for radiologists. DESIGN: This prospective real-world multicentre study involved a group of radiologists using the model in their daily reporting workflow to report consecutive CXRs and recording their feedback on level of agreement with the model findings and whether this significantly affected their reporting. SETTING: The study took place at radiology clinics and hospitals within a large radiology network in Australia between November and December 2020. PARTICIPANTS: Eleven consultant diagnostic radiologists of varying levels of experience participated in this study. PRIMARY AND SECONDARY OUTCOME MEASURES: Proportion of CXR cases where use of the AI model led to significant material changes to the radiologist report, to patient management, or to imaging recommendations. Additionally, level of agreement between radiologists and the model findings, and radiologist attitudes towards the model were assessed. RESULTS: Of 2972 cases reviewed with the model, 92 cases (3.1%) had significant report changes, 43 cases (1.4%) had changed patient management and 29 cases (1.0%) had further imaging recommendations. In terms of agreement with the model, 2569 cases showed complete agreement (86.5%). 390 (13%) cases had one or more findings rejected by the radiologist. There were 16 findings across 13 cases (0.5%) deemed to be missed by the model. Nine out of 10 radiologists felt their accuracy was improved with the model and were more positive towards AI poststudy. CONCLUSIONS: Use of an AI model in a real-world reporting environment significantly improved radiologist reporting and showed good agreement with radiologists, highlighting the potential for AI diagnostic support to improve clinical practice.

Do comprehensive deep learning algorithms suffer from hidden stratification? A retrospective study on pneumothorax detection in chest radiography.

OBJECTIVES: To evaluate the ability of a commercially available comprehensive chest radiography deep convolutional neural network (DCNN) to detect simple and tension pneumothorax, as stratified by the following subgroups: the presence of an intercostal drain; rib, clavicular, scapular or humeral fractures or rib resections; subcutaneous emphysema and erect versus non-erect positioning. The hypothesis was that performance would not differ significantly in each of these subgroups when compared with the overall test dataset. DESIGN: A retrospective case-control study was undertaken. SETTING: Community radiology clinics and hospitals in Australia and the USA. PARTICIPANTS: A test dataset of 2557 chest radiography studies was ground-truthed by three subspecialty thoracic radiologists for the presence of simple or tension pneumothorax as well as each subgroup other than positioning. Radiograph positioning was derived from radiographer annotations on the images. OUTCOME MEASURES: DCNN performance for detecting simple and tension pneumothorax was evaluated over the entire test set, as well as within each subgroup, using the area under the receiver operating characteristic curve (AUC). A difference in AUC of more than 0.05 was considered clinically significant. RESULTS: When compared with the overall test set, performance of the DCNN for detecting simple and tension pneumothorax was statistically non-inferior in all subgroups. The DCNN had an AUC of 0.981 (0.976-0.986) for detecting simple pneumothorax and 0.997 (0.995-0.999) for detecting tension pneumothorax. CONCLUSIONS: Hidden stratification has significant implications for potential failures of deep learning when applied in clinical practice. This study demonstrated that a comprehensively trained DCNN can be resilient to hidden stratification in several clinically meaningful subgroups in detecting pneumothorax.