Development and External Validation of an Artificial Intelligence Model for Identifying Radiology Reports Containing Recommendations for Additional Imaging.

BACKGROUND. Reported rates of recommendations for additional imaging (RAIs) in radiology reports are low. Bidirectional encoder representations from transformers (BERT), a deep learning model pretrained to understand language context and ambiguity, has potential for identifying RAIs and thereby assisting large-scale quality improvement efforts. OBJECTIVE. The purpose of this study was to develop and externally validate an artificial intelligence (AI)-based model for identifying radiology reports containing RAIs. METHODS. This retrospective study was performed at a multisite health center. A total of 6300 radiology reports generated at one site from January 1, 2015, to June 30, 2021, were randomly selected and split by 4:1 ratio to create training (n = 5040) and test (n = 1260) sets. A total of 1260 reports generated at the center's other sites (including academic and community hospitals) from April 1 to April 30, 2022, were randomly selected as an external validation group. Referring practitioners and radiologists of varying sub-specialties manually reviewed report impressions for presence of RAIs. A BERT-based technique for identifying RAIs was developed by use of the training set. Performance of the BERT-based model and a previously developed traditional machine learning (TML) model was assessed in the test set. Finally, performance was assessed in the external validation set. The code for the BERT-based RAI model is publicly available. RESULTS. Among a total of 7419 unique patients (4133 women, 3286 men; mean age, 58.8 years), 10.0% of 7560 reports contained RAI. In the test set, the BERT-based model had 94.4% precision, 98.5% recall, and an F1 score of 96.4%. In the test set, the TML model had 69.0% precision, 65.4% recall, and an F1 score of 67.2%. In the test set, accuracy was greater for the BERT-based than for the TML model (99.2% vs 93.1%, p < .001). In the external validation set, the BERT-based model had 99.2% precision, 91.6% recall, an F1 score of 95.2%, and 99.0% accuracy. CONCLUSION. The BERT-based AI model accurately identified reports with RAIs, outperforming the TML model. High performance in the external validation set suggests the potential for other health systems to adapt the model without requiring institution-specific training. CLINICAL IMPACT. The model could potentially be used for real-time EHR monitoring for RAIs and other improvement initiatives to help ensure timely performance of clinically necessary recommended follow-up.

Patient Complaints in Image-Guided Interventions: Evaluation of Multifactorial Issues Using a Coding Taxonomy.

OBJECTIVE: The purpose of this article is to classify complaints from patients undergoing image-guided interventions using a coding taxonomy and to test whether single-coded complaints were resolved satisfactorily compared with multifactorial (multiple codes) complaints. MATERIALS AND METHODS: In this retrospective study, patients' complaint narratives between April 1999 and December 2012 were reviewed and categorized according to a three-level taxonomy into domains and codes. Resolutions were categorized as satisfactory or unsatisfactory to the patient and were classified as follows: clarification, apology, manager notification, change of provider, reimbursement, and quality review. Complaints were classified as single coded (only one code identified in the patients' description) and multifactorial (multiple codes identified). Statistical analysis was performed with the Fisher test, with the significance level set at 0.05. A run chart with the distribution of complaints by domains (relationships, management, and clinical) by year was performed. RESULTS: A total of 146 codes were extracted from 71 narratives (2.06 codes/complaint) and were classified into the following domains: clinical (52%; n = 76), management (24%; n = 35), and relationships (24%; n = 35). The most common codes included quality of care, safety, and communication breakdown issues. A run chart found a decline in absolute numbers of complaints over the years in the domains studied. The frequency of satisfactory resolution was 86% for multifactorial versus 81% for single-coded complaints with no statistically significant differences observed (p = 0.72). Over 50% of complaints were resolved by providing clarification to patients (n = 36). CONCLUSION: There were no statistically significant differences between multifactorial and single-coded complaints. Clinical codes and communication breakdown were the most common reasons for patient-reported complaint, with most complaints successfully resolved with clarification.