Comparing Artificial Intelligence Approaches to Retrieve Clinical Reports Documenting Implantable Devices Posing MRI Safety Risks.

OBJECTIVE: Assess sensitivity, specificity, and accuracy of two approaches to identify patients with implantable devices that pose safety risks for MRI-an expert-derived approach and an ontology-derived natural language processing (NLP). Determine the proportion of clinical data that identify these implantable devices. METHODS: This Institutional Review Board-approved retrospective study was performed at a 793-bed academic hospital. The expert-derived approach used an open-source software with a list of curated terms to query for implantable devices posing high safety risk ("MRI-Red") in patients undergoing MRI. The ontology-derived approach used an NLP system with terms mapped to Systematized Nomenclature of Medicine-Clinical Terms. Queries were performed in three clinical data types-25,000 radiology reports, 174,769 emergency department (ED) notes, and 41,085 other clinical reports (eg, cardiology, operating room, physician notes, radiology reports, pathology reports, patient letters). Sensitivity, specificity, and accuracy of both methods against manual review of a randomly sampled 465 reports were assessed and tested for significant differences between expert-derived and ontology-derived approaches using t test. RESULTS: Accuracy, sensitivity, and specificity of expert-versus ontology-derived approaches were similar (0.83 versus 0.91, P = .080; 0.88 versus 0.96, P = .178; 0.82 versus 0.92, P = .110). The proportion of radiology reports, ED notes, and other clinical reports retrieved containing implantable devices with high safety risks for MRI ranged from 1.47% to 1.88%. DISCUSSION: Artificial intelligence approaches such as expert-driven NLP and ontology-driven NLP have similar accuracy in identifying patients with implantable devices that pose high safety risks for MRI.