Detection of allergic transfusion-related adverse events from electronic medical records.

BACKGROUND: Transfusion-related adverse events can be unrecognized and unreported. As part of the US Food and Drug Administration's Center for Biologics Evaluation and Research Biologics Effectiveness and Safety initiative, we explored whether machine learning methods, such as natural language processing (NLP), can identify and report transfusion allergic reactions (ARs) from electronic health records (EHRs). STUDY DESIGN AND METHODS: In a 4-year period, all 146 reported transfusion ARs were pulled from a database of 86,764 transfusions in an academic health system, along with a random sample of 605 transfusions without reported ARs. Structured and unstructured EHR data were retrieved, including demographics, new symptoms, medications, and lab results. In unstructured data, evidence from clinicians' notes, test results, and prescriptions fields identified transfusion ARs, which were used to extract NLP features. Clinician reviews of selected validation cases assessed and confirmed model performance. RESULTS: Clinician reviews of selected validation cases yielded a sensitivity of 67.9% and a specificity of 97.5% at a threshold of 0.9, with a positive predictive value (PPV) of 84%, estimated to 4.5% when extrapolated to match transfusion AR incidence in the full transfusion dataset. A higher threshold achieved sensitivity of 43% with specificity/PPV of 100% in our validation set. Essential features predicting ARs were recognized transfusion reactions, administration of antihistamines or glucocorticoids, and skin symptoms (e.g., hives and itching). Removal of NLP features decreased model performance. DISCUSSION: NLP algorithms can identify transfusion reactions from the EHR with a reasonable level of precision for subsequent clinician review and confirmation.

Development of Interoperable Computable Phenotype Algorithms for Adverse Events of Special Interest to Be Used for Biologics Safety Surveillance: Validation Study.

BACKGROUND: Adverse events associated with vaccination have been evaluated by epidemiological studies and more recently have gained additional attention with the emergency use authorization of several COVID-19 vaccines. As part of its responsibility to conduct postmarket surveillance, the US Food and Drug Administration continues to monitor several adverse events of special interest (AESIs) to ensure vaccine safety, including for COVID-19. OBJECTIVE: This study is part of the Biologics Effectiveness and Safety Initiative, which aims to improve the Food and Drug Administration's postmarket surveillance capabilities while minimizing public burden. This study aimed to enhance active surveillance efforts through a rules-based, computable phenotype algorithm to identify 5 AESIs being monitored by the Center for Disease Control and Prevention for COVID-19 or other vaccines: anaphylaxis, Guillain-Barré syndrome, myocarditis/pericarditis, thrombosis with thrombocytopenia syndrome, and febrile seizure. This study examined whether these phenotypes have sufficiently high positive predictive value (PPV) to ensure that the cases selected for surveillance are reasonably likely to be a postbiologic adverse event. This allows patient privacy, and security concerns for the data sharing of patients who had nonadverse events can be properly accounted for when evaluating the cost-benefit aspect of our approach. METHODS: AESI phenotype algorithms were developed to apply to electronic health record data at health provider organizations across the country by querying for standard and interoperable codes. The codes queried in the rules represent symptoms, diagnoses, or treatments of the AESI sourced from published case definitions and input from clinicians. To validate the performance of the algorithms, we applied them to electronic health record data from a US academic health system and provided a sample of cases for clinicians to evaluate. Performance was assessed using PPV. RESULTS: With a PPV of 93.3%, our anaphylaxis algorithm performed the best. The PPVs for our febrile seizure, myocarditis/pericarditis, thrombocytopenia syndrome, and Guillain-Barré syndrome algorithms were 89%, 83.5%, 70.2%, and 47.2%, respectively. CONCLUSIONS: Given our algorithm design and performance, our results support continued research into using interoperable algorithms for widespread AESI postmarket detection.

The Food and Drug Administration Biologics Effectiveness and Safety Initiative Facilitates Detection of Vaccine Administrations From Unstructured Data in Medical Records Through Natural Language Processing.

Introduction: The Food and Drug Administration Center for Biologics Evaluation and Research conducts post-market surveillance of biologic products to ensure their safety and effectiveness. Studies have found that common vaccine exposures may be missing from structured data elements of electronic health records (EHRs), instead being captured in clinical notes. This impacts monitoring of adverse events following immunizations (AEFIs). For example, COVID-19 vaccines have been regularly administered outside of traditional medical settings. We developed a natural language processing (NLP) algorithm to mine unstructured clinical notes for vaccinations not captured in structured EHR data. Methods: A random sample of 1,000 influenza vaccine administrations, representing 995 unique patients, was extracted from a large U.S. EHR database. NLP techniques were used to detect administrations from the clinical notes in the training dataset [80% (N = 797) of patients]. The algorithm was applied to the validation dataset [20% (N = 198) of patients] to assess performance. Full medical charts for 28 randomly selected administration events in the validation dataset were reviewed by clinicians. The NLP algorithm was then applied across the entire dataset (N = 995) to quantify the number of additional events identified. Results: A total of 3,199 administrations were identified in the structured data and clinical notes combined. Of these, 2,740 (85.7%) were identified in the structured data, while the NLP algorithm identified 1,183 (37.0%) administrations in clinical notes; 459 were not also captured in the structured data. This represents a 16.8% increase in the identification of vaccine administrations compared to using structured data alone. The validation of 28 vaccine administrations confirmed 27 (96.4%) as "definite" vaccine administrations; 18 (64.3%) had evidence of a vaccination event in the structured data, while 10 (35.7%) were found solely in the unstructured notes. Discussion: We demonstrated the utility of an NLP algorithm to identify vaccine administrations not captured in structured EHR data. NLP techniques have the potential to improve detection of vaccine administrations not otherwise reported without increasing the analysis burden on physicians or practitioners. Future applications could include refining estimates of vaccine coverage and detecting other exposures, population characteristics, and outcomes not reliably captured in structured EHR data.