Enhancing Postmarketing Surveillance of Medical Products With Large Language Models.

Importance: The Sentinel System is a key component of the US Food and Drug Administration (FDA) postmarketing safety surveillance commitment and uses clinical health care data to conduct analyses to inform drug labeling and safety communications, FDA advisory committee meetings, and other regulatory decisions. However, observational data are frequently deemed insufficient for reliable evaluation of safety concerns owing to limitations in underlying data or methodology. Advances in large language models (LLMs) provide new opportunities to address some of these limitations. However, careful consideration is necessary for how and where LLMs can be effectively deployed for these purposes. Observations: LLMs may provide new avenues to support signal-identification activities to identify novel adverse event signals from narrative text of electronic health records. These algorithms may be used to support epidemiologic investigations examining the causal relationship between exposure to a medical product and an adverse event through development of probabilistic phenotyping of health outcomes of interest and extraction of information related to important confounding factors. LLMs may perform like traditional natural language processing tools by annotating text with controlled vocabularies with additional tailored training activities. LLMs offer opportunities for enhancing information extraction from adverse event reports, medical literature, and other biomedical knowledge sources. There are several challenges that must be considered when leveraging LLMs for postmarket surveillance. Prompt engineering is needed to ensure that LLM-extracted associations are accurate and specific. LLMs require extensive infrastructure to use, which many health care systems lack, and this can impact diversity, equity, and inclusion, and result in obscuring significant adverse event patterns in some populations. LLMs are known to generate nonfactual statements, which could lead to false positive signals and downstream evaluation activities by the FDA and other entities, incurring substantial cost. Conclusions and Relevance: LLMs represent a novel paradigm that may facilitate generation of information to support medical product postmarket surveillance activities that have not been possible. However, additional work is required to ensure LLMs can be used in a fair and equitable manner, minimize false positive findings, and support the necessary rigor of signal detection needed for regulatory activities.

Leveraging structured and unstructured electronic health record data to detect reasons for suboptimal statin therapy use in patients with atherosclerotic cardiovascular disease.

OBJECTIVE: To determine whether natural language processing (NLP) of unstructured medical text can improve identification of ASCVD patients not using high-intensity statin therapy (HIST) due to statin-associated side effects (SASEs) and other reasons. METHODS: Reviewers annotated reasons for not prescribing HIST in notes of 1152 randomly selected patients from across the VA healthcare system treated for ASCVD but not receiving HIST. Developers used reviewer annotations to train the Canary NLP tool to detect and extract notes containing one or more of these reasons. Negative predictive value (NPV), sensitivity, specificity and Area Under the Curve (AUC) were used to assess accuracy at detecting documents containing reasons when using structured data, NLP-extracted unstructured data, or both data sources combined. RESULTS: At least one documented reason for not prescribing HIST occurred in 47% of notes. The most frequent reasons were SASEs (41%) and general intolerance (20%). When identifying notes containing any documented reason for not using HIST, adding NLP-extracted, unstructured data significantly (p<0.05) increased sensitivity (0.69 (95% confidence interval [CI] 0.60-0.76) to 0.89 (95% CI 0.81-0.93)), NPV (0.90 (95% CI 0.87 to 0.93) to 0.96 (95% CI 0.93-0.98)), and AUC (0.84 (95% confidence interval [CI] 0.81-0.88) to 0.91 (95% CI 0.90-0.93)) compared to structured data alone. CONCLUSIONS: NLP extraction of data from unstructured text can improve identification of reasons for patients not being on HIST over structured data alone. The additional information provided through NLP of unstructured free text should help in tailoring and implementing system-level interventions to improve HIST use in patients with ASCVD.