Overview of the 2022 n2c2 shared task on contextualized medication event extraction in clinical notes.

BACKGROUND: An accurate medication history, foundational for providing quality medical care, requires understanding of medication change events documented in clinical notes. However, extracting medication changes without the necessary clinical context is insufficient for real-world applications. METHODS: To address this need, Track 1 of the 2022 National NLP Clinical Challenges focused on extracting the context for medication changes documented in clinical notes using the Contextualized Medication Event Dataset. Track 1 consisted of 3 subtasks: extracting medication mentions from clinical notes (NER), determining whether a medication change is being discussed (Event), and determining the action, negation, temporality, certainty, and actor for any change events (Context). Participants were allowed to participate in any one or more of the subtasks. RESULTS: A total of 32 teams with participants from 19 countries submitted a total of 211 systems across all subtasks. Most teams formulated NER as a token classification task and Event and Context as multi-class classification tasks, using transformer-based large language models. Overall, performance for NER was high across submitted systems. However, performance for Event and Context were much lower, often due to indirectly stated change events with no clear action verb, events requiring farther textual clues for understanding, and medication mentions with multiple change events. CONCLUSIONS: This shared task showed that while NLP research on medication extraction is relatively mature, understanding of contextual information surrounding medication events in clinical notes is still an open problem requiring further research to achieve the end goal of supporting real-world clinical applications.

Extracting medication changes in clinical narratives using pre-trained language models.

An accurate and detailed account of patient medications, including medication changes within the patient timeline, is essential for healthcare providers to provide appropriate patient care. Healthcare providers or the patients themselves may initiate changes to patient medication. Medication changes take many forms, including prescribed medication and associated dosage modification. These changes provide information about the overall health of the patient and the rationale that led to the current care. Future care can then build on the resulting state of the patient. This work explores the automatic extraction of medication change information from free-text clinical notes. The Contextual Medication Event Dataset (CMED) is a corpus of clinical notes with annotations that characterize medication changes through multiple change-related attributes, including the type of change (start, stop, increase, etc.), initiator of the change, temporality, change likelihood, and negation. Using CMED, we identify medication mentions in clinical text and propose three novel high-performing BERT-based systems that resolve the annotated medication change characteristics. We demonstrate that our proposed systems improve medication change classification performance over the initial work exploring CMED.

Capturing Individual-level Social Determinants from Clinical Text.

Knowledge of social determinants of health (SDOH), which refer to nonmedical factors influencing health outcomes, can help providers improve patient care. However, SDOH are often documented in unstructured notes, making them more inaccessible. Although previous works have attempted SDOH extraction from clinical notes, most efforts defined SDOH more narrowly and focused on the note's social history (SH) section, where social factors are traditionally documented. Here, we introduce a new SDOH dataset covering a broad range of SDOH content that is annotated over entire notes. We characterize what, where, and how SDOH information is documented in clinical text, present baseline systems using a token classification and generative approach, and investigate whether training only on the SH section can effectively extract SDOH from the entire note. The final dataset, consisting of 2,007 annotations covering 7 open-ended SDOH domains over 500 notes, will be publicly released to encourage further research in this area.

Hierarchy-aware Adverse Reaction Embeddings for Signal Detection.

Post-market drug surveillance monitors new and evolving treatments for their effectiveness and safety in real-world conditions. A large amount of drug safety surveillance data is captured by spontaneous reporting systems such as the FAERS. Developing automated methods to identify actionable safety signals from these databases is an active area of research. In this paper, we propose two novel network representation learning methods (HARE and T-HARE) for signal detection that jointly utilize association information between drugs and medical outcomes from the FAERS and ancestral information in medical ontologies. We evaluate these methods using two publicly available reference datasets, EU-ADR and OMOP corpus. Experimental results showed that the proposed methods significantly outper-formed standard methodologies based on disproportionality metrics and the existing state-of-the-art aer2vec method with statistically significant improvements on both EU-ADR and OMOP datasets. Through quantitative and qualitative analysis, we demonstrate the potential of the proposed methods for effective signal detection.

AI-assisted tracking of worldwide non-pharmaceutical interventions for COVID-19.

The Coronavirus disease 2019 (COVID-19) global pandemic has transformed almost every facet of human society throughout the world. Against an emerging, highly transmissible disease, governments worldwide have implemented non-pharmaceutical interventions (NPIs) to slow the spread of the virus. Examples of such interventions include community actions, such as school closures or restrictions on mass gatherings, individual actions including mask wearing and self-quarantine, and environmental actions such as cleaning public facilities. We present the Worldwide Non-pharmaceutical Interventions Tracker for COVID-19 (WNTRAC), a comprehensive dataset consisting of over 6,000 NPIs implemented worldwide since the start of the pandemic. WNTRAC covers NPIs implemented across 261 countries and territories, and classifies NPIs into a taxonomy of 16 NPI types. NPIs are automatically extracted daily from Wikipedia articles using natural language processing techniques and then manually validated to ensure accuracy and veracity. We hope that the dataset will prove valuable for policymakers, public health leaders, and researchers in modeling and analysis efforts to control the spread of COVID-19.

Reducing Physicians' Cognitive Load During Chart Review: A Problem-Oriented Summary of the Patient Electronic Record.

Overabundance of information within electronic health records (EHRs) has resulted in a need for automated systems to mitigate the cognitive burden on physicians utilizing today's EHR systems. We present ProSPER, a Problem-oriented Summary of the Patient Electronic Record that displays a patient summary centered around an auto-generated problem list and disease-specific views for chronic conditions. ProSPER was developed using 1,500 longitudinal patient records from two large multi-specialty medical groups in the United States, and leverages multiple natural language processing (NLP) components targeting various fundamental (e.g. syntactic analysis), clinical (e.g. adverse drug event extraction) and summarizing (e.g. problem list generation) tasks. We report evaluation results for each component and discuss how specific components address existing physician challenges in reviewing EHR data. This work demonstrates the need to leverage holistic information in EHRs to build a comprehensive summarization application, and the potential for NLP-based applications to support physicians and improve clinical care.

Toward Understanding Clinical Context of Medication Change Events in Clinical Narratives.

Understanding medication events in clinical narratives is essential to achieving a complete picture of a patient's medication history. While prior research has explored identification of medication changes in clinical notes, due to the longitudinal and narrative nature of clinical documentation, extraction of medication change alone without the necessary clinical context is insufficient for use in real-world applications, such as medication timeline generation and medication reconciliation. Here, we present a framework to capture multi-dimensional context of medication changes documented in clinical notes. We define specific contextual aspects pertinent to medication change events (i.e. Action, Negation, Temporality, Certainty, and Actor), describe the annotation process and challenges encountered while creating the dataset, and explore models based on state-of-the-art transformers to automate the task. The resulting dataset, Contextualized Medication Event Dataset (CMED), consisting of 9,013 medications annotated over 500 clinical notes, will be released to the community as a shared task in 2021-2022.

Identification of Semantically Similar Sentences in Clinical Notes: Iterative Intermediate Training Using Multi-Task Learning.

BACKGROUND: Although electronic health records (EHRs) have been widely adopted in health care, effective use of EHR data is often limited because of redundant information in clinical notes introduced by the use of templates and copy-paste during note generation. Thus, it is imperative to develop solutions that can condense information while retaining its value. A step in this direction is measuring the semantic similarity between clinical text snippets. To address this problem, we participated in the 2019 National NLP Clinical Challenges (n2c2)/Open Health Natural Language Processing Consortium (OHNLP) clinical semantic textual similarity (ClinicalSTS) shared task. OBJECTIVE: This study aims to improve the performance and robustness of semantic textual similarity in the clinical domain by leveraging manually labeled data from related tasks and contextualized embeddings from pretrained transformer-based language models. METHODS: The ClinicalSTS data set consists of 1642 pairs of deidentified clinical text snippets annotated in a continuous scale of 0-5, indicating degrees of semantic similarity. We developed an iterative intermediate training approach using multi-task learning (IIT-MTL), a multi-task training approach that employs iterative data set selection. We applied this process to bidirectional encoder representations from transformers on clinical text mining (ClinicalBERT), a pretrained domain-specific transformer-based language model, and fine-tuned the resulting model on the target ClinicalSTS task. We incrementally ensembled the output from applying IIT-MTL on ClinicalBERT with the output of other language models (bidirectional encoder representations from transformers for biomedical text mining [BioBERT], multi-task deep neural networks [MT-DNN], and robustly optimized BERT approach [RoBERTa]) and handcrafted features using regression-based learning algorithms. On the basis of these experiments, we adopted the top-performing configurations as our official submissions. RESULTS: Our system ranked first out of 87 submitted systems in the 2019 n2c2/OHNLP ClinicalSTS challenge, achieving state-of-the-art results with a Pearson correlation coefficient of 0.9010. This winning system was an ensembled model leveraging the output of IIT-MTL on ClinicalBERT with BioBERT, MT-DNN, and handcrafted medication features. CONCLUSIONS: This study demonstrates that IIT-MTL is an effective way to leverage annotated data from related tasks to improve performance on a target task with a limited data set. This contribution opens new avenues of exploration for optimized data set selection to generate more robust and universal contextual representations of text in the clinical domain.

Extraction of Information Related to Drug Safety Surveillance From Electronic Health Record Notes: Joint Modeling of Entities and Relations Using Knowledge-Aware Neural Attentive Models.

BACKGROUND: An adverse drug event (ADE) is commonly defined as "an injury resulting from medical intervention related to a drug." Providing information related to ADEs and alerting caregivers at the point of care can reduce the risk of prescription and diagnostic errors and improve health outcomes. ADEs captured in structured data in electronic health records (EHRs) as either coded problems or allergies are often incomplete, leading to underreporting. Therefore, it is important to develop capabilities to process unstructured EHR data in the form of clinical notes, which contain a richer documentation of a patient's ADE. Several natural language processing (NLP) systems have been proposed to automatically extract information related to ADEs. However, the results from these systems showed that significant improvement is still required for the automatic extraction of ADEs from clinical notes. OBJECTIVE: This study aims to improve the automatic extraction of ADEs and related information such as drugs, their attributes, and reason for administration from the clinical notes of patients. METHODS: This research was conducted using discharge summaries from the Medical Information Mart for Intensive Care III (MIMIC-III) database obtained through the 2018 National NLP Clinical Challenges (n2c2) annotated with drugs, drug attributes (ie, strength, form, frequency, route, dosage, duration), ADEs, reasons, and relations between drugs and other entities. We developed a deep learning-based system for extracting these drug-centric concepts and relations simultaneously using a joint method enhanced with contextualized embeddings, a position-attention mechanism, and knowledge representations. The joint method generated different sentence representations for each drug, which were then used to extract related concepts and relations simultaneously. Contextualized representations trained on the MIMIC-III database were used to capture context-sensitive meanings of words. The position-attention mechanism amplified the benefits of the joint method by generating sentence representations that capture long-distance relations. Knowledge representations were obtained from graph embeddings created using the US Food and Drug Administration Adverse Event Reporting System database to improve relation extraction, especially when contextual clues were insufficient. RESULTS: Our system achieved new state-of-the-art results on the n2c2 data set, with significant improvements in recognizing crucial drug-reason (F1=0.650 versus F1=0.579) and drug-ADE (F1=0.490 versus F1=0.476) relations. CONCLUSIONS: This study presents a system for extracting drug-centric concepts and relations that outperformed current state-of-the-art results and shows that contextualized embeddings, position-attention mechanisms, and knowledge graph embeddings effectively improve deep learning-based concepts and relation extraction. This study demonstrates the potential for deep learning-based methods to help extract real-world evidence from unstructured patient data for drug safety surveillance.

Timely and Efficient AI Insights on EHR: System Design.

A patient's electronic health record (EHR) contains extensive documentation of the patient's medical history but is difficult for clinicians to review and find what they are looking for under the time constraints of the clinical setting. Although recent advances in artificial intelligence (AI) in healthcare have shown promise in enhancing clinical diagnosis and decision-making in clinicians' day-to-day tasks, the problem of how to implement and scale such computationally expensive analytics remains an open issue. In this work, we present a system architecture that generates AI-based insights from analysis of the entire patient medical record for a multispecialty outpatient facility of over 700,000 patients. Our resulting system is able to generate insights efficiently while handling complexities of scheduling to deliver the results in a timely manner, and handle more than 30,000 updates per day while achieving desirable operating cost-performance goals.

Ground Truth Creation for Complex Clinical NLP Tasks - an Iterative Vetting Approach and Lessons Learned.

Natural language processing (NLP) holds the promise of effectively analyzing patient record data to reduce cognitive load on physicians and clinicians in patient care, clinical research, and hospital operations management. A critical need in developing such methods is the "ground truth" dataset needed for training and testing the algorithms. Beyond localizable, relatively simple tasks, ground truth creation is a significant challenge because medical experts, just as physicians in patient care, have to assimilate vast amounts of data in EHR systems. To mitigate potential inaccuracies of the cognitive challenges, we present an iterative vetting approach for creating the ground truth for complex NLP tasks. In this paper, we present the methodology, and report on its use for an automated problem list generation task, its effect on the ground truth quality and system accuracy, and lessons learned from the effort.

Automated problem list generation and physicians perspective from a pilot study.

OBJECTIVE: An accurate, comprehensive and up-to-date problem list can help clinicians provide patient-centered care. Unfortunately, problem lists created and maintained in electronic health records by providers tend to be inaccurate, duplicative and out of date. With advances in machine learning and natural language processing, it is possible to automatically generate a problem list from the data in the EHR and keep it current. In this paper, we describe an automated problem list generation method and report on insights from a pilot study of physicians' assessment of the generated problem lists compared to existing providers-curated problem lists in an institution's EHR system. MATERIALS AND METHODS: The natural language processing and machine learning-based Watson1 method models clinical thinking in identifying a patient's problem list using clinical notes and structured data. This pilot study assessed the Watson method and included 15 randomly selected, de-identified patient records from a large healthcare system that were each planned to be reviewed by at least two internal medicine physicians. The physicians created their own problem lists, and then evaluated the overall usefulness of their own problem lists (P), Watson generated problem lists (W), and the existing EHR problem lists (E) on a 10-point scale. The primary outcome was pairwise comparisons of P, W, and E. RESULTS: Six out of the 10 invited physicians completed 27 assessments of P, W, and E, and in process evaluated 732 Watson generated problems and 444 problems in the EHR system. As expected, physicians rated their own lists, P, highest. However, W was rated higher than E. Among 89% of assessments, Watson identified at least one important problem that physicians missed. CONCLUSION: Cognitive computing systems like this Watson system hold the potential for accurate, problem-list-centered summarization of patient records, potentially leading to increased efficiency, better clinical decision support, and improved quality of patient care.