Improving Methods of Identifying Anaphylaxis for Medical Product Safety Surveillance Using Natural Language Processing and Machine Learning.

We sought to determine whether machine learning and natural language processing (NLP) applied to electronic medical records could improve performance of automated health-care claims-based algorithms to identify anaphylaxis events using data on 516 patients with outpatient, emergency department, or inpatient anaphylaxis diagnosis codes during 2015-2019 in 2 integrated health-care institutions in the Northwest United States. We used one site's manually reviewed gold-standard outcomes data for model development and the other's for external validation based on cross-validated area under the receiver operating characteristic curve (AUC), positive predictive value (PPV), and sensitivity. In the development site 154 (64%) of 239 potential events met adjudication criteria for anaphylaxis compared with 180 (65%) of 277 in the validation site. Logistic regression models using only structured claims data achieved a cross-validated AUC of 0.58 (95% CI: 0.54, 0.63). Machine learning improved cross-validated AUC to 0.62 (0.58, 0.66); incorporating NLP-derived covariates further increased cross-validated AUCs to 0.70 (0.66, 0.75) in development and 0.67 (0.63, 0.71) in external validation data. A classification threshold with cross-validated PPV of 79% and cross-validated sensitivity of 66% in development data had cross-validated PPV of 78% and cross-validated sensitivity of 56% in external data. Machine learning and NLP-derived data improved identification of validated anaphylaxis events.

Validation of Acute Pancreatitis Among Adults in an Integrated Healthcare System.

BACKGROUND: Acute pancreatitis is a serious gastrointestinal disease that is an important target for drug safety surveillance. Little is known about the accuracy of ICD-10 codes for acute pancreatitis in the United States, or their performance in specific clinical settings. We conducted a validation study to assess the accuracy of acute pancreatitis ICD-10 diagnosis codes in inpatient, emergency department (ED), and outpatient settings. METHODS: We reviewed electronic medical records for encounters with acute pancreatitis diagnosis codes in an integrated healthcare system from October 2015 to December 2019. Trained abstractors and physician adjudicators determined whether events met criteria for acute pancreatitis. RESULTS: Out of 1,844 eligible events, we randomly sampled 300 for review. Across all clinical settings, 182 events met validation criteria for an overall positive predictive value (PPV) of 61% (95% confidence intervals [CI] = 55, 66). The PPV was 87% (95% CI = 79, 92%) for inpatient codes, but only 45% for ED (95% CI = 35, 54%) and outpatient (95% CI = 34, 55%) codes. ED and outpatient encounters accounted for 43% of validated events. Acute pancreatitis codes from any encounter type with lipase >3 times the upper limit of normal had a PPV of 92% (95% CI = 86, 95%) and identified 85% of validated events (95% CI = 79, 89%), while codes with lipase <3 times the upper limit of normal had a PPV of only 22% (95% CI = 16, 30%). CONCLUSIONS: These results suggest that ICD-10 codes accurately identified acute pancreatitis in the inpatient setting, but not in the ED and outpatient settings. Laboratory data substantially improved algorithm performance.

Development of an algorithm to identify inpatient opioid-related overdoses and oversedation using electronic data.

PURPOSE: To facilitate surveillance and evaluate interventions addressing opioid-related overdoses, algorithms are needed for use in large health care databases to identify and differentiate community-occurring opioid-related overdoses from inpatient-occurring opioid-related overdose/oversedation. METHODS: Data were from Kaiser Permanente Northwest (KPNW), a large integrated health plan. We iteratively developed and evaluated an algorithm for electronically identifying inpatient overdose/oversedation in KPNW hospitals from 1 January 2008 to 31 December 2014. Chart audits assessed accuracy; data sources included administrative and clinical records. RESULTS: The best-performing algorithm used these rules: (1) Include events with opioids administered in an inpatient setting (including emergency department/urgent care) followed by naloxone administration within 275 hours of continuous inpatient stay; (2) exclude events with electroconvulsive therapy procedure codes; and (3) exclude events in which an opioid was administered prior to hospital discharge and followed by readmission with subsequent naloxone administration. Using this algorithm, we identified 870 suspect inpatient overdose/oversedation events and chart audited a random sample of 235. Of the random sample, 185 (78.7%) were deemed overdoses/oversedation, 37 (15.5%) were not, and 13 (5.5%) were possible cases. The number of hours between time of opioid and naloxone administration did not affect algorithm accuracy. When "possible" overdoses/oversedations were included with confirmed events, overall positive predictive value (PPV) was very good (PPV = 84.0%). Additionally, PPV was reasonable when evaluated specifically for hospital stays with emergency/urgent care admissions (PPV = 77.0%) and excellent for elective surgery admissions (PPV = 97.0%). CONCLUSIONS: Algorithm performance was reasonable for identifying inpatient overdose/oversedation with best performance among elective surgery patients.

Using natural language processing of clinical text to enhance identification of opioid-related overdoses in electronic health records data.

PURPOSE: To enhance automated methods for accurately identifying opioid-related overdoses and classifying types of overdose using electronic health record (EHR) databases. METHODS: We developed a natural language processing (NLP) software application to code clinical text documentation of overdose, including identification of intention for self-harm, substances involved, substance abuse, and error in medication usage. Using datasets balanced with cases of suspected overdose and records of individuals at elevated risk for overdose, we developed and validated the application using Kaiser Permanente Northwest data, then tested portability of the application using Kaiser Permanente Washington data. Datasets were chart-reviewed to provide a gold standard for comparison and evaluation of the automated method. RESULTS: The method performed well in identifying overdose (sensitivity = 0.80, specificity = 0.93), intentional overdose (sensitivity = 0.81, specificity = 0.98), and involvement of opioids (excluding heroin, sensitivity = 0.72, specificity = 0.96) and heroin (sensitivity = 0.84, specificity = 1.0). The method performed poorly at identifying adverse drug reactions and overdose due to patient error and fairly at identifying substance abuse in opioid-related unintentional overdose (sensitivity = 0.67, specificity = 0.96). Evaluation using validation datasets yielded significant reductions, in specificity and negative predictive values only, for many classifications mentioned above. However, these measures remained above 0.80, thus, performance observed during development was largely maintained during validation. Similar results were obtained when evaluating portability, although there was a significant reduction in sensitivity for unintentional overdose that was attributed to missing text clinical notes in the database. CONCLUSIONS: Methods that process text clinical notes show promise for improving accuracy and fidelity at identifying and classifying overdoses according to type using EHR data.

Identifying and classifying opioid-related overdoses: A validation study.

PURPOSE: The study aims to develop and validate algorithms to identify and classify opioid overdoses using claims and other coded data, and clinical text extracted from electronic health records using natural language processing (NLP). METHODS: Primary data were derived from Kaiser Permanente Northwest (2008-2014), an integrated health care system (~n > 475 000 unique individuals per year). Data included International Classification of Diseases, Ninth Revision (ICD-9) codes for nonfatal diagnoses, International Classification of Diseases, Tenth Revision (ICD-10) codes for fatal events, clinical notes, and prescription medication records. We assessed sensitivity, specificity, positive predictive value, and negative predictive value for algorithms relative to medical chart review and conducted assessments of algorithm portability in Kaiser Permanente Washington, Tennessee State Medicaid, and Optum. RESULTS: Code-based algorithm performance was excellent for opioid-related overdoses (sensitivity = 97.2%, specificity = 84.6%) and classification of heroin-involved overdoses (sensitivity = 91.8%, specificity = 99.0%). Performance was acceptable for code-based suicide/suicide attempt classifications (sensitivity = 70.7%, specificity = 90.5%); sensitivity improved with NLP (sensitivity = 78.7%, specificity = 91.0%). Performance was acceptable for the code-based substance abuse-involved classification (sensitivity = 75.3%, specificity = 79.5%); sensitivity improved with the NLP-enhanced algorithm (sensitivity = 80.5%, specificity = 76.3%). The opioid-related overdose algorithm performed well across portability assessment sites, with sensitivity greater than 96% and specificity greater than 84%. Cross-site sensitivity for heroin-involved overdose was greater than 87%, specificity greater than or equal to 99%. CONCLUSIONS: Code-based algorithms developed to detect opioid-related overdoses and classify them according to heroin involvement perform well. Algorithms for classifying suicides/attempts and abuse-related opioid overdoses perform adequately for use for research, particularly given the complexity of classifying such overdoses. The NLP-enhanced algorithms for suicides/suicide attempts and abuse-related overdoses perform significantly better than code-based algorithms and are appropriate for use in settings that have data and capacity to use NLP.

Assessing Trends in Tobacco Cessation in Diverse Patient Populations.

INTRODUCTION: This study examined change in tobacco use over 4 years among the general population of patients in six diverse health care organizations using electronic medical record data. METHODS: The study cohort (N = 34 393) included all patients age 18 years or older who were identified as smokers in 2007, and who then had at least one primary care visit in each of the following 4 years. RESULTS: In the 4 years following 2007, this patient cohort had a median of 13 primary care visits, and 38.6% of the patients quit smoking at least once. At the end of the fourth follow-up year, 15.4% had stopped smoking for 1 year or more. Smokers were more likely to become long-term quitters if they were 65 or older (OR = 1.32, 95% CI = [1.16, 1.49]), or had a diagnoses of cancer (1.26 [1.12, 1.41]), cardiovascular disease (1.22 [1.09, 1.37]), asthma (1.15 [1.06, 1.25]), or diabetes (1.17 [1.09, 1.27]). Characteristics associated with lower likelihood of becoming a long-term quitter were female gender (0.90 [0.84, 0.95]), black race (0.84 [0.75, 0.94]) and those identified as non-Hispanic (0.50 [0.43, 0.59]). CONCLUSIONS: Among smokers who regularly used these care systems, one in seven had achieved long-term cessation after 4 years. This study shows the practicality of using electronic medical records for monitoring patient smoking status over time. Similar methods could be used to assess tobacco use in any health care organization to evaluate the impact of environmental and organizational programs.

A general framework for developing computable clinical phenotype algorithms.

OBJECTIVE: To present a general framework providing high-level guidance to developers of computable algorithms for identifying patients with specific clinical conditions (phenotypes) through a variety of approaches, including but not limited to machine learning and natural language processing methods to incorporate rich electronic health record data. MATERIALS AND METHODS: Drawing on extensive prior phenotyping experiences and insights derived from 3 algorithm development projects conducted specifically for this purpose, our team with expertise in clinical medicine, statistics, informatics, pharmacoepidemiology, and healthcare data science methods conceptualized stages of development and corresponding sets of principles, strategies, and practical guidelines for improving the algorithm development process. RESULTS: We propose 5 stages of algorithm development and corresponding principles, strategies, and guidelines: (1) assessing fitness-for-purpose, (2) creating gold standard data, (3) feature engineering, (4) model development, and (5) model evaluation. DISCUSSION AND CONCLUSION: This framework is intended to provide practical guidance and serve as a basis for future elaboration and extension.

A survey of informatics platforms that enable distributed comparative effectiveness research using multi-institutional heterogenous clinical data.

Comparative effectiveness research (CER) has the potential to transform the current health care delivery system by identifying the most effective medical and surgical treatments, diagnostic tests, disease prevention methods, and ways to deliver care for specific clinical conditions. To be successful, such research requires the identification, capture, aggregation, integration, and analysis of disparate data sources held by different institutions with diverse representations of the relevant clinical events. In an effort to address these diverse demands, there have been multiple new designs and implementations of informatics platforms that provide access to electronic clinical data and the governance infrastructure required for interinstitutional CER. The goal of this manuscript is to help investigators understand why these informatics platforms are required and to compare and contrast 6 large-scale, recently funded, CER-focused informatics platform development efforts. We utilized an 8-dimension, sociotechnical model of health information technology to help guide our work. We identified 6 generic steps that are necessary in any distributed, multi-institutional CER project: data identification, extraction, modeling, aggregation, analysis, and dissemination. We expect that over the next several years these projects will provide answers to many important, and heretofore unanswerable, clinical research questions.

Measuring problem prescription opioid use among patients receiving long-term opioid analgesic treatment: development and evaluation of an algorithm for use in EHR and claims data.

Objective: Opioid surveillance in response to the opioid epidemic will benefit from scalable, automated algorithms for identifying patients with clinically documented signs of problem prescription opioid use. Existing algorithms lack accuracy. We sought to develop a high-sensitivity, high-specificity classification algorithm based on widely available structured health data to identify patients receiving chronic extended-release/long-acting (ER/LA) therapy with evidence of problem use to support subsequent epidemiologic investigations. Methods: Outpatient medical records of a probability sample of 2,000 Kaiser Permanente Washington patients receiving ≥60 days' supply of ER/LA opioids in a 90-day period from 1 January 2006 to 30 June 2015 were manually reviewed to determine the presence of clinically documented signs of problem use and used as a reference standard for algorithm development. Using 1,400 patients as training data, we constructed candidate predictors from demographic, enrollment, encounter, diagnosis, procedure, and medication data extracted from medical claims records or the equivalent from electronic health record (EHR) systems, and we used adaptive least absolute shrinkage and selection operator (LASSO) regression to develop a model. We evaluated this model in a comparable 600-patient validation set. We compared this model to ICD-9 diagnostic codes for opioid abuse, dependence, and poisoning. This study was registered with ClinicalTrials.gov as study NCT02667262 on 28 January 2016. Results: We operationalized 1,126 potential predictors characterizing patient demographics, procedures, diagnoses, timing, dose, and location of medication dispensing. The final model incorporating 53 predictors had a sensitivity of 0.582 at positive predictive value (PPV) of 0.572. ICD-9 codes for opioid abuse, dependence, and poisoning had a sensitivity of 0.390 at PPV of 0.599 in the same cohort. Conclusions: Scalable methods using widely available structured EHR/claims data to accurately identify problem opioid use among patients receiving long-term ER/LA therapy were unsuccessful. This approach may be useful for identifying patients needing clinical evaluation.

Distributed cognition: an alternative model of cognition for medical informatics.

BACKGROUND: Medical informatics has been guided by an individual-centered model of human cognition, inherited from classical theory of mind, in which knowledge, problem-solving, and information-processing responsible for intelligent behavior all derive from the inner workings of an individual agent. OBJECTIVES AND RESULTS: In this paper we argue that medical informatics commitment to the classical model of cognition conflates the processing performed by the minds of individual agents with the processing performed by the larger distributed activity systems within which individuals operate. We review trends in cognitive science that seek to close the gap between general-purpose models of cognition and applied considerations of real-world human performance. One outcome is the theory of distributed cognition, in which the unit of analysis for understanding performance is the activity system which comprises a group of human actors, their tools and environment, and is organized by a particular history of goal-directed action and interaction. CONCLUSION: We describe and argue for the relevance of distributed cognition to medical informatics, both for the study of human performance in healthcare and for the design of technologies meant to enhance this performance.

Long-Term Outcomes From Repeated Smoking Cessation Assistance in Routine Primary Care.

PURPOSE: To test the association between repeated clinical smoking cessation support and long-term cessation. DESIGN: Retrospective, observational cohort study using structured and free-text data from electronic health records. SETTING: Six diverse health systems in the United States. PARTICIPANTS: Patients aged ≥18 years who were smokers in 2007 and had ≥1 primary care visit in each of the following 4 years (N = 33 691). MEASURES: Primary exposure was a composite categorical variable (comprised of documentation of smoking cessation medication, counseling, or referral) classifying the proportions of visits for which patients received any cessation assistance (<25% (reference), 25%-49%, 50%-74%, and ≥75% of visits). The dependent variable was long-term quit (LTQ; yes/no), defined as no indication of being a current smoker for ≥365 days following a visit where nonsmoker or former smoker was indicated. ANALYSIS: Mixed effects logistic regression analysis adjusted for age, sex, race, and comorbidities, with robust standard error estimation to account for within site correlation. RESULTS: Overall, 20% of the cohort achieved LTQ status. Patients with ≥75% of visits with any assistance had almost 3 times the odds of achieving LTQ status compared to those with <25% visits with assistance (odds ratio = 2.84; 95% confidence interval: 1.50-5.37). Results were similar for specific assistance types. CONCLUSIONS: These findings provide support for the importance of repeated assistance at primary care visits to increase long-term smoking cessation.

Distributed cognition in the heart room: how situation awareness arises from coordinated communications during cardiac surgery.

To help ensure successful outcomes of open-heart surgery, surgeon and perfusionist must coordinate their activities during management of cardioplegia. This research aims to understand the basis for this coordination. We employed the framework of distributed cognition and the methodology of cognitive ethnography to describe how cognitive resources are configured and utilized to accomplish successful cardioplegia management. Analysis identified six types of surgeon-perfusionist verbal exchange which collectively enable robust system performance through (a) making the current situation clear and mutually understood; (b) making goals and envisioned future situations clear and thereby anticipated; and (c) expanding upon the activity system's knowledge base through discovery and sharing of experience. We argue for the "activity system" as the appropriate unit of analysis, and distributed cognition as a powerful theoretical framework for studying the socio-technical work of health care.

Smoking-Cessation Assistance: Before and After Stage 1 Meaningful Use Implementation.

INTRODUCTION: Brief smoking-cessation interventions in primary care settings are effective, but delivery of these services remains low. The Centers for Medicare and Medicaid Services' Meaningful Use (MU) of Electronic Health Record (EHR) Incentive Program could increase rates of smoking assessment and cessation assistance among vulnerable populations. This study examined whether smoking status assessment, cessation assistance, and odds of being a current smoker changed after Stage 1 MU implementation. METHODS: EHR data were extracted from 26 community health centers with an EHR in place by June 15, 2009. AORs were computed for each binary outcome (smoking status assessment, counseling given, smoking-cessation medications ordered/discussed, current smoking status), comparing 2010 (pre-MU), 2012 (MU preparation), and 2014 (MU fully implemented) for pregnant and non-pregnant patients. RESULTS: Non-pregnant patients had decreased odds of current smoking over time; odds for all other outcomes increased except for medication orders from 2010 to 2012. Among pregnant patients, odds of assessment and counseling increased across all years. Odds of discussing or ordering of cessation medications increased from 2010 compared with the other 2 study years; however, medication orders alone did not change over time, and current smoking only decreased from 2010 to 2012. Compared with non-pregnant patients, a lower percentage of pregnant patients were provided counseling. CONCLUSIONS: Findings suggest that incentives for MU of EHRs increase the odds of smoking assessment and cessation assistance, which could lead to decreased smoking rates among vulnerable populations. Continued efforts for provision of cessation assistance among pregnant patients is warranted.

MediClass: A system for detecting and classifying encounter-based clinical events in any electronic medical record.

MediClass is a knowledge-based system that processes both free-text and coded data to automatically detect clinical events in electronic medical records (EMRs). This technology aims to optimize both clinical practice and process control by automatically coding EMR contents regardless of data input method (e.g., dictation, structured templates, typed narrative). We report on the design goals, implemented functionality, generalizability, and current status of the system. MediClass could aid both clinical operations and health services research through enhancing care quality assessment, disease surveillance, and adverse event detection.

Natural language processing in the electronic medical record: assessing clinician adherence to tobacco treatment guidelines.

BACKGROUND: Comprehensively assessing care quality with electronic medical records (EMRs) is not currently possible because much data reside in clinicians' free-text notes. METHODS: We evaluated the accuracy of MediClass, an automated, rule-based classifier of the EMR that incorporates natural language processing, in assessing whether clinicians: (1) asked if the patient smoked; (2) advised them to stop; (3) assessed their readiness to quit; (4) assisted them in quitting by providing information or medications; and (5) arranged for appropriate follow-up care (i.e., the 5A's of smoking-cessation care). DESIGN: We analyzed 125 medical records of known smokers at each of four HMOs in 2003 and 2004. One trained abstractor at each HMO manually coded all 500 records according to whether or not each of the 5A's of smoking cessation care was addressed during routine outpatient visits. MEASUREMENTS: For each patient's record, we compared the presence or absence of each of the 5A's as assessed by each human coder and by MediClass. We measured the chance-corrected agreement between the human raters and MediClass using the kappa statistic. RESULTS: For "ask" and "assist," agreement among human coders was indistinguishable from agreement between humans and MediClass (p>0.05). For "assess" and "advise," the human coders agreed more with each other than they did with MediClass (p<0.01); however, MediClass performance was sufficient to assess quality in these areas. The frequency of "arrange" was too low to be analyzed. CONCLUSIONS: MediClass performance appears adequate to replace human coders of the 5A's of smoking-cessation care, allowing for automated assessment of clinician adherence to one of the most important, evidence-based guidelines in preventive health care.

CER Hub: An informatics platform for conducting comparative effectiveness research using multi-institutional, heterogeneous, electronic clinical data.

OBJECTIVES: Comparative effectiveness research (CER) requires the capture and analysis of data from disparate sources, often from a variety of institutions with diverse electronic health record (EHR) implementations. In this paper we describe the CER Hub, a web-based informatics platform for developing and conducting research studies that combine comprehensive electronic clinical data from multiple health care organizations. METHODS: The CER Hub platform implements a data processing pipeline that employs informatics standards for data representation and web-based tools for developing study-specific data processing applications, providing standardized access to the patient-centric electronic health record (EHR) across organizations. RESULTS: The CER Hub is being used to conduct two CER studies utilizing data from six geographically distributed and demographically diverse health systems. These foundational studies address the effectiveness of medications for controlling asthma and the effectiveness of smoking cessation services delivered in primary care. DISCUSSION: The CER Hub includes four key capabilities: the ability to process and analyze both free-text and coded clinical data in the EHR; a data processing environment supported by distributed data and study governance processes; a clinical data-interchange format for facilitating standardized extraction of clinical data from EHRs; and a library of shareable clinical data processing applications. CONCLUSION: CER requires coordinated and scalable methods for extracting, aggregating, and analyzing complex, multi-institutional clinical data. By offering a range of informatics tools integrated into a framework for conducting studies using EHR data, the CER Hub provides a solution to the challenges of multi-institutional research using electronic medical record data.