Beyond the override: Using evidence of previous drug tolerance to suppress drug allergy alerts; a retrospective study of opioid alerts.

OBJECTIVE: Despite the extensive literature exploring alert fatigue, most studies have focused on describing the phenomenon, but not on fixing it. The authors aimed to identify data useful to avert clinically irrelevant alerts to inform future research on clinical decision support (CDS) design. METHODS: We conducted a retrospective observational study of opioid drug allergy alert (DAA) overrides for the calendar year of 2019 at a large academic medical center, to identify data elements useful to find irrelevant alerts to be averted. RESULTS: Overall, 227,815 DAAs were fired in 2019, with an override rate of 91 % (n = 208196). Opioids represented nearly two-thirds of these overrides (n = 129063; 62 %) and were the drug class with the highest override rate (96 %). On average, 29 opioid DAAs were overridden per patient. While most opioid alerts (97.1 %) are fired for a possible match (the drug class of the allergen matches the drug class of the prescribed drug), they are overridden significantly less frequently for definite match (exact match between allergen and prescribed drug) (88 % vs. 95.9 %, p < 0.001). When comparing the triggering drug with previously administered drugs, override rates were equally high for both definite match (95.9 %), no match (95.5 %), and possible match (95.1 %). Likewise, when comparing to home medications, overrides were excessively high for possible match (96.3 %), no match (96 %), and definite match (94.4 %). CONCLUSION: We estimate that 74.5% of opioid DAAs (46.4% of all DAAs) at our institution could be relatively safely averted, since they either have a definite match for previous inpatient administrations suggesting drug tolerance or are fired as possible match with low risk of cross-sensitivity. Future research should focus on identifying other relevant data elements ideally with automated methods and use of emerging standards to empower CDS systems to suppress false-positive alerts while avoiding safety hazards.

Unintended Consequences of Nationwide Electronic Health Record Adoption: Challenges and Opportunities in the Post-Meaningful Use Era.

The US health system has recently achieved widespread adoption of electronic health record (EHR) systems, primarily driven by financial incentives provided by the Meaningful Use (MU) program. Although successful in promoting EHR adoption and use, the program, and other contributing factors, also produced important unintended consequences (UCs) with far-reaching implications for the US health system. Based on our own experiences from large health information technology (HIT) adoption projects and a collection of key studies in HIT evaluation, we discuss the most prominent UCs of MU: failed expectations, EHR market saturation, innovation vacuum, physician burnout, and data obfuscation. We identify challenges resulting from these UCs and provide recommendations for future research to empower the broader medical and informatics communities to realize the full potential of a now digitized health system. We believe that fixing these unanticipated effects will demand efforts from diverse players such as health care providers, administrators, HIT vendors, policy makers, informatics researchers, funding agencies, and outside developers; promotion of new business models; collaboration between academic medical centers and informatics research departments; and improved methods for evaluations of HIT.

Comprehensive methodology to monitor longitudinal change patterns during EHR implementations: a case study at a large health care delivery network.

OBJECTIVE: To test a systematic methodology to monitor longitudinal change patterns on quality, productivity, and safety outcomes during a large-scale commercial Electronic Health Record (EHR) implementation. MATERIALS AND METHODS: Our method combines an interrupted time-series design with control sites and 41 consensus outcomes including quality (11 measures), productivity (20 measures), and safety (10 measures). The intervention consisted of a phased commercial EHR implementation at a large health care delivery network. Four medium-size hospitals and 39 clinics from 5 geographic regions implementing the new EHR were compared against a parallel control consisting of one medium-size and one large hospital and 10 clinics that had not implemented the new EHR at the time of this study. We collected monthly data from February 2013 to July 2017. RESULTS: The proposed methodology was successfully implemented and significant changes were observed in most measured variables. A significant change attributable to the intervention was observed in 12 (29%) measures in three or more regions; in 32 (78%) measures in two or more regions; and in 40 (98%) measures in at least one region. A similar pattern (i.e., same impact in three or more regions) was detected for nine (22%) measures, a mixed pattern (i.e., same impact in two regions, and different impact in other regions) was detected for nine (22%) measures, and an inconsistent pattern (i.e., did not detect the same impact across regions) was detected for 23 (56%) measures. DISCUSSION: Using a formal methodology to assess changes in a set of consensus measures, we detected various patterns of impact and mixed time-sensitive effects. With an increasing adoption of EHR systems, it is critical for health care organizations to systematically monitor their EHR implementations. The proposed method provides a robust and consistent approach to monitor EHR implementations longitudinally allowing for continuous monitoring after the system becomes stable in order to avoid unexpected effects. CONCLUSION: Our results and methodology can guide the broader medical and informatics communities by informing what and how to continuously monitor EHR impact on quality, productivity, and safety.

Development and classification of a robust inventory of near real-time outcome measurements for assessing information technology interventions in health care.

OBJECTIVE: To develop and classify an inventory of near real-time outcome measures for assessing information technology (IT) interventions in health care and assess their relevance as perceived by experts in the field. MATERIALS AND METHODS: To verify the robustness and coverage of a previously published inventory of measures and taxonomy, we conducted semi-structured interviews with clinical and administrative leaders from a large care delivery system to collect suggestions of outcome measures that can be calculated with data available in electronic format for near real-time monitoring of EHR implementations. We combined these measures with the most commonly reported in the literature. We then conducted two online surveys with subject-matter experts to collect their perceptions of the relevance of the measures, and identify other potentially relevant measures. RESULTS: With input from experienced health care leaders and informaticists, we developed an inventory of 102 outcome measures. These measures were classified into a taxonomy of commonly used measures around the categories of quality, productivity, and safety. Safety measures were rated as most relevant by subject-matter experts, especially those measuring medication processes. Clinician satisfaction and measures assessing mean time to complete tasks and time spent on electronic documentation were also rated as highly relevant. DISCUSSION: By expanding the coverage of our previously published inventory and taxonomy, we expect to help providers, health IT vendors and researchers to more effectively and consistently monitor the impact of EHR implementations in near real-time, and report more standardized outcomes in future studies. We identified several measures not commonly assessed by previous studies of IT implementations, especially those of safety and productivity, which deserve more attention from the broader informatics community. CONCLUSION: Our inventory of measures and taxonomy will help researchers identify gaps in their measurement approaches and report more standardized measurements of IT interventions that could be shared among researchers, hopefully facilitating comparison across future studies and increasing our understanding of the impact of IT interventions in health care.

Health information technology adoption: Understanding research protocols and outcome measurements for IT interventions in health care.

OBJECTIVE: To classify and characterize the variables commonly used to measure the impact of Information Technology (IT) adoption in health care, as well as settings and IT interventions tested, and to guide future research. MATERIALS AND METHODS: We conducted a descriptive study screening a sample of 236 studies from a previous systematic review to identify outcome measures used and the availability of data to calculate these measures. We also developed a taxonomy of commonly used measures and explored setting characteristics and IT interventions. RESULTS: Clinical decision support is the most common intervention tested, primarily in non-hospital-based clinics and large academic hospitals. We identified 15 taxa representing the 79 most commonly used measures. Quality of care was the most common category of these measurements with 62 instances, followed by productivity (11 instances) and patient safety (6 instances). Measures used varied according to type of setting, IT intervention and targeted population. DISCUSSION: This study provides an inventory and a taxonomy of commonly used measures that will help researchers select measures in future studies as well as identify gaps in their measurement approaches. The classification of the other protocol components such as settings and interventions will also help researchers identify underexplored areas of research on the impact of IT interventions in health care. CONCLUSION: A more robust and standardized measurement system and more detailed descriptions of interventions and settings are necessary to enable comparison between studies and a better understanding of the impact of IT adoption in health care settings.

Semantically oriented EHR navigation with a patient specific knowledge base and a clinical context ontology.

Widespread adoption of electronic health records (EHR) in the U.S. has been followed by unintended consequences, overexposing clinicians to widely reported EHR limitations. As an attempt to fixing the EHR, we propose the use of a clinical context ontology (CCO), applied to turn implicit contextual statements into formally represented data in the form of concept-relationship-concept tuples. These tuples form what we call a patient specific knowledge base (PSKB), a collection of formally defined tuples containing facts about the patient's care context. We report the process to create a CCO, which guides annotation of structured and narrative patient data to produce a PSKB. We also present an application of our PSKB using real patient data displayed on a semantically oriented patient summary to improve EHR navigation. Our approach can potentially save precious time spent by clinicians using today's EHRs, by showing a chronological view of the patient's record along with contextual statements needed for care decisions with minimum effort. We propose several other applications of a PSKB to improve multiple EHR functions to guide future research.

Clinical Decision Support Systems: Contributions from 2021.

OBJECTIVES: To summarize significant research contributions published in 2021 in the field of clinical decision support (CDS) systems and select the best papers for the Decision Support section of the International Medical Informatics Association (IMIA) Yearbook. METHODS: The authors searched the MEDLINE® database for papers focused on clinical decision support (CDS) systems. From search results, section editors established a list of candidate best papers, which were then peer-reviewed by at least three external reviewers. The IMIA Yearbook editorial committee selected the best papers on the basis of all reviews including the section editors' evaluation. RESULTS: A total of 337 articles were retrieved from which 13 candidate papers were identified. Finally, from the candidate papers, the top three papers were selected. The first paper introduces an innovative evaluation approach to CDS systems, the second compares six health institutions on how they are measuring CDS alert fatigue and the last one adds new evidence on how CDS can help to reduce unnecessary interventions.

Physicians' perceptions about a semantically integrated display for chart review: A Multi-Specialty survey.

OBJECTIVE: To assess physicians' perceptions about integrated displays for chart review based on a formal representation of patients' care context. METHODS: We iteratively designed a conceptual prototype of an integrated patient summary and conducted an online survey with a multi-specialty panel of outpatient physicians from a large health system to collect their perceptions of the usefulness of our prototype. Survey questions were responded with a 7-point Likert scale and include two open-ended questions for comments on challenges and suggestions related to electronic health record (EHR) navigation, with which a thematic analysis was performed. RESULTS: Forty-nine physicians completed the survey. The usefulness of our integrated display was rated slightly positive, and respondents did not consider it confusing. Challenges related to EHR navigation frequently reported by physicians included the need to navigate between multiple functionalities and to manually search for relevant data. The most common suggestions were related to facilitating integration of data from multiple parts of the record to facilitate data visualization and comprehension. CONCLUSION: Physicians' rating of usefulness was slightly positive, and several insights to improve EHR navigation were derived from their comments. More effective EHR navigation may be achieved through facilitating integration of data from multiple parts of the record to simplify data retrieval and synthesis.

Physicians' perceptions about narrative note sections format and content: A multi-specialty survey.

OBJECTIVE: To assess physicians' perceptions about narrative note sections format and content commonly reported in visit notes to inform future research and EHR development. METHODS: We conducted two online surveys with a multi-specialty panel of outpatient physicians from a large health system to collect their perceptions of the usefulness of three narrative formats and the relevance of content reported in the note sections history of present illness (HPI) and assessment and plan (AP). Survey questions were responded with a 7-point Likert scale and include two open-ended questions for comments on challenges and suggestions related to electronic clinical documentation. RESULTS: Eighty-eight physicians completed the surveys. The most preferred format for HPI was story (i.e., coherent paragraph), followed by list without categories (i.e., non-categorized sentences) and list with categories (i.e., categorized sentences). The most preferred format for AP was list with categories, followed by story and list without categories. The most relevant type of content in HPI was temporal information and finding/condition. The most relevant type of content reported in AP was intervention and reasons and justifications. Challenges frequently mentioned include suboptimal note creation interfaces and bloated notes, and the most common suggestions for improvements are related to note entry facilitators and organizational improvements. CONCLUSION: Physicians' input is extremely valuable to inform improvements to EHRs. More effective clinical documentation systems should include less intrusive, more intuitive and automated user interfaces for note creation, smarter autopoluation functionality and linkage between note content and data from other parts of the record.

Twilighted Homegrown Systems: The Experience of Six Traditional Electronic Health Record Developers in the Post-Meaningful Use Era.

OBJECTIVES: This study aimed to understand if and how homegrown electronic health record (EHR) systems are used in the post-Meaningful Use (MU) era according to the experience of six traditional EHR developers. METHODS: We invited informatics leaders from a convenience sample of six health care organizations that have recently replaced their long used homegrown systems with commercial EHRs. Participants were asked to complete a written questionnaire with open-ended questions designed to explore if and how their homegrown system(s) is being used and maintained after adoption of a commercial EHR. We used snowball sampling to identify other potential respondents and institutions. RESULTS: Participants from all six organizations included in our initial sample completed the questionnaire and provided referrals to four other organizations; from these, two did not respond to our invitations and two had not yet replaced their system and were excluded. Two organizations (Columbia University and University of Alabama at Birmingham) still use their homegrown system for direct patient care and as a downtime system. Four organizations (Intermountain Healthcare, Partners Healthcare, Regenstrief Institute, and Vanderbilt University) kept their systems primarily to access historical data. All organizations reported the need to continue to develop or maintain local applications despite having adopted a commercial EHR. The most common applications developed include display and visualization tools and clinical decision support systems. CONCLUSION: Homegrown EHR systems continue to be used for different purposes according to the experience of six traditional homegrown EHR developers. The annual cost to maintain these systems varies from $21,000 to over 1 million. The collective experience of these organizations indicates that commercial EHRs have not been able to provide all functionality needed for patient care and local applications are often developed for multiple purposes, which presents opportunities for future research and EHR development.

Formal representation of patients' care context data: the path to improving the electronic health record.

OBJECTIVE: To develop a collection of concept-relationship-concept tuples to formally represent patients' care context data to inform electronic health record (EHR) development. MATERIALS AND METHODS: We reviewed semantic relationships reported in the literature and developed a manual annotation schema. We used the initial schema to annotate sentences extracted from narrative note sections of cardiology, urology, and ear, nose, and throat (ENT) notes. We audio recorded ENT visits and annotated their parsed transcripts. We combined the results of each annotation into a consolidated set of concept-relationship-concept tuples. We then compared the tuples used within and across the multiple data sources. RESULTS: We annotated a total of 626 sentences. Starting with 8 relationships from the literature, we annotated 182 sentences from 8 inpatient consult notes (initial set of tuples = 43). Next, we annotated 232 sentences from 10 outpatient visit notes (enhanced set of tuples = 75). Then, we annotated 212 sentences from transcripts of 5 outpatient visits (final set of tuples = 82). The tuples from the visit transcripts covered 103 (74%) concepts documented in the notes of their respective visits. There were 20 (24%) tuples used across all data sources, 10 (12%) used only in inpatient notes, 15 (18%) used only in visit notes, and 7 (9%) used only in the visit transcripts. CONCLUSIONS: We produced a robust set of 82 tuples useful to represent patients' care context data. We propose several applications of our tuples to improve EHR navigation, data entry, learning health systems, and decision support.

Capturing Clinician Reasoning in Electronic Health Records: An Exploratory Study of Under-Treated Essential Hypertension.

Monitoring response to antihypertensive medications is a frequent reason for outpatient visits. Blood pressure (BP) is often documented as elevated, but no change in medication occurs (Medication Non-adjustment or MNA). We studied the frequency of MNA, reasons for non-adjustment, how reasons (including reasons for patient nonadherence) were documented, and whether they could be represented in a clinical care context ontology. We examined 129 visit notes with MNA occurring in 80 cases (59%). We coded MNA as Conscious Maintenance (patient adherent but clinician continues therapy for stated reason), Nonadherence (clinician attributes BP elevation to patient nonadherence), and Finding Not Addressed (clinician does not indicate reasoning for MNA). We characterized Conscious Maintenance with 11 subcodes and Nonadherence with 6 subcodes. Our ontology successfully represented relationships between concepts and reasoning, supporting the feasibility of formal representation of clinical care contexts for patient care, decision support and research.

The anatomy of clinical documentation: an assessment and classification of narrative note sections format and content.

Introduction. We systematically analyzed the most commonly used narrative note formats and content found in primary and specialty care visit notes to inform future research and electronic health record (EHR) development. Methods. We extracted data from the history of present illness (HPI) and impression and plan (IP) sections of 80 primary and specialty care visit notes. Two authors iteratively classified the format of the sections and compared the size of each section and the overall note size between primary and specialty care notes. We then annotated the content of these sections to develop a taxonomy of types of data communicated in the narrative note sections. Results. Both HPI and IP were significantly longer in primary care when compared to specialty care notes (HPI: n = 187 words, SD[130] vs. n = 119 words, SD [53]; p = 0.004 / IP: n = 270 words, SD [145] vs. n = 170 words, SD [101]; p < 0.001). Although we did not find a significant difference in the overall note size between the two groups, the proportion of HPI and IP content in relation to the total note size was significantly higher in primary care notes (40%, SD [13] vs. 28%, SD [11]; p < 0.001). We identified five combinations of format of HPI + IP sections respectively: (A) story + list with categories; (B) story + story; (C) list without categories + list with categories; (D) list with categories + list with categories; and (E) list with categories + story. HPI and IP content was significantly smaller in combination C compared to combination A (-172 words, [95% Conf. -326, -17.89]; p = 0.02). We identified seven taxa representing 45 different types of data: finding/condition documented (n = 14), intervention documented (n = 9), general descriptions and definitions (n = 7), temporal information (n = 6), reasons and justifications (n = 4), participants and settings (n = 4), and clinical documentation (n = 1). Conclusion. We identified commonly used narrative note section formats and developed a taxonomy of narrative note content to help researchers to tailor their efforts and design more efficient clinical documentation systems.

Health information technology as a learning health system: Call for a national monitoring system.

After over half a century of computer application development in medicine, the US health system has gone digital with an enthusiastic confidence for rapid improvements in care outcomes, especially those of quality of care, safety, and productivity. The bad news is that evidence for the justification of the hype around health information technology (HIT) is conflicting, and the expected benefits of a digital health system have not yet materialized. We propose a national system for monitoring HIT impact based on the paradigm of the learning health system (LHS): learning from practical experience through high-quality, ongoing monitoring of care outcomes. Our proposal aims at leveraging current de facto standard research data repositories used to support large-scale clinical studies by incorporating data needed for more robust HIT assessments and application of rigorous research designs that are now feasible on a large scale.

Using clinical reasoning ontologies to make smarter clinical decision support systems: a systematic review and data synthesis.

OBJECTIVE: The study sought to describe the literature describing clinical reasoning ontology (CRO)-based clinical decision support systems (CDSSs) and identify and classify the medical knowledge and reasoning concepts and their properties within these ontologies to guide future research. METHODS: MEDLINE, Scopus, and Google Scholar were searched through January 30, 2019, for studies describing CRO-based CDSSs. Articles that explored the development or application of CROs or terminology were selected. Eligible articles were assessed for quality features of both CDSSs and CROs to determine the current practices. We then compiled concepts and properties used within the articles. RESULTS: We included 38 CRO-based CDSSs for the analysis. Diversity of the purpose and scope of their ontologies was seen, with a variety of knowledge sources were used for ontology development. We found 126 unique medical knowledge concepts, 38 unique reasoning concepts, and 240 unique properties (137 relationships and 103 attributes). Although there is a great diversity among the terms used across CROs, there is a significant overlap based on their descriptions. Only 5 studies described high quality assessment. CONCLUSION: We identified current practices used in CRO development and provided lists of medical knowledge concepts, reasoning concepts, and properties (relationships and attributes) used by CRO-based CDSSs. CRO developers reason that the inclusion of concepts used by clinicians' during medical decision making has the potential to improve CDSS performance. However, at present, few CROs have been used for CDSSs, and high-quality studies describing CROs are sparse. Further research is required in developing high-quality CDSSs based on CROs.

Clinicians' reasoning as reflected in electronic clinical note-entry and reading/retrieval: a systematic review and qualitative synthesis.

Objective: To describe the literature exploring the use of electronic health record (EHR) systems to support creation and use of clinical documentation to guide future research. Materials and Methods: We searched databases including MEDLINE, Scopus, and CINAHL from inception to April 20, 2018, for studies applying qualitative or mixed-methods examining EHR use to support creation and use of clinical documentation. A qualitative synthesis of included studies was undertaken. Results: Twenty-three studies met the inclusion criteria and were reviewed in detail. We briefly reviewed 9 studies that did not meet the inclusion criteria but provided recommendations for EHR design. We identified 4 key themes: purposes of electronic clinical notes, clinicians' reasoning for note-entry and reading/retrieval, clinicians' strategies for note-entry, and clinicians' strategies for note-retrieval/reading. Five studies investigated note purposes and found that although patient care is the primary note purpose, non-clinical purposes have become more common. Clinicians' reasoning studies (n = 3) explored clinicians' judgement about what to document and represented clinicians' thought process in cognitive pathways. Note-entry studies (n = 6) revealed that what clinicians document is affected by EHR interfaces. Lastly, note-retrieval studies (n = 12) found that "assessment and plan" is the most read note section and what clinicians read is affected by external stimuli, care/information goals, and what they know about the patient. Conclusion: Despite the widespread adoption of EHRs, their use to support note-entry and reading/retrieval is still understudied. Further research is needed to investigate approaches to capture and represent clinicians' reasoning and improve note-entry and retrieval/reading.

Looking Behind the Curtain: Identifying Factors Contributing to Changes on Care Outcomes During a Large Commercial EHR Implementation.

OBJECTIVE: To identify factors contributing to changes on quality, productivity, and safety outcomes during a large commercial electronic health record (EHR) implementation and to guide future research. METHODS: We conducted a mixed-methods study assessing the impact of a commercial EHR implementation. The method consisted of a quantitative longitudinal evaluation followed by qualitative semi-structured, in-depth interviews with clinical employees from the same implementation. Fourteen interviews were recorded and transcribed. Three authors independently coded interview narratives and via consensus identified factors contributing to changes on 15 outcomes of quality, productivity, and safety. RESULTS: We identified 14 factors that potentially affected the outcomes previously monitored. Our findings demonstrate that several factors related to the implementation (e.g., incomplete data migration), partially related (e.g., intentional decrease in volume of work), and not related (e.g., health insurance changes) may affect outcomes in different ways. DISCUSSION: This is the first study to investigate factors contributing to changes on a broad set of quality, productivity, and safety outcomes during an EHR implementation guided by the results of a large longitudinal evaluation. The diversity of factors identified indicates that the need for organizational adaptation to take full advantage of new technologies is as important for health care as it is for other services sectors. CONCLUSIONS: We recommend continuous identification and monitoring of these factors in future evaluations to hopefully increase our understanding of the full impact of health information technology interventions.

Evaluation of a systematic methodology to detect in near real-time performance changes during electronic health record system implementations: a longitudinal study.

Introduction. Although Electronic Health Record (EHR) adoption has increased in the U.S., our understanding of how it affects health care organizations is still limited. Current literature has produced mixed-results due to the use of simple, non-standardized measurements and poor research designs. Methods. We propose the use of a systematic methodology that combines measures of quality, productivity and safety processes, tracked over time using an interrupted time-series design with multiple control sites. Results. Our methodology successfully detected performance changes during an EHR implementation on 17 (77%) outcomes, including a significant increase in Emergency Department length of stay immediately after go live by 0.19 hours [95%CI (0.12, 0.27), p<0.001], and an improvement in time to complete radiology tests, which significantly decreased per month by 0.19 minutes [95%CI (-0.26, -0.12), p<0.001]. Conclusion. The proposed methodology was able to detect several changes immediately after an EHR implementation and over time. The method is a promising and robust approach to assessing the impact of EHR implementations on a wide range of health care quality, productivity, and safety care processes.