Active Learning Pipeline to Identify Candidate Terms for a CDSS Ontology.

Ontology is essential for achieving health information and information technology application interoperability in the biomedical fields and beyond. Traditionally, ontology construction is carried out manually by human domain experts (HDE). Here, we explore an active learning approach to automatically identify candidate terms from publications, with manual verification later as a part of a deep learning model training and learning process. We introduce the overall architecture of the active learning pipeline and present some preliminary results. This work is a critical and complementary component in addition to manually building the ontology, especially during the long-term maintenance stage.

Patient-centered clinical decision support challenges and opportunities identified from workflow execution models.

OBJECTIVE: To use workflow execution models to highlight new considerations for patient-centered clinical decision support policies (PC CDS), processes, procedures, technology, and expertise required to support new workflows. METHODS: To generate and refine models, we used (1) targeted literature reviews; (2) key informant interviews with 6 external PC CDS experts; (3) model refinement based on authors' experience; and (4) validation of the models by a 26-member steering committee. RESULTS AND DISCUSSION: We identified 7 major issues that provide significant challenges and opportunities for healthcare systems, researchers, administrators, and health IT and app developers. Overcoming these challenges presents opportunities for new or modified policies, processes, procedures, technology, and expertise to: (1) Ensure patient-generated health data (PGHD), including patient-reported outcomes (PROs), are documented, reviewed, and managed by appropriately trained clinicians, between visits and after regular working hours. (2) Educate patients to use connected medical devices and handle technical issues. (3) Facilitate collection and incorporation of PGHD, PROs, patient preferences, and social determinants of health into existing electronic health records. (4) Troubleshoot erroneous data received from devices. (5) Develop dashboards to display longitudinal patient-reported data. (6) Provide reimbursement to support new models of care. (7) Support patient engagement with remote devices. CONCLUSION: Several new policies, processes, technologies, and expertise are required to ensure safe and effective implementation and use of PC CDS. As we gain more experience implementing and working with PC CDS, we should be able to begin realizing the long-term positive impact on patient health that the patient-centered movement in healthcare promises.

Implementation of a health information technology safety classification system in the Veterans Health Administration's Informatics Patient Safety Office.

OBJECTIVE: Implement the 5-type health information technology (HIT) patient safety concern classification system for HIT patient safety issues reported to the Veterans Health Administration's Informatics Patient Safety Office. MATERIALS AND METHODS: A team of informatics safety analysts retrospectively classified 1 year of HIT patient safety issues by type of HIT patient safety concern using consensus discussions. The processes established during retrospective classification were then applied to incoming HIT safety issues moving forward. RESULTS: Of 140 issues retrospectively reviewed, 124 met the classification criteria. The majority were HIT failures (eg, software defects) (33.1%) or configuration and implementation problems (29.8%). Unmet user needs and external system interactions accounted for 20.2% and 10.5%, respectively. Absence of HIT safety features accounted for 2.4% of issues, and 4% did not have enough information to classify. CONCLUSION: The 5-type HIT safety concern classification framework generated actionable categories helping organizations effectively respond to HIT patient safety risks.

Leveraging explainable artificial intelligence to optimize clinical decision support.

OBJECTIVE: To develop and evaluate a data-driven process to generate suggestions for improving alert criteria using explainable artificial intelligence (XAI) approaches. METHODS: We extracted data on alerts generated from January 1, 2019 to December 31, 2020, at Vanderbilt University Medical Center. We developed machine learning models to predict user responses to alerts. We applied XAI techniques to generate global explanations and local explanations. We evaluated the generated suggestions by comparing with alert's historical change logs and stakeholder interviews. Suggestions that either matched (or partially matched) changes already made to the alert or were considered clinically correct were classified as helpful. RESULTS: The final dataset included 2 991 823 firings with 2689 features. Among the 5 machine learning models, the LightGBM model achieved the highest Area under the ROC Curve: 0.919 [0.918, 0.920]. We identified 96 helpful suggestions. A total of 278 807 firings (9.3%) could have been eliminated. Some of the suggestions also revealed workflow and education issues. CONCLUSION: We developed a data-driven process to generate suggestions for improving alert criteria using XAI techniques. Our approach could identify improvements regarding clinical decision support (CDS) that might be overlooked or delayed in manual reviews. It also unveils a secondary purpose for the XAI: to improve quality by discovering scenarios where CDS alerts are not accepted due to workflow, education, or staffing issues.

A lifecycle framework illustrates eight stages necessary for realizing the benefits of patient-centered clinical decision support.

The design, development, implementation, use, and evaluation of high-quality, patient-centered clinical decision support (PC CDS) is necessary if we are to achieve the quintuple aim in healthcare. We developed a PC CDS lifecycle framework to promote a common understanding and language for communication among researchers, patients, clinicians, and policymakers. The framework puts the patient, and/or their caregiver at the center and illustrates how they are involved in all the following stages: Computable Clinical Knowledge, Patient-specific Inference, Information Delivery, Clinical Decision, Patient Behaviors, Health Outcomes, Aggregate Data, and patient-centered outcomes research (PCOR) Evidence. Using this idealized framework reminds key stakeholders that developing, deploying, and evaluating PC-CDS is a complex, sociotechnical challenge that requires consideration of all 8 stages. In addition, we need to ensure that patients, their caregivers, and the clinicians caring for them are explicitly involved at each stage to help us achieve the quintuple aim.

A guide to mitigating audit log-related risk in medical professional liability cases.

Following the American Recovery and Reinvestment Act in 2009, use of electronic health records (EHRs) has become ubiquitous. Accordingly, one should expect most medical professional liability cases to involve review of patient records produced from EHRs. When questions arise regarding who was involved in care of a patient, what they knew and when, or the meaning, completeness, integrity, validity, timeliness, confidentiality, accuracy, or legitimacy of data, or ways that the EHR's user interface or automated clinical decision support tools may have contributed to the alleged events, one often turns to the EHR and its audit log. This manuscript discusses lines of defense incorporated into the design, development, implementation, and use of EHRs to ensure their integrity and the types of EHR transaction logs (e.g., audit log) that exist. Using these logs can help one answer questions that often arise in medical malpractice cases. Finally, there are "best practices" surrounding EHR audit logs that health care organizations should implement. When used appropriately, EHRs and their audit logs provide another source of information to help hospital risk managers, legal counsel, and EHR expert witnesses to investigate adverse incidents and, if needed, prosecute or defend clinicians and/or health care organizations involved in the patient's care.

Keyphrase Identification Using Minimal Labeled Data with Hierarchical Context and Transfer Learning.

Interoperable clinical decision support system (CDSS) rules provide a pathway to interoperability, a well-recognized challenge in health information technology. Building an ontology facilitates creating interoperable CDSS rules, which can be achieved by identifying the keyphrases (KP) from the existing literature. However, KP identification for data labeling requires human expertise, consensus, and contextual understanding. This paper aims to present a semi-supervised KP identification framework using minimal labeled data based on hierarchical attention over the documents and domain adaptation. Our method outperforms the prior neural architectures by learning through synthetic labels for initial training, document-level contextual learning, language modeling, and fine-tuning with limited gold standard label data. To the best of our knowledge, this is the first functional framework for the CDSS sub-domain to identify KPs, which is trained on limited labeled data. It contributes to the general natural language processing (NLP) architectures in areas such as clinical NLP, where manual data labeling is challenging, and light-weighted deep learning models play a role in real-time KP identification as a complementary approach to human experts' effort.

Developing electronic clinical quality measures to assess the cancer diagnostic process.

OBJECTIVE: Measures of diagnostic performance in cancer are underdeveloped. Electronic clinical quality measures (eCQMs) to assess quality of cancer diagnosis could help quantify and improve diagnostic performance. MATERIALS AND METHODS: We developed 2 eCQMs to assess diagnostic evaluation of red-flag clinical findings for colorectal (CRC; based on abnormal stool-based cancer screening tests or labs suggestive of iron deficiency anemia) and lung (abnormal chest imaging) cancer. The 2 eCQMs quantified rates of red-flag follow-up in CRC and lung cancer using electronic health record data repositories at 2 large healthcare systems. Each measure used clinical data to identify abnormal results, evidence of appropriate follow-up, and exclusions that signified follow-up was unnecessary. Clinicians reviewed 100 positive and 20 negative randomly selected records for each eCQM at each site to validate accuracy and categorized missed opportunities related to system, provider, or patient factors. RESULTS: We implemented the CRC eCQM at both sites, while the lung cancer eCQM was only implemented at the VA due to lack of structured data indicating level of cancer suspicion on most chest imaging results at Geisinger. For the CRC eCQM, the rate of appropriate follow-up was 36.0% (26 746/74 314 patients) in the VA after removing clinical exclusions and 41.1% at Geisinger (1009/2461 patients; P < .001). Similarly, the rate of appropriate evaluation for lung cancer in the VA was 61.5% (25 166/40 924 patients). Reviewers most frequently attributed missed opportunities at both sites to provider factors (84 of 157). CONCLUSIONS: We implemented 2 eCQMs to evaluate the diagnostic process in cancer at 2 large health systems. Health care organizations can use these eCQMs to monitor diagnostic performance related to cancer.

Using AI-generated suggestions from ChatGPT to optimize clinical decision support.

OBJECTIVE: To determine if ChatGPT can generate useful suggestions for improving clinical decision support (CDS) logic and to assess noninferiority compared to human-generated suggestions. METHODS: We supplied summaries of CDS logic to ChatGPT, an artificial intelligence (AI) tool for question answering that uses a large language model, and asked it to generate suggestions. We asked human clinician reviewers to review the AI-generated suggestions as well as human-generated suggestions for improving the same CDS alerts, and rate the suggestions for their usefulness, acceptance, relevance, understanding, workflow, bias, inversion, and redundancy. RESULTS: Five clinicians analyzed 36 AI-generated suggestions and 29 human-generated suggestions for 7 alerts. Of the 20 suggestions that scored highest in the survey, 9 were generated by ChatGPT. The suggestions generated by AI were found to offer unique perspectives and were evaluated as highly understandable and relevant, with moderate usefulness, low acceptance, bias, inversion, redundancy. CONCLUSION: AI-generated suggestions could be an important complementary part of optimizing CDS alerts, can identify potential improvements to alert logic and support their implementation, and may even be able to assist experts in formulating their own suggestions for CDS improvement. ChatGPT shows great potential for using large language models and reinforcement learning from human feedback to improve CDS alert logic and potentially other medical areas involving complex, clinical logic, a key step in the development of an advanced learning health system.

Assessing the Value of ChatGPT for Clinical Decision Support Optimization.

Objective: To determine if ChatGPT can generate useful suggestions for improving clinical decision support (CDS) logic and to assess noninferiority compared to human-generated suggestions. Methods: We supplied summaries of CDS logic to ChatGPT, an artificial intelligence (AI) tool for question answering that uses a large language model, and asked it to generate suggestions. We asked human clinician reviewers to review the AI-generated suggestions as well as human-generated suggestions for improving the same CDS alerts, and rate the suggestions for their usefulness, acceptance, relevance, understanding, workflow, bias, inversion, and redundancy. Results: Five clinicians analyzed 36 AI-generated suggestions and 29 human-generated suggestions for 7 alerts. Of the 20 suggestions that scored highest in the survey, 9 were generated by ChatGPT. The suggestions generated by AI were found to offer unique perspectives and were evaluated as highly understandable and relevant, with moderate usefulness, low acceptance, bias, inversion, redundancy. Conclusion: AI-generated suggestions could be an important complementary part of optimizing CDS alerts, can identify potential improvements to alert logic and support their implementation, and may even be able to assist experts in formulating their own suggestions for CDS improvement. ChatGPT shows great potential for using large language models and reinforcement learning from human feedback to improve CDS alert logic and potentially other medical areas involving complex, clinical logic, a key step in the development of an advanced learning health system.

A multi-site randomized trial of a clinical decision support intervention to improve problem list completeness.

OBJECTIVE: To improve problem list documentation and care quality. MATERIALS AND METHODS: We developed algorithms to infer clinical problems a patient has that are not recorded on the coded problem list using structured data in the electronic health record (EHR) for 12 clinically significant heart, lung, and blood diseases. We also developed a clinical decision support (CDS) intervention which suggests adding missing problems to the problem list. We evaluated the intervention at 4 diverse healthcare systems using 3 different EHRs in a randomized trial using 3 predetermined outcome measures: alert acceptance, problem addition, and National Committee for Quality Assurance Healthcare Effectiveness Data and Information Set (NCQA HEDIS) clinical quality measures. RESULTS: There were 288 832 opportunities to add a problem in the intervention arm and the problem was added 63 777 times (acceptance rate 22.1%). The intervention arm had 4.6 times as many problems added as the control arm. There were no significant differences in any of the clinical quality measures. DISCUSSION: The CDS intervention was highly effective at improving problem list completeness. However, the improvement in problem list utilization was not associated with improvement in the quality measures. The lack of effect on quality measures suggests that problem list documentation is not directly associated with improvements in quality measured by National Committee for Quality Assurance Healthcare Effectiveness Data and Information Set (NCQA HEDIS) quality measures. However, improved problem list accuracy has other benefits, including clinical care, patient comprehension of health conditions, accurate CDS and population health, and for research. CONCLUSION: An EHR-embedded CDS intervention was effective at improving problem list completeness but was not associated with improvement in quality measures.

Identifying a Clinical Informatics or Electronic Health Record Expert Witness for Medical Professional Liability Cases.

BACKGROUND: The health care field is experiencing widespread electronic health record (EHR) adoption. New medical professional liability (i.e., malpractice) cases will likely involve the review of data extracted from EHRs as well as EHR workflows, audit logs, and even the potential role of the EHR in causing harm. OBJECTIVES: Reviewing printed versions of a patient's EHRs can be difficult due to differences in printed versus on-screen presentations, redundancies, and the way printouts are often grouped by document or information type rather than chronologically. Simply recreating an accurate timeline often requires experts with training and experience in designing, developing, using, and reviewing EHRs and audit logs. Additional expertise is required if questions arise about data's meaning, completeness, accuracy, and timeliness or ways that the EHR's user interface or automated clinical decision support tools may have contributed to alleged events. Such experts often come from the sociotechnical field of clinical informatics that studies the design, development, implementation, use, and evaluation of information and communications technology, specifically, EHRs. Identifying well-qualified EHR experts to aid a legal team is challenging. METHODS: Based on literature review and experience reviewing cases, we identified seven criteria to help in this assessment. RESULTS: The criteria are education in clinical informatics; clinical informatics knowledge; experience with EHR design, development, implementation, and use; communication skills; academic publications on clinical informatics; clinical informatics certification; and membership in informatics-related professional organizations. CONCLUSION: While none of these criteria are essential, understanding the breadth and depth of an individual's qualifications in each of these areas can help identify a high-quality, clinical informatics expert witness.

Computer clinical decision support that automates personalized clinical care: a challenging but needed healthcare delivery strategy.

How to deliver best care in various clinical settings remains a vexing problem. All pertinent healthcare-related questions have not, cannot, and will not be addressable with costly time- and resource-consuming controlled clinical trials. At present, evidence-based guidelines can address only a small fraction of the types of care that clinicians deliver. Furthermore, underserved areas rarely can access state-of-the-art evidence-based guidelines in real-time, and often lack the wherewithal to implement advanced guidelines. Care providers in such settings frequently do not have sufficient training to undertake advanced guideline implementation. Nevertheless, in advanced modern healthcare delivery environments, use of eActions (validated clinical decision support systems) could help overcome the cognitive limitations of overburdened clinicians. Widespread use of eActions will require surmounting current healthcare technical and cultural barriers and installing clinical evidence/data curation systems. The authors expect that increased numbers of evidence-based guidelines will result from future comparative effectiveness clinical research carried out during routine healthcare delivery within learning healthcare systems.

Clinical decision support malfunctions related to medication routes: a case series.

OBJECTIVE: To identify common medication route-related causes of clinical decision support (CDS) malfunctions and best practices for avoiding them. MATERIALS AND METHODS: Case series of medication route-related CDS malfunctions from diverse healthcare provider organizations. RESULTS: Nine cases were identified and described, including both false-positive and false-negative alert scenarios. A common cause was the inclusion of nonsystemically available medication routes in value sets (eg, eye drops, ear drops, or topical preparations) when only systemically available routes were appropriate. DISCUSSION: These value set errors are common, occur across healthcare provider organizations and electronic health record (EHR) systems, affect many different types of medications, and can impact the accuracy of CDS interventions. New knowledge management tools and processes for auditing existing value sets and supporting the creation of new value sets can mitigate many of these issues. Furthermore, value set issues can adversely affect other aspects of the EHR, such as quality reporting and population health management. CONCLUSION: Value set issues related to medication routes are widespread and can lead to CDS malfunctions. Organizations should make appropriate investments in knowledge management tools and strategies, such as those outlined in our recommendations.

i-CLIMATE: a "clinical climate informatics" action framework to reduce environmental pollution from healthcare.

Addressing environmental pollution and climate change is one of the biggest sociotechnical challenges of our time. While information technology has led to improvements in healthcare, it has also contributed to increased energy usage, destructive natural resource extraction, piles of e-waste, and increased greenhouse gases. We introduce a framework "Information technology-enabled Clinical cLimate InforMAtics acTions for the Environment" (i-CLIMATE) to illustrate how clinical informatics can help reduce healthcare's environmental pollution and climate-related impacts using 5 actionable components: (1) create a circular economy for health IT, (2) reduce energy consumption through smarter use of health IT, (3) support more environmentally friendly decision-making by clinicians and health administrators, (4) mobilize healthcare workforce environmental stewardship through informatics, and (5) Inform policies and regulations for change. We define Clinical Climate Informatics as a field that applies data, information, and knowledge management principles to operationalize components of the i-CLIMATE Framework.

Translating electronic health record-based patient safety algorithms from research to clinical practice at multiple sites.

INTRODUCTION: Researchers are increasingly developing algorithms that impact patient care, but algorithms must also be implemented in practice to improve quality and safety. OBJECTIVE: We worked with clinical operations personnel at two US health systems to implement algorithms to proactively identify patients without timely follow-up of abnormal test results that warrant diagnostic evaluation for colorectal or lung cancer. We summarise the steps involved and lessons learned. METHODS: Twelve sites were involved across two health systems. Implementation involved extensive software documentation, frequent communication with sites and local validation of results. Additionally, we used automated edits of existing code to adapt it to sites' local contexts. RESULTS: All sites successfully implemented the algorithms. Automated edits saved sites significant work in direct code modification. Documentation and communication of changes further aided sites in implementation. CONCLUSION: Patient safety algorithms developed in research projects were implemented at multiple sites to monitor for missed diagnostic opportunities. Automated algorithm translation procedures can produce more consistent results across sites.

The Technology Landscape of Patient-Centered Clinical Decision Support - Where Are We and What Is Needed?

Patient Centered Outcomes Research (PCOR) and health care delivery system transformation require investments in development of tools and techniques for rapid dissemination of clinical and operational best practices. This paper explores the current technology landscape for patient-centered clinical decision support (PC CDS) and what is needed to make it more shareable, standards-based, and publicly available with the goal of improving patient care and clinical outcomes. The landscape assessment used three sources of information: (1) a 22-member technical expert panel; (2) a literature review of peer-reviewed and grey literature; and (3) key informant interviews with PC CDS stakeholders. We identified ten salient technical considerations that span all phases of PC CDS development; our findings suggest there has been significant progress in the development and implementation of PC CDS but challenges remain.

Reporting Outcomes of Pediatric Intensive Care Unit Patients to Referring Physicians via an Electronic Health Record-Based Feedback System.

BACKGROUND: Many critically ill children are initially evaluated in front-line settings by clinicians with variable pediatric training before they are transferred to a pediatric intensive care unit (PICU). Because clinicians learn from past performance, communicating outcomes of patients back to front-line clinicians who provide pediatric emergency care could be valuable; however, referring clinicians do not consistently receive this important feedback. OBJECTIVES: Our aim was to determine the feasibility, usability, and clinical relevance of a semiautomated electronic health record (EHR)-supported system developed at a single institution to deliver timely and relevant PICU patient outcome feedback to referring emergency department (ED) physicians. METHODS: Guided by the Health Information Technology Safety Framework, we iteratively designed, implemented, and evaluated a semiautomated electronic feedback system leveraging the EHR in one institution. After conducting interviews and focus groups with stakeholders to understand the PICU-ED health care work system, we designed the EHR-supported feedback system by translating stakeholder, organizational, and usability objectives into feedback process and report requirements. Over 6 months, we completed three cycles of implementation and evaluation, wherein we analyzed EHR access logs, reviewed feedback reports sent, performed usability testing, and conducted physician interviews to determine the system's feasibility, usability, and clinical relevance. RESULTS: The EHR-supported feedback process is feasible with timely delivery and receipt of feedback reports. Usability testing revealed excellent Systems Usability Scale scores. According to physicians, the process was well-integrated into their clinical workflows and conferred minimal additional workload. Physicians also indicated that delivering and receiving consistent feedback was relevant to their clinical practice. CONCLUSION: An EHR-supported system to deliver timely and relevant PICU patient outcome feedback to referring ED physicians was feasible, usable, and important to physicians. Future work is needed to evaluate impact on clinical practice and patient outcomes and to investigate applicability to other clinical settings involved in similar care transitions.