Enhancing the value to users of machine learning-based clinical decision support tools: A framework for iterative, collaborative development and implementation.

BACKGROUND: Health care organizations are integrating a variety of machine learning (ML)-based clinical decision support (CDS) tools into their operations, but practitioners lack clear guidance regarding how to implement these tools so that they assist end users in their work. PURPOSE: We designed this study to identify how health care organizations can facilitate collaborative development of ML-based CDS tools to enhance their value for health care delivery in real-world settings. METHODOLOGY/APPROACH: We utilized qualitative methods, including 37 interviews in a large, multispecialty health system that developed and implemented two operational ML-based CDS tools in two of its hospital sites. We performed thematic analyses to inform presentation of an explanatory framework and recommendations. RESULTS: We found that ML-based CDS tool development and implementation into clinical workflows proceeded in four phases: iterative solution coidentification, iterative coengagement, iterative coapplication, and iterative corefinement. Each phase is characterized by a collaborative back-and-forth process between the technology's developers and users, through which both users' activities and the technology itself are transformed. CONCLUSION: Health care organizations that anticipate iterative collaboration to be an integral aspect of their ML-based CDS tools' development and implementation process may have more success in deploying ML-based CDS tools that assist end users in their work than organizations that expect a traditional technology innovation process. PRACTICE IMPLICATIONS: Managers developing and implementing ML-based CDS tools should frame the work as a collaborative learning opportunity for both users and the technology itself and should solicit constructive feedback from users on potential changes to the technology, in addition to potential changes to user workflows, in an ongoing, iterative manner.

A Pilot Study to Address Tolerance of Uncertainty Among Family Medicine Residents.

BACKGROUND AND OBJECTIVES: Because patients often present to their family physicians with undifferentiated medical problems, uncertainty is common. Family medicine residents must manage both the ambiguity inherent in the field as well as the very real uncertainty of learning to become a skilled physician with little experience to serve as a guide. The purpose of this analysis was to assess the impact of a new curriculum on family medicine residents' tolerance of ambiguity. METHODS: We conducted an exploratory quasi-experimental study to assess the impact of a novel curriculum designed to improve family medicine residents' tolerance of ambiguity. Four different surveys were administered to 25 family medicine residents at different stages in their training prior to and immediately and 6 months after the new curriculum. RESULTS: Although many constructs remained unchanged with the intervention, one important construct, namely perceived threats of ambiguity, showed significant and sustained improvement relative to before undertaking this curriculum (score of 26.2 prior to the intervention, 22.1 immediately after, and 22.0 6 months after the intervention). CONCLUSIONS: A new curriculum designed to improve tolerance to ambiguity appears to reduce the perceived threats of ambiguity in this small exploratory study.

The Electronic Health Record Objective Structured Clinical Examination: Assessing Student Competency in Patient Interactions While Using the Electronic Health Record.

PROBLEM: Because many medical students do not have access to electronic health records (EHRs) in the clinical environment, simulated EHR training is necessary. Explicitly training medical students to use EHRs appropriately during patient encounters equips them to engage patients while also attending to the accuracy of the record and contributing to a culture of information safety. APPROACH: Faculty developed and successfully implemented an EHR objective structured clinical examination (EHR-OSCE) for clerkship students at two institutions. The EHR-OSCE objectives include assessing EHR-related communication and data management skills. OUTCOMES: The authors collected performance data for students (n = 71) at the first institution during academic years 2011-2013 and for students (n = 211) at the second institution during academic year 2013-2014. EHR-OSCE assessment checklist scores showed that students performed well in EHR-related communication tasks, such as maintaining eye contact and stopping all computer work when the patient expresses worry. Findings indicated student EHR skill deficiencies in the areas of EHR data management including medical history review, medication reconciliation, and allergy reconciliation. Most students' EHR skills failed to improve as the year progressed, suggesting that they did not gain the EHR training and experience they need in clinics and hospitals. NEXT STEPS: Cross-institutional data comparisons will help determine whether differences in curricula affect students' EHR skills. National and institutional policies and faculty development are needed to ensure that students receive adequate EHR education, including hands-on experience in the clinic as well as simulated EHR practice.