Applying the master adaptive learner framework to just-in-time training of procedures.

Background: Just-in-time training (JITT) occurs in the clinical context when learners need immediate guidance for procedures due to a lack of proficiency or the need for knowledge refreshment. The master adaptive learner (MAL) framework presents a comprehensive model of transforming learners into adaptive experts, proficient not only in their current tasks but also in the ongoing development of lifelong skills. With the evolving landscape of procedural competence in emergency medicine (EM), trainees must develop the capacity to acquire and master new techniques consistently. This concept paper will discuss using JITT to support the development of MALs in the emergency department. Methods: In May 2023, an expert panel from the Society for Academic Emergency Medicine (SAEM) Medical Educator's Boot Camp delivered a comprehensive half-day preconference session entitled "Be the Best Teacher" at the society's annual meeting. A subgroup within this panel focused on applying the MAL framework to JITT. This subgroup collaboratively developed a practical guide that underwent iterative review and refinement. Results: The MAL-JITT framework integrates the learner's past experiences with the educator's proficiency, allowing the educational experience to address the unique requirements of each case. We outline a structured five-step process for applying JITT, utilizing the lumbar puncture procedure as an example of integrating the MAL stages of planning, learning, assessing, and adjusting. This innovative approach facilitates prompt procedural competence and cultivates a positive learning environment that fosters acquiring adaptable learning skills with enduring benefits throughout the learner's career trajectory. Conclusions: JITT for procedures holds the potential to cultivate a dynamic learning environment conducive to nurturing the development of MALs in EM.

Second- and third-year medical students' clinical encounters in the emergency department.

Background: Experiential learning theory suggests that direct clinical experiences facilitate learning. Previous literature has focused primarily on the experiences of fourth-year medical students. As more students gain early clinical exposure, it is important to understand the types of patients seen by junior students. Objectives: This study aims to categorize the clinical experiences of early (M2 and M3) students in the emergency department (ED). Methods: A retrospective review of the electronic health record of patients seen by M2s and M3s on a 2-week emergency medicine rotation at a single urban academic ED in the Midwest was performed. Data elements extracted included total number of patients seen, Emergency Severity Index (ESI), disposition, and chief complaint. Students were not mandated to see any particular patients. Results: Medical students (248) saw 2994 total patients from 2018 to 2022. The median number of patients seen by each student was 12.0 (range 1-32). Pediatric patients made up 6.5% (n = 194) of total patients. Encounters were primarily ESI 2 or 3, which accounted for 89.4% of all patients (n = 2676). The most encountered complaints were abdominal pain, chest pain, and dyspnea, making up 15.6% (n = 467), 8.7% (n = 260), and 5.5% (n = 165), respectively, of total cases. Obstetrics/gynecology, hematologic, and environmental disorders were the least frequently encountered domains. No students saw all Clerkship Directors in Emergency Medicine (CDEM)-recommended complaints. Conclusions: There is significant variability in the ED encounters of M2s and M3s, with wide ranges of patient volume and presentations. This study provides some evidence that early students may not be meeting CDEM recommendations.

Better together: A multistakeholder approach to developing specialty-wide entrustable professional activities in emergency medicine.

Purpose: Entrustable professional activities (EPAs) are a widely used framework for curriculum and assessment, yet the variability in emergency medicine (EM) training programs mandates the development of EPAs that meet the needs of the specialty as a whole. This requires eliciting and incorporating the perspectives of multiple stakeholders (i.e., faculty, residents, and patients) in the development of EPAs. Without a shared understanding of what a resident must be able to do upon graduation, we run the risk of advancing ill-prepared residents that may provide inconsistent care. Methods: In an effort to address these challenges, beginning in February 2020, the authors assembled an advisory board of 25 EM faculty to draft and reach consensus on a final list of EPAs that can be used across all training programs within the specialty of EM. Using modified Delphi methodology, the authors came to consensus on an initial list of 22 EPAs. The authors presented these EPAs to faculty supervisors, residents, and patients for refinement. The authors collated and analyzed feedback from focus groups of residents and patients using thematic analysis. The EPAs were subsequently refined based on this feedback. Results: Stakeholders in EM residency training endorsed a final revised list of 22 EPAs. Stakeholder focus groups highlighted two main thematic considerations that helped shape the finalized list of EM EPAs: attention to the meaningful nuances of EPA language and contextualizing the EPAs and viewing them developmentally. Conclusions: To foreground all key stakeholders within the EPA process for EM, the authors chose within the development process to draft; come to consensus; and refine EPAs for EM in collaboration with relevant faculty, patient, and resident stakeholders. Each stakeholder group contributed meaningfully to the content and intended implementation of the EPAs. This process may serve as a model for others in developing stakeholder-responsive EPAs.

Measuring and Predicting Faculty Consensus Rankings of Standardized Letters of Evaluation.

Background Standardized letters of evaluation (SLOE) are becoming more widely incorporated into the residency application process to make the letter of recommendation, an already critical component in a residency application packet, more objective. However, it is not currently known if the reviewers of these letters share consensus regarding the strength of an applicant determined by their SLOE. Objective We measured the level of faculty agreement regarding applicant competitiveness as determined by SLOEs and the ability of 2 algorithms to predict faculty consensus rankings. Methods Using data from the 2021-2022 Match cycle from the Council of Residency Directors in Emergency Medicine SLOE Database as a blueprint, authors created 50 fictional SLOEs representative of the national data. Seven faculty then rated these SLOEs in order of applicant competitiveness, defined as suggested rank position. Consensus was evaluated using cutoffs established a priori, and 2 prediction models, a point-based system and a linear regression model, were tested to determine their ability to predict consensus rankings. Results There was strong faculty consensus regarding the interpretation of SLOEs. Within narrow windows of agreement, faculty demonstrated similar ranking patterns with 83% and 93% agreement for "close" and "loose" agreement, respectively. Predictive models yielded a strong correlation with the consensus ranking (point-based system r=0.97, linear regression r=0.97). Conclusions Faculty displayed strong consensus regarding the competitiveness of applicants via SLOEs, adding further support to the use of SLOEs for selection and advising. Two models predicted consensus competitiveness rankings with a high degree of accuracy.

What You Didn't Learn in Residency: A Collective Curriculum for New Academic EM Faculty and Fellows.

Audience and Type of Curriculum: This curriculum is designed for emergency medicine fellows and first-year junior faculty. The curriculum covers core topics related to academic and professional success for an early career faculty member. Length of Curriculum: The curriculum is designed as quarterly sessions over the course of one academic year. Introduction: An increasing number of emergency medicine graduates are pursuing fellowship after completion of residency.1 Fellowship can be challenging as newly minted graduates begin to explore their academic niche, refine their clinical practice, and define their personal and professional spheres. We propose a structured curriculum to help guide fellows and new faculty to mitigate these challenges. Educational Goals: The aim of this curriculum is to develop relevant skills to promote academic success for fellows and first-year faculty at the start of their academic career and which could be completed during a one-year training timeline. We included topics relevant to all fellow and new faculty's expected personal and professional journey during this first year, including time management, academic productivity, resilience/wellness, and developing a national reputation. Educational Methods: The educational strategies used in this curriculum consist primarily of lecture seminars. There is one short individual activity associated with the lectures and one small group discussion. Research Methods: The course was assessed with pre- and post-test surveys following each lecture. Surveys assessed participants' reaction, learning, and behavior for each session. Evaluations were completed based on a 5-point Likert scale (1=strongly disagree, 5=strongly agree). Results: Fifteen participants attended the seminar series encompassing fellows and first-year faculty/post-fellows from ten different fellowship subspecialities. Average pre-assessment scores were low for many of the self-reported skills and confidence throughout the seminar series. Overall, participants reported increased confidence on the post-test for each of the seminar topics. In addition, participants reported that they learned new skills and planned to use the new ideas presented. All participants reported they would recommend these seminars to someone else on their same career path. Discussion: Overall, participants reported increased confidence, new skills, and plans to use the ideas presented in the seminar series. The content appears applicable to this learner set since all reported they would recommend the series to others on their career path. In conclusion, we believe our seminar series will build skills for fellows and first-year faculty which will promote academic success. Topics: Academic success, professional development, early career development.

Emergency Medicine Obstetrics and Gynecology: A Case-Based Curriculum for Residents.

Introduction: Millions of patients present to US emergency departments every year with OB/GYN concerns. Emergency medicine trainees must be adequately prepared to care for this population, regardless of how commonly they appear in the training environment. We used active learning and gamification principles in this curriculum to increase learner engagement and participation in the material. Methods: We chose OB/GYN topics based on review of Tintinalli's OB/GYN content and the American Board of Emergency Medicine's Model of Clinical Practice. Each session comprised a case-based lecture and review questions using the game-based Kahoot! online software. Pre- and postcurriculum surveys assessed residents' confidence in caring for emergent OB/GYN pathologies on a 5-point Likert scale. We designed survey questions assessing the first level of Kirkpatrick's levels of training evaluation; these questions were reviewed and revised by the department's Medical Education Scholarship Committee for validity. Results: A mean of 18 residents attended each session. Seventy-six percent of residents (26 of 34) completed the precurriculum survey, 67% (23 of 34) completed the postcurriculum survey, and 44% (15 of 34) completed both. For all respondents, mean reported confidence with curriculum topics increased from 3.5 to 4.0 (p < .05). For residents completing both surveys, confidence increased from 3.4 to 4.0 (p < .01). Discussion: Application of this curriculum significantly improved learner confidence in targeted OB/GYN topics. Future directions could include evaluating curricular impact at higher levels in the Kirkpatrick model, extending sessions to include more time for interaction, and adding suggested readings.

Deployment of Real-time Natural Language Processing and Deep Learning Clinical Decision Support in the Electronic Health Record: Pipeline Implementation for an Opioid Misuse Screener in Hospitalized Adults.

BACKGROUND: The clinical narrative in electronic health records (EHRs) carries valuable information for predictive analytics; however, its free-text form is difficult to mine and analyze for clinical decision support (CDS). Large-scale clinical natural language processing (NLP) pipelines have focused on data warehouse applications for retrospective research efforts. There remains a paucity of evidence for implementing NLP pipelines at the bedside for health care delivery. OBJECTIVE: We aimed to detail a hospital-wide, operational pipeline to implement a real-time NLP-driven CDS tool and describe a protocol for an implementation framework with a user-centered design of the CDS tool. METHODS: The pipeline integrated a previously trained open-source convolutional neural network model for screening opioid misuse that leveraged EHR notes mapped to standardized medical vocabularies in the Unified Medical Language System. A sample of 100 adult encounters were reviewed by a physician informaticist for silent testing of the deep learning algorithm before deployment. An end user interview survey was developed to examine the user acceptability of a best practice alert (BPA) to provide the screening results with recommendations. The planned implementation also included a human-centered design with user feedback on the BPA, an implementation framework with cost-effectiveness, and a noninferiority patient outcome analysis plan. RESULTS: The pipeline was a reproducible workflow with a shared pseudocode for a cloud service to ingest, process, and store clinical notes as Health Level 7 messages from a major EHR vendor in an elastic cloud computing environment. Feature engineering of the notes used an open-source NLP engine, and the features were fed into the deep learning algorithm, with the results returned as a BPA in the EHR. On-site silent testing of the deep learning algorithm demonstrated a sensitivity of 93% (95% CI 66%-99%) and specificity of 92% (95% CI 84%-96%), similar to published validation studies. Before deployment, approvals were received across hospital committees for inpatient operations. Five interviews were conducted; they informed the development of an educational flyer and further modified the BPA to exclude certain patients and allow the refusal of recommendations. The longest delay in pipeline development was because of cybersecurity approvals, especially because of the exchange of protected health information between the Microsoft (Microsoft Corp) and Epic (Epic Systems Corp) cloud vendors. In silent testing, the resultant pipeline provided a BPA to the bedside within minutes of a provider entering a note in the EHR. CONCLUSIONS: The components of the real-time NLP pipeline were detailed with open-source tools and pseudocode for other health systems to benchmark. The deployment of medical artificial intelligence systems in routine clinical care presents an important yet unfulfilled opportunity, and our protocol aimed to close the gap in the implementation of artificial intelligence-driven CDS. TRIAL REGISTRATION: ClinicalTrials.gov NCT05745480; https://www.clinicaltrials.gov/ct2/show/NCT05745480.

Optimizing efficiency of a custom clinical decision support tool improves adult congenital heart disease care.

Study objective: Improve the efficiency of an inpatient clinical decision support tool (CDS) for patients with adult congenital heart disease (ACHD). Design: The efficiency of a CDS was evaluated across two time periods and compared. Setting: An academic, tertiary care center. Participants: ACHD patients roomed in an inpatient setting. Intervention: Plan-Do-Study-Act (PDSA) methods were applied starting in 2021 and included refinement of diagnostic codes and the addition of department encounter codes. Main outcome measures: True positive and false positive CDS alerts. Results: Baseline data from 2017 had a median (IQR) of 38 (17) and 2019 baseline data had 65 (19) total alerts per month. Combining both baseline data years, the median true positive CDS alerts was 47.3 %. There were 71 (6) total alerts per month for the 2021-2022 time period and with ongoing PDSA cycles and optimization in the CDS the true positive alerts improved substantially resulting in a shifting of the median to 78.9 % within 9 months. Conclusion: CDS can efficiently notify providers when an ACHD patient is encountered. The use of ICD 10 codes alone to identify ACHD patients has limited accuracy with a high proportion of false positives. Ongoing revision of the CDS system methods is important to improving efficiency and minimizing provider alert fatigue.

From intern to "accepted": A guide to preparing for fellowship in emergency medicine.

Introduction: The number of fellowship options for emergency medicine (EM) physicians continues to expand. While guides exist to help residents explore individual fellowship pathways, we aimed to create a comprehensive guide for all residents considering fellowship. Methods: At the direction of the Society for Academic Emergency Medicine (SAEM) Board, 9 members of the Fellowship Guide Workgroup, including members of the Fellowship Approval Committee, and 2 members of SAEM Residents and Medical Students (RAMS) group collaboratively developed the guide using available evidence and expert opinion when high-quality evidence was unavailable. The guide was reviewed and approved by all members. Results: The guide offers advice to EM residents on how to conceptualize key aspects of their training with respect to preparation for fellowship, including scholarship, teaching, leadership, and electives. Additionally, it offers perspective on selecting a fellowship that matches the resident's interests and goals and successfully applying. Conclusion: This fellowship guide for EM residents considering fellowship summarizes the best currently available advice for residents considering fellowship training after residency.

From the ground up: Creating and leading fellowship programs in emergency medicine.

Background: A methodical and evidence-based approach to the creation and implementation of fellowship programs is not well described in the graduate medical education literature. The Society for Academic Emergency Medicine (SAEM) convened an expert panel to promote standardization and excellence in fellowship training. The purpose of the expert panel was to develop a fellowship guide to give prospective fellowship directors the necessary skills to successfully implement and maintain a fellowship program. Methods: Under direction of the SAEM Board of Directors, SAEM Education Committee, and SAEM Fellowship Approval Committee, a panel of content experts convened to develop a fellowship guide using an evidence-based approach and best practices content method. The resource guide was iteratively reviewed by all panel members. Results: Utilizing Kern's six-step model as a conceptual framework, the fellowship guide summarizes the construction, implementation, evaluation, and dissemination of a novel fellowship curriculum to meet the needs of trainees, educators, and sponsoring institutions. Other key areas addressed include Accreditation Council for Graduate Medical Education and nonaccredited fellowships, programmatic assessment, finances, and recruitment. Conclusions: The fellowship guide summarizes the conceptual framework, best practices, and strategies to create and implement a new fellowship program.

Documentation from trained medical students has a low rate of relative downcoding for emergency medicine encounters.

Background: Since 2018, the Centers for Medicare & Medicaid Services (CMS) guidelines have allowed teaching physicians to bill for evaluation and management services based on medical student documentation. Limited previous data suggest that medical student documentation suffers from a high rate of downcoding relative to faculty documentation. We sought to compare the coding outcomes of documentation performed by medical students, and not edited by faculty, with documentation edited and submitted by faculty. Methods: A total of 104 randomly selected notes from real patient encounters written by senior medical students were compared to the revised notes submitted by faculty. The note pairs were then split and reviewed by blinded professional coders and assigned level of service (LoS) codes 1-5 (corresponding to E&M CPT codes 99281-99285). Results: We found that the LoS agreement between student and faculty note versions was 63%, with 23% of all student notes receiving lower LoS compared to faculty notes (downcoded). This was found to be similar to baseline variability in professional coder LoS designations. Conclusions: Notes from medical students who have completed a focused documentation curriculum have less LoS downcoding than in previous reports.

Emergency Medicine Residents' "Just World" Bias Is Not Associated with a Biased Case Mix.

INTRODUCTION: Belief in a just world is the cognitive bias that "one gets what they deserve." Stronger belief in a just world for others (BJW-O) has been associated with discrimination against individuals with low socioeconomic status (SES) or poor health status, as they may be perceived to have "deserved" their situation. Emergency medicine (EM) residents have been shown to "cherry pick" patients; in this study we sought to determine whether BJW-O is associated with a biased case mix seen in residency. METHODS: We assessed EM residents on their BJW-O using a scale with previous validity evidence and behavioral correlates. We identified chief complaints that residents may associate with low SES or poor health status, including psychiatric disease, substance use disorder (SUD); and patients with multidisciplinary care plans due to frequent ED visits. We then calculated the percentage of each of these patient types seen by each resident as well as correlations and a multiple linear regression. RESULTS: 38 of 48 (79%) residents completed the BJW-O, representing 98,825 total patient encounters. The median BJW-O score was 3.25 (interquartile range 2.81-3.75). There were no significant correlations observed between BJW-O and the percentage of patients with multidisciplinary care plans who were seen, or patients with psychiatric, SUD, dental or sickle cell chief complaints seen; and a multiple linear regression showed no significant association. CONCLUSION: Higher BJW-O scores in EM residents are not significantly associated with a biased case mix of patients seen in residency.

A Model Curriculum for an Emergency Medicine Residency Rotation in Clinical Informatics.

Audience: This curriculum is designed for emergency medicine residents at all levels of training. The curriculum covers basic foundations in clinical informatics for improving patient care and outcomes, utilizing data, and leading improvements in emergency medicine. Length of Curriculum: The curriculum is designed for a four-week rotation. Introduction: The American College of Graduate Medical Education (ACGME) mandated that all Emergency Medicine (EM) residents receive specific training in the use of information technology.1,2 To our knowledge, a clinical informatics curriculum for EM residents does not exist. We propose the following standardized and reproducible educational curriculum for EM residents. Educational Goals: The aim of this curriculum is to teach informatics skills to emergency physicians to improve patient care and outcomes, utilize data, and develop projects to lead change.3 These goals will be achieved by providing a foundational informatics elective for EM residents that follows the delineation of practice for Clinical Informatics outlined by the American Medical Informatics Association (AMIA) and the American Board of Preventive Medicine (ABPM).4-6. Educational Methods: The educational strategies used in this curriculum include asynchronous learning via books, papers, videos, and websites. Residents attend administrative sessions (meetings), develop a project proposal, and participate in small group discussions.The rotation emphasizes the basic concepts surrounding clinical informatics with an emphasis on improving care delivery and outcomes, information systems, data governance and analytics, as well as leadership and professionalism. The course focuses on the practical application of these concepts, including implementation, clinical decision support, workflow analysis, privacy and security, information technology across the patient care continuum, health information exchange, data analytics, and leading change through stakeholder engagement. Research Methods: An initial version of the curriculum was introduced to two separate institutions and was completed by three rotating resident physicians and one rotating resident pharmacist. A brief course evaluation as well as qualitative feedback was solicited from elective participants by the course director, via email following the completion of the course, regarding the effectiveness of the course content. Learner feedback was used to influence the development of this complete curriculum. Results: The curriculum was graded by learners on a 5-point Likert scale (1=strongly disagree, 5 = strongly agree). The mean response to, "This course was a valuable use of my elective time," was 5 (sd=0). The mean response to, "I achieved the learning objectives," and "This rotation helped me understand Clinical Informatics," were both 4.75 (sd=0.5). Discussion: Overall, participants reported that the content was effective for achieving the learning objectives. During initial implementation, we found that the preliminary asynchronous learning component worked less effectively than we anticipated due to a lower volume of content. In response to this, as well as resident feedback, we added significantly more educational content.In conclusion, this model curriculum provides a structured process for an informatics rotation for the emergency medicine resident that utilizes the core competencies established by the governing bodies of the clinical informatics specialty and ACGME. Topics: Clinical informatics key concepts, including definitions, fundamental terminology, history, policy and regulations, ethical considerations, clinical decision support, health information systems, data governance and analytics, process improvement, stakeholder engagement and change management.

Number of Patient Encounters in Emergency Medicine Residency Does Not Correlate with In-Training Exam Domain Scores.

INTRODUCTION: Emergency medicine (EM) residents take the American Board of Emergency Medicine (ABEM) In-Training Examination (ITE) every year. This examination is based on the ABEM Model of Clinical Practice (Model). The purpose of this study was to determine whether a relationship exists between the number of patient encounters a resident sees within a specific clinical domain and their ITE performance on questions that are related to that domain. METHODS: Chief complaint data for each patient encounter was taken from the electronic health record for EM residents graduating in three consecutive years between 2016-2021. We excluded patient encounters without an assigned resident or a listed chief complaint. Chief complaints were then categorized into one of 20 domains based on the 2016 Model. We calculated correlations between the total number of encounters seen by a resident for all clinical years and their ITE performance for the corresponding clinical domain from their third year of training. RESULTS: Available for analysis were a total of 232,625 patient encounters and 69 eligible residents who treated the patients. We found no statistically significant correlations following Bonferroni correction for multiple analyses. CONCLUSION: There was no correlation between the number of patient encounters a resident has within a clinical domain and their ITE performance on questions corresponding to that domain. This suggests the need for separate but parallel educational missions to achieve success in both the clinical environment and standardized testing.

Changing Medical Education When Change Is Hard: Implementing an Interdepartmental Entrustable Professional Activity.

BACKGROUND: Entrustable professional activities (EPAs) are a new tool for assessing learners that represents a significant movement in graduate medical education (GME) toward competency-based assessment and serves as a bridge between milestones and clinical practice. Whenever a major change is implemented to any system, resistance to change is expected. Many change management models have been proposed to overcome this resistance; a newer model is outlined in the book Switch. The objective was to describe the change management principles used to institute an EPA. METHODS: The model introduced in Switch was used as a framework for implementing a trauma resuscitation EPA in a joint effort between the departments of surgery and emergency medicine (EM) at the University of Wisconsin Hospitals and Clinics. The Department of Emergency Medicine used the principles of change management, completing 295 faculty evaluations of trauma resuscitations compared to the Department of Surgery, whose faculty completed 50 evaluations between the study period of July 2018 through October 2019. A survey completed winter 2019 of EM faculty was used to determine the most important principles toward successful implementation. RESULTS: Twenty-seven of 35 (78% response rate) of EM faculty identified key concepts from Switch as being instrumental in the successful implementation. Internal discussion of the implementation approach used by the Department of Surgery identified several limitations that would have been overcome by using these same change management principles. CONCLUSION: We conclude that the principles of change management provide a useful framework for successfully implementing EPAs into GME.