Precision medicine within health professions education: Defining a research agenda for emergency medicine using a foresight and strategy technique (FaST) review.

Background: Precision medicine, sometimes referred to as personalized medicine, is rapidly changing the possibilities for how people will engage health care in the near future. As technology to support precision medicine exponentially develops, there is an urgent need to proactively improve our understanding of precision medicine and pose important research questions (RQs) related to its inclusion in the education and training of future emergency physicians. Methods: A seven-step process was employed to develop a research agenda exploring the intersection of precision and emergency medicine education/training. A literature search of articles about precision medicine was conducted first, which informed the creation of future four scenarios in which trainees and practicing physicians regularly discuss and incorporate precision medicine tools into their discussions and work. Based on these futurist narratives, potential education RQs were generated by an expert panel. A total of 59 initial questions were subsequently categorized and refined to a priority list through a nominal group voting method. The top/priority questions were presented at the 2023 SAEM Consensus Conference on Precision Medicine, Austin, Texas, for further input. Results: Eight high-value education RQs were developed, reflecting a holistic view of the challenges and opportunities for precision medicine education in the knowledge, skills, and attitudes relevant to emergency medicine. These questions contend with topics such as most effective pedagogical methods; intended resulting outcomes and behaviors; the generational differences between practicing emergency physicians, educators, and future trainees; and the desires and expectations of patients. Conclusions: Emergency medicine and emergency physicians must be prepared to understand precision medicine and incorporate this information into their "toolbox" of thinking, problem solving, and communication with patients and colleagues. This research agenda on how best to educate future emergency physicians in the use of personalized data to provide optimal health care is the focus of this article.

Development of entrustable professional activities for emergency medicine medical education fellowships: A modified Delphi study.

Background: It is essential that medical education (MedEd) fellows achieve desired outcomes prior to graduation. Despite the increase in postgraduate MedEd fellowships in emergency medicine (EM), there is no consistently applied competency framework. We sought to develop entrustable professional activities (EPAs) for EM MedEd fellows. Methods: From 2021 to 2022, we used a modified Delphi method to achieve consensus for EPAs. EM education experts generated an initial list of 173 EPAs after literature review. In each Delphi round, panelists were asked to make a binary choice of whether to include the EPA. We determined an inclusion threshold of 70% agreement a priori. After the first round, given the large number of EPAs meeting inclusion threshold, panelists were instructed to vote whether each EPA should be included in the "20 most important" EPAs for a MedEd fellowship. Modifications were made between rounds based on expert feedback. We calculated descriptive statistics. Results: Seventeen experts completed four Delphi rounds each with 100% response. After Round 1, 87 EPAs were eliminated and two were combined. Following Round 2, 46 EPAs were eliminated, seven were combined, and three were included in the final list. After the third round, one EPA was eliminated and 13 were included. After the fourth round, 11 EPAs were eliminated. The final list consisted of 16 EPAs in domains of career development, education theory and methods, research and scholarship, and educational program administration. Conclusions: We developed a list of 16 EPAs for EM MedEd fellowships, the first step in implementing competency-based MedEd.

Better together: A multistakeholder approach to developing specialty-wide entrustable professional activities in emergency medicine.

Purpose: Entrustable professional activities (EPAs) are a widely used framework for curriculum and assessment, yet the variability in emergency medicine (EM) training programs mandates the development of EPAs that meet the needs of the specialty as a whole. This requires eliciting and incorporating the perspectives of multiple stakeholders (i.e., faculty, residents, and patients) in the development of EPAs. Without a shared understanding of what a resident must be able to do upon graduation, we run the risk of advancing ill-prepared residents that may provide inconsistent care. Methods: In an effort to address these challenges, beginning in February 2020, the authors assembled an advisory board of 25 EM faculty to draft and reach consensus on a final list of EPAs that can be used across all training programs within the specialty of EM. Using modified Delphi methodology, the authors came to consensus on an initial list of 22 EPAs. The authors presented these EPAs to faculty supervisors, residents, and patients for refinement. The authors collated and analyzed feedback from focus groups of residents and patients using thematic analysis. The EPAs were subsequently refined based on this feedback. Results: Stakeholders in EM residency training endorsed a final revised list of 22 EPAs. Stakeholder focus groups highlighted two main thematic considerations that helped shape the finalized list of EM EPAs: attention to the meaningful nuances of EPA language and contextualizing the EPAs and viewing them developmentally. Conclusions: To foreground all key stakeholders within the EPA process for EM, the authors chose within the development process to draft; come to consensus; and refine EPAs for EM in collaboration with relevant faculty, patient, and resident stakeholders. Each stakeholder group contributed meaningfully to the content and intended implementation of the EPAs. This process may serve as a model for others in developing stakeholder-responsive EPAs.

The Next Era of Assessment: Building a Trustworthy Assessment System.

Assessment in medical education has evolved through a sequence of eras each centering on distinct views and values. These eras include measurement (e.g., knowledge exams, objective structured clinical examinations), then judgments (e.g., workplace-based assessments, entrustable professional activities), and most recently systems or programmatic assessment, where over time multiple types and sources of data are collected and combined by competency committees to ensure individual learners are ready to progress to the next stage in their training. Significantly less attention has been paid to the social context of assessment, which has led to an overall erosion of trust in assessment by a variety of stakeholders including learners and frontline assessors. To meaningfully move forward, the authors assert that the reestablishment of trust should be foundational to the next era of assessment. In our actions and interventions, it is imperative that medical education leaders address and build trust in assessment at a systems level. To that end, the authors first review tenets on the social contextualization of assessment and its linkage to trust and discuss consequences should the current state of low trust continue. The authors then posit that trusting and trustworthy relationships can exist at individual as well as organizational and systems levels. Finally, the authors propose a framework to build trust at multiple levels in a future assessment system; one that invites and supports professional and human growth and has the potential to position assessment as a fundamental component of renegotiating the social contract between medical education and the health of the public.

The Inconspicuous Learner Handover: An Exploratory Study of U.S. Emergency Medicine Program Directors' Perceptions of Learner Handovers from Medical School to Residency.

Phenomenon: Central to competency-based medical education is the need for a seamless developmental continuum of training and practice. Trainees currently experience significant discontinuity in the transition from undergraduate (UME) to graduate medical education (GME). The learner handover is intended to smooth this transition, but little is known about how well this is working from the GME perspective. In an attempt to gather preliminary evidence, this study explores U.S. program directors (PDs) perspective of the learner handover from UME to GME. Approach: Using exploratory qualitative methodology, we conducted semi-structured interviews with 12 Emergency Medicine PDs within the U.S. from October to November, 2020. We asked participants to describe their current perception of the learner handover from UME to GME. Then we performed thematic analysis using an inductive approach. Findings: We identified two main themes: The inconspicuous learner handover and barrier to creating a successful UME to GME learner handover. PDs described the current state of the learner handover as "nonexistent," yet acknowledged that information is transmitted from UME to GME. Participants also highlighted key challenges preventing a successful learner handover from UME to GME. These included: conflicting expectations, issues of trust and transparency, and a dearth of assessment data to actually hand over. Insights: PDs highlight the inconspicuous nature of learner handovers, suggesting that assessment information is not shared in the way it should be in the transition from UME to GME. Challenges with the learner handover demonstrate a lack of trust, transparency, and explicit communication between UME and GME. Our findings can inform how national organizations establish a unified approach to transmitting growth-oriented assessment data and formalize transparent learner handovers from UME to GME.

Lessons From Organic Chemistry: The Case for Considering Both High Standards and Equity in Assessment.

ABSTRACT: In this commentary, the authors explore the tension of balancing high performance standards in medical education with the acceptability of those standards to stakeholders (e.g., learners and patients). The authors then offer a lens through which this tension might be considered and ways forward that focus on both patient outcomes and learner needs.In examining this phenomenon, the authors argue that high performance standards are often necessary. Societal accountability is key to medical education, with the public demanding that training programs prepare physicians to provide high-quality care. Medical schools and residency programs, therefore, require rigorous standards to ensure graduates are ready to care for patients. At the same time, learners' experience is important to consider. Making sure that performance standards are acceptable to stakeholders supports the validity of assessment decisions.Equity should also be central to program evaluation and validity arguments when considering performance standards. Currently, learners across the continuum are variably prepared for the next phase in training and often face inequities in resource availability to meet high passing standards, which may lead to learner attrition. Many students who face these inequities come from underrepresented or disadvantaged backgrounds and are essential to ensuring a diverse medical workforce to meet the needs of patients and society. When these students struggle, it contributes to the leaky pipeline of more socioeconomically and racially diverse applicants.The authors posit that 4 key factors can balance the tension between high performance standards and stakeholder acceptability: standards that are acceptable and defensible, progression that is time variable, requisite support structures that are uniquely tailored for each learner, and assessment systems that are equitably designed.

Finding Medicine's Moneyball: How Lessons From Major League Baseball Can Advance Assessment in Precision Education.

ABSTRACT: Precision education (PE) leverages longitudinal data and analytics to tailor educational interventions to improve patient, learner, and system-level outcomes. At present, few programs in medical education can accomplish this goal as they must develop new data streams transformed by analytics to drive trainee learning and program improvement. Other professions, such as Major League Baseball (MLB), have already developed extremely sophisticated approaches to gathering large volumes of precise data points to inform assessment of individual performance.In this perspective, the authors argue that medical education-whose entry into precision assessment is fairly nascent-can look to MLB to learn the possibilities and pitfalls of precision assessment strategies. They describe 3 epochs of player assessment in MLB: observation, analytics (sabermetrics), and technology (Statcast). The longest tenured approach, observation, relies on scouting and expert opinion. Sabermetrics brought new approaches to analyzing existing data in a way that better predicted which players would help the team win. Statcast created precise, granular data about highly attributable elements of player performance while helping to account for nonplayer factors that confound assessment such as weather, ballpark dimensions, and the performance of other players. Medical education is progressing through similar epochs marked by workplace-based assessment, learning analytics, and novel measurement technologies. The authors explore how medical education can leverage intersectional concepts of MLB player and medical trainee assessment to inform present and future directions of PE.

Postgraduate Selection in Medical Education: A Scoping Review of Current Priorities and Values.

PURPOSE: The process of screening and selecting trainees for postgraduate training has evolved significantly in recent years, yet remains a daunting task. Postgraduate training directors seek ways to feasibly and defensibly select candidates, which has resulted in an explosion of literature seeking to identify root causes for the problems observed in postgraduate selection and generate viable solutions. The authors therefore conducted a scoping review to analyze the problems and priorities presented within the postgraduate selection literature to explore practical implications and present a research agenda. METHOD: Between May 2021 and February 2022, the authors searched PubMed, EMBASE, Web of Science, ERIC, and Google Scholar for English language literature published after 2000. Articles that described postgraduate selection were eligible for inclusion. 2,273 articles were ultimately eligible for inclusion. Thematic analysis was performed on a subset of 100 articles examining priorities and problems within postgraduate selection. Articles were sampled to ensure broad thematic and geographical variation across the breadth of articles that were eligible for inclusion. RESULTS: Five distinct perspectives or value statements were identified in the thematic analysis: (1) Using available metrics to predict performance in postgraduate training; (2) identifying the best applicants via competitive comparison; (3) seeking alignment between applicant and program in the selection process; (4) ensuring diversity, mitigation of bias, and equity in the selection process; and (5) optimizing the logistics or mechanics of the selection process. CONCLUSIONS: This review provides insight into the framing and value statements authors use to describe postgraduate selection within the literature. The identified value statements provide a window into the assumptions and subsequent implications of viewing postgraduate selection through each of these lenses. Future research must consider the outcomes and consequences of the value statement chosen and the impact on current and future approaches to postgraduate selection.

Educator's blueprint: A primer on consensus methods in medical education research.

Consensus methods such as the Delphi and nominal group techniques are increasingly utilized within medical education research. This educator's blueprint paper provides practical strategies regarding five key steps for ensuring best practices when using consensus methods. These strategies include deciding which consensus method is best, developing the initial questionnaire, identifying the participants, determining the number of rounds and consensus threshold, and describing and justifying any modifications. These strategies will help guide education researchers on their next study using consensus methods.

Patient handoffs and multi-specialty trainee perspectives across an institution: informing recommendations for health systems and an expanded conceptual framework for handoffs.

BACKGROUND: Safe and effective physician-to-physician patient handoffs are integral to patient safety. Unfortunately, poor handoffs continue to be a major cause of medical errors. Developing a better understanding of challenges faced by health care providers is critical to address this continued patient safety threat. This study addresses the gap in the literature exploring broad, cross-specialty trainee perspectives around handoffs and provides a set of trainee-informed recommendations for both training programs and institutions. METHODS: Using a constructivist paradigm, the authors conducted a concurrent/embedded mixed method study to investigate trainees' experiences with patient handoffs across Stanford University Hospital, a large academic medical center. The authors designed and administered a survey instrument including Likert-style and open-ended questions to solicit information about trainee experiences from multiple specialties. The authors performed a thematic analysis of open-ended responses. RESULTS: 687/1138 (60.4%) of residents and fellows responded to the survey, representing 46 training programs and over 30 specialties. There was wide variability in handoff content and process, most notably code status not being consistently mentioned a third of the time for patients who were not full code. Supervision and feedback about handoffs were inconsistently provided. Trainees identified multiple health-systems level issues that complicated handoffs and suggested solutions to these threats. Our thematic analysis identified five important aspects of handoffs: (1) handoff elements, (2) health-systems-level factors, (3) impact of the handoff, (4) agency (duty), and (5) blame and shame. CONCLUSIONS: Health systems, interpersonal, and intrapersonal issues affect handoff communication. The authors propose an expanded theoretical framework for effective patient handoffs and provide a set of trainee-informed recommendations for training programs and sponsoring institutions. Cultural and health-systems issues must be prioritized and addressed, as an undercurrent of blame and shame permeates the clinical environment.

TRainee Attributable & Automatable Care Evaluations in Real-time (TRACERs): A Scalable Approach for Linking Education to Patient Care.

Competency-based medical education (CBME) is an outcomes-based approach to education and assessment that focuses on what competencies trainees need to learn in order to provide effective patient care. Despite this goal of providing quality patient care, trainees rarely receive measures of their clinical performance. This is problematic because defining a trainee's learning progression requires measuring their clinical performance. Traditional clinical performance measures (CPMs) are often met with skepticism from trainees given their poor individual-level attribution. Resident-sensitive quality measures (RSQMs) are attributable to individuals, but lack the expeditiousness needed to deliver timely feedback and can be difficult to automate at scale across programs. In this eye opener, the authors present a conceptual framework for a new type of measure - TRainee Attributable & Automatable Care Evaluations in Real-time (TRACERs) - attuned to both automation and trainee attribution as the next evolutionary step in linking education to patient care. TRACERs have five defining characteristics: meaningful (for patient care and trainees), attributable (sufficiently to the trainee of interest), automatable (minimal human input once fully implemented), scalable (across electronic health records [EHRs] and training environments), and real-time (amenable to formative educational feedback loops). Ideally, TRACERs optimize all five characteristics to the greatest degree possible. TRACERs are uniquely focused on measures of clinical performance that are captured in the EHR, whether routinely collected or generated using sophisticated analytics, and are intended to complement (not replace) other sources of assessment data. TRACERs have the potential to contribute to a national system of high-density, trainee-attributable, patient-centered outcome measures.

Competency-Based Medical Education in a Norm-Referenced World: A Root Cause Analysis of Challenges to the Competency-Based Paradigm in Medical School.

Competency-based medical education (CBME) requires a criterion-referenced approach to assessment. However, despite best efforts to advance CBME, there remains an implicit, and at times, explicit, demand for norm-referencing, particularly at the junction of undergraduate medical education (UME) and graduate medical education (GME). In this manuscript, the authors perform a root cause analysis to determine the underlying reasons for continued norm-referencing in the context of the movement toward CBME. The root cause analysis consisted of 2 processes: (1) identification of potential causes and effects organized into a fishbone diagram and (2) identification of the 5 whys. The fishbone diagram identified 2 primary drivers: the false notion that measures such as grades are truly objective and the importance of different incentives for different key constituents. From these drivers, the importance of norm-referencing for residency selection was identified as a critical component. Exploration of the 5 whys further detailed the reasons for continuation of norm-referenced grading to facilitate selection, including the need for efficient screening in residency selection, dependence upon rank-order lists, perception that there is a best outcome to the match, lack of trust between residency programs and medical schools, and inadequate resources to support progression of trainees. Based on these findings, the authors argue that the implied purpose of assessment in UME is primarily stratification for residency selection. Because stratification requires comparison, a norm-referenced approach is needed. To advance CBME, the authors recommend reconsideration of the approach to assessment in UME to maintain the purpose of selection while also advancing the purpose of rendering a competency decision. Changing the approach will require a collaboration between national organizations, accrediting bodies, GME programs, UME programs, students, and patients/societies. Details are provided regarding the specific approaches required of each key constituent group.

The next generation of researchers: One-year outcome data from the SAEM Advanced Research Methodology Evaluation and Design in Medical Education (ARMED MedEd) program.

Introduction: As the field of medical education evolves, there is a need to increase the quality of education scholarship and develop a cadre of research scholars; however, clinician educators (CEs) considering this career transition have limited formal training in education research methodology to heed this call. Therefore, a program that provides more advanced training in education scholarship for CEs without the financial and resource barriers of fellowships and masters programs is needed. Methods: The SAEM Advanced Research Methodology Evaluation and Design in Medical Education (ARMED MedEd) program is a longitudinal program for the beyond-beginner CE, seeking advanced training in education research. The program was created using a comprehensive needs assessment and included longitudinal training; small-group projects; dedicated project mentors; and integrated diversity, equity, and inclusion initiatives. Program participants applied for a grant upon program completion. Results: Twenty-one participants completed the course with 100% completing the baseline survey and 67% (14/21) completing the end-of-program survey. Participants reported improved perception of knowledge across all of the topics with a medium to large effect size, ranging from 0.40 to 0.62. When asked about impact on their network of potential collaborators, participants reported a median of 7 (interquartile range [IQR] 5-8) out of 9. When asked about the impact on their community of practice, participants reported a median of 7 (IQR 5-7) out of 9. When asked about the impact on their professional identity, participants reported a median of 7 (IQR 4-9) out of 9. Participants also reported an increase in both the quantity (mean of 2 ± 1 new mentors) and the quality (median score 7 [IQR 5-8] out of 9) of new research mentorship as a result of the program. Open-ended feedback was generally positive, with 100% reporting they would advise others to take this program. Conclusions: The SAEM ARMED MedEd program represents a proof of concept for an advanced education research program seeking to fill the research training gap for the beyond-beginner Clinician educators.

Bridging the gap: Development of an experiential learning-based health disparities curriculum.

Background: The increasing number of vulnerable populations served by the emergency department (ED) calls for developing and implementing curricula aimed at training residents to deliver quality care for the most marginalized groups. Objective: We aimed to address this by developing and piloting a curriculum to introduce the unique challenges and disparities encountered by our diverse ED patient population using an experiential learning approach. Methods: We engaged community partners in designing and implementing a curriculum for incoming interns. This curriculum addresses specific populations encountered within the ED including patients with psychiatric illness, patients with cognitive disabilities, intoxicated patients, violent patients, patients of various cultural backgrounds, non-English-speaking patients, and LGBTQ patients. Experiential and active learning sessions with content experts and site visits to area organizations were arranged. Pre-, post-, and time-delayed surveys were deployed to evaluate the pilot of this curriculum. Results: Thirteen incoming interns participated in the orientation curriculum. Residents' comfort with each of these various patient populations as well as familiarity with community and ED resources was assessed before, after, and 1 year delayed from each session (response 13/13, 100%). Their scores increased significantly from baseline in the postsurvey and were maintained 1 year later (p < 0.05). Residents additionally provided narrative responses regarding what they learned and what was most helpful after completing their intern year. Conclusions: This pilot curriculum demonstrates that implementing an experiential learning curriculum and engaging community leaders and resources is key to training residents to address health disparities within their unique ED patient population. As such, it is imperative that we seek to immersively introduce trainees to the unique needs of the patient population they will serve early in training.

Filling the Core EPA 10 assessment void: A framework for individual assessment of Core Entrustable professional activity 10 competencies in medical students.

Objectives: The goal of this study was to develop and evaluate a novel curriculum and assessment tool for Core Entrustable Professional Activity (EPA) 10 competencies and entrustment scoring in a cohort of medical students in their emergency medicine (EM) clerkship using a framework of individualized, ad hoc, formative assessment. Core EPA 10 is an observable workplace-based activity for graduating medical students to recognize a patient requiring urgent or emergent care and initiate evaluation and management. Methods: This is a prospective, pretest-posttest study of medical students during their EM clerkship. Using the Thomas and Kern framework, we created a curriculum of simulation cases about chest pain/cardiac arrest and respiratory distress, which included novel assessment checklists, and instructional videos about recognizing and managing emergencies. Students were individually pretested on EPA 10 competencies using the simulation cases. Two raters scored students using standardized checklists. Students then watched instructional videos, underwent a posttest with the simulation cases, and were scored again by the two raters using the checklists. Differences between pretest and posttest scores were analyzed using paired t-tests and Wilcoxon signed-rank tests. Results: Seventy-three out of 85 (86%) students completed the curriculum. Mean scores from pretest to final posttest in the chest pain/cardiac arrest and respiratory distress cases significantly improved from 14.8/19 (SD 1.91), to 17.1/19 (SD = 1.00), t(68) = 10.56, p < 0.001, and 8.5/13 (SD 1.79), to 11.1/13(SD 0.89), t(67) = 11.15, p < 0.001, respectively. The kappa coefficients were 0.909 (n = 2698, p < 0.001) and 0.931 (n = 1872, p < 0.001). Median modified Chen entrustment scores improved from 1b (i.e., "Watch me do this") to 2b (i.e., "I'll watch you") for the chest pain/cardiac arrest case (p < 0.001) and 1b/2a (i.e., "Watch me do this"/ "Let's do this together") to 3a (i.e. "You go ahead, and I'll double-check all of your findings") for the respiratory distress case (p < 0.001). Conclusion: A new directed curriculum of standardized simulation cases and asynchronous instructional videos improved medical student performance in EPA 10 competencies and entrustment scores. This study provides a curricular framework to support formative individualized assessments for EPA 10.

The Purpose, Design, and Promise of Medical Education Research Labs.

Medical education researchers are often subject to challenges that include lack of funding, collaborators, study subjects, and departmental support. The construct of a research lab provides a framework that can be employed to overcome these challenges and effectively support the work of medical education researchers; however, labs are relatively uncommon in the medical education field. Using case examples, the authors describe the organization and mission of medical education research labs contrasted with those of larger research team configurations, such as research centers, collaboratives, and networks. They discuss several key elements of education research labs: the importance of lab identity, the signaling effect of a lab designation, required infrastructure, and the training mission of a lab. The need for medical education researchers to be visionary and strategic when designing their labs is emphasized, start-up considerations and the likelihood of support for medical education labs is considered, and the degree to which department leaders should support such labs is questioned.

A mixed-methods needs assessment to identify pharmacology education objectives for emergency medicine residents.

Objectives: Medication errors represent a significant threat to patient safety. Pharmacotherapy is one of the 23 Accreditation Council of Graduate Medical Education milestones for emergency medicine, yet there is minimal understanding of what content should be prioritized during training. The study aim was to develop objectives for a patient-safety focused pharmacology curriculum for emergency medicine residents. Methods: We incorporated data from a de-identified safety event database and survey responses of 30 faculty and clinical pharmacists at a single-site suburban university hospital with 24-hour emergency medicine pharmacists and an annual volume of approximately 70,000. We reviewed the database to quantify types and severity of medication errors over a 5-year period for a total of 370 errors. Anonymous surveys included categorical items that we analyzed with descriptive statistics and short answer questions that underwent thematic analysis by 2 coders. We summarized all data sources to identify curriculum gaps. Results: Common medication errors reported in our database were wrong dose (43%) and computer order entry errors (14%). Knowledge gaps were medication cost (63%), pregnancy risk information (60%), antibiotic stewardship (53%), interactions (47%), and side effects (47%). Qualitative analysis revealed the need to optimize computer order entry, understand the scope of critical medications, use references, and consult pharmacists. Integration of data suggested specific medications should be covered in curricular efforts, including antibiotics, analgesics, sedatives, and insulin. Conclusion: We developed objectives of pharmacology topics to prioritize during emergency medicine training to enhance prescribing safety. This study is limited due to its small sample and single institution source of data. Future studies should investigate the impact of pharmacology curriculum on minimizing clinical errors.