Mental Practice: Applying Successful Strategies in Sports to the Practice of Emergency Medicine.

Emergency physicians are expected to learn and maintain a large and varied set of competencies for clinical practice. These include high acuity, low occurrence procedures that may not be encountered frequently in the clinical environment and are difficult to practice with high fidelity and frequency in a simulated environment. Mental practice is a form of a cognitive walk-through that has been shown to be an effective method for improving motor and cognitive skills, with literature in sports science and emerging evidence supporting its use in medicine. In this article, we review the literature on mental practice in sports and medicine as well as the underlying neuroscientific theories that support its use. We review best-known practices and provide a framework to design and use mental imagery scripts to augment learning and maintaining the competencies necessary for physicians at all levels of training and clinical environments in the practice of emergency medicine.

How well do workplace-based assessments support summative entrustment decisions? A multi-institutional generalisability study.

BACKGROUND: Assessment of the Core Entrustable Professional Activities for Entering Residency requires direct observation through workplace-based assessments (WBAs). Single-institution studies have demonstrated mixed findings regarding the reliability of WBAs developed to measure student progression towards entrustment. Factors such as faculty development, rater engagement and scale selection have been suggested to improve reliability. The purpose of this investigation was to conduct a multi-institutional generalisability study to determine the influence of specific factors on reliability of WBAs. METHODS: The authors analysed WBA data obtained for clerkship-level students across seven institutions from 2018 to 2020. Institutions implemented a variety of strategies including selection of designated assessors, altered scales and different EPAs. Data were aggregated by these factors. Generalisability theory was then used to examine the internal structure validity evidence of the data. An unbalanced cross-classified random-effects model was used to decompose variance components. A phi coefficient of >0.7 was used as threshold for acceptable reliability. RESULTS: Data from 53 565 WBAs were analysed, and a total of 77 generalisability studies were performed. Most data came from EPAs 1 (n = 17 118, 32%) 2 (n = 10 237, 19.1%), and 6 (n = 6000, 18.5%). Low variance attributed to the learner (<10%) was found for most (59/77, 76%) analyses, resulting in a relatively large number of observations required for reasonable reliability (range = 3 to >560, median = 60). Factors such as DA, scale or EPA were not consistently associated with improved reliability. CONCLUSION: The results from this study describe relatively low reliability in the WBAs obtained across seven sites. Generalisability for these instruments may be less dependent on factors such as faculty development, rater engagement or scale selection. When used for formative feedback, data from these instruments may be useful. However, such instruments do not consistently provide reasonable reliability to justify their use in high-stakes summative entrustment decisions.

Utilizing natural language processing to analyze student narrative reflections for medical curriculum improvement.

MOTIVATION: Medical curricula improvement is an ongoing process to keep material relevant and improve the student's learning experience to better prepare them for patient care. Many programs utilize end-of-year evaluations, but these frequently have low response rates and lack actionable feedback. We hypothesized that student reflections written during a fourth year Sub-Internship could be used retrospectively to mine additional information as feedback for future curriculum adjustments. However, reflections contain a large amount of narrative content that would require a cumbersome and essentially infeasible manual review process for busy medical education faculty. METHODS: We developed a Natural Language Processing (NLP) pipeline to automatically identify common themes and topics present in the set of reflective writings that could be used to improve the curriculum. The dataset contains required responses to a faculty issued question submitted between August 2016 and July 2018 about challenges experienced during the medical students fourth year Sub-Internship. RESULTS: Eleven distinct topics were identified, with several being subsequently addressed in future iterations of the curriculum. CONCLUSION: Utilizing NLP on reflective writings was able to identify areas of curriculum improvement, and the NLP results provided a quick and easy way to explore the main themes and challenges expressed by students.

Understanding resident wellness: A path analysis of the clinical learning environment at three institutions.

PURPOSE: The clinical learning environment (CLE) affects resident physician well-being. This study assessed how aspects of the learning environment affected the level of resident job stress and burnout. MATERIALS AND METHODS: Three institutions surveyed residents assessing aspects of the CLE and well-being via anonymous survey in fall of 2020 during COVID. Psychological safety (PS) and perceived organizational support (POS) were used to capture the CLE, and the Mini-Z Scale was used to assess resident job stress and burnout. A total of 2,196 residents received a survey link; 889 responded (40% response rate). Path analysis explored both direct and indirect relationships between PS, POS, resident stress, and resident burnout. RESULTS: Both POS and PS had significant negative relationships with experiencing a great deal of job stress; the relationship between PS and stress was noticeably stronger than POS and stress (POS: B= -0.12, p=.025; PS: B= -0.37, p<.001). The relationship between stress and residents' level of burnout was also significant (B = 0.38, p<.001). The overall model explained 25% of the variance in resident burnout. CONCLUSIONS: Organizational support and psychological safety of the learning environment is associated with resident burnout. It is important for educational leaders to recognize and mitigate these factors.

A mixed methods exploration of the emergence of goal orientation in medical students' individualized learning plans.

PURPOSE: Goal orientation (GO) describes an individual's approach to different achievement situations. Understanding the motivations and approach to achieving goals of medical students is vital with the increasing emphasis on self-directed learning. The purpose of this study was to identify themes in self-improvement reflections that relate to each GO dimension (learning, performance-prove, and performance-avoid). METHODS: A sequential explanatory mixed methods design was used. GO data was used to categorize students into groups aligning with the GO dimension identified in the previous stage of quantitative analysis. Individualized learning plans (ILPs) for each GO dimension group were coded inductively to identify emergent themes related to goal setting and achievement. RESULTS: The learning GO group was the largest of the three GOs. Five themes were identified from inductive analysis: importance of practice, identifying elements that helped, identifying structural barriers, opportunities for improvement, and acknowledging experience. While these themes occur across GO, patterns exist within their ILPs based on GO. CONCLUSIONS: We identified common themes for motivations of medical students, and these motivations might differ depending on their GO. Further exploration into the themes over the course of their training will provide additional insights on what factors may be involved in student motivations towards learning and achievement. Educators can use this information to individualize feedback and students can better understand their motivations towards achieving goals.

Competency-based medical education across the continuum: How well aligned are medical school EPAs to residency milestones?

INTRODUCTION: Competency-based medical education (CBME) provides a framework for describing learner progression throughout training. However, specific approaches to CBME implementation vary widely across educational settings. Alignment between various methods used across the continuum is critical to support transitions and assess learner performance. The purpose of this study was to investigate alignment between CBME frameworks used in undergraduate medical education (UME) and graduate medical education (GME) settings using the US context as a model. METHOD: The authors analyzed content from the core entrustable professional activities for entering residency (Core EPAs; UME model) and residency milestones (GME model). From that analysis, they performed a series of cross-walk activities to investigate alignment between frameworks. After independent review, authors discussed findings until consensus was reached. RESULTS: Some alignment was found for activities associated with history taking, physical examination, differential diagnosis, patient safety, and interprofessional care; however, there were far more examples of misalignment. CONCLUSIONS: These findings highlight challenges creating alignment of assessment frameworks across the continuum of training. The importance of these findings includes implications for assessment and persistence of the educational gap across UME and GME. The authors provide four next steps to improve upon the continuum of education.

Social network analysis of publication collaboration of accelerating change in MedEd consortium.

INTRODUCTION: The American Medical Association formed the Accelerating Change in Medical Education Consortium through grants to effect change in medical education. The dissemination of educational innovations through scholarship was a priority. The objective of this study was to explore the patterns of collaboration of educational innovation through the consortium's publications. METHOD: Publications were identified from grantee schools' semi-annual reports. Each publication was coded for the number of citations, Altmetric score, domain of scholarship, and collaboration with other institutions. Social network analysis explored relationships at the midpoint and end of the grant. RESULTS: Over five years, the 32 Consortium institutions produced 168 publications, ranging from 38 papers from one institution to no manuscripts from another. The two most common domains focused on health system science (92 papers) and competency-based medical education (30 papers). Articles were published in 54 different journals. Forty percent of publications involved more than one institution. Social network analysis demonstrated rich publishing relationships within the Consortium members as well as beyond the Consortium schools. In addition, there was growth of the network connections and density over time. CONCLUSION: The Consortium fostered a scholarship network disseminating a broad range of educational innovations through publications of individual school projects and collaborations.

Teaching Toolbox: Breaking Bad News with Virtual Technology in the Time of COVID.

Breaking bad news is a key component of the physicians' work. Traditionally, breaking bad news has been encouraged to be performed in person whenever possible (Monden et al. Proc (Bayl Univ Med Cent) 29(1):101-102, 2016; Nickson 2019). The common practice prior to the pandemic can be summarized by "The first rule of breaking bad news is: do not do it over the phone." It is important to be present with the family and provide support through compassion and empathy. Until recently, virtual communication technology for serious medical discussions was rare and primarily used when compelled by circumstances such as distance. The COVID-19 pandemic has transformed our ability to deliver news in person and has required the medical community to increase the utilization of telephone and video conferencing to communicate with patients and their family members. Breaking bad news through virtual media is a new skill in need of further guidance and education regarding how to set up the conversation, provide empathy, and lend support (Wolf et al., Oncologist 25(6):e879-e880, 2020). Therefore, we have created a teaching toolbox to help educate healthcare providers on how to deliver bad news by phone or video.

When Safety Event Reporting Is Seen as Punitive: "I've Been PSN-ed!"

STUDY OBJECTIVE: Reporting systems are designed to identify patient care issues so changes can be made to improve safety. However, a culture of blame discourages event reporting, and reporting seen as punitive can inhibit individual and system performance in patient safety. This study aimed to determine the frequency and factors related to punitive patient safety event report submissions, referred to as Patient Safety Net reports, or PSNs. METHODS: Three subject matter experts reviewed 513 PSNs submitted between January and June 2019. If the PSN was perceived as blaming an individual, it was coded as punitive. The experts had high agreement (κ=0.84 to 0.92), and identified relationships between PSN characteristics and punitive reporting were described. RESULTS: A total of 25% of PSNs were punitive, 7% were unclear, and 68% were designated nonpunitive. Punitive (vs nonpunitive) PSNs more likely focused on communication (41% vs 13%), employee behavior (38% vs 2%), and patient assessment issues (17% vs 4%). Nonpunitive (vs punitive) PSNs were more likely for equipment (19% vs 4%) and patient or family behavior issues (8% vs 2%). Punitive (vs nonpunitive) PSNs were more common with adverse reactions or complications (21% vs 10%), communication failures (25% vs 16%), and noncategorized events (19% vs 8%), and nonpunitive (vs punitive) PSNs were more frequent in falls (5% vs 0%) and radiology or laboratory events (17% vs 7%). CONCLUSION: Punitive reports have important implications for reporting systems because they may reflect a culture of blame and a failure to recognize system influences on behaviors. Nonpunitive wording better identifies factors contributing to safety concerns. Reporting systems should focus on patient outcomes and learning from systems issues, not blaming individuals.

What is the association between student well-being and high-stakes examination scores?

INTRODUCTION: As educators seek to improve medical student well-being, it is essential to understand the interplay between distress and important outcomes. Performance on Step 1 of the United States Medical Licensing Examination has played a significant role in selection for postgraduate residency positions in the United States and consequently has been a source of great stress for medical students. The purpose of this study was to examine whether student well-being correlates with performance on a high stakes licensing examination. METHODS: Between 2014 and 2016, three sequential cohorts of medical students at the University of Michigan Medical School completed the Medical Student Well-Being Index (MSWBI) at the end of their 2nd-year curriculum, shortly before taking Step 1. Associations between well-being and Step 1 scores were investigated while adjusting for MCAT scores and cumulative second-year course scores. RESULTS: In total, 354 students were included in the analysis (68.1% of potential responders). On bivariate analysis, poor student well-being (0 = low distress [high well-being], 7 = high distress [poor well-being]) was associated with lower Step 1 examination scores (slope = -2.10, P < .01), and well-being accounted for 5% of overall Step 1 score variability (R2  = .05). However, after adjustment for MCAT scores and cumulative GPA (full model R2  = .51), the relationship between well-being and Step 1 score was no longer significant (slope = -0.70, P-value = .06). CONCLUSIONS: When controlling for metrics of academic performance, student well-being prior to taking Step 1 was not associated with how well students performed on Step 1 for the study sample.

Does Delaying the United States Medical Licensing Examination Step 1 to After Clerkships Affect Student Performance on Clerkship Subject Examinations?

Phenomenon: Schools are considering the optimal timing of Step 1 of the United States Medical Licensing Examination (USMLE). Two primary reasons for moving Step 1 after the core clerkships are to promote deeper, more integrated basic science learning in clinical contexts and to better prepare students for the increasingly clinical focus of Step 1. Positioning Step 1 after the core clerkships leverages a major national assessment to drive learning, encouraging students to deepen their basic science knowledge while in the clinical setting. Previous studies demonstrated small increases in Step 1 scores, reductions in failure rates, and similar Step 2 Clinical Knowledge scores when Step 1 was after the clerkships. Some schools that have moved Step 1 reported declines in clinical subject examination (CSE) performance. This may be due to shortened pre-clerkship curricula, the absence of the Step 1 study period for knowledge consolidation, or exposure to fewer National Board of Medical Examiners type questions prior to taking CSEs. This multi-institutional study aimed to determine whether student performance on CSEs was affected by moving Step 1 after the core clerkships. Approach: CSE scores for students from eight schools that moved Step 1 after core clerkships between 2012 and 2016 were analyzed in a pre-post format. Hierarchical linear modeling was used to quantify the effect of the curriculum on CSE performance. Additional analysis determined if clerkship order impacted clinical subject exam performance and whether the curriculum change resulted in more students scoring in the lowest percentiles (as defined as below the national fifth percentile) before and after the curricular change. Findings: After moving Step 1 to after the clerkships, collectively these eight schools demonstrated statistically significant lower performance on four CSEs (Medicine, Neurology, Pediatrics, and Surgery) but not Obstetrics/Gynecology or Psychiatry. Comparing performance within the three years pre and post Step 1 change, differences across all clerkships ranged from 0.3 to -2.0 points, with an average difference of -1.1. CSE performance in clerkships taken early in the sequence was more affected by the curricular change, and differences gradually disappeared with subsequent examinations. Medicine and Neurology showed the largest average differences between curricular-group when taken early in the clinical year. Finally, there was a slightly higher chance of scoring below the national fifth percentile in four of the clinical subject exams (Medicine, Neurology, Pediatrics, and Psychiatry) for the cohort with Step 1 after the clerkships. Insights: Moving Step 1 after core clerkships had a small impact on CSE scores overall, with decreased scores for exams early in the clerkship sequence and an increased number of students below the fifth percentile. Score differences have minor effects on clerkship grades, but overall the size of the effect is unlikely to be educationally meaningful. Schools can use a variety of mitigation strategies to address CSE performance and Step 1 preparation in the clerkship phase.

Beyond advising and mentoring: Competencies for coaching in medical education.

BACKGROUND: Coaching supports academic goals, professional development and wellbeing in medical education. Scant literature exists on training and assessing coaches and evaluating coaching programs. To begin filling this gap, we created a set of coach competencies for medical education using a modified Delphi approach. METHODS: An expert team assembled, comprised of seven experts in the field of coaching. A modified Delphi approach was utilized to develop competencies. RESULTS: Fifteen competencies in five domains resulted: coaching process and structure, relational skills, coaching skills, coaching theories and models, and coach development. CONCLUSION: These competencies delineate essential features of a coach in medical education. Next steps include creating faculty development and assessment tools for coaching.

Building reliable and generalizable clerkship competency assessments: Impact of 'hawk-dove' correction.

PURPOSE: Systematic differences among raters' approaches to student assessment may result in leniency or stringency of assessment scores. This study examines the generalizability of medical student workplace-based competency assessments including the impact of rater-adjusted scores for leniency and stringency. METHODS: Data were collected from summative clerkship assessments completed for 204 students during 2017-2018 the clerkship at a single institution. Generalizability theory was used to explore variance attributed to different facets (rater, learner, item, and competency domain) through three unbalanced random-effects models by clerkship including applying assessor stringency-leniency adjustments. RESULTS: In the original assessments, only 4-8% of the variance was attributed to the student with the remainder being rater variance and error. Aggregating items to create a composite score increased variability attributable to the student (5-13% of variance). Applying a stringency-leniency ('hawk-dove') correction substantially increased the variance attributed to the student (14.8-17.8%) and reliability. Controlling for assessor leniency/stringency reduced measurement error, decreasing the number of assessments required for generalizability from 16-50 to 11-14. CONCLUSIONS: Similar to prior research, most of the variance in competency assessment scores was attributable to raters, with only a small proportion attributed to the student. Making stringency-leniency corrections using rater-adjusted scores improved the psychometric characteristics of assessment scores.

What faculty write versus what students see? Perspectives on multiple-choice questions using Bloom's taxonomy.

BACKGROUND: Using revised Bloom's taxonomy, some medical educators assume they can write multiple choice questions (MCQs) that specifically assess higher (analyze, apply) versus lower-order (recall) learning. The purpose of this study was to determine whether three key stakeholder groups (students, faculty, and education assessment experts) assign MCQs the same higher- or lower-order level. METHODS: In Phase 1, stakeholders' groups assigned 90 MCQs to Bloom's levels. In Phase 2, faculty wrote 25 MCQs specifically intended as higher- or lower-order. Then, 10 students assigned these questions to Bloom's levels. RESULTS: In Phase 1, there was low interrater reliability within the student group (Krippendorf's alpha = 0.37), the faculty group (alpha = 0.37), and among three groups (alpha = 0.34) when assigning questions as higher- or lower-order. The assessment team alone had high interrater reliability (alpha = 0.90). In Phase 2, 63% of students agreed with the faculty as to whether the MCQs were higher- or lower-order. There was low agreement between paired faculty and student ratings (Cohen's Kappa range .098-.448, mean .256). DISCUSSION: For many questions, faculty and students did not agree whether the questions were lower- or higher-order. While faculty may try to target specific levels of knowledge or clinical reasoning, students may approach the questions differently than intended.

Incidence of resident mistreatment in the learning environment across three institutions.

INTRODUCTION: Mistreatment in the learning environment is associated with negative outcomes for trainees. While the Association of American Medical Colleges (AAMC) annual Graduation Questionnaire (GQ) has collected medical student reports of mistreatment for a decade, there is not a similar nationally benchmarked survey for residents. The objective of this study is to explore the prevalence of resident experiences with mistreatment. METHODS: Residents at three academic institutions were surveyed using questions similar to the GQ in 2018. Quantitative data were analyzed based on frequency and Mann-Whitney U tests to detect gender differences. RESULTS: Nine hundred ninety-six of 2682 residents (37.1%) responded to the survey. Thirty-nine percent of residents reported experiencing at least one incident of mistreatment. The highest reported incidents were public humiliation (23.7%) and subject to offensive sexist remarks/comments (16.0%). Female residents indicated experiencing significantly more incidents of public embarrassment, public humiliation, offensive sexist remarks, lower evaluations based on gender, denied opportunities for training or rewards, and unwanted sexual advances. Faculty were the most frequent instigators of mistreatment (66.4%). Of trainees who reported experiencing mistreatment, less than one-quarter reported the behavior. CONCLUSION: Mistreatment in the academic learning environment is a concern in residency programs. There is increased frequency among female residents.

Versatility in multiple mini-interview implementation: Rater background does not significantly influence assessment scoring.

The medical school admissions process seeks to assess a core set of cognitive and non-cognitive competencies that reflect professional readiness and institutional mission alignment. The standardized format of multiple mini-interviews (MMIs) can enhance assessments, and thus many medical schools have switched to this for candidate interviews. However, because MMIs are resource-intensive, admissions deans use a variety of interviewers from different backgrounds/professions. Here, we analyze the MMI process for the 2018 admissions cycle at the VCU School of Medicine, where 578 applicants were interviewed by 126 raters from five distinct backgrounds: clinical faculty, basic science faculty, medical students, medical school administrative staff, and community members. We found that interviewer background did not significantly influence MMI evaluative performance scoring, which eliminates a potential concern about the consistency and reliability of assessment.

The educators' experience: Learning environments that support the master adaptive learner.

BACKGROUND: The Master Adaptive Learner (MAL) theoretical framework describes an integrated approach to learning that combines features of educational theory on self-regulated learning and aspects of quality improvement. In order to develop MAL students, it is important to pay attention to the learning environment. PURPOSE: To describe educators' perspectives about the learning environment needed to promote the development of master adaptive learners. METHODS: Thematic analysis of reports by medical educators who were workshop participants at a national presentation on creating effective learning environment to develop MAL in undergraduate medical education. RESULTS: Three themes educators considered important in the development of the Master Adaptive Learner were Adaptive Educator, Support for Learning, and Institutional Commitment. These findings suggest that in order to support the MAL, an educational setting should provide faculty who can be flexible and adapt to the developing MAL, learning experiences that support active learning, focused on groups as well as developing individual learners. Leaders in the educational setting should demonstrate a commitment to creating a culture to support learning and provide appropriate resources to that end. CONCLUSION: Learning environments to develop master adaptive learners need to have adaptive educators, teaching, learning, and institutional culture to support challenge and grow Master Adaptive Learners.

Burnout in OB/GYN Students and Residents.

In this article we address the concept of burnout, first in the medical student setting, and then in the residency setting. We will review the prevalence followed by a discussion of risk factors, consequences, and finally thoughts on prevention and intervention.

Beyond mistreatment: Learner neglect in the clinical teaching environment.

Medical student mistreatment has been recognized as a deterrent to education as it interferes with the learning process and contributes to student burnout and attrition. Medical schools and leaders in undergraduate medical education have expended tremendous effort in addressing this phenomenon in hopes of eradicating mistreatment. However, there is a spectrum of behaviors that negatively impact the learning environment beyond that which is considered frank mistreatment. In this conceptual article, the authors propose the concept of learner neglect for the consideration by educators and researchers. This is a term for a range of behaviors exhibited intentionally or unintentionally by a supervisor that prevent a learner from reaching his or her potential. While the behaviors may overlap with mistreatment, they do not always fit within the definition of mistreatment. This concept is illustrated in the context of optimal and suboptimal teaching behaviors that commonly occur within the ecosystem of clinical education. Descriptions and examples are provided for both intentional and unintentional learner neglect. The authors hypothesize possible short- and long-term impacts of learner neglect, describe contributors to its prevalence, and offer questions for key stakeholders to consider in an effort to recognize, study, and ameliorate this issue within medical education programs.

The influence of MCAT and GPA preadmission academic metrics on interview scores.

Medical school admissions interviews are used to assess applicants' nonacademic characteristics as advocated by the Association of American Medical Colleges' Advancing Holistic Review Initiative. The objective of this study is to determine whether academic metrics continue to significantly influence interviewers' scores in holistic processes by blinding interviewers to applicants' undergraduate grade point averages (uGPA) and Medical College Admission Test (MCAT). This study examines academic and demographic predictors of interview scores for two applicant cohorts at the University of Michigan Medical School. In 2012, interviewers were provided applicants' uGPA and MCAT scores; in 2013, these academic metrics were withheld from interviewers' files. Hierarchical regression analysis was conducted to examine the influence of academic and demographic variables on overall cohort interview scores. When interviewers were provided uGPA and MCAT scores, academic metrics explained more variation in interview scores (7.9%) than when interviewers were blinded to these metrics (4.1%). Further analysis showed a statistically significant interaction between cohort and uGPA, indicating that the association between uGPA and interview scores was significantly stronger for the 2012 unblinded cohort compared to the 2013 blinded cohort (ß = .573, P < .05). By contrast, MCAT scores had no interactive effects on interviewer scores. While MCAT scores accounted for some variation in interview scores for both cohorts, only access to uGPA significantly influenced interviewers' scores when looking at interaction effects. Withholding academic metrics from interviewers' files may promote assessment of nonacademic characteristics independently from academic metrics.