"What's Next in My Arc of Development?": An Exploratory Study of What Medical Students Need to Care for Patients of Different Backgrounds.

PHENOMENON: Medical schools must equip future physicians to provide equitable patient care. The best approach, however, is mainly dependent on a medical school's context. Graduating students from our institution have reported feeling ill-equipped to care for patients from "different backgrounds" on the Association of American Medical Colleges' Graduation Questionnaire. We explored how medical students interpret "different patient backgrounds" and what they need to feel prepared to care for diverse patients. APPROACH: We conducted an exploratory qualitative case study using focus groups with 11, Year 2 (MS2) and Year 4 (MS4) medical students at our institution. Focus groups were recorded, transcribed, and coded using thematic analysis. We used Bobbie Harro's cycles of socialization and liberation to understand how the entire medical school experience, not solely the curriculum, informs how medical students learn to interact with all patients. FINDINGS: We organized our findings into four major themes to characterize students' medical education experience when learning to care for patients of different backgrounds: (1) Understandings of different backgrounds (prior to medical school); (2) Admissions process; (3) Curricular socialization; and (4) Co-curricular (or environmental) socialization. We further divided themes 2, 3, and 4 into two subthemes when learning how to care for patients of different backgrounds: (a) the current state and (b) proposed changes. We anticipate that following the proposed changes will help students feel more prepared to care for patients of differing backgrounds. INSIGHTS: Our findings show that preparing medical students to care for diverse patient populations requires a multitude of intentional changes throughout medical students' education. Using Harro's cycles of socialization and liberation as an analytic lens, we identified multiple places throughout medical students' educational experience that are barriers to learning how to care for diverse populations. We propose changes within medical students' education that build upon each other to adequately prepare students to care for patients of diverse backgrounds. Each proposed change culminates into a systemic shift within an academic institution and requires an intentional commitment by administration, faculty, admissions, curriculum, and student affairs.

Three key issues for determining competence in a system of assessment.

There is widespread agreement that medical education should include multi-source, multi-method, and multi-purpose forms of assessment and thus should move towards cohesive systems of assessment. One possibility that fits comfortably with a system of assessment framework is to organize assessments around a competency based medical education model. However conceptually appealing a competency based medical education model is, discussions are sparse regarding the details of determining competence (or the pass/fail point) within each competency. In an effort to make discussions more concrete, we put forth three key issues relevant to implementation of competency-based assessment: (1) each competency is measured with multiple assessments, (2) not all assessments produce a score for a competency as a good portion of assessment in medical school is narrative, and (3) competence decisions re-occur as assessments cumulate. We agree there are a host of other issues to consider, but think the practical action-oriented issues we set forth will be helpful in putting form into what is now largely abstract discussions.

Milestone Implementation's Impact on Narrative Comments and Perception of Feedback for Internal Medicine Residents: a Mixed Methods Study.

BACKGROUND: Feedback is a critical element of graduate medical education. Narrative comments on evaluation forms are a source of feedback for residents. As a shared mental model for performance, milestone-based evaluations may impact narrative comments and resident perception of feedback. OBJECTIVE: To determine if milestone-based evaluations impacted the quality of faculty members' narrative comments on evaluations and, as an extension, residents' perception of feedback. DESIGN: Concurrent mixed methods study, including qualitative analysis of narrative comments and survey of resident perception of feedback. PARTICIPANTS: Seventy internal medicine residents and their faculty evaluators at the University of Utah. APPROACH: Faculty narrative comments from 248 evaluations pre- and post-milestone implementation were analyzed for quality and Accreditation Council for Graduate Medical Education competency by area of strength and area for improvement. Seventy residents were surveyed regarding quality of feedback pre- and post-milestone implementation. KEY RESULTS: Qualitative analysis of narrative comments revealed nearly all evaluations pre- and post-milestone implementation included comments about areas of strength but were frequently vague and not related to competencies. Few evaluations included narrative comments on areas for improvement, but these were of higher quality compared to areas of strength (p < 0.001). Overall resident perception of quality of narrative comments was low and did not change following milestone implementation (p = 0.562) for the 86% of residents (N = 60/70) who completed the pre- and post-surveys. CONCLUSIONS: The quality of narrative comments was poor, and there was no evidence of improved quality following introduction of milestone-based evaluations. Comments on areas for improvement were of higher quality than areas of strength, suggesting an area for targeted intervention. Residents' perception of feedback quality did not change following implementation of milestone-based evaluations, suggesting that in the post-milestone era, internal medicine educators need to utilize additional interventions to improve quality of feedback.

To What Degree Are the 13 Entrustable Professional Activities Already Incorporated Into Physicians' Performance Schemas for Medical Students?

Phenomenon: There is an abundance of literature on Entrustable Professional Activities (EPAs) in theory, but there are few studies on the EPAs in practice for undergraduate clinical education. In addition, little is known about the degree to which the EPAs are or are not aligned with physician assessors' performance schemas of the clerkship student. Investigating the degree to which physician assessors' performance schemas are already aligned with the activities described by the EPAs is critical for effective workplace assessment design. Approach: We sampled 1,032 areas of strength (strength) and areas for improvement (improvement) written evaluation comments by 423 physician assessors for clerkship students' performance in academic years 2014-15 and 2015-16 at the University of Utah School of Medicine. Two researchers independently categorized each comment by EPA and/or coded by non-EPA topic. The proportion of comment types was compared between strength comments and improvement comments with the Wilcoxon Signed-Rank Test. Findings: The most frequently mentioned EPAs in comments were about history gathering/physical exam, differential diagnosis, documentation, presentation, and interprofessional collaboration; few mentioned diagnostic tests, patient handovers, recognition of urgent patient care, and patient safety, and none mentioned orders/prescriptions and informed consent. The most frequent non-EPA topics were about medical knowledge, need to read more, learning attitude, work ethic, professionalism/maturity, and receptiveness to feedback. The proportion of comments aligned with an EPA only, a non-EPA topic only, or both an EPA and non-EPA topic was significantly different for clerkship students' strength compared to improvement. Insights: Physician assessors' performance schemas for clerkship students were aligned with EPAs to varying degrees depending on the specific EPA and whether describing strength or improvement. Of interest, the frequently mentioned non-EPA comments represented some of the competencies that contribute to effectively performing particular EPAs and are Accreditation Council for Graduate Medical Education (ACGME) core competencies (e.g., medical knowledge, professionalism), used in residency programs. Because physician assessors for undergraduate medical education often also participate in graduate medical education, the frequency of non-EPA topics aligned to ACGME competencies may suggest influence of graduate medical education evaluative frameworks on performance schemas for clerkship students; this could be important when considering implementation of EPAs in undergraduate medical education.

What's in a Transition? An Integrative Perspective on Transitions in Medical Education.

This Conversation Starters article presents a selected research abstract from the 2016 Association of American Medical Colleges Western Region Group on Educational Affairs annual spring meeting. The abstract is paired with the integrative commentary of three experts who shared their thoughts stimulated by the needs assessment study. These thoughts explore how the general theoretical mechanisms of transition may be integrated with cognitive load theory in order to design interventions and environments that foster transition.

"Am I cut out for this?" Understanding the experience of doubt among first-year medical students.

PURPOSE: Existing research shows that medical students experience high levels of distress. The purpose of this study was to understand how medical students experience doubt, and how doubt relates to distress. METHODS: A mixed-methods study was conducted among first-year students at the Johns Hopkins University School of Medicine in June 2012. Students answered survey questions and participated in focus groups about doubt and other forms of distress. RESULTS: Ninety-four percent (112) of students responded to the survey, with 49% reporting a moderate or high degree of doubt. Compared to those reporting no or low doubt, students with moderate/high doubt were significantly more likely to question their purpose and identity, struggle to cope with doubt, and experience depression and emotional hardening. Twenty-eight percent of students (34/112) participated in focus groups to explore their doubt, and three themes emerged: types of doubt, ways of coping with doubt, and impact of doubt. CONCLUSIONS: Doubt is highly prevalent among first-year medical students, affects their identity and purpose, and has positive and negative consequences. Doubt among medical students merits awareness and further study, as it may be an important mediator of students' emerging identity and sense of well-being.

Learning environment assessments of a single curriculum being taught at two medical schools 10,000 miles apart.

BACKGROUND: Perdana University Graduate School of Medicine (PUGSOM), the first graduate-entry medical school in Malaysia, was established in 2011 in collaboration with Johns Hopkins University School of Medicine (JHUSOM), an American medical school. This study compared learning environments (LE) at these two schools, which shared the same overarching curriculum, along with a comparator Malaysian medical school, Cyberjaya University College of Medical Sciences (CUCMS). As a secondary aim, we compared 2 LE assessment tools - the widely-used Dundee Ready Educational Environment Measure (DREEM) and the newer Johns Hopkins Learning Environment Scale (JHLES). METHODS: Students responded anonymously at the end of their first year of medical school to surveys which included DREEM, JHLES, single-item global LE assessment variables, and demographics questions. RESULTS: Respondents included 24/24 (100 %) students at PUGSOM, 100/120 (83 %) at JHUSOM, and 79/83 (95 %) at CUCMS. PUGSOM had the highest overall LE ratings (p < 0.05) [DREEM 155.3 (SD 21.3); JHLES 116.5 (SD 12.2)], followed by JHUSOM [DREEM 143.3 (SD 22.5); JHLES 111.7 (SD 12.0)] and CUCMS [DREEM 138.5 (SD 22.4); JHLES 106.4 (SD 14.5)]. PUGSOM's overall high LE ratings were driven by responses in "perception of teaching," "meaningful engagement," and "acceptance and safety" domains. JHLES detected significant differences across schools in 5/7 domains and had stronger correlations than DREEM to each global LE assessment variable. CONCLUSIONS: The inaugural class of medical students at PUGSOM rated their LE exceptionally highly, providing evidence that transporting a medical school curriculum may be successful. The JHLES showed promise as a LE assessment tool for use in international settings.

Integrating a self-directed ultrasound curriculum for the internal medicine clerkship.

BACKGROUND: Incorporating ultrasound into the clinical curriculum of undergraduate medical education has been limited by a need for faculty support. Without integration into the clinical learning environment, ultrasound skills become a stand-alone skill and may decline by the time of matriculation into residency. A less time intensive ultrasound curriculum is needed to preserve skills acquired in preclinical years. We aimed to create a self-directed ultrasound curriculum to support and assess students' ability to acquire ultrasound images and to utilize ultrasound to inform clinical decision-making. METHODS: Third year students completed the self-directed ultrasound curriculum during their required internal medicine clerkship. Students used Butterfly iQ+ portable ultrasound probes. The curriculum included online modules that focused on clinical application of ultrasound as well as image acquisition technique. Students were graded on image acquisition quality and setting, a patient write-up focused on clinical decision-making, and a multiple-choice quiz. Student feedback was gathered with an end-of-course survey. Faculty time was tracked. RESULTS: One hundred and ten students participated. Students averaged 1.79 (scale 0-2; SD = 0.21) on image acquisition, 78% (SD = 15%) on the quiz, and all students passed the patient write-up. Most reported the curriculum improved their clinical reasoning (72%), learning of pathophysiology (69%), and patient care (55%). Faculty time to create the curriculum was approximately 45 h. Faculty time to grade student assignments was 38.5 h per year. CONCLUSIONS: Students were able to demonstrate adequate image acquisition, use of ultrasound to aid in clinical decision-making, and interpretation of ultrasound pathology with no in-person faculty instruction. Additionally, students reported improved learning of pathophysiology, clinical reasoning, and rapport with patients. The self-directed curriculum required less faculty time than prior descriptions of ultrasound curricula in the clinical years and could be considered at institutions that have limited faculty support.

Warm-up on a simulator improves residents' performance in laparoscopic surgery: a randomized trial.

INTRODUCTION AND HYPOTHESIS: Our aim was to assess the impact of immediate preoperative laparoscopic warm-up using a simulator on intraoperative laparoscopic performance by gynecologic residents. METHODS: Eligible laparoscopic cases performed for benign, gynecologic indications were randomized to be performed with or without immediate preoperative warm-up. Residents randomized to warm-up performed a brief set of standardized exercises on a laparoscopic trainer immediately before surgery. Intraoperative performance was scored using previously validated global rating scales. Assessment was made immediately after surgery by attending faculty who were blinded to the warm-up randomization. RESULTS: We randomized 237 residents to 47 minor laparoscopic cases (adnexal/ tubal surgery) and 44 to major laparoscopic cases (hysterectomy). Overall, attendings rated upper-level resident performances (postgraduate year [PGY-3, 4]) significantly higher on global rating scales than lower-level resident performances (PGY-1, 2). Residents who performed warm-up exercises prior to surgery were rated significantly higher on all subscales within each global rating scale, irrespective of the difficulty of the surgery. Most residents felt that performing warm-up exercises helped their intraoperative performances. CONCLUSION: Performing a brief warm-up exercise before a major or minor laparoscopic procedure significantly improved the intraoperative performance of residents irrespective of the difficulty of the case.

Revisiting effects of contextual strength on the subordinate bias effect: evidence from eye movements.

In this study, we examined two issues regarding the role of context in ambiguity resolution: whether access to the contextually appropriate meaning is exhaustive or selective, and whether the contextually inappropriate meaning is inhibited. Participants read texts in which a biased ambiguous word was encountered twice while their eye movements were measured. The context preceding the first encounter varied in the extent to which the subordinate meaning was supported; the context preceding the second encounter always supported the dominant meaning. The findings suggest that lexical access is exhaustive but can be influenced by context, and that the subsequent accessibility of the contextually inappropriate meaning is unaffected by previous selection processes. The results were interpreted in terms of the assumptions of the reordered-access model and activation mechanisms that operate during reading.

Academic Performance of Clerkship Students Working on a Longitudinal Inpatient Pediatrics Team.

OBJECTIVE: To determine if the academic performance of students who worked on a longitudinal inpatient team in the pediatric clerkship differed from students on traditional teams. We hypothesized that working on the longitudinal team would be associated with improved performance. METHODS: We retrospectively identified students who rotated in the pediatric clerkship at a single institution from 2017 through 2021. We used multiple linear and multiple ordered logistic regression to examine whether working on a longitudinal inpatient team in which the majority of students work with the same senior resident and attending for the entire inpatient block and function without interns was associated with improved academic performance. RESULTS: We included data from 463 students, 316 in the longitudinal team group and 147 in the traditional team group. Working on the longitudinal team was associated with a higher inpatient preceptor rating (adjusted mean rating 3, 95% confidence interval [CI] 2.97 to 3.03 vs 2.85, 95% CI 2.81 to 2.90; P = .02; on a scale of 0 to 4) and an increased probability of achieving a higher final grade in the pediatric clerkship (adjusted probability of achieving honors 22%, 95% CI 17% to 28% vs 11%, 95% CI 6% to 16%; P = .003). These differences did not persist in the clerkship immediately after pediatrics. CONCLUSIONS: Compared with a traditional inpatient team, working on a longitudinal team was associated with achieving a higher preceptor rating and final pediatric clerkship grade. Implementing similar models within clinical clerkships may help foster optimal student performance.

Validity Study of an End-of-Clerkship Oral Examination in Obstetrics and Gynecology.

OBJECTIVE: Surgical clerkships frequently include oral exams to assess students' ability to critically analyze data and utilize clinical judgment during common scenarios. Limited guidance exists for the interpretation of oral exam score validity, thus making improvements difficult to target. We examined the development, administration, and scoring of a clerkship oral exam from a validity evidence framework. DESIGN: This was a retrospective study of a third-year, end-of-clerkship oral exam in obstetrics and gynecology (OBGYN). Content, response process, internal structure, and relationship to other variables validity evidence was collected and evaluated for 5 versions of the oral exam. SETTING: Albert Einstein College of Medicine, Bronx, New York City. PARTICIPANTS: Participants were 186 third-year medical students who completed the OBGYN clerkship in the academic year 2020 to 2021. RESULTS: The average number of objectives assessed per oral exam version were uniform, but the distribution of questions per Bloom's level of cognition was uneven. Student scores on all questions regardless of Bloom's level of cognition were >87%, and reliability (Cronbach's alpha) of item scores varied from 0.58 to 0.74. There was a moderate, positive correlation (Spearman's rho) between the oral exam scores and national shelf exam scores (0.35). There were low correlations between oral exam scores and (a) clinical performance ratings (0.14) and (b) formal presentation scores (-0.19). CONCLUSIONS: This study provides an example of how to examine the validity of oral exam scores for targeted improvements. Further modifications are needed before using scores for high stakes decisions. The authors provide recommendations for additional sources of validity evidence to collect in order to better meet the goals of any surgical clerkship oral exam.

Capturing growth curves of medical students' clinical skills performance.

INTRODUCTION: A benefit of a milestone or Entrustable Professional Activity (EPA) assessment framework is the ability to capture longitudinal performance with growth curves using multi-level modelling (MLM). Growth curves can inform curriculum design and individualised learning. Residency programmes have found growth curves to vary by resident and by milestone. Only one study has analysed medical students' growth curves for EPAs. Analysis of EPA growth curves is critical because no change in performance raises concerns for EPAs as an assessment framework. METHODS: Spencer Fox Eccles School of Medicine-University of Utah students' workplace-based assessment ratings for 7 EPAs were captured at 3 time-points in years 3-4 of AY2017-2018 to AY2020-2021. MLM was used to capture EPA growth curves and determine if variation in growth curves was explained by internal medicine (IM) clerkship order. FINDINGS: A curvilinear slope significantly captured 256 students' average ratings overtime for EPA1a-history-taking, EPA2-clinical reasoning, EPA3-diagnostics, EPA5-documentation and EPA6-presentation, and a linear slope significantly captured EPA9-teamwork ratings, p ≤ 0.001. Growth curves were steepest for EPA2-clinical reasoning and EPA3-diagnostics. Growth curves varied by students, p < 0.05 for all EPA ratings, but IM clerkship rotation order did not significantly explain the variance, p > 0.05. DISCUSSION: The increase in ratings from Year 3 to Year 4 provides validity evidence for use of EPAs in an assessment framework. Students may benefit from more curriculum/skills practice for EPA2-clinical reasoning and EPA3-diagnostics prior to year 3. Variation in student's growth curves is important for coaching and skill development; a one size fits all approach may not suffice.

Promotion of a Mastery Orientation to Learning in Medical School: Implementation of the Not Yet Pass Grade.

PROBLEM: Using pass/fail (P/F) course grades may motivate students to perform well enough to earn a passing grade, giving them a false sense of competence and not motivating them to remediate deficiencies. The authors explored whether adding a not yet pass (NYP) grade to a P/F scale would promote students' mastery orientation toward learning. APPROACH: The authors captured student outcomes and data on time and cost of implementing the NYP grade in 2021 at the University of Utah School of Medicine. One cohort of medical students, who had experienced both P/F and P/NYP/F scales in years 1 and 2, completed an adapted Achievement Goal Questionnaire-Revised (AGQ-R) in fall 2021 to measure how well the P/NYP/F grading scale compared with the P/F scale promoted mastery orientation and performance orientation goals. Students who received an NYP grade provided feedback on the NYP process. OUTCOMES: Students reported that the P/NYP/F scale increased their achievement of both mastery and performance orientation goals, with significantly higher ratings for mastery orientation goals than for performance orientation goals on the AGQ-R (response rate = 124/125 [99%], P ≤ .001, effect size = 0.31). Thirty-eight students received 48 NYP grades in 7 courses during 2021, and 3 (2%) failed a subsequent course after receiving an NYP grade. Most NYP students reported the NYP process enabled them to identify and correct a deficiency (32/36 [89%]) and made them feel supported (28/36 [78%]). The process was time intensive (897 hours total for 48 NYP grades), but no extra funding was budgeted. NEXT STEPS: The findings suggest mastery orientation can be increased with an NYP grade. Implementing a P/NYP/F grading scale for years 1 and/or 2 may help students transition to programmatic assessment or no grading later in medical school, which may better prepare graduates for lifelong learning.

Digging Deeper, Zooming Out: Reimagining Legacies in Medical Education.

Although the wide-scale disruption precipitated by the COVID-19 pandemic has somewhat subsided, there are many questions about the implications of such disruptions for the road ahead. This year's Research in Medical Education (RIME) supplement may provide a window of insight. Now, more than ever, researchers are poised to question long-held assumptions while reimagining long-established legacies. Themes regarding the boundaries of professional identity, approaches to difficult conversations, challenges of power and hierarchy, intricacies of selection processes, and complexities of learning climates appear to be the most salient and critical to understand. In this commentary, the authors use the relationship between legacies and assumptions as a framework to gain a deeper understanding about the past, present, and future of RIME.

Development of a novel hospitalist advanced practice provider assessment instrument: A pilot study.

Advanced practice providers (APPs) graduate from school with variable hospitalist experience. While hospitalist-specific onboarding is recommended for hospitalist APPs, no standard method currently exists to assess their readiness for practice. We created a 17-item instrument called the Cardin Hospitalist Advanced Practice Provider-Readiness Assessment (CHAPP-RA) to assess APPs'; readiness for practice using a milestones-based scale. We piloted CHAPP-RA at a single site where 11 APPs with varied experience were rated by 30 supervising physicians. Supervisors also provided global ratings for overall performance. We investigated the feasibility of CHAPP-RA and collected validity evidence for the interpretation of scores. The mean time to complete one CHAPP-RA was 10.5 min. Supervisors rated novice APPs lower than more experienced APPs, p ≤ .001. CHAPP-RA ratings also correlated strongly with global ratings. CHAPP-RA is feasible to implement and has initial validity evidence.

"Am I even a med-student anymore?" A Mixed-Methods Study of the Impact of the Initial Disruptions Caused by the COVID-19 Pandemic on Medical Student Professional Identity Formation.

Purpose: Developing a professional identity requires learners to integrate themselves into the medical profession and take on the role of doctor. The impact of COVID-19 on medical education has been widely investigated, but little attention has been paid to the impact of students' professional identify formation (PIF). The goal of this study was to investigate the impact that the onset of the COVID-19 pandemic had on medical students' PIF. Materials and Methods: An embedded mixed-methods design was utilized. Focus groups were conducted with a subset of year 1-4 students and coded using thematic analysis. Year 1-2 students were surveyed about their professional identity integration in the spring of 2020. Responses were analyzed using descriptive statistics and Wilcoxon signed rank and Mann-Whitney U tests. Results: Qualitative data were organized into six themes that touched on losses and challenges, reflection, and reevaluation of the physician career. Roughly 50% of MS1s and MS2s reported a change in their professional identity integration, but this was not statistically significant. Conclusions: Medical education does not occur in isolation and is influenced by disruptive local and global events. Students perceived challenges when in-person community interaction and hands-on clinical experiences were interrupted. Additionally, students reflected upon their own role and their future career goals. Supplementary Information: The online version contains supplementary material available at 10.1007/s40670-022-01652-4.

Assessment for Learning with Ungraded and Graded Assessments.

Introduction: Assessment for learning has many benefits, but learners will still encounter high-stakes decisions about their performance throughout training. It is unknown if assessment for learning can be promoted with a combination model where scores from some assessments are factored into course grades and scores from other assessments are not used for course grading. Methods: At the University of Utah School of Medicine, year 1-2 medical students (MS) completed multiple-choice question quiz assessments and final examinations in six systems-based science courses. Quiz and final examination performance counted toward course grades for MS2017-MS2018. Starting with the MS2020 cohort, quizzes no longer counted toward course grades. Quiz, final examination, and Step 1 scores were compared between ungraded quiz and graded quiz cohorts with independent samples t-tests. Student and faculty feedback was collected. Results: Quiz performance was not different for the ungraded and graded cohorts (p = 0.173). Ungraded cohorts scored 4% higher on final examinations than graded cohorts (p ≤ 0.001, d = 0.88). Ungraded cohorts scored above the national average and 11 points higher on Step 1 compared to graded cohorts, who had scored below the national average (p ≤ 0.001, d = 0.64). During the study period, Step 1 scores increased by 2 points nationally. Student feedback was positive, and faculty felt it improved their relationship with students. Discussion: The change to ungraded quizzes did not negatively affect final examination or Step 1 performance, suggesting a combination of ungraded and graded assessments can effectively promote assessment for learning.