Racial and ethnic bias in letters of recommendation in academic medicine: a systematic review.

PURPOSE: Letters of recommendations (LORs) are key components of academic medicine applications. Given that bias against students and trainees underrepresented in medicine (UIM) has been demonstrated across assessment, achievement, and advancement domains, the authors reviewed studies on LORs to assess racial, ethnic, and UIM differences in LORs. Standardized LORs (SLORs), an increasingly common form of LORs, were also assessed for racial and ethnic differences. METHOD: A systematic review was conducted for English-language studies that assessed racial or ethnic differences in LORs in academic medicine published from database inception to July 16, 2023. Studies evaluating SLORs underwent data abstraction to evaluate their impact on the given race or ethnicity comparison and outcome variables. RESULTS: Twenty-three studies describing 19,012 applicants and 41,925 LORs were included. Nineteen studies (82.6%) assessed LORs for residency, 4 (17.4%) assessed LORs for fellowship, and none evaluated employment or promotion. Fifteen of 17 studies (88.2%) assessing linguistic differences reported a significant difference in a particular race or ethnicity comparison. Of the 7 studies assessing agentic language (e.g., "strong," "confident"), 1 study found fewer agentic terms used for Black and Latinx applicants, and 1 study reported higher agency scores for Asian applicants and applicants of races other than White. There were mixed results for the use of communal and grindstone language in UIM and non-UIM comparisons. Among 6 studies, 4 (66.7%) reported that standout language (e.g., "exceptional," "outstanding") was less likely to be ascribed to UIM applicants. Doubt-raising language was more frequently used for UIM trainees. When SLORs and unstructured LORs were compared, fewer linguistic differences were found in SLORs. CONCLUSIONS: There is a moderate bias against UIM candidates in the domains of linguistic differences, doubt-raising language, and topics discussed in LORs, which has implications for perceptions of competence and ability in the high-stakes residency and fellowship application process.

Accuracy of Entrustment-Based Assessment: Implications for Programs and Patients.

Background Although entrustment-supervision ratings are more intuitive compared to other rating scales, it is not known whether their use accurately assesses the appropriateness of care provided by a resident. Objective To determine the frequency of incorrect entrustment ratings assigned by faculty and whether accuracy of an entrustment-supervision scale differed by resident performance when the scripted resident performance level is known. Methods Faculty participants rated standardized residents in 10 videos using a 4-point entrustment-supervision scale. We calculated the frequency of rating a resident incorrectly. We performed generalizability (G) and decision (D) studies for all 10 cases (768 ratings) and repeated the analysis using only cases with an entrustment score of 2. Results The mean score by 77 raters for all videos was 2.87 (SD=0.86) with a mean of 2.37 (SD=0.72), 3.11 (SD=0.67) and 3.78 (SD=0.43) for the scripted levels of 2, 3, and 4. Faculty ratings differed from the scripted score for 331of 768 (43%) ratings. Most errors were ratings higher than the scripted score (223, 67%). G studies estimated the variance proportions of rater and case to be 4.99% and 54.29%. D studies estimated that 3 raters would need to watch 10 cases. The variance proportion of rater was 8.5% when the analysis was restricted to level 2 entrustment, requiring 15 raters to watch 5 cases. Conclusions Participants underestimated residents' potential need for greater supervision. Overall agreement between raters and scripted scores were low.

Gender Differences in Work-Based Assessment Scores and Narrative Comments After Direct Observation.

BACKGROUND: While some prior studies of work-based assessment (WBA) numeric ratings have not shown gender differences, they have been unable to account for the true performance of the resident or explore narrative differences by gender. OBJECTIVE: To explore gender differences in WBA ratings as well as narrative comments (when scripted performance was known). DESIGN: Secondary analysis of WBAs obtained from a randomized controlled trial of a longitudinal rater training intervention in 2018-2019. Participating faculty (n = 77) observed standardized resident-patient encounters and subsequently completed rater assessment forms (RAFs). SUBJECTS: Participating faculty in longitudinal rater training. MAIN MEASURES: Gender differences in mean entrustment ratings (4-point scale) were assessed with multivariable regression (adjusted for scripted performance, rater and resident demographics, and the interaction between study arm and time period [pre- versus post-intervention]). Using pre-specified natural language processing categories (masculine, feminine, agentic, and communal words), multivariable linear regression was used to determine associations of word use in the narrative comments with resident gender, race, and skill level, faculty demographics, and interaction between the study arm and the time period (pre- versus post-intervention). KEY RESULTS: Across 1527 RAFs, there were significant differences in entrustment ratings between women and men standardized residents (2.29 versus 2.54, respectively, p < 0.001) after correction for resident skill level. As compared to men, feminine terms were more common for comments of what the resident did poorly among women residents (ß 0.45, CI 0.12-0.78, p 0.01). This persisted despite adjusting for the faculty's entrustment ratings. There were no other significant linguistic differences by gender. CONCLUSIONS: Contrasting prior studies, we found entrustment rating differences in a simulated WBA which persisted after adjusting for the resident's scripted performance. There were also linguistic differences by gender after adjusting for entrustment ratings, with feminine terms being used more frequently in comments about women in some, but not all narrative comments.

Preclerkship Medical Students' Use of Third-Party Learning Resources.

Importance: Undergraduate medical education increasingly relies on asynchronous, virtual learning; and medical educators have observed students engaging in self-directed learning outside of their institutional curriculum using widely available third-party resources. If medical educators better understand how students are learning, they may uncover novel opportunities to improve preclerkship education. Objective: To explore how and why preclerkship medical students use third-party learning resources. Design, Setting, and Participants: This qualitative study recruited second-year medical students from 7 public and private allopathic US medical schools and conducted 7 virtual focus groups (1 per institution) from September 2022 to January 2023, exploring how and why students use third-party resources. Data were iteratively analyzed in parallel with focus groups using constructivist grounded theory methodology. Data analysis was performed from October 2022 to February 2023. Results: Fifty-eight second-year US medical students who had used a third-party resource at least once participated; 36 (61%) identified as women; 13 (23%) identified as Asian, 6 (11%) as Black, 30 (53%) as White, 6 (11%) as multiracial, and 4 (7%) as other; 6 (10%) identified as Hispanic, Latino, or Spanish origin, and 52 (90%) identified as non-Hispanic, Latino, or Spanish origin; 48 (83%) were aged 23 to 25 years. Participants described engaging in a cyclical process of deciding whether and how to use third-party resources. Four broad themes were identified: (1) hearing about resources, (2) selecting resources, (3) using resources, and (4) tensions and possible solutions. Participants largely heard about third-party resources from peers and turned to resources out of dissatisfaction with some aspect of their medical school curriculum. Students used resources in various ways that were user-dependent and context-dependent. Participants endorsed multiple benefits over their in-house curricula, particularly efficiency, clarity, and concision. Tensions included navigating resource drawbacks and the perception of an antagonistic relationship between medical schools and third-party resources. Participants suggested that medical schools examine the resources, recommend specific ones, integrate them into the curriculum, and subsidize their cost. Conclusions and Relevance: In this qualitative study of preclerkship medical student use of third-party resources, participants perceived that the resources had numerous benefits for learning and suggested that medical schools should more formally acknowledge and integrate their use.

Development of a student-created internal medicine frameworks website for healthcare trainees.

OBJECTIVES: Describe medical student perspectives on framework learning and develop a free, online, mobile-friendly framework website. METHODS: Internal medicine clerkship students were surveyed at a single U.S. medical school regarding how they learn frameworks. We used Draw.io to create frameworks, which were edited by expert clinicians. Frameworks were hosted online through an academic server, and Google analytics was used to track website activity. RESULTS: Most medical students report learning frameworks from attending clinicians. We developed 87 frameworks on the "Penn Frameworks'' website, which was visited by 9,539 unique users from 124 countries over three years. CONCLUSIONS: Most medical students perceive that they learn frameworks during clinical rotations from attending clinicians. We found that it is feasible to develop a low-cost, expert-curated, mobile-friendly resource to supplement in-person learning.

The Urgency of Now: Rethinking and Improving Assessment Practices in Medical Education Programs.

Assessment is essential to professional development. Assessment provides the information needed to give feedback, support coaching and the creation of individualized learning plans, inform progress decisions, determine appropriate supervision levels, and, most importantly, help ensure patients and families receive high-quality, safe care in the training environment. While the introduction of competency-based medical education has catalyzed advances in assessment, much work remains to be done. First, becoming a physician (or other health professional) is primarily a developmental process, and assessment programs must be designed using a developmental and growth mindset. Second, medical education programs must have integrated programs of assessment that address the interconnected domains of implicit, explicit and structural bias. Third, improving programs of assessment will require a systems-thinking approach. In this paper, the authors first address these overarching issues as key principles that must be embraced so that training programs may optimize assessment to ensure all learners achieve desired medical education outcomes. The authors then explore specific needs in assessment and provide suggestions to improve assessment practices. This paper is by no means inclusive of all medical education assessment challenges or possible solutions. However, there is a wealth of current assessment research and practice that medical education programs can use to improve educational outcomes and help reduce the harmful effects of bias. The authors' goal is to help improve and guide innovation in assessment by catalyzing further conversations.

Natural Language Processing of Learners' Evaluations of Attendings to Identify Professionalism Lapses.

Unprofessional faculty behaviors negatively impact the well-being of trainees yet are infrequently reported through established reporting systems. Manual review of narrative faculty evaluations provides an additional avenue for identifying unprofessional behavior but is time- and resource-intensive, and therefore of limited value for identifying and remediating faculty with professionalism concerns. Natural language processing (NLP) techniques may provide a mechanism for streamlining manual review processes to identify faculty professionalism lapses. In this retrospective cohort study of 15,432 narrative evaluations of medical faculty by medical trainees, we identified professionalism lapses using automated analysis of the text of faculty evaluations. We used multiple NLP approaches to develop and validate several classification models, which were evaluated primarily based on the positive predictive value (PPV) and secondarily by their calibration. A NLP-model using sentiment analysis (quantifying subjectivity of the text) in combination with key words (using the ensemble technique) had the best performance overall with a PPV of 49% (CI 38%-59%). These findings highlight how NLP can be used to screen narrative evaluations of faculty to identify unprofessional faculty behaviors. Incorporation of NLP into faculty review workflows enables a more focused manual review of comments, providing a supplemental mechanism to identify faculty professionalism lapses.

Faculty Perceptions of Frame of Reference Training to Improve Workplace-Based Assessment.

Background: Workplace-based assessment (WBA) is a key assessment strategy in competency-based medical education. However, its full potential has not been actualized secondary to concerns with reliability, validity, and accuracy. Frame of reference training (FORT), a rater training technique that helps assessors distinguish between learner performance levels, can improve the accuracy and reliability of WBA, but the effect size is variable. Understanding FORT benefits and challenges help improve this rater training technique. Objective: To explore faculty's perceptions of the benefits and challenges associated with FORT. Methods: Subjects were internal medicine and family medicine physicians (n=41) who participated in a rater training intervention in 2018 consisting of in-person FORT followed by asynchronous online spaced learning. We assessed participants' perceptions of FORT in post-workshop focus groups and an end-of-study survey. Focus groups and survey free text responses were coded using thematic analysis. Results: All subjects participated in 1 of 4 focus groups and completed the survey. Four benefits of FORT were identified: (1) opportunity to apply skills frameworks via deliberate practice; (2) demonstration of the importance of certain evidence-based clinical skills; (3) practice that improved the ability to discriminate between resident skill levels; and (4) highlighting the importance of direct observation and the dangers using proxy information in assessment. Challenges included time constraints and task repetitiveness. Conclusions: Participants believe that FORT training serves multiple purposes, including helping them distinguish between learner skill levels while demonstrating the impact of evidence-based clinical skills and the importance of direct observation.

Gender-Based Differences in Language Used by Students to Describe Their Noteworthy Characteristics in Medical Student Performance Evaluations.

PURPOSE: The noteworthy characteristic (NC) section of the medical student performance evaluation (MSPE) was introduced to facilitate holistic review of residency applications and mitigate biases. The student-written aspect of the characteristics, however, may introduce biases resulting from gender differences in self-promotion behaviors. The authors conducted an exploratory analysis of potential gender-based differences in language used in NCs. METHOD: The authors performed a single-center cohort analysis of all student-written NCs at the Perelman School of Medicine (2018-2022). NCs were converted into single words and characterized into word categories: ability (e.g., "talent"), standout ("best"), grindstone ("meticulous"), communal ("caring"), or agentic ("ambitious"). The authors qualitatively analyzed NC topic characteristics (i.e., focused on scholarship, community service). Logistic regression was used to identify gender differences in word categories and topics used in NCs. RESULTS: The cohort included 2,084 characteristics from 783 MSPEs (47.5%, n = 371 written by women). After adjusting for underrepresented in medicine status, honor society membership, and intended specialty, men were more likely to use standout (OR = 2.00; 95% confidence interval [CI] = 1.35, 2.96; P = .001) and communal (OR = 1.40; 95% CI = 1.03, 1.90; P = .03) words in their NCs compared with women but less likely to use grindstone words (OR = 0.72; 95% CI = 0.53, 0.98; P = .04). Men were more likely than women to discuss scholarship (OR = 2.03; 95% CI = 1.27, 3.23; P = .003), hobbies (OR = 1.45; 95% CI = 1.07, 1.96; P = .02), and/or awards (OR = 1.59; 95% CI = 1.16, 2.16; P = .004) and less likely to highlight community service (OR = 0.66; 95% CI = 0.48, 0.92; P = .02). CONCLUSIONS: The self-written nature of NCs permits language differences that may contribute to gender bias in residency application.

Can Rater Training Improve the Quality and Accuracy of Workplace-Based Assessment Narrative Comments and Entrustment Ratings? A Randomized Controlled Trial.

PURPOSE: Prior research evaluating workplace-based assessment (WBA) rater training effectiveness has not measured improvement in narrative comment quality and accuracy, nor accuracy of prospective entrustment-supervision ratings. The purpose of this study was to determine whether rater training, using performance dimension and frame of reference training, could improve WBA narrative comment quality and accuracy. A secondary aim was to assess impact on entrustment rating accuracy. METHOD: This single-blind, multi-institution, randomized controlled trial of a multifaceted, longitudinal rater training intervention consisted of in-person training followed by asynchronous online spaced learning. In 2018, investigators randomized 94 internal medicine and family medicine physicians involved with resident education. Participants assessed 10 scripted standardized resident-patient videos at baseline and follow-up. Differences in holistic assessment of narrative comment accuracy and specificity, accuracy of individual scenario observations, and entrustment rating accuracy were evaluated with t tests. Linear regression assessed impact of participant demographics and baseline performance. RESULTS: Seventy-seven participants completed the study. At follow-up, the intervention group (n = 41), compared with the control group (n = 36), had higher scores for narrative holistic specificity (2.76 vs 2.31, P < .001, Cohen V = .25), accuracy (2.37 vs 2.06, P < .001, Cohen V = .20) and mean quantity of accurate (6.14 vs 4.33, P < .001), inaccurate (3.53 vs 2.41, P < .001), and overall observations (2.61 vs 1.92, P = .002, Cohen V = .47). In aggregate, the intervention group had more accurate entrustment ratings (58.1% vs 49.7%, P = .006, Phi = .30). Baseline performance was significantly associated with performance on final assessments. CONCLUSIONS: Quality and specificity of narrative comments improved with rater training; the effect was mitigated by inappropriate stringency. Training improved accuracy of prospective entrustment-supervision ratings, but the effect was more limited. Participants with lower baseline rating skill may benefit most from training.

The Context of "Confidence": Analyzing the Term Confidence in Resident Evaluations.

BACKGROUND: Despite similar performance metrics, women medical trainees routinely self-assess their own skills lower than men. The phenomenon of a "confidence gap" between genders, where women report lower self-confidence independent of actual ability or competency, may have an important interaction with gender differences in assessment. Identifying whether there are gender-based differences in how confidence is mentioned in written evaluations is a necessary step to understand the interaction between evaluation and the gender-based confidence gap. OBJECTIVE: To analyze faculty evaluations of internal medicine (IM) residents for gender-based patterns in the use of iterations of "confidence." DESIGN: We performed a retrospective cohort study of all inpatient faculty evaluations of University of Pennsylvania IM residents from 2018 to 2021. We performed n-gram text-mining to identify evaluations containing the terms "confident," "confidence," or "confidently." We performed univariable and multivariable logistic regression to determine the association between resident gender and references to confidence (including comments reflecting too little confidence), adjusting for faculty gender, post-graduate year (PGY), numeric rating, and service. SUBJECTS: University of Pennsylvania IM residents from 2018 to 2021. KEY RESULTS: There were 5416 evaluations of IM residents (165 women [51%], 156 men [49%]) submitted by 356 faculty members (149 women [51%]), of which 7.1 % (n=356) contained references to confidence. There was a significant positive association between the mention of confidence and women resident gender (OR 1.54, CI 1.23-1.92; p<0.001), which persisted after adjustment for faculty gender, numeric rating, and PGY level. Eighty evaluations of the cohort explicitly mentioned the resident having "too little confidence," which was also associated with women resident gender (OR 1.66, CI 1.05-2.62; p=0.031). CONCLUSION: Narrative evaluations of women residents were more likely to contain references to confidence, after adjustment for numerical score, PGY level, and faculty gender, which may perpetuate the gender-based confidence gap, introduce bias, and ultimately impact professional identity development.

Amplifying the Student Voice: Medical Student Perceptions of AΩA.

BACKGROUND: Recent literature has suggested racial disparities in Alpha Omega Alpha Honor Medical Society (AΩA) selection and raised concerns about its effects on the learning environment. Internal reviews at multiple institutions have led to changes in selection practices or suspension of student chapters; in October 2020, the national AΩA organization provided guidance to address these concerns. OBJECTIVE: This study aimed to better understand student opinions of AΩA. DESIGN: An anonymous survey using both multiple response option and free response questions. PARTICIPANTS: Medical students at the Perelman School of Medicine at the University of Pennsylvania. MAIN MEASURES: Descriptive statistics and logistic regressions were used to examine predictors of student opinion towards AΩA. Free responses were analyzed by two independent coders to identify key themes. KEY RESULTS: In total, 70% of the student body (n = 547) completed the survey. Sixty-three percent had a negative opinion of AΩA, and 57% felt AΩA should not exist at the student level. Thirteen percent believed AΩA membership appropriately reflects the student body; 8% thought selection processes were fair. On multivariate analysis, negative predictors of a student's preference to continue AΩA at the student level included belief that AΩA membership does not currently mirror class composition (OR: 0.45, [95% CI: 0.23-0.89]) and that AΩA selection processes were unfair (OR: 0.20 [0.08-0.47]). Self-perception as not competitive for AΩA selection was also a negative predictor (OR: 0.44 [0.22-0.88]). Major qualitative themes included equity, impact on the learning environment, transparency, and positive aspects of AΩA. CONCLUSIONS: This single-institution survey demonstrated significant student concerns regarding AΩA selection fairness and effects on the learning environment. Many critiques extended beyond AΩA itself, instead focusing on the perceived magnification of existing disparities in the learning environment. As the national conversation about AΩA continues, engaging student voices in the discussion is critical.

Will Any Road Get You There? Examining Warranted and Unwarranted Variation in Medical Education.

Undergraduate and graduate medical education have long embraced uniqueness and variability in curricular and assessment approaches. Some of this variability is justified (warranted or necessary variation), but a substantial portion represents unwarranted variation. A primary tenet of outcomes-based medical education is ensuring that all learners acquire essential competencies to be publicly accountable to meet societal needs. Unwarranted variation in curricular and assessment practices contributes to suboptimal and variable educational outcomes and, by extension, risks graduates delivering suboptimal health care quality. Medical education can use lessons from the decades of study on unwarranted variation in health care as part of efforts to continuously improve the quality of training programs. To accomplish this, medical educators will first need to recognize the difference between warranted and unwarranted variation in both clinical care and educational practices. Addressing unwarranted variation will require cooperation and collaboration between multiple levels of the health care and educational systems using a quality improvement mindset. These efforts at improvement should acknowledge that some aspects of variability are not scientifically informed and do not support desired outcomes or societal needs. This perspective examines the correlates of unwarranted variation of clinical care in medical education and the need to address the interdependency of unwarranted variation occurring between clinical and educational practices. The authors explore the challenges of variation across multiple levels: community, institution, program, and individual faculty members. The article concludes with recommendations to improve medical education by embracing the principles of continuous quality improvement to reduce the harmful effect of unwarranted variation.

The Dissolution of the Step 2 Clinical Skills Examination and the Duty of Medical Educators to Step Up the Effectiveness of Clinical Skills Assessment.

In this Invited Commentary, the authors explore the implications of the dissolution of the Step 2 Clinical Skills Examination (Step 2 CS) for medical student clinical skills assessment. The authors describe the need for medical educators (at both the undergraduate and graduate levels) to work collaboratively to improve medical student clinical skills assessment to assure the public that medical school graduates have the requisite skills to begin residency training. The authors outline 6 specific recommendations for how to capitalize on the discontinuation of Step 2 CS to improve clinical skills assessment: (1) defining national, end-of-clerkship, and transition-to-residency standards for required clinical skills and for levels of competence; (2) creating a national resource for standardized patient, augmented reality, and virtual reality assessments; (3) improving workplace-based assessment through local collaborations and national resources; (4) improving learner engagement in and coproduction of assessments; (5) requiring, as a new standard for accreditation, medical schools to establish and maintain competency committees; and (6) establishing a national registry of assessment data for research and evaluation. Together, these actions will help the medical education community earn the public's trust by enhancing the rigor of assessment to ensure the mastery of skills that are essential to providing safe, high-quality care for patients.

Idiosyncrasy in Assessment Comments: Do Faculty Have Distinct Writing Styles When Completing In-Training Evaluation Reports?

PURPOSE: Written comments are gaining traction as robust sources of assessment data. Compared with the structure of numeric scales, what faculty choose to write is ad hoc, leading to idiosyncratic differences in what is recorded. This study offers exploration of what aspects of writing styles are determined by the faculty offering comment and what aspects are determined by the trainee being commented upon. METHOD: The authors compiled in-training evaluation report comment data, generated from 2012 to 2015 by 4 large North American Internal Medicine training programs. The Linguistic Index and Word Count (LIWC) was used to categorize and quantify the language contained. Generalizability theory was used to determine whether faculty could be reliably discriminated from one another based on writing style. Correlations and ANOVAs were used to determine what styles were related to faculty or trainee demographics. RESULTS: Datasets contained 23-142 faculty who provided 549-2,666 assessments on 161-989 trainees. Faculty could easily be discriminated from one another using a variety of LIWC metrics including word count, words per sentence, and the use of "clout" words. These patterns appeared person specific and did not reflect demographic factors such as gender or rank. These metrics were similarly not consistently associated with trainee factors such as postgraduate year or gender. CONCLUSIONS: Faculty seem to have detectable writing styles that are relatively stable across the trainees they assess, which may represent an under-recognized source of construct irrelevance. If written comments are to meaningfully contribute to decision making, we need to understand and account for idiosyncratic writing styles.

Sparking Change: How a Shift to Step 1 Pass/Fail Scoring Could Promote the Educational and Catalytic Effects of Assessment in Medical Education.

This Invited Commentary explores the implications of the upcoming shift to pass/fail scoring for the United States Medical Licensing Examination Step 1 that was announced in February 2020. Using the frameworks of validity and growth/mastery mindset, the authors describe how this change in Step 1 score reporting presents an opportunity to advance the meaningful assessment of medical students and improve the transition from undergraduate medical education (UME) to graduate medical education (GME). The authors explore the potential risks and consequences associated with a narrow implementation of pass/fail scoring. Alternatively, with a shift in scoring, educators can leverage consequential validity, particularly the educational and catalytic effects of assessment. The authors emphasize the importance of assessment that motivates learners to prepare in a fashion that benefits their learning and that provides results and feedback that motivate all stakeholders to create, enhance, and support education. The authors suggest that this use of assessment will require a culture change from fixed mindset to growth mindset among both students and educators, a shift from static to dynamic testing, and a transition to assessment that focuses on a learner's trajectory. The authors conclude by emphasizing the importance of work-based assessment and the need for improved transparency across the UME-GME continuum.

A coach of my own.

This personal view about medical student self-feedback is based on personal experiences as a third-year medical student with insights from a director of undergraduate medical education for a department of medicine. We highlight the importance of adult-learning theory in the clinical arena, and suggest that the application of self-regulated learning theory during clinical clerkships may contribute to increased student engagement and partnership in learning.