Gender Differences in Work-Based Assessment Scores and Narrative Comments After Direct Observation.

BACKGROUND: While some prior studies of work-based assessment (WBA) numeric ratings have not shown gender differences, they have been unable to account for the true performance of the resident or explore narrative differences by gender. OBJECTIVE: To explore gender differences in WBA ratings as well as narrative comments (when scripted performance was known). DESIGN: Secondary analysis of WBAs obtained from a randomized controlled trial of a longitudinal rater training intervention in 2018-2019. Participating faculty (n = 77) observed standardized resident-patient encounters and subsequently completed rater assessment forms (RAFs). SUBJECTS: Participating faculty in longitudinal rater training. MAIN MEASURES: Gender differences in mean entrustment ratings (4-point scale) were assessed with multivariable regression (adjusted for scripted performance, rater and resident demographics, and the interaction between study arm and time period [pre- versus post-intervention]). Using pre-specified natural language processing categories (masculine, feminine, agentic, and communal words), multivariable linear regression was used to determine associations of word use in the narrative comments with resident gender, race, and skill level, faculty demographics, and interaction between the study arm and the time period (pre- versus post-intervention). KEY RESULTS: Across 1527 RAFs, there were significant differences in entrustment ratings between women and men standardized residents (2.29 versus 2.54, respectively, p < 0.001) after correction for resident skill level. As compared to men, feminine terms were more common for comments of what the resident did poorly among women residents (ß 0.45, CI 0.12-0.78, p 0.01). This persisted despite adjusting for the faculty's entrustment ratings. There were no other significant linguistic differences by gender. CONCLUSIONS: Contrasting prior studies, we found entrustment rating differences in a simulated WBA which persisted after adjusting for the resident's scripted performance. There were also linguistic differences by gender after adjusting for entrustment ratings, with feminine terms being used more frequently in comments about women in some, but not all narrative comments.

The Context of "Confidence": Analyzing the Term Confidence in Resident Evaluations.

BACKGROUND: Despite similar performance metrics, women medical trainees routinely self-assess their own skills lower than men. The phenomenon of a "confidence gap" between genders, where women report lower self-confidence independent of actual ability or competency, may have an important interaction with gender differences in assessment. Identifying whether there are gender-based differences in how confidence is mentioned in written evaluations is a necessary step to understand the interaction between evaluation and the gender-based confidence gap. OBJECTIVE: To analyze faculty evaluations of internal medicine (IM) residents for gender-based patterns in the use of iterations of "confidence." DESIGN: We performed a retrospective cohort study of all inpatient faculty evaluations of University of Pennsylvania IM residents from 2018 to 2021. We performed n-gram text-mining to identify evaluations containing the terms "confident," "confidence," or "confidently." We performed univariable and multivariable logistic regression to determine the association between resident gender and references to confidence (including comments reflecting too little confidence), adjusting for faculty gender, post-graduate year (PGY), numeric rating, and service. SUBJECTS: University of Pennsylvania IM residents from 2018 to 2021. KEY RESULTS: There were 5416 evaluations of IM residents (165 women [51%], 156 men [49%]) submitted by 356 faculty members (149 women [51%]), of which 7.1 % (n=356) contained references to confidence. There was a significant positive association between the mention of confidence and women resident gender (OR 1.54, CI 1.23-1.92; p<0.001), which persisted after adjustment for faculty gender, numeric rating, and PGY level. Eighty evaluations of the cohort explicitly mentioned the resident having "too little confidence," which was also associated with women resident gender (OR 1.66, CI 1.05-2.62; p=0.031). CONCLUSION: Narrative evaluations of women residents were more likely to contain references to confidence, after adjustment for numerical score, PGY level, and faculty gender, which may perpetuate the gender-based confidence gap, introduce bias, and ultimately impact professional identity development.

Amplifying the Student Voice: Medical Student Perceptions of AΩA.

BACKGROUND: Recent literature has suggested racial disparities in Alpha Omega Alpha Honor Medical Society (AΩA) selection and raised concerns about its effects on the learning environment. Internal reviews at multiple institutions have led to changes in selection practices or suspension of student chapters; in October 2020, the national AΩA organization provided guidance to address these concerns. OBJECTIVE: This study aimed to better understand student opinions of AΩA. DESIGN: An anonymous survey using both multiple response option and free response questions. PARTICIPANTS: Medical students at the Perelman School of Medicine at the University of Pennsylvania. MAIN MEASURES: Descriptive statistics and logistic regressions were used to examine predictors of student opinion towards AΩA. Free responses were analyzed by two independent coders to identify key themes. KEY RESULTS: In total, 70% of the student body (n = 547) completed the survey. Sixty-three percent had a negative opinion of AΩA, and 57% felt AΩA should not exist at the student level. Thirteen percent believed AΩA membership appropriately reflects the student body; 8% thought selection processes were fair. On multivariate analysis, negative predictors of a student's preference to continue AΩA at the student level included belief that AΩA membership does not currently mirror class composition (OR: 0.45, [95% CI: 0.23-0.89]) and that AΩA selection processes were unfair (OR: 0.20 [0.08-0.47]). Self-perception as not competitive for AΩA selection was also a negative predictor (OR: 0.44 [0.22-0.88]). Major qualitative themes included equity, impact on the learning environment, transparency, and positive aspects of AΩA. CONCLUSIONS: This single-institution survey demonstrated significant student concerns regarding AΩA selection fairness and effects on the learning environment. Many critiques extended beyond AΩA itself, instead focusing on the perceived magnification of existing disparities in the learning environment. As the national conversation about AΩA continues, engaging student voices in the discussion is critical.

A coach of my own.

This personal view about medical student self-feedback is based on personal experiences as a third-year medical student with insights from a director of undergraduate medical education for a department of medicine. We highlight the importance of adult-learning theory in the clinical arena, and suggest that the application of self-regulated learning theory during clinical clerkships may contribute to increased student engagement and partnership in learning.

Overnight Hospital Experiences for Medical Students: Results of the 2014 Clerkship Directors in Internal Medicine National Survey.

BACKGROUND: Since the 2011 Accreditation Council of Graduate Medical Education (ACGME) work hour rules for residents were implemented, 24-30 h call for interns has been replaced by shift work, including night-float. The impact of these changes on undergraduate medical education experiences in internal medicine has not been described. OBJECTIVE: We aimed to determine the current status of medical students' overnight experiences in Internal Medicine clerkships and sub-internships, and to assess internal medicine educators' perceptions of the importance of overnight work during internal medicine rotations. DESIGN AND PARTICIPANTS: In May 2014, the Clerkship Directors in Internal Medicine (CDIM) conducted its annual survey. Twenty-eight questions about student participation in overnight work and perceptions of the importance of overnight work (rated on 1-5 Likert scale, 1 = very unimportant and 5 = ery important) were included. Descriptive statistics were used to summarize responses. Free text results were analyzed qualitatively. KEY RESULTS: The response rate was 78 %. A minority of respondents reported students having any overnight experience during the clerkship (38.7 %) or the sub-internship (40.7 %). Only 5 % of respondents reported having students assigned to night-float rotations outside of clerkships or sub-internships. Respondents agreed that overnight experiences were more important during the sub-internship than the clerkship, 4.0 ± 1.1 vs. 3.2 ± 1.2, p < 0.001. Admitting new patients, following their course and responding to emergencies were rated as important overnight tasks for both clerkship and sub-internship students. CONCLUSIONS: Overnight experiences offer students additional educational opportunities. Clerkship directors felt that the overnight experience for the sub-intern in particular was an important chance to practice providing emergency cross coverage and other intern roles. In the era of ACGME duty hours, there is a need to further examine whether there is a role for increased overnight hospital experiences for medical students.

How faculty members experience workplace-based assessment rater training: a qualitative study.

CONTEXT: Direct observation of clinical skills is a common approach in workplace-based assessment (WBA). Despite widespread use of the mini-clinical evaluation exercise (mini-CEX), faculty development efforts are typically required to improve assessment quality. Little consensus exists regarding the most effective training methods, and few studies explore faculty members' reactions to rater training. OBJECTIVES: This study was conducted to qualitatively explore the experiences of faculty staff with two rater training approaches - performance dimension training (PDT) and a modified approach to frame of reference training (FoRT) - to elucidate how such faculty development can be optimally designed. METHODS: In a qualitative study of a multifaceted intervention using complex intervention principles, 45 out-patient resident faculty preceptors from 26 US internal medicine residency programmes participated in a rater training faculty development programme. All participants were interviewed individually and in focus groups during and after the programme to elicit how the training influenced their approach to assessment. A constructivist grounded theory approach was used to analyse the data. RESULTS: Many participants perceived that rater training positively influenced their approach to direct observation and feedback, their ability to use entrustment as the standard for assessment, and their own clinical skills. However, barriers to implementation and change included: (i) a preference for holistic assessment over frameworks; (ii) challenges in defining competence; (iii) difficulty in changing one's approach to assessment, and (iv) concerns about institutional culture and buy-in. CONCLUSIONS: Rater training using PDT and a modified approach to FoRT can provide faculty staff with assessment skills that are congruent with principles of criterion-referenced assessment and entrustment, and foundational principles of competency-based education, while providing them with opportunities to reflect on their own clinical skills. However, multiple challenges to incorporating new forms of training exist. Ongoing efforts to improve WBA are needed to address institutional and cultural contexts, and systems of care delivery.

How clinical supervisors develop trust in their trainees: a qualitative study.

CONTEXT: Clinical supervisors oversee trainees' performance while granting them increasing opportunities to work independently. Although the factors contributing to supervisors' trust in their trainees to conduct clinical work have been identified, how the development of trust is shaped by these factors remains less clear. OBJECTIVES: This study was designed to determine how supervisors develop and experience trust in resident (postgraduate years 2 and 3) trainees in the clinical workplace. METHODS: Internal medicine in-patient supervisors at two institutions were interviewed about the meaning and experience of developing trust in resident trainees. Transcribed data were coded and analysed using a phenomenographic approach. RESULTS: Forty-three supervisors participated. Supervisors characterised the meaning of trust from the perspectives of trainee competence and leadership or from their own perspective of needing to provide more or less supervision. Supervisors initially considered trust to be usually independent of prior knowledge of the resident, and then used sources of information about trust to develop their judgements of trust. Sources, which incorporated inference, included supervisors' comparisons with a standard, direct observation of the trainee as a team leader or care provider, and stakeholder input from team members, patients and families. Barriers against and accelerators to trust formation related to the resident, supervisor, resident-supervisor relationship, context and task. Trust formation had implications for supervisors' roles, residents' increasingly independent provision of care, and team functioning. CONCLUSIONS: From a general starting point, supervisors develop trust in residents informed by observation, inference and information gathered from the team and patients. Judgements of trust yield outcomes defined by supervisors' changing roles, the increasingly independent provision of care by residents, and team functioning. The implications of these findings for graded resident autonomy aligned with learning needs can inform the design of training environments to enable readiness for unsupervised practice.

The effect of resident duty-hours restrictions on internal medicine clerkship experiences: surveys of medical students and clerkship directors.

UNLABELLED: PHENOMENON: Medical students receive much of their inpatient teaching from residents who now experience restructured teaching services to accommodate the 2011 duty-hour regulations (DHR). The effect of DHR on medical student educational experiences is unknown. We examined medical students' and clerkship directors' perceptions of the effects of the 2011 DHR on internal medicine clerkship students' experiences with teaching, feedback and evaluation, and patient care. APPROACH: Students at 14 institutions responded to surveys after their medicine clerkship or subinternship. Students who completed their clerkship (n = 839) and subinternship (n = 228) March to June 2011 (pre-DHR historical controls) were compared to clerkship students (n = 895) and subinterns (n = 377) completing these rotations March to June 2012 (post-DHR). Z tests for proportions correcting for multiple comparisons were performed to assess attitude changes. The Clerkship Directors in Internal Medicine annual survey queried institutional members about the 2011 DHR just after implementation. FINDINGS: Survey response rates were 64% and 50% for clerkship students and 60% and 48% for subinterns in 2011 and 2012 respectively, and 82% (99/121) for clerkship directors. Post-DHR, more clerkship students agreed that attendings (p =.011) and interns (p =.044) provided effective teaching. Clerkship students (p =.013) and subinterns (p =.001) believed patient care became more fragmented. The percentage of holdover patients clerkship students (p =.001) and subinterns (p =.012) admitted increased. Clerkship directors perceived negative effects of DHR for students on all survey items. Most disagreed that interns (63.1%), residents (67.8%), or attendings (71.1%) had more time to teach. Most disagreed that students received more feedback from interns (56.0%) or residents (58.2%). Fifty-nine percent felt that students participated in more patient handoffs. INSIGHTS: Students perceive few adverse consequences of the 2011 DHR on their internal medicine experiences, whereas their clerkship director educators have negative perceptions. Future research should explore the impact of fragmented patient care on the student-patient relationship and students' clinical skills acquisition.

Content and timing of feedback and reflection: a multi-center qualitative study of experienced bedside teachers.

BACKGROUND: Competency-based medical education increasingly recognizes the importance of observation, feedback, and reflection for trainee development. Although bedside rounds provide opportunities for authentic workplace-based implementation of feedback and team-based reflection strategies, this relationship has not been well described. The authors sought to understand the content and timing of feedback and team-based reflection provided by bedside teachers in the context of patient-centered bedside rounds. METHODS: The authors conducted a thematic analysis qualitative study using transcripts from audio-recorded, semi-structured telephone interviews with internal medicine attending physicians (n= 34) identified as respected bedside teachers from 10 academic US institutions (2010-2011). RESULTS: Half of the respondents (50%) were associate/full professors, with an average of 14 years of academic experience. In the context of bedside encounters, bedside teachers reported providing feedback on history-taking, physical-examination, and case-presentation skills, patient-centered communication, clinical decision-making, leadership, teaching skills, and professionalism. Positive feedback about physical-exam skills or clinical decision-making occurred during encounters, positive or constructive team-based feedback occurred immediately following encounters, and individualized constructive feedback occurred in one-on-one settings following rounding sessions. Compared to less frequent, emotionally-charged events, bedside teachers initiated team-based reflection on commonplace "teachable moments" related to patient characteristics or emotions, trainee actions and emotions, and attending physician role modeling. CONCLUSIONS: Bedside teachers use bedside rounds as a workplace-based method to provide assessment, feedback, and reflection, which are aligned with the goals of competency-based medical education. Embedded in patient-centered activities, clinical teachers should be encouraged to incorporate these content- and timing-related feedback and reflection strategies into their bedside teaching.

Racial and ethnic bias in letters of recommendation in academic medicine: a systematic review.

PURPOSE: Letters of recommendations (LORs) are key components of academic medicine applications. Given that bias against students and trainees underrepresented in medicine (UIM) has been demonstrated across assessment, achievement, and advancement domains, the authors reviewed studies on LORs to assess racial, ethnic, and UIM differences in LORs. Standardized LORs (SLORs), an increasingly common form of LORs, were also assessed for racial and ethnic differences. METHOD: A systematic review was conducted for English-language studies that assessed racial or ethnic differences in LORs in academic medicine published from database inception to July 16, 2023. Studies evaluating SLORs underwent data abstraction to evaluate their impact on the given race or ethnicity comparison and outcome variables. RESULTS: Twenty-three studies describing 19,012 applicants and 41,925 LORs were included. Nineteen studies (82.6%) assessed LORs for residency, 4 (17.4%) assessed LORs for fellowship, and none evaluated employment or promotion. Fifteen of 17 studies (88.2%) assessing linguistic differences reported a significant difference in a particular race or ethnicity comparison. Of the 7 studies assessing agentic language (e.g., "strong," "confident"), 1 study found fewer agentic terms used for Black and Latinx applicants, and 1 study reported higher agency scores for Asian applicants and applicants of races other than White. There were mixed results for the use of communal and grindstone language in UIM and non-UIM comparisons. Among 6 studies, 4 (66.7%) reported that standout language (e.g., "exceptional," "outstanding") was less likely to be ascribed to UIM applicants. Doubt-raising language was more frequently used for UIM trainees. When SLORs and unstructured LORs were compared, fewer linguistic differences were found in SLORs. CONCLUSIONS: There is a moderate bias against UIM candidates in the domains of linguistic differences, doubt-raising language, and topics discussed in LORs, which has implications for perceptions of competence and ability in the high-stakes residency and fellowship application process.

Accuracy of Entrustment-Based Assessment: Implications for Programs and Patients.

Background Although entrustment-supervision ratings are more intuitive compared to other rating scales, it is not known whether their use accurately assesses the appropriateness of care provided by a resident. Objective To determine the frequency of incorrect entrustment ratings assigned by faculty and whether accuracy of an entrustment-supervision scale differed by resident performance when the scripted resident performance level is known. Methods Faculty participants rated standardized residents in 10 videos using a 4-point entrustment-supervision scale. We calculated the frequency of rating a resident incorrectly. We performed generalizability (G) and decision (D) studies for all 10 cases (768 ratings) and repeated the analysis using only cases with an entrustment score of 2. Results The mean score by 77 raters for all videos was 2.87 (SD=0.86) with a mean of 2.37 (SD=0.72), 3.11 (SD=0.67) and 3.78 (SD=0.43) for the scripted levels of 2, 3, and 4. Faculty ratings differed from the scripted score for 331of 768 (43%) ratings. Most errors were ratings higher than the scripted score (223, 67%). G studies estimated the variance proportions of rater and case to be 4.99% and 54.29%. D studies estimated that 3 raters would need to watch 10 cases. The variance proportion of rater was 8.5% when the analysis was restricted to level 2 entrustment, requiring 15 raters to watch 5 cases. Conclusions Participants underestimated residents' potential need for greater supervision. Overall agreement between raters and scripted scores were low.

Using peers to assess handoffs: a pilot study.

BACKGROUND: Handoffs among post-graduate year 1 (PGY1) trainees occur with high frequency. Peer assessment of handoff competence would add a new perspective on how well the handoff information helped them to provide optimal patient care. OBJECTIVE: The goals of this study were to test the feasibility of the approach of an instrument for peer assessment of handoffs by meeting criteria of being able to use technology to capture evaluations in real time, exhibiting strong psychometric properties, and having high PGY1 satisfaction scores. DESIGN: An iPad® application was built for a seven-item handoff instrument. Over a two-month period, post-call PGY1s completed assessments of three co-PGY1s from whom they received handoffs the prior evening. PARTICIPANTS: Internal Medicine PGY1s at the University of Pennsylvania. MAIN MEASURES: ANOVA was used to explore interperson score differences (validity). Generalizability analyses provided estimates of score precision (reproducibility). PGY1s completed satisfaction surveys about the process. KEY RESULTS: Sixty-two PGY1s (100 %) participated in the study. 59 % of the targeted evaluations were completed. The major limitations were network connectivity and inability to find the post-call trainee. PGY1 scores on the single item of "overall competency" ranged from 4 to 9 with a mean of 7.31 (SD 1.09). Generalizability coefficients approached 0.60 for 10 evaluations per PGY1 for a single rotation and 12 evaluations per PGY1 across multiple rotations. The majority of PGY1s believed that they could adequately assess handoff competence and that the peer assessment process was valuable (70 and 77 %, respectively). CONCLUSION: Psychometric properties of an instrument for peer assessment of handoffs are encouraging. Obtaining 10 or 12 evaluations per PGY1 allowed for reliable assessment of handoff skills. Peer evaluations of handoffs using mobile technology were feasible, and were well received by PGY1s.

The art of bedside rounds: a multi-center qualitative study of strategies used by experienced bedside teachers.

BACKGROUND: Medical educators believe bedside rounds are effective for the delivery of patient-centered care, and are necessary in helping trainees acquire competence in clinical care. Although recommendations for bedside rounds have been reported, a recent, systematic assessment of strategies used by current-day bedside teachers was needed to advance knowledge of this teaching method. OBJECTIVE: To identify and understand bedside teachers' 1) preparatory steps, 2) patient selection, and 3) role allocation during the process of bedside rounds. DESIGN: A qualitative inductive thematic analysis using transcripts from audio-recorded, semi-structured telephone interviews. PARTICIPANTS: Internal medicine physicians (n = 34) who perform bedside rounds from ten academic US institutions (2010-2011). APPROACH: A purposive sampling strategy was utilized to identify physicians who were active inpatient attending physicians and met specific inclusion criteria for "bedside rounds." A total of 34 interviews were completed, and each was recorded and transcribed verbatim. A thematic analysis extracted key themes pertaining to the three objectives. KEY RESULTS: Most respondents (51 %) were associate or full professors, with an average of 14 years of academic experience. Attending physicians prepared using trainee-specific, patient-specific and disease-specific information, while also mentally preparing for bedside rounds. They sought trainee buy-in and learning objectives, reviewed expectations and methods to ensure patient comfort, and provided early guidance with bedside encounters. Patients were selected if they required immediate care, were new to the service, or had a high educational value, while patients were deferred if unavailable, unwilling, or unable to communicate. The team members' roles during bedside rounds varied, with trainees being given graduated autonomy with increased experience. CONCLUSIONS: Bedside teachers' methods for preparation, patient selection, and role allocation during bedside rounds enhance trainees' education within the workplace. Strategies used by experienced bedside teachers can be used for faculty development efforts aimed at promoting this activity.

Realizing the promise and importance of performance-based assessment.

Work-based assessment (WBA) is the assessment of trainees and physicians across the educational continuum of day-to-day competencies and practices in authentic, clinical environments. What distinguishes WBA from other assessment modalities is that it enables the evaluation of performance in context. In this perspective, we describe the growing importance, relevance, and evolution of WBA as it relates to competency-based medical education, supervision, and entrustment. Although a systematic review is beyond the purview of this perspective, we highlight specific methods and needed shifts to WBA that (a) consider patient outcomes, (b) use nonphysician assessors, and (c) assess the care provided to populations of patients. We briefly describe strategies for the effective implementation of WBA and identify outstanding research questions related to its use.

The value of bedside rounds: a multicenter qualitative study.

BACKGROUND: Bedside rounds have decreased on teaching services, raising concern about trainees' clinical skills and patient-physician relationships. PURPOSE: We sought to identify recognized bedside teachers' perceived value of bedside rounds to assist in the promotion of bedside rounds on teaching services. METHODS: Authors used a grounded theory, qualitative study design of telephone semistructured interviews with bedside teachers (n = 34) from 10 U.S. institutions (2010-2011). Main outcomes were characteristics of participants, themes pertaining to the perceived value of bedside rounds, and quotations highlighting each respective theme. RESULTS: The mean years in academic medicine was 13.7, and 51% were associate or full professors. Six main themes emerged: (a) skill development for learners (e.g., physical examination, communication, and clinical decision-making skills); (b) observation and feedback; (c) role-modeling; (d) team building among trainees, attending, and patient; (e) improved patient care delivery through combined clinical decision-making and team consensus; and (f) the culture of medicine as patient-centered care, which was embodied in all themes. CONCLUSIONS: Bedside teachers identify potential benefits of bedside rounds, many of which align with national calls to change our approach to medical education. The practice of bedside rounds enables activities essential to high-quality patient care and education.

Gender-Based Differences in Language Used by Students to Describe Their Noteworthy Characteristics in Medical Student Performance Evaluations.

PURPOSE: The noteworthy characteristic (NC) section of the medical student performance evaluation (MSPE) was introduced to facilitate holistic review of residency applications and mitigate biases. The student-written aspect of the characteristics, however, may introduce biases resulting from gender differences in self-promotion behaviors. The authors conducted an exploratory analysis of potential gender-based differences in language used in NCs. METHOD: The authors performed a single-center cohort analysis of all student-written NCs at the Perelman School of Medicine (2018-2022). NCs were converted into single words and characterized into word categories: ability (e.g., "talent"), standout ("best"), grindstone ("meticulous"), communal ("caring"), or agentic ("ambitious"). The authors qualitatively analyzed NC topic characteristics (i.e., focused on scholarship, community service). Logistic regression was used to identify gender differences in word categories and topics used in NCs. RESULTS: The cohort included 2,084 characteristics from 783 MSPEs (47.5%, n = 371 written by women). After adjusting for underrepresented in medicine status, honor society membership, and intended specialty, men were more likely to use standout (OR = 2.00; 95% confidence interval [CI] = 1.35, 2.96; P = .001) and communal (OR = 1.40; 95% CI = 1.03, 1.90; P = .03) words in their NCs compared with women but less likely to use grindstone words (OR = 0.72; 95% CI = 0.53, 0.98; P = .04). Men were more likely than women to discuss scholarship (OR = 2.03; 95% CI = 1.27, 3.23; P = .003), hobbies (OR = 1.45; 95% CI = 1.07, 1.96; P = .02), and/or awards (OR = 1.59; 95% CI = 1.16, 2.16; P = .004) and less likely to highlight community service (OR = 0.66; 95% CI = 0.48, 0.92; P = .02). CONCLUSIONS: The self-written nature of NCs permits language differences that may contribute to gender bias in residency application.

Faculty Perceptions of Frame of Reference Training to Improve Workplace-Based Assessment.

Background: Workplace-based assessment (WBA) is a key assessment strategy in competency-based medical education. However, its full potential has not been actualized secondary to concerns with reliability, validity, and accuracy. Frame of reference training (FORT), a rater training technique that helps assessors distinguish between learner performance levels, can improve the accuracy and reliability of WBA, but the effect size is variable. Understanding FORT benefits and challenges help improve this rater training technique. Objective: To explore faculty's perceptions of the benefits and challenges associated with FORT. Methods: Subjects were internal medicine and family medicine physicians (n=41) who participated in a rater training intervention in 2018 consisting of in-person FORT followed by asynchronous online spaced learning. We assessed participants' perceptions of FORT in post-workshop focus groups and an end-of-study survey. Focus groups and survey free text responses were coded using thematic analysis. Results: All subjects participated in 1 of 4 focus groups and completed the survey. Four benefits of FORT were identified: (1) opportunity to apply skills frameworks via deliberate practice; (2) demonstration of the importance of certain evidence-based clinical skills; (3) practice that improved the ability to discriminate between resident skill levels; and (4) highlighting the importance of direct observation and the dangers using proxy information in assessment. Challenges included time constraints and task repetitiveness. Conclusions: Participants believe that FORT training serves multiple purposes, including helping them distinguish between learner skill levels while demonstrating the impact of evidence-based clinical skills and the importance of direct observation.

Can Rater Training Improve the Quality and Accuracy of Workplace-Based Assessment Narrative Comments and Entrustment Ratings? A Randomized Controlled Trial.

PURPOSE: Prior research evaluating workplace-based assessment (WBA) rater training effectiveness has not measured improvement in narrative comment quality and accuracy, nor accuracy of prospective entrustment-supervision ratings. The purpose of this study was to determine whether rater training, using performance dimension and frame of reference training, could improve WBA narrative comment quality and accuracy. A secondary aim was to assess impact on entrustment rating accuracy. METHOD: This single-blind, multi-institution, randomized controlled trial of a multifaceted, longitudinal rater training intervention consisted of in-person training followed by asynchronous online spaced learning. In 2018, investigators randomized 94 internal medicine and family medicine physicians involved with resident education. Participants assessed 10 scripted standardized resident-patient videos at baseline and follow-up. Differences in holistic assessment of narrative comment accuracy and specificity, accuracy of individual scenario observations, and entrustment rating accuracy were evaluated with t tests. Linear regression assessed impact of participant demographics and baseline performance. RESULTS: Seventy-seven participants completed the study. At follow-up, the intervention group (n = 41), compared with the control group (n = 36), had higher scores for narrative holistic specificity (2.76 vs 2.31, P < .001, Cohen V = .25), accuracy (2.37 vs 2.06, P < .001, Cohen V = .20) and mean quantity of accurate (6.14 vs 4.33, P < .001), inaccurate (3.53 vs 2.41, P < .001), and overall observations (2.61 vs 1.92, P = .002, Cohen V = .47). In aggregate, the intervention group had more accurate entrustment ratings (58.1% vs 49.7%, P = .006, Phi = .30). Baseline performance was significantly associated with performance on final assessments. CONCLUSIONS: Quality and specificity of narrative comments improved with rater training; the effect was mitigated by inappropriate stringency. Training improved accuracy of prospective entrustment-supervision ratings, but the effect was more limited. Participants with lower baseline rating skill may benefit most from training.

Development of a student-created internal medicine frameworks website for healthcare trainees.

OBJECTIVES: Describe medical student perspectives on framework learning and develop a free, online, mobile-friendly framework website. METHODS: Internal medicine clerkship students were surveyed at a single U.S. medical school regarding how they learn frameworks. We used Draw.io to create frameworks, which were edited by expert clinicians. Frameworks were hosted online through an academic server, and Google analytics was used to track website activity. RESULTS: Most medical students report learning frameworks from attending clinicians. We developed 87 frameworks on the "Penn Frameworks'' website, which was visited by 9,539 unique users from 124 countries over three years. CONCLUSIONS: Most medical students perceive that they learn frameworks during clinical rotations from attending clinicians. We found that it is feasible to develop a low-cost, expert-curated, mobile-friendly resource to supplement in-person learning.

The Urgency of Now: Rethinking and Improving Assessment Practices in Medical Education Programs.

Assessment is essential to professional development. Assessment provides the information needed to give feedback, support coaching and the creation of individualized learning plans, inform progress decisions, determine appropriate supervision levels, and, most importantly, help ensure patients and families receive high-quality, safe care in the training environment. While the introduction of competency-based medical education has catalyzed advances in assessment, much work remains to be done. First, becoming a physician (or other health professional) is primarily a developmental process, and assessment programs must be designed using a developmental and growth mindset. Second, medical education programs must have integrated programs of assessment that address the interconnected domains of implicit, explicit and structural bias. Third, improving programs of assessment will require a systems-thinking approach. In this paper, the authors first address these overarching issues as key principles that must be embraced so that training programs may optimize assessment to ensure all learners achieve desired medical education outcomes. The authors then explore specific needs in assessment and provide suggestions to improve assessment practices. This paper is by no means inclusive of all medical education assessment challenges or possible solutions. However, there is a wealth of current assessment research and practice that medical education programs can use to improve educational outcomes and help reduce the harmful effects of bias. The authors' goal is to help improve and guide innovation in assessment by catalyzing further conversations.