Gender Differences in Work-Based Assessment Scores and Narrative Comments After Direct Observation.

BACKGROUND: While some prior studies of work-based assessment (WBA) numeric ratings have not shown gender differences, they have been unable to account for the true performance of the resident or explore narrative differences by gender. OBJECTIVE: To explore gender differences in WBA ratings as well as narrative comments (when scripted performance was known). DESIGN: Secondary analysis of WBAs obtained from a randomized controlled trial of a longitudinal rater training intervention in 2018-2019. Participating faculty (n = 77) observed standardized resident-patient encounters and subsequently completed rater assessment forms (RAFs). SUBJECTS: Participating faculty in longitudinal rater training. MAIN MEASURES: Gender differences in mean entrustment ratings (4-point scale) were assessed with multivariable regression (adjusted for scripted performance, rater and resident demographics, and the interaction between study arm and time period [pre- versus post-intervention]). Using pre-specified natural language processing categories (masculine, feminine, agentic, and communal words), multivariable linear regression was used to determine associations of word use in the narrative comments with resident gender, race, and skill level, faculty demographics, and interaction between the study arm and the time period (pre- versus post-intervention). KEY RESULTS: Across 1527 RAFs, there were significant differences in entrustment ratings between women and men standardized residents (2.29 versus 2.54, respectively, p < 0.001) after correction for resident skill level. As compared to men, feminine terms were more common for comments of what the resident did poorly among women residents (ß 0.45, CI 0.12-0.78, p 0.01). This persisted despite adjusting for the faculty's entrustment ratings. There were no other significant linguistic differences by gender. CONCLUSIONS: Contrasting prior studies, we found entrustment rating differences in a simulated WBA which persisted after adjusting for the resident's scripted performance. There were also linguistic differences by gender after adjusting for entrustment ratings, with feminine terms being used more frequently in comments about women in some, but not all narrative comments.

The Impact of Trainee and Standardized Patient Race and Gender on Internal Medicine Resident Communication Assessment Scores.

BACKGROUND: Standardized patient (SP) encounters are commonly used to assess communication skills in medical training. The impact of SP and resident demographics on the standardized communication ratings in residents has not been evaluated. OBJECTIVE: To examine the impact of gender and race on SP assessments of internal medicine (IM) residents' communication skills during postgraduate year (PGY) 1. METHODS: We performed a retrospective cohort study of all SP assessments of IM PGY-1 residents for a standardized communication exercise from 2012 to 2018. We performed descriptive analyses of numeric communication SP ratings by gender, race, and age (for residents and SPs). A generalized estimating equation model, clustered on individual SP, was used to determine the association of gender (among SP and residents) with communication ratings. A secondary analysis was performed to determine the impact of residents and SP racial concordance in communication scores. RESULTS: There were 1356 SP assessments of 379 IM residents (199 male residents [53%] and 178 female residents [47%]). There were significant differences in average numeric communication rating (mean 3.40 vs 3.34, P = .009) by gender of resident, with higher scores in female residents. There were no significant interactions between SP and resident gender across the communication domains. There were no significant interactions noted with racial concordance between interns and SPs. CONCLUSIONS: Our data demonstrate an association of resident gender on ratings in standardized communication exercises, across multiple communication skills. There was not an interaction impact for gender or racial concordance between SPs and interns.

Longitudinal Milestone Assessment Extending Through Subspecialty Training: The Relationship Between ACGME Internal Medicine Residency Milestones and Subsequent Pulmonary and Critical Care Fellowship Milestones.

PURPOSE: Accreditation Council for Graduate Medical Education (ACGME) milestones were implemented across medical subspecialties in 2015. Although milestones were proposed as a longitudinal assessment tool potentially providing opportunities for early implementation of individualized fellowship learning plans, the association of subspecialty fellowship ratings with prior residency ratings remains unclear. This study aimed to assess the relationship between internal medicine (IM) residency milestones and pulmonary and critical care medicine (PCCM) fellowship milestones. METHOD: A multicenter retrospective cohort analysis was conducted for all PCCM trainees in ACGME-accredited PCCM fellowship programs, 2017-2018, who had complete prior IM milestone ratings from 2014 to 2017. Only professionalism and interpersonal and communication skills (ICS) were included based on shared anchors between IM and PCCM milestones. Using a generalized estimating equations model, the association of PCCM milestones ≤ 2.5 during the first fellowship year with corresponding IM subcompetencies was assessed at each time point, nested by program. Statistical significance was determined using logistic regression. RESULTS: The study included 354 unique PCCM fellows. For ICS and professionalism subcompetencies, fellows with higher IM ratings were less likely to obtain PCCM ratings ≤ 2.5 during the first fellowship year. Each ICS subcompetency was significantly associated with future lapses in fellowship (ICS01: ß = -0.67, P = .003; ICS02: ß = -0.70, P = .001; ICS03: ß = -0.60, P = .004) at various residency time points. Similar associations were noted for PROF03 (ß = -0.57, P = .007). CONCLUSIONS: Findings demonstrated an association between IM milestone ratings and low milestone ratings during PCCM fellowship. IM trainees with low ratings in several professionalism and ICS subcompetencies were more likely to be rated ≤ 2.5 during the first PCCM fellowship year. This highlights a potential use of longitudinal milestones to target educational gaps at the beginning of PCCM fellowship.

Miles to Go Before We Sleep. Reforming the Pulmonary and Critical Care Milestones to Improve Trainee Assessment.

The Accreditation Council for Graduate Medical Education (ACGME) Milestones are a systematic assessment framework for medical trainees within the six core competencies of practice. Their use by internal medicine subspecialties, including semiannual reports to the ACGME, was mandated beginning in 2014. The Milestones, which were based on specific, observable behaviors, improved upon the prior subjective, global comparisons of each fellow with an "average" fellow in his or her field and served the goals of competency-based medical education. However, the original set of Milestones has proven challenging to apply and interpret. Part of the challenge stems from the use of identical Milestones across all medicine subspecialties, which led to unclear relevance of the patient care and medical knowledge domains to the practice of pulmonary and critical care. This also precluded their use for individualized feedback or development of a learning plan for fellows. In addition, verbose behavioral descriptors, which were designed to provide specificity, ultimately led to rater fatigue among assessors and clinical competency committees. Therefore, the ACGME convened committees for each of the medical subspecialties to revise the original Milestones in an effort to improve subspecialty relevance, minimize educational jargon, and simplify the current iteration. New patient care and medical knowledge Milestones were created to be subspecialty specific and improve utility. The remaining four Milestones were developed as a common set of shorter Milestones, harmonized across specialties. For pulmonary, critical care, and combined fellowship programs, the resulting Milestones 2.0 aims to simplify the use, implementation, and interpretation of this framework for program directors, trainees, and society.

Assessment of Gender-based Qualitative Differences within Trainee Evaluations of Faculty.

Rationale: Recent studies of trainee evaluations of medical faculty have demonstrated subtle gender-based word choice differences. However, it is not known if this manifests in major contextual differences in written comments.Objectives: To characterize qualitative differences in narrative evaluations of female and male pulmonary and critical care medicine faculty.Methods: We performed a single-center, retrospective cohort analysis of trainee narrative evaluations of pulmonary and critical care medicine faculty at the Hospital of the University of Pennsylvania, written from 2015 to 2016. Directive and summative content analysis was performed by four raters. Major contextual themes were identified using constant comparative techniques. Thematic differences based on faculty gender were identified, with statistical significance determined using χ2 analysis (P < 0.05). Effect sizes were calculated using Cramér's V.Results: A total of 1,216 total narrative evaluations were analyzed, representing 62 faculty members (17 women and 45 men), with analysis continued for the full cohort beyond achieving saturation of themes. Five overarching themes emerged: teaching skills, clinical skills, supervision, interpersonal and communication skills, and leadership skills. Within subthemes, we found no significant gender differences in reference to general teaching skills, learning environment, enthusiasm for teaching, or interpersonal concern. We identified subtheme differences between male and female faculty evaluations in regard to mentions of learner autonomy, clinical learning environment, humor, and motivating the trainee for patient care. Although the mention of constructive criticisms did not differ between genders, constructive criticisms of female faculty were more likely to mention demeanor (P = 0.06).Conclusions: Our data suggest minimal thematic differences in trainee narrative evaluations of male and female pulmonary and critical care medicine faculty. However, we noted several subtheme contextual differences between male and female faculty, which warrants further investigation.

Assessment of Gender-Based Linguistic Differences in Physician Trainee Evaluations of Medical Faculty Using Automated Text Mining.

Importance: Women are underrepresented at higher ranks in academic medicine. However, the factors contributing to this disparity have not been fully elucidated. Implicit bias and unconscious mental attitudes toward a person or group may be factors. Although academic medical centers use physician trainee evaluations of faculty to inform promotion decisions, little is known about gender bias in these evaluations. To date, no studies have examined narrative evaluations of medical faculty by physician trainees for differences based on gender. Objective: To characterize gender-associated linguistic differences in narrative evaluations of medical faculty written by physician trainees. Design, Setting, and Participants: This retrospective cohort study included all faculty teaching evaluations completed for the department of medicine faculty by medical students, residents, and fellows at a large academic center in Pennsylvania from July 1, 2015, through June 30, 2016. Data analysis was performed from June 1, 2018, through July 31, 2018. Main Outcomes and Measures: Word use in faculty evaluations was quantified using automated text mining by converting free-text comments into unique 1- and 2-word phrases. Mixed-effects logistic regression analysis was performed to assess associations of faculty gender with frequencies of specific words and phrases present in a physician trainee evaluation. Results: A total of 7326 unique evaluations were collected for 521 faculty (325 men [62.4%] and 196 women [37.6%]). The individual words art (odds ratio [OR], 7.78; 95% CI, 1.01-59.89), trials (OR, 4.43; 95% CI, 1.34-14.69), master (OR, 4.24; 95% CI, 1.69-10.63), and humor (OR, 2.32; 95% CI, 1.44-3.73) were significantly associated with evaluations of male faculty, whereas the words empathetic (OR, 4.34; 95% CI, 1.56-12.07), delight (OR, 4.26; 95% CI, 1.35-13.40), and warm (OR, 3.45; 95% CI, 1.83-6.49) were significantly associated with evaluations of female faculty. Two-word phrases associated with male faculty evaluations included run rounds (OR, 7.78; 95% CI, 1.01-59.84), big picture (OR, 7.15; 95% CI, 1.68-30.42), and master clinician (OR, 4.02; 95% CI, 1.21-13.36), whereas evaluations of female faculty were more likely to be associated with model physician (OR, 7.75; 95% CI, 1.70-35.39), just right (OR, 6.97; 95% CI, 1.51-32.30), and attention (to) detail (OR, 4.26; 95% CI, 1.36-13.40). Conclusions and Relevance: The data showed quantifiable linguistic differences between free-text comments about male and female faculty in physician trainee evaluations. Further evaluation of these differences, particularly in association with ongoing gender disparities in faculty promotion and retention, may be warranted.