Gender Differences in Work-Based Assessment Scores and Narrative Comments After Direct Observation.

BACKGROUND: While some prior studies of work-based assessment (WBA) numeric ratings have not shown gender differences, they have been unable to account for the true performance of the resident or explore narrative differences by gender. OBJECTIVE: To explore gender differences in WBA ratings as well as narrative comments (when scripted performance was known). DESIGN: Secondary analysis of WBAs obtained from a randomized controlled trial of a longitudinal rater training intervention in 2018-2019. Participating faculty (n = 77) observed standardized resident-patient encounters and subsequently completed rater assessment forms (RAFs). SUBJECTS: Participating faculty in longitudinal rater training. MAIN MEASURES: Gender differences in mean entrustment ratings (4-point scale) were assessed with multivariable regression (adjusted for scripted performance, rater and resident demographics, and the interaction between study arm and time period [pre- versus post-intervention]). Using pre-specified natural language processing categories (masculine, feminine, agentic, and communal words), multivariable linear regression was used to determine associations of word use in the narrative comments with resident gender, race, and skill level, faculty demographics, and interaction between the study arm and the time period (pre- versus post-intervention). KEY RESULTS: Across 1527 RAFs, there were significant differences in entrustment ratings between women and men standardized residents (2.29 versus 2.54, respectively, p < 0.001) after correction for resident skill level. As compared to men, feminine terms were more common for comments of what the resident did poorly among women residents (ß 0.45, CI 0.12-0.78, p 0.01). This persisted despite adjusting for the faculty's entrustment ratings. There were no other significant linguistic differences by gender. CONCLUSIONS: Contrasting prior studies, we found entrustment rating differences in a simulated WBA which persisted after adjusting for the resident's scripted performance. There were also linguistic differences by gender after adjusting for entrustment ratings, with feminine terms being used more frequently in comments about women in some, but not all narrative comments.

The Context of "Confidence": Analyzing the Term Confidence in Resident Evaluations.

BACKGROUND: Despite similar performance metrics, women medical trainees routinely self-assess their own skills lower than men. The phenomenon of a "confidence gap" between genders, where women report lower self-confidence independent of actual ability or competency, may have an important interaction with gender differences in assessment. Identifying whether there are gender-based differences in how confidence is mentioned in written evaluations is a necessary step to understand the interaction between evaluation and the gender-based confidence gap. OBJECTIVE: To analyze faculty evaluations of internal medicine (IM) residents for gender-based patterns in the use of iterations of "confidence." DESIGN: We performed a retrospective cohort study of all inpatient faculty evaluations of University of Pennsylvania IM residents from 2018 to 2021. We performed n-gram text-mining to identify evaluations containing the terms "confident," "confidence," or "confidently." We performed univariable and multivariable logistic regression to determine the association between resident gender and references to confidence (including comments reflecting too little confidence), adjusting for faculty gender, post-graduate year (PGY), numeric rating, and service. SUBJECTS: University of Pennsylvania IM residents from 2018 to 2021. KEY RESULTS: There were 5416 evaluations of IM residents (165 women [51%], 156 men [49%]) submitted by 356 faculty members (149 women [51%]), of which 7.1 % (n=356) contained references to confidence. There was a significant positive association between the mention of confidence and women resident gender (OR 1.54, CI 1.23-1.92; p<0.001), which persisted after adjustment for faculty gender, numeric rating, and PGY level. Eighty evaluations of the cohort explicitly mentioned the resident having "too little confidence," which was also associated with women resident gender (OR 1.66, CI 1.05-2.62; p=0.031). CONCLUSION: Narrative evaluations of women residents were more likely to contain references to confidence, after adjustment for numerical score, PGY level, and faculty gender, which may perpetuate the gender-based confidence gap, introduce bias, and ultimately impact professional identity development.

Gender-Based Differences in Language Used by Students to Describe Their Noteworthy Characteristics in Medical Student Performance Evaluations.

PURPOSE: The noteworthy characteristic (NC) section of the medical student performance evaluation (MSPE) was introduced to facilitate holistic review of residency applications and mitigate biases. The student-written aspect of the characteristics, however, may introduce biases resulting from gender differences in self-promotion behaviors. The authors conducted an exploratory analysis of potential gender-based differences in language used in NCs. METHOD: The authors performed a single-center cohort analysis of all student-written NCs at the Perelman School of Medicine (2018-2022). NCs were converted into single words and characterized into word categories: ability (e.g., "talent"), standout ("best"), grindstone ("meticulous"), communal ("caring"), or agentic ("ambitious"). The authors qualitatively analyzed NC topic characteristics (i.e., focused on scholarship, community service). Logistic regression was used to identify gender differences in word categories and topics used in NCs. RESULTS: The cohort included 2,084 characteristics from 783 MSPEs (47.5%, n = 371 written by women). After adjusting for underrepresented in medicine status, honor society membership, and intended specialty, men were more likely to use standout (OR = 2.00; 95% confidence interval [CI] = 1.35, 2.96; P = .001) and communal (OR = 1.40; 95% CI = 1.03, 1.90; P = .03) words in their NCs compared with women but less likely to use grindstone words (OR = 0.72; 95% CI = 0.53, 0.98; P = .04). Men were more likely than women to discuss scholarship (OR = 2.03; 95% CI = 1.27, 3.23; P = .003), hobbies (OR = 1.45; 95% CI = 1.07, 1.96; P = .02), and/or awards (OR = 1.59; 95% CI = 1.16, 2.16; P = .004) and less likely to highlight community service (OR = 0.66; 95% CI = 0.48, 0.92; P = .02). CONCLUSIONS: The self-written nature of NCs permits language differences that may contribute to gender bias in residency application.

Natural Language Processing of Learners' Evaluations of Attendings to Identify Professionalism Lapses.

Unprofessional faculty behaviors negatively impact the well-being of trainees yet are infrequently reported through established reporting systems. Manual review of narrative faculty evaluations provides an additional avenue for identifying unprofessional behavior but is time- and resource-intensive, and therefore of limited value for identifying and remediating faculty with professionalism concerns. Natural language processing (NLP) techniques may provide a mechanism for streamlining manual review processes to identify faculty professionalism lapses. In this retrospective cohort study of 15,432 narrative evaluations of medical faculty by medical trainees, we identified professionalism lapses using automated analysis of the text of faculty evaluations. We used multiple NLP approaches to develop and validate several classification models, which were evaluated primarily based on the positive predictive value (PPV) and secondarily by their calibration. A NLP-model using sentiment analysis (quantifying subjectivity of the text) in combination with key words (using the ensemble technique) had the best performance overall with a PPV of 49% (CI 38%-59%). These findings highlight how NLP can be used to screen narrative evaluations of faculty to identify unprofessional faculty behaviors. Incorporation of NLP into faculty review workflows enables a more focused manual review of comments, providing a supplemental mechanism to identify faculty professionalism lapses.

Build It and They Shall Come: Medical Education Communities of Practice.

Background: Producing scholarship in education is essential to the career development of a clinician-educator. Challenges to scholarly production include a lack of resources, time, expertise, and collaborators. Objective: To develop communities of practice for education scholarship through an international society to increase community and academic productivity. Methods: We developed multi-institutional scholarship pods within the American Thoracic Society through the creation of a working group (2017-2019). Pods met virtually, and meetings were goal focused to advance education scholarship within their area of interest. To understand the impact of these scholarship pods, we surveyed pod leaders and members in 2021 and analyzed the academic productivity of each pod via a survey of pod leaders and a review of the PubMed index. Results: Nine pods were created, each with an assigned educational topic. The survey had a response rate of 76.6%. The perceived benefits were the opportunity to meet colleagues with similar interests at other institutions, production of scholarly work, and engagement in new experiences. The main challenges were difficulty finding times to meet because of competing clinical demands and aligning times among pod members. Regarding academic productivity, eight publications, four conference presentations, and one webinar/podcast were produced by six of the nine pods. Conclusion: The development of communities of practice resulted in increased multi-site collaboration, with boosted academic productivity as well as an enhanced sense of belonging. Multiple challenges remain but can likely be overcome with accountability, early discussion of roles and expectations, and clear delegation of tasks and authorship.

The Impact of Trainee and Standardized Patient Race and Gender on Internal Medicine Resident Communication Assessment Scores.

BACKGROUND: Standardized patient (SP) encounters are commonly used to assess communication skills in medical training. The impact of SP and resident demographics on the standardized communication ratings in residents has not been evaluated. OBJECTIVE: To examine the impact of gender and race on SP assessments of internal medicine (IM) residents' communication skills during postgraduate year (PGY) 1. METHODS: We performed a retrospective cohort study of all SP assessments of IM PGY-1 residents for a standardized communication exercise from 2012 to 2018. We performed descriptive analyses of numeric communication SP ratings by gender, race, and age (for residents and SPs). A generalized estimating equation model, clustered on individual SP, was used to determine the association of gender (among SP and residents) with communication ratings. A secondary analysis was performed to determine the impact of residents and SP racial concordance in communication scores. RESULTS: There were 1356 SP assessments of 379 IM residents (199 male residents [53%] and 178 female residents [47%]). There were significant differences in average numeric communication rating (mean 3.40 vs 3.34, P = .009) by gender of resident, with higher scores in female residents. There were no significant interactions between SP and resident gender across the communication domains. There were no significant interactions noted with racial concordance between interns and SPs. CONCLUSIONS: Our data demonstrate an association of resident gender on ratings in standardized communication exercises, across multiple communication skills. There was not an interaction impact for gender or racial concordance between SPs and interns.

Longitudinal Milestone Assessment Extending Through Subspecialty Training: The Relationship Between ACGME Internal Medicine Residency Milestones and Subsequent Pulmonary and Critical Care Fellowship Milestones.

PURPOSE: Accreditation Council for Graduate Medical Education (ACGME) milestones were implemented across medical subspecialties in 2015. Although milestones were proposed as a longitudinal assessment tool potentially providing opportunities for early implementation of individualized fellowship learning plans, the association of subspecialty fellowship ratings with prior residency ratings remains unclear. This study aimed to assess the relationship between internal medicine (IM) residency milestones and pulmonary and critical care medicine (PCCM) fellowship milestones. METHOD: A multicenter retrospective cohort analysis was conducted for all PCCM trainees in ACGME-accredited PCCM fellowship programs, 2017-2018, who had complete prior IM milestone ratings from 2014 to 2017. Only professionalism and interpersonal and communication skills (ICS) were included based on shared anchors between IM and PCCM milestones. Using a generalized estimating equations model, the association of PCCM milestones ≤ 2.5 during the first fellowship year with corresponding IM subcompetencies was assessed at each time point, nested by program. Statistical significance was determined using logistic regression. RESULTS: The study included 354 unique PCCM fellows. For ICS and professionalism subcompetencies, fellows with higher IM ratings were less likely to obtain PCCM ratings ≤ 2.5 during the first fellowship year. Each ICS subcompetency was significantly associated with future lapses in fellowship (ICS01: ß = -0.67, P = .003; ICS02: ß = -0.70, P = .001; ICS03: ß = -0.60, P = .004) at various residency time points. Similar associations were noted for PROF03 (ß = -0.57, P = .007). CONCLUSIONS: Findings demonstrated an association between IM milestone ratings and low milestone ratings during PCCM fellowship. IM trainees with low ratings in several professionalism and ICS subcompetencies were more likely to be rated ≤ 2.5 during the first PCCM fellowship year. This highlights a potential use of longitudinal milestones to target educational gaps at the beginning of PCCM fellowship.

Miles to Go Before We Sleep. Reforming the Pulmonary and Critical Care Milestones to Improve Trainee Assessment.

The Accreditation Council for Graduate Medical Education (ACGME) Milestones are a systematic assessment framework for medical trainees within the six core competencies of practice. Their use by internal medicine subspecialties, including semiannual reports to the ACGME, was mandated beginning in 2014. The Milestones, which were based on specific, observable behaviors, improved upon the prior subjective, global comparisons of each fellow with an "average" fellow in his or her field and served the goals of competency-based medical education. However, the original set of Milestones has proven challenging to apply and interpret. Part of the challenge stems from the use of identical Milestones across all medicine subspecialties, which led to unclear relevance of the patient care and medical knowledge domains to the practice of pulmonary and critical care. This also precluded their use for individualized feedback or development of a learning plan for fellows. In addition, verbose behavioral descriptors, which were designed to provide specificity, ultimately led to rater fatigue among assessors and clinical competency committees. Therefore, the ACGME convened committees for each of the medical subspecialties to revise the original Milestones in an effort to improve subspecialty relevance, minimize educational jargon, and simplify the current iteration. New patient care and medical knowledge Milestones were created to be subspecialty specific and improve utility. The remaining four Milestones were developed as a common set of shorter Milestones, harmonized across specialties. For pulmonary, critical care, and combined fellowship programs, the resulting Milestones 2.0 aims to simplify the use, implementation, and interpretation of this framework for program directors, trainees, and society.

Development and Implementation of a Novel Learner-driven Precepting Model for Pulmonary Fellowship.

Background: Ambulatory education is currently underemphasized in pulmonary and critical care medicine (PCCM) fellowship training. Existing clinic precepting models, originally developed for students and residents, do not meet the unique needs of fellow-level trainees. Objective: We aimed to develop and implement a novel fellow-led precepting model to improve ambulatory education for PCCM trainees. Methods: We performed a mixed-methods needs assessment, including surveys, focus groups, and direct observations, to explore perceptions of ambulatory training, define current precepting practices, and identify target areas for improvement. On the basis of these findings, we developed, implemented, and evaluated a novel model for PCCM outpatient precepting. Results: A targeted needs assessment identified that current precepting practices did not meet fellows' needs for graduated autonomy, development of assessment and management skills, and self-directed learning. We developed and implemented a novel, learner-driven precepting model (Set the Stage, Tell the Story, Educational Goals, Preliminary Plan, Uncertainties, Plan Recap [STEP-UP]), designed to prioritize clinical reasoning skills and self-identification of learning goals. Implementation of the STEP-UP model improved perceptions of overall outpatient training and precepting. However, we faced several barriers to uptake of the new model, including increased cognitive burden of integrating a new process. Conclusion: A robust assessment of ambulatory education at a single PCCM fellowship program identified a need to align precepting processes with the unique goals of advanced trainees. We developed a learner-driven precepting model focused on development of clinical reasoning skills and self-directed educational objectives. Additional study is warranted to refine, adapt, and test the model in different setting.

Assessment of Gender-based Qualitative Differences within Trainee Evaluations of Faculty.

Rationale: Recent studies of trainee evaluations of medical faculty have demonstrated subtle gender-based word choice differences. However, it is not known if this manifests in major contextual differences in written comments.Objectives: To characterize qualitative differences in narrative evaluations of female and male pulmonary and critical care medicine faculty.Methods: We performed a single-center, retrospective cohort analysis of trainee narrative evaluations of pulmonary and critical care medicine faculty at the Hospital of the University of Pennsylvania, written from 2015 to 2016. Directive and summative content analysis was performed by four raters. Major contextual themes were identified using constant comparative techniques. Thematic differences based on faculty gender were identified, with statistical significance determined using χ2 analysis (P < 0.05). Effect sizes were calculated using Cramér's V.Results: A total of 1,216 total narrative evaluations were analyzed, representing 62 faculty members (17 women and 45 men), with analysis continued for the full cohort beyond achieving saturation of themes. Five overarching themes emerged: teaching skills, clinical skills, supervision, interpersonal and communication skills, and leadership skills. Within subthemes, we found no significant gender differences in reference to general teaching skills, learning environment, enthusiasm for teaching, or interpersonal concern. We identified subtheme differences between male and female faculty evaluations in regard to mentions of learner autonomy, clinical learning environment, humor, and motivating the trainee for patient care. Although the mention of constructive criticisms did not differ between genders, constructive criticisms of female faculty were more likely to mention demeanor (P = 0.06).Conclusions: Our data suggest minimal thematic differences in trainee narrative evaluations of male and female pulmonary and critical care medicine faculty. However, we noted several subtheme contextual differences between male and female faculty, which warrants further investigation.

ACGME Milestones Within Subspecialty Training Programs: One Institution's Experience.

BACKGROUND: The Accreditation Council for Graduate Medical Education Milestones were created as a criterion-based framework to promote competency-based education during graduate medical education. Despite widespread implementation across subspecialty programs, extensive validity evidence supporting the use of milestones within fellowship training is lacking. OBJECTIVE: We assessed the construct and response process validity of milestones in subspecialty fellowship programs in an academic medical center. METHODS: From 2014-2016, we performed a single center retrospective cohort analysis of milestone data from fellows across 5 programs. We analyzed summary statistics and performed multivariable linear regression to assess change in milestone ratings by training year and variability in ratings across fellowship programs. Finally, we examined a subset of Professionalism and Interpersonal and Communication Skills subcompetencies from the first 6 months of training to identify the proportion of fellows deemed "ready for independent practice" in these domains. RESULTS: Milestone data were available for 68 fellows, with 75 933 unique subcompetency ratings. Multivariable linear regression, adjusted for subcompetency and subspecialty, revealed an increase of 0.17 (0.16-0.19) in ratings with each postgraduate year level increase (P < .005), as well as significant variation in milestone ratings across subspecialties. For the Professionalism and Interpersonal and Communication Skills domains, mean ratings within the first 6 months of training were 3.78 and 3.95, respectively. CONCLUSIONS: We noted a minimal upward trend of milestone ratings in subspecialty training programs, and significant variability in implementing milestones across differing subspecialties. This may suggest possible difficulties with the construct validity and response process of the milestone system in certain medical subspecialties.

Assessment of Gender-Based Linguistic Differences in Physician Trainee Evaluations of Medical Faculty Using Automated Text Mining.

Importance: Women are underrepresented at higher ranks in academic medicine. However, the factors contributing to this disparity have not been fully elucidated. Implicit bias and unconscious mental attitudes toward a person or group may be factors. Although academic medical centers use physician trainee evaluations of faculty to inform promotion decisions, little is known about gender bias in these evaluations. To date, no studies have examined narrative evaluations of medical faculty by physician trainees for differences based on gender. Objective: To characterize gender-associated linguistic differences in narrative evaluations of medical faculty written by physician trainees. Design, Setting, and Participants: This retrospective cohort study included all faculty teaching evaluations completed for the department of medicine faculty by medical students, residents, and fellows at a large academic center in Pennsylvania from July 1, 2015, through June 30, 2016. Data analysis was performed from June 1, 2018, through July 31, 2018. Main Outcomes and Measures: Word use in faculty evaluations was quantified using automated text mining by converting free-text comments into unique 1- and 2-word phrases. Mixed-effects logistic regression analysis was performed to assess associations of faculty gender with frequencies of specific words and phrases present in a physician trainee evaluation. Results: A total of 7326 unique evaluations were collected for 521 faculty (325 men [62.4%] and 196 women [37.6%]). The individual words art (odds ratio [OR], 7.78; 95% CI, 1.01-59.89), trials (OR, 4.43; 95% CI, 1.34-14.69), master (OR, 4.24; 95% CI, 1.69-10.63), and humor (OR, 2.32; 95% CI, 1.44-3.73) were significantly associated with evaluations of male faculty, whereas the words empathetic (OR, 4.34; 95% CI, 1.56-12.07), delight (OR, 4.26; 95% CI, 1.35-13.40), and warm (OR, 3.45; 95% CI, 1.83-6.49) were significantly associated with evaluations of female faculty. Two-word phrases associated with male faculty evaluations included run rounds (OR, 7.78; 95% CI, 1.01-59.84), big picture (OR, 7.15; 95% CI, 1.68-30.42), and master clinician (OR, 4.02; 95% CI, 1.21-13.36), whereas evaluations of female faculty were more likely to be associated with model physician (OR, 7.75; 95% CI, 1.70-35.39), just right (OR, 6.97; 95% CI, 1.51-32.30), and attention (to) detail (OR, 4.26; 95% CI, 1.36-13.40). Conclusions and Relevance: The data showed quantifiable linguistic differences between free-text comments about male and female faculty in physician trainee evaluations. Further evaluation of these differences, particularly in association with ongoing gender disparities in faculty promotion and retention, may be warranted.