Finding the Needle in the Haystack: Can Natural Language Processing of Students' Evaluations of Teachers Identify Teaching Concerns?

BACKGROUND: Institutions rely on student evaluations of teaching (SET) to ascertain teaching quality. Manual review of narrative comments can identify faculty with teaching concerns but can be resource and time-intensive. AIM: To determine if natural language processing (NLP) of SET comments completed by learners on clinical rotations can identify teaching quality concerns. SETTING AND PARTICIPANTS: Single institution retrospective cohort analysis of SET (n = 11,850) from clinical rotations between July 1, 2017, and June 30, 2018. PROGRAM DESCRIPTION: The performance of three NLP dictionaries created by the research team was compared to an off-the-shelf Sentiment Dictionary. PROGRAM EVALUATION: The Expert Dictionary had an accuracy of 0.90, a precision of 0.62, and a recall of 0.50. The Qualifier Dictionary had lower accuracy (0.65) and precision (0.16) but similar recall (0.67). The Text Mining Dictionary had an accuracy of 0.78 and a recall of 0.24. The Sentiment plus Qualifier Dictionary had good accuracy (0.86) and recall (0.77) with a precision of 0.37. DISCUSSION: NLP methods can identify teaching quality concerns with good accuracy and reasonable recall, but relatively low precision. An existing, free, NLP sentiment analysis dictionary can perform nearly as well as dictionaries requiring expert coding or manual creation.

Gender Differences in Work-Based Assessment Scores and Narrative Comments After Direct Observation.

BACKGROUND: While some prior studies of work-based assessment (WBA) numeric ratings have not shown gender differences, they have been unable to account for the true performance of the resident or explore narrative differences by gender. OBJECTIVE: To explore gender differences in WBA ratings as well as narrative comments (when scripted performance was known). DESIGN: Secondary analysis of WBAs obtained from a randomized controlled trial of a longitudinal rater training intervention in 2018-2019. Participating faculty (n = 77) observed standardized resident-patient encounters and subsequently completed rater assessment forms (RAFs). SUBJECTS: Participating faculty in longitudinal rater training. MAIN MEASURES: Gender differences in mean entrustment ratings (4-point scale) were assessed with multivariable regression (adjusted for scripted performance, rater and resident demographics, and the interaction between study arm and time period [pre- versus post-intervention]). Using pre-specified natural language processing categories (masculine, feminine, agentic, and communal words), multivariable linear regression was used to determine associations of word use in the narrative comments with resident gender, race, and skill level, faculty demographics, and interaction between the study arm and the time period (pre- versus post-intervention). KEY RESULTS: Across 1527 RAFs, there were significant differences in entrustment ratings between women and men standardized residents (2.29 versus 2.54, respectively, p < 0.001) after correction for resident skill level. As compared to men, feminine terms were more common for comments of what the resident did poorly among women residents (ß 0.45, CI 0.12-0.78, p 0.01). This persisted despite adjusting for the faculty's entrustment ratings. There were no other significant linguistic differences by gender. CONCLUSIONS: Contrasting prior studies, we found entrustment rating differences in a simulated WBA which persisted after adjusting for the resident's scripted performance. There were also linguistic differences by gender after adjusting for entrustment ratings, with feminine terms being used more frequently in comments about women in some, but not all narrative comments.

Build It and They Shall Come: Medical Education Communities of Practice.

Background: Producing scholarship in education is essential to the career development of a clinician-educator. Challenges to scholarly production include a lack of resources, time, expertise, and collaborators. Objective: To develop communities of practice for education scholarship through an international society to increase community and academic productivity. Methods: We developed multi-institutional scholarship pods within the American Thoracic Society through the creation of a working group (2017-2019). Pods met virtually, and meetings were goal focused to advance education scholarship within their area of interest. To understand the impact of these scholarship pods, we surveyed pod leaders and members in 2021 and analyzed the academic productivity of each pod via a survey of pod leaders and a review of the PubMed index. Results: Nine pods were created, each with an assigned educational topic. The survey had a response rate of 76.6%. The perceived benefits were the opportunity to meet colleagues with similar interests at other institutions, production of scholarly work, and engagement in new experiences. The main challenges were difficulty finding times to meet because of competing clinical demands and aligning times among pod members. Regarding academic productivity, eight publications, four conference presentations, and one webinar/podcast were produced by six of the nine pods. Conclusion: The development of communities of practice resulted in increased multi-site collaboration, with boosted academic productivity as well as an enhanced sense of belonging. Multiple challenges remain but can likely be overcome with accountability, early discussion of roles and expectations, and clear delegation of tasks and authorship.

Natural Language Processing of Learners' Evaluations of Attendings to Identify Professionalism Lapses.

Unprofessional faculty behaviors negatively impact the well-being of trainees yet are infrequently reported through established reporting systems. Manual review of narrative faculty evaluations provides an additional avenue for identifying unprofessional behavior but is time- and resource-intensive, and therefore of limited value for identifying and remediating faculty with professionalism concerns. Natural language processing (NLP) techniques may provide a mechanism for streamlining manual review processes to identify faculty professionalism lapses. In this retrospective cohort study of 15,432 narrative evaluations of medical faculty by medical trainees, we identified professionalism lapses using automated analysis of the text of faculty evaluations. We used multiple NLP approaches to develop and validate several classification models, which were evaluated primarily based on the positive predictive value (PPV) and secondarily by their calibration. A NLP-model using sentiment analysis (quantifying subjectivity of the text) in combination with key words (using the ensemble technique) had the best performance overall with a PPV of 49% (CI 38%-59%). These findings highlight how NLP can be used to screen narrative evaluations of faculty to identify unprofessional faculty behaviors. Incorporation of NLP into faculty review workflows enables a more focused manual review of comments, providing a supplemental mechanism to identify faculty professionalism lapses.

Gender-Based Differences in Language Used by Students to Describe Their Noteworthy Characteristics in Medical Student Performance Evaluations.

PURPOSE: The noteworthy characteristic (NC) section of the medical student performance evaluation (MSPE) was introduced to facilitate holistic review of residency applications and mitigate biases. The student-written aspect of the characteristics, however, may introduce biases resulting from gender differences in self-promotion behaviors. The authors conducted an exploratory analysis of potential gender-based differences in language used in NCs. METHOD: The authors performed a single-center cohort analysis of all student-written NCs at the Perelman School of Medicine (2018-2022). NCs were converted into single words and characterized into word categories: ability (e.g., "talent"), standout ("best"), grindstone ("meticulous"), communal ("caring"), or agentic ("ambitious"). The authors qualitatively analyzed NC topic characteristics (i.e., focused on scholarship, community service). Logistic regression was used to identify gender differences in word categories and topics used in NCs. RESULTS: The cohort included 2,084 characteristics from 783 MSPEs (47.5%, n = 371 written by women). After adjusting for underrepresented in medicine status, honor society membership, and intended specialty, men were more likely to use standout (OR = 2.00; 95% confidence interval [CI] = 1.35, 2.96; P = .001) and communal (OR = 1.40; 95% CI = 1.03, 1.90; P = .03) words in their NCs compared with women but less likely to use grindstone words (OR = 0.72; 95% CI = 0.53, 0.98; P = .04). Men were more likely than women to discuss scholarship (OR = 2.03; 95% CI = 1.27, 3.23; P = .003), hobbies (OR = 1.45; 95% CI = 1.07, 1.96; P = .02), and/or awards (OR = 1.59; 95% CI = 1.16, 2.16; P = .004) and less likely to highlight community service (OR = 0.66; 95% CI = 0.48, 0.92; P = .02). CONCLUSIONS: The self-written nature of NCs permits language differences that may contribute to gender bias in residency application.

The Context of "Confidence": Analyzing the Term Confidence in Resident Evaluations.

BACKGROUND: Despite similar performance metrics, women medical trainees routinely self-assess their own skills lower than men. The phenomenon of a "confidence gap" between genders, where women report lower self-confidence independent of actual ability or competency, may have an important interaction with gender differences in assessment. Identifying whether there are gender-based differences in how confidence is mentioned in written evaluations is a necessary step to understand the interaction between evaluation and the gender-based confidence gap. OBJECTIVE: To analyze faculty evaluations of internal medicine (IM) residents for gender-based patterns in the use of iterations of "confidence." DESIGN: We performed a retrospective cohort study of all inpatient faculty evaluations of University of Pennsylvania IM residents from 2018 to 2021. We performed n-gram text-mining to identify evaluations containing the terms "confident," "confidence," or "confidently." We performed univariable and multivariable logistic regression to determine the association between resident gender and references to confidence (including comments reflecting too little confidence), adjusting for faculty gender, post-graduate year (PGY), numeric rating, and service. SUBJECTS: University of Pennsylvania IM residents from 2018 to 2021. KEY RESULTS: There were 5416 evaluations of IM residents (165 women [51%], 156 men [49%]) submitted by 356 faculty members (149 women [51%]), of which 7.1 % (n=356) contained references to confidence. There was a significant positive association between the mention of confidence and women resident gender (OR 1.54, CI 1.23-1.92; p<0.001), which persisted after adjustment for faculty gender, numeric rating, and PGY level. Eighty evaluations of the cohort explicitly mentioned the resident having "too little confidence," which was also associated with women resident gender (OR 1.66, CI 1.05-2.62; p=0.031). CONCLUSION: Narrative evaluations of women residents were more likely to contain references to confidence, after adjustment for numerical score, PGY level, and faculty gender, which may perpetuate the gender-based confidence gap, introduce bias, and ultimately impact professional identity development.

The Impact of Trainee and Standardized Patient Race and Gender on Internal Medicine Resident Communication Assessment Scores.

BACKGROUND: Standardized patient (SP) encounters are commonly used to assess communication skills in medical training. The impact of SP and resident demographics on the standardized communication ratings in residents has not been evaluated. OBJECTIVE: To examine the impact of gender and race on SP assessments of internal medicine (IM) residents' communication skills during postgraduate year (PGY) 1. METHODS: We performed a retrospective cohort study of all SP assessments of IM PGY-1 residents for a standardized communication exercise from 2012 to 2018. We performed descriptive analyses of numeric communication SP ratings by gender, race, and age (for residents and SPs). A generalized estimating equation model, clustered on individual SP, was used to determine the association of gender (among SP and residents) with communication ratings. A secondary analysis was performed to determine the impact of residents and SP racial concordance in communication scores. RESULTS: There were 1356 SP assessments of 379 IM residents (199 male residents [53%] and 178 female residents [47%]). There were significant differences in average numeric communication rating (mean 3.40 vs 3.34, P = .009) by gender of resident, with higher scores in female residents. There were no significant interactions between SP and resident gender across the communication domains. There were no significant interactions noted with racial concordance between interns and SPs. CONCLUSIONS: Our data demonstrate an association of resident gender on ratings in standardized communication exercises, across multiple communication skills. There was not an interaction impact for gender or racial concordance between SPs and interns.

Longitudinal Milestone Assessment Extending Through Subspecialty Training: The Relationship Between ACGME Internal Medicine Residency Milestones and Subsequent Pulmonary and Critical Care Fellowship Milestones.

PURPOSE: Accreditation Council for Graduate Medical Education (ACGME) milestones were implemented across medical subspecialties in 2015. Although milestones were proposed as a longitudinal assessment tool potentially providing opportunities for early implementation of individualized fellowship learning plans, the association of subspecialty fellowship ratings with prior residency ratings remains unclear. This study aimed to assess the relationship between internal medicine (IM) residency milestones and pulmonary and critical care medicine (PCCM) fellowship milestones. METHOD: A multicenter retrospective cohort analysis was conducted for all PCCM trainees in ACGME-accredited PCCM fellowship programs, 2017-2018, who had complete prior IM milestone ratings from 2014 to 2017. Only professionalism and interpersonal and communication skills (ICS) were included based on shared anchors between IM and PCCM milestones. Using a generalized estimating equations model, the association of PCCM milestones ≤ 2.5 during the first fellowship year with corresponding IM subcompetencies was assessed at each time point, nested by program. Statistical significance was determined using logistic regression. RESULTS: The study included 354 unique PCCM fellows. For ICS and professionalism subcompetencies, fellows with higher IM ratings were less likely to obtain PCCM ratings ≤ 2.5 during the first fellowship year. Each ICS subcompetency was significantly associated with future lapses in fellowship (ICS01: ß = -0.67, P = .003; ICS02: ß = -0.70, P = .001; ICS03: ß = -0.60, P = .004) at various residency time points. Similar associations were noted for PROF03 (ß = -0.57, P = .007). CONCLUSIONS: Findings demonstrated an association between IM milestone ratings and low milestone ratings during PCCM fellowship. IM trainees with low ratings in several professionalism and ICS subcompetencies were more likely to be rated ≤ 2.5 during the first PCCM fellowship year. This highlights a potential use of longitudinal milestones to target educational gaps at the beginning of PCCM fellowship.

Miles to Go Before We Sleep. Reforming the Pulmonary and Critical Care Milestones to Improve Trainee Assessment.

The Accreditation Council for Graduate Medical Education (ACGME) Milestones are a systematic assessment framework for medical trainees within the six core competencies of practice. Their use by internal medicine subspecialties, including semiannual reports to the ACGME, was mandated beginning in 2014. The Milestones, which were based on specific, observable behaviors, improved upon the prior subjective, global comparisons of each fellow with an "average" fellow in his or her field and served the goals of competency-based medical education. However, the original set of Milestones has proven challenging to apply and interpret. Part of the challenge stems from the use of identical Milestones across all medicine subspecialties, which led to unclear relevance of the patient care and medical knowledge domains to the practice of pulmonary and critical care. This also precluded their use for individualized feedback or development of a learning plan for fellows. In addition, verbose behavioral descriptors, which were designed to provide specificity, ultimately led to rater fatigue among assessors and clinical competency committees. Therefore, the ACGME convened committees for each of the medical subspecialties to revise the original Milestones in an effort to improve subspecialty relevance, minimize educational jargon, and simplify the current iteration. New patient care and medical knowledge Milestones were created to be subspecialty specific and improve utility. The remaining four Milestones were developed as a common set of shorter Milestones, harmonized across specialties. For pulmonary, critical care, and combined fellowship programs, the resulting Milestones 2.0 aims to simplify the use, implementation, and interpretation of this framework for program directors, trainees, and society.