Exploring the Use of Natural Language Processing to Understand Emotions of Trainees and Faculty Regarding Entrustable Professional Activity Assessments.

Background In medical education, artificial intelligence techniques such as natural language processing (NLP) are starting to be used to capture and analyze emotions through written text. Objective To explore the application of NLP techniques to understand resident and faculty emotions related to entrustable professional activity (EPA) assessments. Methods Open-ended text data from a survey on emotions toward EPA assessments were analyzed. Respondents were residents and faculty from pediatrics (Peds), general surgery (GS), and emergency medicine (EM), recruited for a larger emotions study in 2023. Participants wrote about their emotions related to receiving/completing EPA assessments. We analyzed the frequency of words rated as positive via a validated sentiment lexicon used in NLP studies. Specifically, we were interested if the count of positive words varied as a function of group membership (faculty, resident), specialty (Peds, GS, EM), gender (man, woman, nonbinary), or visible minority status (yes, no, omit). Results A total of 66 text responses (30 faculty, 36 residents) contained text data useful for sentiment analysis. We analyzed the difference in the count of words categorized as positive across group, specialty, gender, and being a visible minority. Specialty was the only category revealing significant differences via a bootstrapped Poisson regression model with GS responses containing fewer positive words than EM responses. Conclusions By analyzing text data to understand emotions of residents and faculty through an NLP approach, we identified differences in EPA assessment-related emotions of residents versus faculty, and differences across specialties.

Understanding Emotions Impacted by New Assessment Mandates Implemented in Medical Education: A Survey of Residents and Faculty Across Multiple Specialties.

Background Previous research findings show that the overall perception of residents regarding the new entrustable professional activity (EPA) assessment mandates is primarily negative. Hence, this study aims to explore the link between EPA assessment experiences and resident and faculty emotions and expectancy of successfully completing residency training. Methods A standardized questionnaire (Medical Emotions Scale (MES)), which measures 20 unique emotions on a 5-point Likert scale, was used to explore the emotions of residents and faculty members regarding EPA assessments and residents' expectancy of success. Data analysis included descriptive statistics and analysis of variance (ANOVA). Results Ninety-one (N=91) participants (46 faculty members and 45 residents) completed the survey. The results revealed that residents have more negative emotions toward EPA assessments compared to faculty. Additionally, resident and faculty emotions regarding EPA assessments vary across specialty and gender. Conclusions These findings will be crucial in providing the Royal College of Physicians and Surgeons of Canada and medical education programs with concrete evidence and guidance in understanding the perspectives and emotions of residents and faculty towards EPA assessments and residents' beliefs about successfully completing their medical training.