Using Natural Language Processing to Visualize Narrative Feedback in a Medical Student Performance Dashboard.

PROBLEM: Clinical competency committees rely on narrative feedback for important insight into learner performance, but reviewing comments can be time-consuming. Techniques such as natural language processing (NLP) could create efficiencies in narrative feedback review. In this study, the authors explored whether using NLP to create a visual dashboard of narrative feedback to preclerkship medical students would improve the competency review efficiency. APPROACH: Preclerkship competency review data collected at the Northwestern University Feinberg School of Medicine from 2014 to 2021 were used to identify relevant features of narrative data associated with review outcome (ready or not ready) and draft visual summary reports of the findings. A user needs analysis was held with experienced reviewers to better understand work processes in December 2019. Dashboards were designed based on this input to help reviewers efficiently navigate large amounts of narrative data. The dashboards displayed the model's prediction of the review outcome along with visualizations of how narratives in a student's portfolio compared with previous students' narratives. Excerpts of the most relevant comments were also provided. Six faculty reviewers who comprised the competency committee in spring 2023 were surveyed on the dashboard's utility. OUTCOMES: Reviewers found the predictive component of the dashboard most useful. Only 1 of 6 reviewers (17%) agreed that the dashboard improved process efficiency. However, 3 (50%) thought the visuals made them more confident in decisions about competence, and 3 (50%) thought they would use the visual summaries for future reviews. The outcomes highlight limitations of visualizing and summarizing narrative feedback in a comprehensive assessment system. NEXT STEPS: Future work will explore how to optimize the dashboards to meet reviewer needs. Ongoing advancements in large language models may facilitate these efforts. Opportunities to collaborate with other institutions to apply the model to an external context will also be sought.

What Do Pediatric Subinterns Say About Their Learning and Assessment? A Qualitative Analysis of Individual Learning Plans.

OBJECTIVE: To perform a qualitative content analysis of learning and assessment strategies that pediatric subinterns describe in Individualized Learning Plans (ILPs) and to explore barriers and facilitators to their learning. METHODS: We analyzed ILPs from medical students enrolled in pediatric subinternships at 10 US medical schools that utilized a standardized curriculum and were recruited to reflect diversity in geographic location, funding, and enrollment. Students used an ILP to record 3 or more selected learning objectives, rationale for selection, and reflection on learning and assessment strategies. Investigators used the constant comparative method to perform a content analysis of the ILPs, grouping codes into themes, and verifying relationships between codes within themes. RESULTS: Two hundred and four ILPs that included student reflections on 850 learning objectives were analyzed. Content was analyzed in 5 categories: rationale for selecting objectives, learning strategies, assessment strategies, challenges to learning, and facilitators of learning. Students showed strong commitment to individualized, self-directed learning, developed a wide range of creative learning strategies, and relied heavily on self-reflection to assess their progress. The learning environment both helped and hindered students' ability to make and assess progress on their selected learning objectives. CONCLUSIONS: Through ILP-guided reflection and a formal curriculum, students can choose well-justified learning objectives and demonstrate resourcefulness and independence in developing self-directed learning and assessment strategies. The strategies that students identified in this study provide a menu of learning and assessment options for subinterns. Identified challenges and facilitators of learning provide guidance for educators who seek to enhance the clinical learning environment.

Utilizing Natural Language Processing of Narrative Feedback to Develop a Predictive Model of Pre-Clerkship Performance: Lessons Learned.

Background: Natural language processing is a promising technique that can be used to create efficiencies in the review of narrative feedback to learners. The Feinberg School of Medicine has implemented formal review of pre-clerkship narrative feedback since 2014 through its portfolio assessment system but this process requires considerable time and effort. This article describes how natural language processing was used to build a predictive model of pre-clerkship student performance that can be utilized to assist competency committee reviews. Approach: The authors took an iterative and inductive approach to the analysis, which allowed them to identify characteristics of narrative feedback that are both predictive of performance and useful to faculty reviewers. Words and phrases were manually grouped into topics that represented concepts illustrating student performance. Topics were reviewed by experienced reviewers, tested for consistency across time, and checked to ensure they did not demonstrate bias. Outcomes: Sixteen topic groups of words and phrases were found to be predictive of performance. The best-fitting model used a combination of topic groups, word counts, and categorical ratings. The model had an AUC value of 0.92 on the training data and 0.88 on the test data. Reflection: A thoughtful, careful approach to using natural language processing was essential. Given the idiosyncrasies of narrative feedback in medical education, standard natural language processing packages were not adequate for predicting student outcomes. Rather, employing qualitative techniques including repeated member checking and iterative revision resulted in a useful and salient predictive model.

Addressing Vaccine Hesitancy Through a Comprehensive Resident Vaccine Curriculum.

Introduction: Vaccine hesitancy can lead to incomplete vaccination, increased risk of vaccine-preventable diseases, and distrust or conflict between physicians and patients. Yet many physicians are uncomfortable navigating vaccine hesitancy and educating vaccine-hesitant patients and families. We developed a vaccine hesitancy curriculum to increase vaccine knowledge, comfort, and communication skills in pediatric residents. Methods: The curriculum consisted of four interactive 40-minute sessions delivered to pediatric residents over 10 months. The first two sessions discussed recommended childhood vaccines, the third session examined common vaccine misconceptions, and the final session reviewed vaccine hesitancy-specific communication skills, incorporating practice through role-playing. Residents completed pre- and posttests assessing knowledge and comfort as well as receiving a standardized patient (SP) assessment of vaccine-specific communication skills after the curriculum. Results: Thirty-five residents were in the educational intervention group and 35 in a control group. Pretest scores did not differ significantly between the groups. The mean knowledge score for the intervention group increased from 47% on the pretest to 66% on the posttest. The mean self-reported comfort score (1 = low comfort, 5 = high comfort) for the intervention group increased from 2.9 on the pretest to 3.8 on the posttest. The control group showed no difference between pre- and posttest scores for knowledge or comfort. The mean postintervention SP assessment score was significantly higher for the intervention group (78%) than the control group (52%). Discussion: Implementation of a comprehensive vaccine hesitancy curriculum resulted in improved vaccine knowledge, self-reported comfort, and communication skills among pediatric residents.

Natural history of burnout, stress, and fatigue in a pediatric resident cohort over three years.

BACKGROUND: Burnout is known to be high amongst physician trainees. Factors such as stress, fatigue, social environment, and resilience could affect burnout. Cross-sectional data describe burnout in pediatric residents, but the trajectory of burnout in a cohort of residents followed longitudinally through the full course of residency training has not been reported. We prospectively examined the prevalence and trajectory of burnout, stress, fatigue, social connectedness, and resilience in a pediatric resident cohort from orientation through three years of residency. The cohort (N = 33) was surveyed six times between 2015-2018 using the Abbreviated Maslach Burnout Inventory (AMBI), Perceived Stress Scale (PSS), Epworth Sleepiness Scale (ESS), Social Connectedness Scale-Revised (SCS-R), and Connor-Davidson Resilience Scale (CD-RISC10). Data were analyzed using repeated measures mixed effects models. Significant change from baseline was considered to be adjusted p < 0.05. Response rate was >50% at each timepoint; 69% of trainees completed surveys ≥4 times. Scores were significantly worse than baseline in all surveys, at every timepoint, with the exception of AMBI-PA (personal accomplishment) at the PGY1/PGY2 transition and SCS-R and CD-RISC10 at the end of training. The most significant changes from baseline occurred mid-PGY1 to mid-PGY2. At least 65% of residents demonstrated worse scores than baseline on 36/40 (90%) follow-up surveys. Furthermore, ≥65% met criteria for emotional exhaustion and moderate stress at every timepoint. SCS-R was the only survey measure to improve at residency completion compared to baseline. CONCLUSION: Within 6 months of starting residency this pediatric resident cohort became burned out, stressed, fatigued, less socially connected, and less resilient. Burnout is only one factor that indicates impaired resident well-being. To fully address this, a comprehensive examination of how residents are trained is needed to identify effective interventions. ABBREVIATIONS: MBI - Maslach Burnout Inventory; AMBI - Abbreviated Maslach Burnout Inventory; AMBI-EE - Emotional Exhaustion; AMBI-D - Depersonalization; AMBI-PA - Personal Accomplishment; AMBI-SAT - Satisfaction with Medicine; LCH - Ann & Robert H. Lurie Children's Hospital of Chicago/Lurie Children's Hospital; P/CN - Pediatrics/Child Neurology; PSS - Perceived Stress Scale; ESS - Epworth Sleepiness Scale; CD-RISC10 - Resilience; SCS-R - Social Connectedness Scale Revised; PGY - Post-Graduate Year.

Evaluation of a National Pediatric Subinternship Curriculum Implemented Through Individual Learning Plans.

OBJECTIVE: The Council on Medical Student Education in Pediatrics and Association of Pediatric Program Directors developed a Pediatric Subinternship (CAPS) curriculum for use with an individualized learning plan (ILP). The authors determined which learning objectives (LOs) pediatric subinterns selected when provided the CAPS curriculum, summarized students' self-reported progress, and determined feasibility of ILPs in subinternship. METHODS: Students from 10 medical schools completed a standardized ILP during pediatric subinternship. Students listed ≥3 LOs using CAPS curriculum as a guide and self-assessed their progress. Students reviewed ILPs with faculty preceptors; preceptors completed questionnaires on time and effort spent. Authors mapped student LOs to CAPS curriculum objectives and grouped in Accreditation Council for Graduate Medical Education competency domains. RESULTS: Two hundred four students documented 850 LOs. Authors mapped student LOs to 61 of the 69 CAPS objectives (88%). Students most commonly chose Patient Care LOs, with the top 3 related to oral presentations, time management, and management plans. Student LOs not in CAPS addressed nutrition, child development, test interpretation, and cost. No students chose LOs related to health disparities, shared decision making, informed consent, or patient safety. Students self-reported significant progress on most LOs (73%). Faculty met with students ≥1 time and 93% met for a total of ≤1 hour. According to faculty, students required little or no help completing ILPs. CONCLUSIONS: Students chose a wide range of LOs when provided the CAPS curriculum. Revision to include additional student-identified LOs would enhance CAPS curriculum's comprehensiveness. Using this curriculum with an ILP during subinternship is feasible, but gaps between educator-identified and student-identified objectives require further exploration.

Oral Health Promotion During Well Visits.

Training pediatric residents in Bright Futures and oral health concepts is critical to improving oral health. This study's objective was to determine the skill level of pediatric residents in integrating oral health promotion during health supervision visits of 12- to 35-month-old children. One hundred forty-three pediatric residents participated in an evaluation of the effectiveness of a Bright Futures oral health curriculum. Competencies assessed preintervention included partnership building, communication, and integration of oral health concepts. Pediatric residents' abilities to integrate oral health promotion into health supervision visits varied considerably. Residents demonstrated greater skill in communication and partnership building compared with oral health promotion behaviors and performance of an oral examination. Further education is needed at a national level if we are to meet Healthy People 2020 goals.

Feasibility and Outcomes of Implementing a Portfolio Assessment System Alongside a Traditional Grading System.

PURPOSE: Portfolios are a powerful tool to collect and evaluate evidence of medical students' competence across time. However, comprehensive portfolio assessment systems that are implemented alongside traditional graded curricula at medical schools in the United States have not been described in the literature. This study describes the development and implementation of a longitudinal competency-based electronic portfolio system alongside a graded curriculum at a relatively large U.S. medical school. METHOD: In 2009, the authors developed a portfolio system that served as a repository for all student assessments organized by competency domain. Five competencies were selected for a preclerkship summative portfolio review. Students submitted reflections on their performance. In 2014, four clinical faculty members participated in standard-setting activities and used expert judgment and holistic review to rate students' competency achievement as "progressing toward competence," "progressing toward competence with some concern," or "progressing toward competence pending remediation." Follow-up surveys measured students' and faculty members' perceptions of the process. RESULTS: Faculty evaluated 156 portfolios and showed high levels of agreement in their ratings. The majority of students achieved the "progressing toward competence" benchmark in all competency areas. However, 31 students received at least one concerning rating, which was not reflected in their course grades. Students' perceptions of the system's ability to foster self-assessment were mixed. CONCLUSIONS: The portfolio review process allowed faculty to identify students with a concerning rating in a behavioral competency who would not have been identified in a traditional grading system. Identification of these students allows for intervention and early remediation.

Cognitive enhancement drug use among future physicians: findings from a multi-institutional census of medical students.

BACKGROUND: Nonmedical use of prescription psychostimulants such as methylphenidate and amphetamine salts for the purpose of cognitive enhancement is a growing trend, particularly in educational environments. To our knowledge, no recent studies have evaluated the use of these psychostimulants in a medical academic setting. OBJECTIVE: To conduct an online census of psychostimulant use among medical students. DESIGN: In 2011, we conducted a multi-institutional census using a 31-48 item online survey regarding use of prescription psychostimulants. PARTICIPANTS: 2,732 actively enrolled medical students at four private and public medical schools in the greater Chicago area. MAIN MEASURES: Prevalence and correlates of psychostimulant use KEY RESULTS: 1,115 (41 %) of students responded to the web-based questionnaire (range 26-47 % among schools). On average, students were 25.1 years of age (SD = 2.7, range 20-49), and single (70 %). Overall, 18 % (198/1,115) of this medical student sample had used prescription psychostimulants at least once in their lifetime, with first use most often in college. Of these, 11 % (117/1,115) of students reported use during medical school (range 7-16 % among schools). Psychostimulant use was significantly correlated with use of barbiturates, ecstasy, and tranquilizers (Pearson's correlation r > 0.5, Student's t-test p < 0.01); male gender (21 % male versus 15 % female, Chi squared p = 0.007); and training at a medical school which by student self-report determined class rank (68 % versus 51 %, Chi-squared p = 0.018). Non-users were more likely to be first year students (Chi-squared p = 0.048) or to have grown up outside of the United States (Chi-squared p = 0.013). CONCLUSIONS: Use of psychostimulants, including use without a prescription, is common among medical students. Further study of the side effects, medical implications, and use during post-graduate medical training and medical practice is needed to inform evidence-based policy.

Handgun safety: what do consumers learn from gun dealers?

OBJECTIVE: To explore the type and quality of handgun safety information a typical consumer would obtain from a licensed gun dealer. METHODS: Semistructured, interactional on-site interviews were conducted with licensed handgun dealers in 2 metropolitan areas. A variety of dealers (gun shops, pawnshops, general merchandise stores, and sporting goods stores) were visited. Investigators posed as customers interested in buying a handgun. During the interview, investigators expressed concern that as the parent of a 4-year-old child, they needed suggestions about keeping their child safe with a gun in the home. Information collected included basic dealer demographics, opinions on whether a 4-year-old child could pull a handgun trigger, handgun safety advice and recommendations, and the type of safety devices and handgun safety educational materials that were available in the store. RESULTS: There were 96 visits made to gun dealers. The typical salesperson was a man who appeared to be older than 40 years. Trigger locks were the most common safety devices available. When asked what a consumer should know about purchasing a handgun, 85% of salespeople did not mention safe storage. Only 9 (9%) offered advice that included all of the following: keeping the gun securely locked, keeping the gun unloaded, and storing the gun separately from the ammunition. One third answered "no" or "don't know" or "uncertain" when asked if a 4-year-old could pull the trigger. The majority (92%) did not have any handgun safe storage educational materials on site. CONCLUSIONS: Salespeople offered potential buyers little or no education about safe storage of handguns. The information provided was often inconsistent with the recommendations of the American Academy of Pediatrics.