Competency-Based Assessments: Leveraging Artificial Intelligence to Predict Subcompetency Content.

PURPOSE: Faculty feedback on trainees is critical to guiding trainee progress in a competency-based medical education framework. The authors aimed to develop and evaluate a Natural Language Processing (NLP) algorithm that automatically categorizes narrative feedback into corresponding Accreditation Council for Graduate Medical Education Milestone 2.0 subcompetencies. METHOD: Ten academic anesthesiologists analyzed 5,935 narrative evaluations on anesthesiology trainees at 4 graduate medical education (GME) programs between July 1, 2019, and June 30, 2021. Each sentence (n = 25,714) was labeled with the Milestone 2.0 subcompetency that best captured its content or was labeled as demographic or not useful. Inter-rater agreement was assessed by Fleiss' Kappa. The authors trained an NLP model to predict feedback subcompetencies using data from 3 sites and evaluated its performance at a fourth site. Performance metrics included area under the receiver operating characteristic curve (AUC), positive predictive value, sensitivity, F1, and calibration curves. The model was implemented at 1 site in a self-assessment exercise. RESULTS: Fleiss' Kappa for subcompetency agreement was moderate (0.44). Model performance was good for professionalism, interpersonal and communication skills, and practice-based learning and improvement (AUC 0.79, 0.79, and 0.75, respectively). Subcompetencies within medical knowledge and patient care ranged from fair to excellent (AUC 0.66-0.84 and 0.63-0.88, respectively). Performance for systems-based practice was poor (AUC 0.59). Performances for demographic and not useful categories were excellent (AUC 0.87 for both). In approximately 1 minute, the model interpreted several hundred evaluations and produced individual trainee reports with organized feedback to guide a self-assessment exercise. The model was built into a web-based application. CONCLUSIONS: The authors developed an NLP model that recognized the feedback language of anesthesiologists across multiple GME programs. The model was operationalized in a self-assessment exercise. It is a powerful tool which rapidly organizes large amounts of narrative feedback.

Milestones 2.0: Otolaryngology Resident Competency in the Postpandemic Era.

The Accreditation Council for Graduate Medical Education and the American Board of Otolaryngology-Head and Neck Surgery Milestones Project grew out of a continued need to document objective outcomes within resident education. Milestones 2.0 began its work in 2016, with an intent to resolve inconsistencies in the original milestones based on an iterative process. Milestones 2.0 retains the original 5 levels of achievement but includes a "not yet assessable" option as well. In addition, Milestones 2.0 has added harmonized milestones across all specialties. Each specialty has incorporated a supplemental guide with examples and resources to improve facility with the tool. There will be further refinement of the Milestones as new research emerges with the ultimate goal of providing programs and trainees with a reliable roadmap that can be used to direct and assess learning.

Twelve Tips for Utilizing Residency Program Social Media Accounts for Modified Residency Recruitment.

This article was migrated. The article was marked as recommended. Social media use across the health professions has significantly expanded in recent years. Specific attention has been paid to both the value of social media use in graduate medical education with residency program twitter accounts. More recently, social media has been examined for its role in supporting the rapid expansion of information exchange and connection across digital and virtual platforms during the COVID-19 pandemic. With the ongoing response to the pandemic, the 2020-2021 residency application cycle is anticipated to be a completely virtual interview process. Here, we draw from our collective experiences managing, maturing, and maximizing social media accounts for residency programs and GME to provide practical tips for using social media for the upcoming virtual interview season.