Development and Validation of a Machine Learning-Based Decision Support Tool for Residency Applicant Screening and Review.

PURPOSE: Residency programs face overwhelming numbers of residency applications, limiting holistic review. Artificial intelligence techniques have been proposed to address this challenge but have not been created. Here, a multidisciplinary team sought to develop and validate a machine learning (ML)-based decision support tool (DST) for residency applicant screening and review. METHOD: Categorical applicant data from the 2018, 2019, and 2020 residency application cycles (n = 8,243 applicants) at one large internal medicine residency program were downloaded from the Electronic Residency Application Service and linked to the outcome measure: interview invitation by human reviewers (n = 1,235 invites). An ML model using gradient boosting was designed using training data (80% of applicants) with over 60 applicant features (e.g., demographics, experiences, academic metrics). Model performance was validated on held-out data (20% of applicants). Sensitivity analysis was conducted without United States Medical Licensing Examination (USMLE) scores. An interactive DST incorporating the ML model was designed and deployed that provided applicant- and cohort-level visualizations. RESULTS: The ML model areas under the receiver operating characteristic and precision recall curves were 0.95 and 0.76, respectively; these changed to 0.94 and 0.72, respectively, with removal of USMLE scores. Applicants' medical school information was an important driver of predictions-which had face validity based on the local selection process-but numerous predictors contributed. Program directors used the DST in the 2021 application cycle to select 20 applicants for interview that had been initially screened out during human review. CONCLUSIONS: The authors developed and validated an ML algorithm for predicting residency interview offers from numerous application elements with high performance-even when USMLE scores were removed. Model deployment in a DST highlighted its potential for screening candidates and helped quantify and mitigate biases existing in the selection process. Further work will incorporate unstructured textual data through natural language processing methods.

Warm Handoffs: a Novel Strategy to Improve End-of-Rotation Care Transitions.

BACKGROUND: Hospitalized medical patients undergoing transition of care by house staff teams at the end of a ward rotation are associated with an increased risk of mortality, yet best practices surrounding this transition are lacking. AIM: To assess the impact of a warm handoff protocol for end-of-rotation care transitions. SETTING: A large, university-based internal medicine residency using three different training sites. PARTICIPANTS: PGY-2 and PGY-3 internal medicine residents. PROGRAM DESCRIPTION: Implementation of a warm handoff protocol whereby the incoming and outgoing residents meet at the hospital to sign out in-person and jointly round at the bedside on sicker patients using a checklist. PROGRAM EVALUATION: An eight-question survey completed by 60 of 99 eligible residents demonstrated that 85% of residents perceived warm handoffs to be safer for patients (p < 0.001), while 98% felt warm handoffs improved their knowledge and comfort level of patients on day 1 of an inpatient rotation (p < 0.001) as compared to prior handoff techniques. Finally, 88% felt warm handoffs were worthwhile despite requiring additional time (p < 0.001). DISCUSSION: A warm handoff protocol represents a novel strategy to potentially mitigate the known risks associated with end-of-rotation care transitions. Additional studies analyzing patient outcomes will be needed to assess the impact of this strategy.