Using artificial intelligence in medical school admissions screening to decrease inter- and intra-observer variability.

Objectives: Inter- and intra-observer variability is a concern for medical school admissions. Artificial intelligence (AI) may present an opportunity to apply a fair standard to all applicants systematically and yet maintain sensitivity to nuances that have been a part of traditional screening methods. Material and Methods: Data from 5 years of medical school applications were retrospectively accrued and analyzed. The applicants (m = 22 258 applicants) were split 60%-20%-20% into a training set (m = 13 354), validation set (m = 4452), and test set (m = 4452). An AI model was trained and evaluated with the ground truth being whether a given applicant was invited for an interview. In addition, a "real-world" evaluation was conducted simultaneously within an admissions cycle to observe how it would perform if utilized. Results: The algorithm had an accuracy of 95% on the training set, 88% on the validation set, and 88% on the test set. The area under the curve of the test set was 0.93. The SHapely Additive exPlanations (SHAP) values demonstrated that the model utilizes features in a concordant manner with current admissions rubrics. By using a combined human and AI evaluation process, the accuracy of the process was demonstrated to be 96% on the "real-world" evaluation with a negative predictive value of 0.97. Discussion and Conclusion: These results demonstrate the feasibility of an AI approach applied to medical school admissions screening decision-making. Model explainability and supplemental analyses help ensure that the model makes decisions as intended.

Comparison of the Multiple Mini-Interview and the Traditional Interview in Medical School Admissions: Lessons Learned Using a Hybrid Model at One Institution.

PURPOSE: Medical school admissions interviews are a critical form of assessment; however, the most effective interview strategy is debated. This study compares the traditional interview (TI) and multiple mini-interview (MMI) within a hybrid TI-MMI model at one medical school to determine whether the interview approaches reveal different information about applicants and whether a hybrid model results in a more diversified applicant pool. METHOD: Admissions data from 3 application cycles at the Donald and Barbara Zucker School of Medicine at Hofstra/Northwell were used. The TI was used in 2017-2018 and the hybrid TI-MMI model in 2018-2019 and 2019-2020. Applicants were scored on a 5-point scale and referred to a voting committee for acceptance consideration if interview scores met threshold criteria. Changes in the number of students referred to the committee using the TI vs the TI-MMI score criteria were analyzed. RESULTS: In 2017-2018 (TI only), 683 applicants were interviewed; in 2018-2019 (TI-MMI), 844 applicants were interviewed; and in 2019-2020 (TI-MMI), 805 applicants were interviewed. Medium correlations were found between total MMI and TI scores in 2018-2019 ( ρ = 0.37, P < .001) and 2019-2020 ( ρ = 0.33, P < .001). No differences were found in TI scores between 2017-2018 and 2018-2019 ( P = .30), but TI scores were significantly lower in 2019-2020 vs 2017-2018 ( P < .001) and 2018-2019 ( P = .002). Overall, a 10% to 18% increase was found in the number of applicants referred to the voting committee when using hybrid criteria, with a 19% to 27% increase in underrepresented in medicine applicants. CONCLUSIONS: The TI-MMI model may allow for a more holistic interview approach and an expanded pool of applicants, particularly underrepresented in medicine applicants, considered for acceptance.