Key factors for residency interview selection from the National Resident Matching Program: analysis of residency Program Director surveys, 2016-2020.

CONTEXT: As the number of medical school graduates continues to outpace the available residency training positions, applying for residency in the United States has become a highly competitive process, often associated with a low rate of selection and invitation for interview. The National Resident Matching Program (NRMP) Program Director survey provides data assessing factors considered by Program Directors (PD) in selecting and inviting candidates for interview. Assessing the evolution of these factors over time is efficacious to inform and guide prospective applicants toward improving preparation for residency application. OBJECTIVES: We aim to synthesize NRMP data showing factors that PDs reported and rated as important in their decision to select and invite applicants for interview. METHODS: Data from residency PD surveys from 2008 to 2021 were accessed, but after applying inclusion/exclusion criteria, only the data from 2016 to 2020 were reviewed and analyzed. The NRMP survey reports provided two metrics that characterized PDs' evaluation of the residency factors for interview, namely, "percent citing factor" and "average rating" on a 0 to 5 Likert-type scale. These two metrics were combined into an aggregate measure of importance (AI), and another measure of relative importance (RI) was constructed from normalizing the AI of each individual factor to the sum of the AI within each survey year. RESULTS: The top ranked factors were United States Medical Licensing Examination (USMLE) Step 1/Comprehensive Osteopathic Medical Licensing Examination (COMLEX) Level 1, Letter of Recommendation (LOR) in the specialty, Medical Student Performance Evaluation (MSPE/Dean's Letter), and USMLE Step 2 Clinical Knowledge (CK)/COMLEX Level 2 Cognitive Exam (CE) score, any failed attempt in USMLE/COMLEX, and perceived commitment to specialty. Factors rising in importance were Audition Elective/Rotation Within Your Department, Personal Statement (PS), Perceived Commitment to Specialty, Perceived Interest in Program, LOR in the Specialty, Other Life Experience, and Personal Prior Knowledge of the Applicant. Factors with declining importance were Interest in Academic Career, Awards or Special Honors in Basic Sciences, Graduate of Highly Regarded US Medical School, Awards or Special Honors in Clinical Clerkships, Lack of Gaps in Medical Education, Awards or Special Honors in Clerkship in Desired Specialty, and Consistency of Grades. Compared to the 2021 PD survey, our findings show continued predictive consistency, particularly related to specialty and program commitment. CONCLUSIONS: The factors identified for the selection of medical school graduates for interview into a residency program reveal that PDs move toward a more integrated approach. Specifically, PDs are placing increasing emphasis on factors that border on subjective qualities more so than the more traditional, quantitative, and objective metrics. Medical students and educators need to continually apprise themselves of the NRMP data to inform students' preparation endeavors throughout medical school to strengthen their application portfolios and enhance their competitiveness for the matching process.

Improving Student Experiences During USMLE Step 1 "Dedicated Preparation Period" via a Course Designed to Holistically Support Academic and Wellness Needs.

Common medical school practice includes an independent "dedicated preparation period" for the USMLE Step 1 exam during which students have reported considerable stress and anxiety. Carle Illinois College of Medicine sought to improve their students' experience during the dedicated preparation period via a unique course, Phase 1 Synthesis and Summary, designed to provide support for academics and wellness. The strengths of the course were elements that maximized student flexibility, autonomy, wellness, and targeted self-study. The course design reported here may provide a model for other schools to design courses to holistically improve students' preparation for Step 1 and other board exams. Supplementary Information: The online version contains supplementary material available at 10.1007/s40670-023-01791-2.

USMLE step 1 and step 2 CK as indicators of resident performance.

BACKGROUND: The purpose of this systematic review was to (1) determine the scope of literature measuring USMLE Step 1 and Step 2 CK as predictors or indicators of quality resident performance across all medical specialties and (2) summarize the ability of Step 1 and Step 2 CK to predict quality resident performance, stratified by ACGME specialties, based on available literature. METHODS: This systematic review was designed according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) [16]. The original search strategy surveyed MEDLINE and was adapted to survey Cochrane Library and Embase. A study was deemed eligible if it provided all three of the following relevant information: (a) Step 1 or Step 2 CK as indicators for (b) resident outcomes in (c) any ACGME accredited specialty training program. RESULTS: A total of 1803 articles were screened from three separate databases. The 92 included studies were stratified by specialty, with Surgery (21.7% [20/92]), Emergency Medicine (13.0% [12/92]), Internal Medicine (10.9% [10/92]), and Orthopedic Surgery (8.7% [8/92]) being the most common. Common resident performance measures included ITE scores, board certification, ACGME milestone ratings, and program director evaluations. CONCLUSIONS: Further studies are imperative to discern the utility of Step 1 and Step 2 CK as predictors of resident performance and as tools for resident recruitment and selection. The results of this systematic review suggest that a scored Step 1 dated prior to January 2022 can be useful as a tool in a holistic review of future resident performance, and that Step 2 CK score performance may be an effective tool in the holistic review process. Given its inherent complexity, multiple tools across many assessment modalities are necessary to assess resident performance comprehensively and effectively.

Vascular surgery integrated resident selection criteria in the pass or fail era.

OBJECTIVES: Vascular surgery integrated residency (VSIR) programs are highly competitive; however, criteria for resident selection remain opaque and non-standardized. The already unclear selection criteria will be further impacted by the impending transition of the United States Medical Licensing Examination (USMLE) Step 1 from numeric scores to a binary pass/fail outcome. The purpose of this study was to investigate the historical and anticipated selection criteria of VSIR applicants. METHODS: This was a cross-sectional, nationwide, 59-item survey that was sent to all VSIR program directors (PDs). Data was analyzed using the Fisher exact test if categorical and the Mann-Whitney U test and the Kruskal-Wallis test if ordinal. RESULTS: Forty of 69 PDs (58%) responded to the survey. University-based programs constituted 85% of responders. Most VSIR PDs (65%) reported reviewing between 101 to 150 applications for 1 to 2 positions annually. Forty-two percent of the responding PDs reported sole responsibility for inviting applicants to interview, whereas 50% had a team of faculty responsible for reviewing applications. On a five-point Likert scale, letters of recommendation (LOR) from vascular surgeons or colleagues (a person the PD knows) were the most important objective criteria. Work within a team structure was rated highest among subjective criteria. The majority of respondents (72%) currently use the Step 1 score as a primary method to screen applicants. Regional differences in use of Step 1 score as a primary screening method were: Midwest (100%), Northeast (76%), South (43%), and West (40%) (P = .01). PDs responded that that they will use USMLE Step 2 score (42%) and LOR (10%) to replace USMLE Step 1 score. The current top ranked selection criteria are letters from a vascular surgeon, USMLE Step 1 score and overall LOR. The proposed top ranked selection criteria after transition of USMLE Step 1 to pass/fail include LOR overall followed by Step 2 score. CONCLUSIONS: This is the first study to evaluate the selection criteria used by PDs for VSIR. The landscape of VSIR selection criteria is shifting and increasing transparency is essential to applicants' understanding of the selection process. The transition of USMLE Step 1 to a pass/fail report will shift the attention to Step 2 scores and elevate the importance of other relatively more subjective criteria. Defining VSIR program selection criteria is an important first step toward establishing holistic review processes that are transparent and equitable.

A retrospective and correlative analysis of academic and nonacademic predictors of COMLEX level 1 performance.

CONTEXT: National licensing exams (NLEs) including the Comprehensive Osteopathic Medical Licensing Examination (COMLEX) Level 1 evaluate student achievement. Scores have historically been utilized to stratify medical student applicants for residency. Grade point average (GPA), number of practice questions completed, and performance on practice exams have been shown to be predictive of NLE performance. Test anxiety and acute stress have been shown to negatively impact NLE performance. The role of study behaviors and other nonacademic factors in COMLEX Level 1 performance is unknown. OBJECTIVES: This study aims to evaluate academic and nonacademic factors and to correlate them with COMLEX Level 1 performance. Additional analysis is conducted to associate COMLEX Level 1 performance with academic and nonacademic factors when controlling for GPA. METHODS: An anonymous online survey was administered to third- (OMS III) and fourth-year (OMS IV) osteopathic medical students at Kansas City University that had completed the COMLEX Level 1 examination. In total, 72 students responded to the survey. Survey results were linked to student records of GPA and COMLEX Level 1 scores, resulting in 59 complete responses for analysis. Independent-sample t-tests and linear ordinary least squares regression were utilized to analyze the results. RESULTS: The majority of participants are male (62.7%) and OMS III (98.3%) with an average age of 27.14 ± 2.58 (mean ± standard deviation). Further demographic data reveal hours per week spent for personal time during dedicated study (n=46, 19.7 ± 18.53), hours of sleep per night during dedicated study (7.34 ± 0.92), and money spent on board preparation ($1,319.12 ± $689.17). High ($1,600-$3,000), average ($1,000-$1,500), and low ($100-$900) spenders do not statistically differ and COMLEX Level 1 performance is not related to the number of resources utilized (F statistics <1; p>0.05). Pearson correlations reveal a statistically significant relationship between COMLEX Level 1 scores with GPA (0.73, p<0.001), number of practice exams completed (0.39, p<0.001), number of questions completed (0.46, p<0.001), number of weeks of study (0.55, p<0.001), and preparation cost (0.28, p<0.05). The regression analysis revealed that money spent on board preparation, number of questions completed, and time spent studying accounted for 75.8% of the variance in COMLEX Level 1 scores after controlling for GPA. CONCLUSIONS: The data show the association of money spent on board preparation, numbers of questions competed, and time spent studying with a student's COMLEX Level 1 score. Additionally, these results highlight the amount of money students spend on extracurricular materials to prepare for COMLEX Level 1, yet the data show that the number of resources that students utilized is not related to a student's COMLEX Level 1 performance.

New Heuristics to Stratify Applicants: Predictors of General Surgery Residency Applicant Step 1 Scores.

OBJECTIVE: In 2022, United States Medical Licensing Examination (USMLE) Step 1 scores will become pass/fail. This may be problematic, as residency programs heavily rely on USMLE Step 1 scores as a metric when determining interview invitations. This study aimed to assess candidate application metrics associated with USMLE Step 1 scores to offer programs new cues for stratifying applicants. DESIGN: Retrospective cohort study analyzing interviewed applicants to one general surgery residency program in 2019 and 2020. Applicant data analyzed included USMLE Step 1 scores, number of publications, clerkship scores, letter of recommendation scores (out of 2, scored by 0.25 interval), interview overall score (out of 5, scored by integer level), and standardized question score (out of 10). Each year, applicant's answers to one standardized behavioral question during their interview were scored by interviewers. SETTING: Tertiary medical center, academic general surgery residency program. PARTICIPANTS: Interviewed applicants at one general surgery residency program whose applications contained complete demographic information (203 out of 247). RESULTS: Multiple Linear Regression revealed that higher surgical clerkship (ß = 0.19, p = 0.006) and higher standardized interview question (ß = 0.32, p < 0.001) scores were positively associated with applicant USMLE Step 1 score (F[7, 195] = 6.61, p < 0.001, R2 = 0.19). Letter of recommendation score, number of peer reviewed publications, gender, race, and applicant type (preliminary/categorical) were not associated with USMLE Step 1 scores. CONCLUSIONS: With USMLE Step 1 scores transitioning to pass/fail, surgical residency programs need new selection heuristics. Surgery clerkship scores and standardized behavioral questions answered by applicants prior to the interview could provide a holistic view of applicants and help programs better stratify candidates without USMLE Step 1 scores.

Overemphasis of USMLE and Its Potential Impact on Diversity in Otolaryngology.

OBJECTIVE: Applicant demographics during the 2019-2020 residency cycle were evaluated to determine if strict utilization of United States Medical Licensing Examination (USMLE) scores in applicant selection could lead to a restriction in diversity. STUDY DESIGN: Cross-sectional study. SETTING: Otolaryngology residency applicants to a single institution. METHODS: A total of 381 applicants were analyzed by age, gender, applicant type, race/ethnicity, USMLE scores, permanent zip code, and graduating medical school. RESULTS: Among applicants, 37% were women; 9% were ≥30 years of age; 12% were underrepresented minorities (URMs); 71% to 81% had above-average socioeconomic surrogate markers; 22% were from a top 25 US News & World Report-ranked institution; and 81% were from an institution with an otolaryngology residency program. There was no increase in applicants who identified as URM from the 2015-2020 cycles. Multivariable regression analysis showed that applicants who were international medical graduates, URMs, and ≥30 years of age had lower Step 1 and Step 2 scores (P < .05). Applicants who identified as women had a lower Step 1 score, and those from top 25 National Institutes of Health-funded institutions had a higher Step 1 score; however, there was no difference when Step 2 scores were compared (P > .05). CONCLUSION: Our data suggest that in the pre-USMLE Step 1 pass/fail setting, strict adherence to USMLE scores may lead to disproportionally low recruitment of applicants who are women, ≥30 years of age, URMs, and from institutions without an otolaryngology residency program. We must implement measures against overemphasizing the absolute values of USMLE scores for a true holistic review of applicants, specifically to prevent an overemphasis on the USMLE Step 2 score.