Effect of United States Medical Licensing Examination Score Cutoffs on Recruitment of Underrepresented Applicants in the Urology Match.

OBJECTIVE: To determine how the use of United States Medical Licensing Examination (USMLE) score cutoffs during the screening process of the Urology Residency Match Program may affect recruitment of applicants who are underrepresented in medicine (URM). MATERIALS AND METHODS: Deidentified data from the Association of American Medical Colleges' (AAMC) Electronic Residency Application Service (ERAS) system was reviewed, representing all applicants to our institution's urology residency program from 2018 to 2022. We analyzed self-reported demographic variables including race/ethnicity, age, sex/gender, as well as USMLE Step 1 and Step 2 scores. Chi-square tests and ANOVA were used to determine the association between race/ethnicity and other sociodemographic factors and academic metrics. Applicants were stratified according to USMLE Step 1 cutoff scores and the distribution of applicants by race/ethnicity was assessed using a Gaussian nonlinear regression fit. RESULTS: A total of 1258 applicants submitted applications to our program during the 5-year period, including 872 males (69.3%) and 386 females (30.7%). Most applicants were White (43.5%), followed by Asian (28.3%), Hispanic/Latino (11.7%), and Black (7.0%). There was an association between race/ethnicity and USMLE scores. Median USMLE Step 1 scores for White, Asian, Hispanic/Latino, and Black applicants were 242, 242, 237, and 232, respectively (P < .001). As cutoff score increases, percentage of URM applicants decreases. CONCLUSION: The use of cutoffs based on USMLE scores disproportionately affects URM applicants. Transitioning from numeric scores to pass/fail may enhance holistic review processes and increase the representation of URM applicants offered interviews at urology residency programs.

USMLE step 1 and step 2 CK as indicators of resident performance.

BACKGROUND: The purpose of this systematic review was to (1) determine the scope of literature measuring USMLE Step 1 and Step 2 CK as predictors or indicators of quality resident performance across all medical specialties and (2) summarize the ability of Step 1 and Step 2 CK to predict quality resident performance, stratified by ACGME specialties, based on available literature. METHODS: This systematic review was designed according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) [16]. The original search strategy surveyed MEDLINE and was adapted to survey Cochrane Library and Embase. A study was deemed eligible if it provided all three of the following relevant information: (a) Step 1 or Step 2 CK as indicators for (b) resident outcomes in (c) any ACGME accredited specialty training program. RESULTS: A total of 1803 articles were screened from three separate databases. The 92 included studies were stratified by specialty, with Surgery (21.7% [20/92]), Emergency Medicine (13.0% [12/92]), Internal Medicine (10.9% [10/92]), and Orthopedic Surgery (8.7% [8/92]) being the most common. Common resident performance measures included ITE scores, board certification, ACGME milestone ratings, and program director evaluations. CONCLUSIONS: Further studies are imperative to discern the utility of Step 1 and Step 2 CK as predictors of resident performance and as tools for resident recruitment and selection. The results of this systematic review suggest that a scored Step 1 dated prior to January 2022 can be useful as a tool in a holistic review of future resident performance, and that Step 2 CK score performance may be an effective tool in the holistic review process. Given its inherent complexity, multiple tools across many assessment modalities are necessary to assess resident performance comprehensively and effectively.

Increasing Diversity in the Physician Workforce: Pathway Programs and Predictive Analytics.

PROBLEM: Lack of diversity in the physician workforce has well-documented negative impacts on health outcomes. Evidence supports the use of pathway or pipeline programs to recruit underrepresented in medicine students. However, data on how a pathway program should deliver instruction are lacking. This report describes a multiyear project to build such a system with the goal of increasing diversity within medical school cohorts and ultimately the physician workforce. APPROACH: In the 2015-2016 academic year, the Ponce Health Sciences University started a 3-phase project to create a data-driven medical school feeder system by coupling a pathway program with predictive analytics. Phase 1 launched the pathway program. Phase 2 developed and validated a predictive model that estimates United States Medical Licensing Examination (USMLE) Step 1 performance. Phase 3 is underway and focuses on adoption, implementation, and support. OUTCOMES: Data analysis compared 2 groups of students (pathway vs direct) across specific factors, including Medical College Admission Test (MCAT) score, undergraduate grade point average (GPA), first-generation status, and Step 1 exam performance. Statistically significant differences were found between the 2 groups on the MCAT exam and undergraduate GPA; however, no significant differences were found between groups for first-generation status and performance on the Step 1 exam. This finding supports the authors' hypothesis that although pathway students have significantly lower mean MCAT exam scores compared with direct students, pathway students perform just as well on the USMLE Step 1 exam. NEXT STEPS: Next steps include expanding the project to another campus, adding more socioeconomic status and first-generation data, and identifying best curricular predictors. The authors recommend that medical school programs use pathway programs and predictive analytics to create a more data-centered approach to accepting students with the goal of increasing physician workforce diversity.

Factors Leading to Successful Performance on U.S. National Licensure Exams for Medical Students: A Scoping Review.

PURPOSE: To synthesize the evidence of the factors leading to successful performance on knowledge-based national licensure exams (NLEs) for medical students. METHOD: The authors conducted a scoping review to summarize the peer-reviewed empiric literature that used United States Medical Licensing Examination (USMLE) Step 1 or Step 2 Clinical Knowledge or Comprehensive Osteopathic Medical Licensing Examination (COMLEX) Level 1 or Level 2 Cognitive Evaluation scores as outcomes. The authors searched PubMed and Scopus without date restrictions through April 30, 2021. Two reviewers independently screened and selected studies for inclusion. Data were summarized narratively and with descriptive statistics. RESULTS: The authors screened 1,185 unique citations and included 233 full-text studies in their review. Of these, 201 (86%) were studies of USMLE exams, 31 (13%) were studies of COMLEX exams, and 1 (0.4%) reported on both. The authors classified 29 studies (12%) as informing NLE preparation, 163 (70%) as attempting to identify predictive variables, and 76 (33%) as using NLE scores for program evaluation. Preparation studies found that the number of practice test items, practice exam scores, and less time in dedicated preparation correlated with higher NLE scores. Use of other commercial resources or study strategies was not consistently associated with higher scores. Predictive studies found the strongest relationships between individuals' performance on past assessments and their NLE scores. CONCLUSIONS: The factors leading to successful performance on knowledge-based NLEs align with well-known principles from the cognitive sciences. Learners build on existing foundations of knowledge (reflected in their prior academic performance) and are likely to learn more efficiently with testing and spaced learning over time. While commercial test preparation resources are ubiquitous, there is no evidence that a single resource gives students a competitive advantage on NLEs. Developing habits of regular and continuous learning is necessary for clinical practice and successful NLE performance.

A Concordance Study of COMLEX-USA and USMLE Scores.

BACKGROUND: Under the single GME accreditation system, residency programs receive applicants from MD- and DO-granting medical schools, each of which have their own set of licensing examinations, making concordance studies increasingly relevant. Previous studies comparing Comprehensive Osteopathic Medical Licensing Examination of the United States (COMLEX-USA) and United States Medical Licensing Examination (USMLE) scores have been limited in sample size and examinee composition and have yielded comparisons that may not be generalizable across all applicants. Some osteopathic medical students take USMLE in addition to COMLEX-USA, often at considerable cost and effort, with the aim of making themselves more desirable to potential residency programs. Having more reliable comparisons of COMLEX-USA and USMLE scores would allow program directors to better estimate a score on the alternate examination. OBJECTIVE: To derive an accurate concordance between COMLEX-USA and USMLE scores, based on a large sample of osteopathic students who took both examinations. METHODS: Five colleges of osteopathic medicine, representing various regions of the United States, participated in this study. The data included demographics and COMLEX-USA and USMLE scores from September 2015 through August 2020 for students who took both examinations. We derived the concordance between COMLEX-USA and USMLE scores using equipercentile matching. RESULTS: Comparisons of demographic characteristics showed only minor differences between the sample and the overall population for COMLEX-USA takers, although scores for the study sample were, on average, greater. CONCLUSIONS: A strong association exists between the scores on the COMLEX-USA and USMLE examinations, allowing prediction of performance on USMLE from COMLEX-USA.

Comprehensive Reform and Greater Equity in Applying to Residency-Trainees' Mixed Responses to a Pass/Fail USMLE Step 1.

BACKGROUND: Pass/fail USMLE Step 1 score reporting may have varying implications for trainees of different demographic and training backgrounds. OBJECTIVE: To characterize the perspectives of a diverse cohort of trainees on the impact of pass/fail Step 1 score reporting. METHODS: In 2020, 197 US and international medical school deans and 822 designated institutional officials were invited to distribute anonymous electronic surveys among their trainees. Separate surveys for medical students and residents/fellows were developed based on the authors' prior work surveying program directors on this topic. Underrepresented in medicine (UiM) was defined in accordance with AAMC definitions. Descriptive and comparative analyses were performed, and results were considered statistically significant with P < .05. RESULTS: A total of 11 633 trainees responded (4379 medical students and 7254 residents/fellows; 3.3% of an estimated 285 000 US trainees). More students favored the score reporting change than residents/fellows (43% vs 31%; P < .001; 95% CI 0-24). Trainees identifying as UiM were more likely to favor the change (50% vs 34%; P < .001; 95% CI 0-32) and to agree it would decrease socioeconomic disparities (44% vs 25%; P < .001; 95% CI 0-38) relative to non-UiM trainees. Nearly twice as many osteopathic and international medical graduate students felt they would be disadvantaged compared to MD students because of pass/fail score reporting (61% vs 31%; P < .001; 95% CI 0-60). CONCLUSIONS: Trainee perspectives regarding USMLE Step 1 score reporting are mixed. UiM trainees were more likely to favor the score reporting change, while osteopathic and international medical students were less in favor of the change.

Assessment of Plastic Surgery Residency Applications without United States Medical Licensing Examination Step 1 Scores.

SUMMARY: The United States Medical Licensing Examination announced the changing of Step 1 score reporting from a three-digit number to pass/fail beginning on January 1, 2022. Plastic surgery residency programs have traditionally used United States Medical Licensing Examination Step 1 scores to compare plastic surgery residency applicants. Without a numerical score, the plastic surgery residency application review process will likely change. This article discusses advantages, disadvantages, and steps forward for residency programs related to the upcoming change. The authors encourage programs to continue to seek innovative methods of objectively and holistically evaluating applications.

Examining concurrent validity between COMLEX-USA Level 2-Cognitive Evaluation and COMLEX-USA Level 2-Performance Evaluation.

CONTEXT: The Comprehensive Osteopathic Medical Licensing Examination of the United States of America (COMLEX-USA) is a three level examination used as a pathway to licensure for students in osteopathic medical education programs. COMLEX-USA Level 2 includes a written assessment of Fundamental Clinical Sciences for Osteopathic Medical Practice (Level 2-Cognitive Evaluation [L2-CE]) delivered in a computer based format and separate performance evaluation (Level 2-Performance Evaluation [L2-PE]) administered through live encounters with standardized patients. L2-PE was designed to augment L2-CE. It is expected that the two examinations measure related yet distinct constructs. OBJECTIVES: To explore the concurrent validity of L2-CE with L2-PE. METHODS: First attempt test scores were obtained from the National Board of Osteopathic Medical Examiners database for 6,639 candidates who took L2-CE between June 2019 and May 2020 and matched to the students' L2-PE scores. The sample represented all colleges of osteopathic medicine and 97.5% of candidates who took L2-CE during the complete 2019-2020 test cycle. We calculated disattenuated correlations between the total score for L2-CE, the L2-CE scores for the seven competency domains (CD1 through CD7), and the L2-PE scores for the Humanistic Domain (HM) and Biomedical/Biomechanical Domain (BM). All scores were on continuous scales. RESULTS: Pearson correlations ranged from 0.10 to 0.88 and were all statically significant (p<0.01). L2-CE total score was most strongly correlated with CD2 (0.88) and CD3 (0.85). Pearson correlations between the L2-CE competency domain subscores ranged from 0.17 to 0.70, and correlations which included either HM or BM ranged from 0.10 to 0.34 with the strongest of those correlations being between BM and L2-CE total score (0.34) as well as between HM and BM (0.28).The largest increase between corresponding Pearson and disattenuated correlations was for pairs of scores with lower reliabilities such as CD5 and CD6, which had a Pearson correlation of 0.17 and a disattenuated correlation of 0.68. The smallest increase in correlations was observed in pairs of scores with larger reliabilities such as L2-CE total score and HM, which had a Pearson correlation of 0.23 and a disattenuated correlation of 0.28. The reliability of L2-CE was 0.87, 0.81 for HM, and 0.73 for BM. The reliabilities for the L2-CE competency domain scores ranged from 0.22 to 0.74. The small to moderate correlations between the L2-CE total score and the two L2-PE support the expectation that these examinations measure related but distinct constructs. The correlations between L2-PE and L2-CE competency domain subscores reflect the distribution of items defined by the L2-PE blueprint, providing evidence that the examinations are performing as designed. CONCLUSIONS: This study provides evidence supporting the validity of the blueprints for constructing COMLEX-USA Levels 2-CE and 2-PE examinations in concert with the purpose and nature of the examinations.

Meaningful use of COMSAE Phase 1 in preparation for COMLEX-USA Level 1.

CONTEXT: The Comprehensive Osteopathic Medical Licensing Examination of the United States (COMLEX-USA) is a three level national standardized licensure examination for the practice of osteopathic medicine. The Comprehensive Medical Self Assessment Examination (COMSAE) is a three phase self assessment tool designed to gauge the base knowledge and ability of candidates preparing for COMLEX-USA. OBJECTIVES: To investigate how COMSAE Phase 1 (Phase 1) was used by candidates and how completing Phase 1 impacted their performance on the COMLEX-USA Level 1 (Level 1) examination. METHODS: Using data from the 2018-2019 administration of Level 1 and Phase 1 examinations, we counted the frequency of the unique Phase 1 forms taken by the candidates and calculated the correlation between the candidates' first attempt Phase 1 scores and the number of Phase 1 forms taken. We then calculated the correlation between the Level 1 scores and the Phase 1 scores. Next, we applied a multilevel regression model to examine the candidates' score improvement on the multiple Phase 1 forms taken. Finally, we investigated the effect of practicing through Phase 1 on the candidates' Level 1 performance using logistic regression models. RESULTS: The majority of candidates took one (2,414; 33.9%) to two (2,196; 30.8%) timed Phase 1 forms prior to the Level 1 examination. There was a significant negative correlation (r=-0.48, t(6,505)=-44.05, p<0.001) between the candidates' first attempt Phase 1 scores and the number of Phase 1 forms taken. There was a strong and positive correlation (r=0.66 to 0.74, p<0.001) between Phase 1 and Level 1 scores. With other variables controlled, on average, candidates' Phase 1 scores increased 23.2 points on one attempt from the previous attempt. Having the most recent Phase 1 score controlled, a greater number of Phase 1 forms taken was associated with an improvement on the Level 1 performance. CONCLUSIONS: The significant correlation between Phase 1 and Level 1 performance provided validity evidence for Phase 1. Moreover, our results suggested that candidates, especially those with lower performance on their initial Phase 1 attempt, might improve their Level 1 performance by taking multiple Phase 1 forms to monitor their academic improvement and gauge their readiness for Level 1.

Early prediction of the risk of scoring lower than 500 on the COMLEX 1.

BACKGROUND: The Comprehensive Osteopathic Medical Licensing Examination of the United States Level 1 (COMLEX 1) is important for medical students to be able to graduate. There is a glaring need to identify students who are at a significant risk of performing poorly on COMLEX 1 as early as possible so that extra assistance can be provided to those students. Our goal is to produce a reliable predictive model to identify students who are at risk of scoring lower than 500 on COMLEX 1 at the earliest possible time. METHODS: Academic data from medical students who matriculated at Rocky Vista University College of Osteopathic Medicine between 2011 and 2017 were obtained. Odds ratios were used to assess the predictors for scoring lower than 500 on COMLEX 1. Correlation with COMLEX 1 scores was assessed with Pearson correlation coefficient. The predictive models were developed by multiple logistic regression, backward logistic regression, and logistic regression with average scores in courses in the first three semesters, and were based on performances on the Medical College Admissions Test (MCAT) before admission, as well as students' performances in preclinical courses during the first three semesters. The models were generated in about 82% of the student performance data and were then validated in the remaining 18% of the data. RESULTS: Odds ratios showed that MCAT scores and final grades in each course in the first three semesters were significant in predicting a score lower than 500 on COMLEX 1. Performances in third-semester courses including Renal System II, Cardiovascular System II, and Respiratory System II were most important in prediction. The three predictive models had sensitivities of 65.8 -71%, and specificities of 83.2 - 88.2% in predicting a score lower than 500 on COMLEX 1. CONCLUSIONS: Lower MCAT scores and lower grades in the first three semesters of medical school predict scoring lower than 500 on COMLEX 1. Students who are identified at risk by our models will have a 65.8 -71% chance of actually scoring lower than 500 on COMLEX 1. Those students will have enough time to receive assistance before taking COMLEX 1.

Admissions Is Not Enough: The Racial Achievement Gap in Medical Education.

The achievement gap is a disparity in academic and standardized test performance that exists between White and underrepresented minority (URM) students that begins as early as preschool and worsens as students progress through the educational system. Medical education is not immune to this inequality. URM medical students are more likely to experience delayed graduation and course failure, even after accounting for science grade point average and Medical College Admission Test performance. Moreover, URM students are more likely to earn lower scores on licensing examinations, which can have a significant impact on their career trajectory, including specialty choice and residency competitiveness. After the release of preliminary recommendations from the Invitational Conference on USMLE Scoring (InCUS) and public commentary on these recommendations, the National Board of Medical Examiners and Federation of State Medical Boards announced that the United States Medical Licensing Examination (USMLE) Step 1 would transition from a 3-digit numeric score to pass/fail scoring. Given that another of InCUS's recommendations was to "minimize racial demographic differences that exist in USMLE performance," it is paramount to consider the impact of this scoring change on URM medical students specifically. Holistic admissions are a step in the right direction of acknowledging that URM students often travel a further distance to reach medical school. However, when residency programs emphasize USMLE performance (or any standardized test score) despite persistent test score gaps, medical education contributes to the disproportionate harm URM students face and bolsters segregation across medical specialties. This Perspective provides a brief explanation of the achievement gap, its psychological consequences, and its consequences in medical education; discusses the potential effect of the Step 1 scoring change on URM medical students; and provides a review of strategies to redress this disparity.

Implications of the United States Medical Licensing Examination Step 1 Examination Transition to Pass/Fail on Medical Students Education and Future Career Opportunities.

BACKGROUND: United States Medical Licensing Examination (USMLE) Step 1 will transition to pass/fail score by 2022. We aim to investigate US medical students' perspectives on the potential implications this transition would have on their education and career opportunities. METHODS: A cross-sectional study investigating US medical students' perspectives on the implications of transition of the USMLE Step 1 exam to pass/fail. Students were asked their preferences regarding various aspects of the USMLE Step 1 examination, including activities, educational opportunities, expenses regarding preparation for the examination, and future career opportunities. RESULTS: 215 medical students responded to the survey, 59.1% were women, 80.9% were allopathic vs. 19.1% osteopathic students. 34.0% preferred the USMLE Step 1 to be graded on a pass/fail score, whereas 53.5% preferred a numeric scale. Osteopathic vs. allopathic students were more likely to report that the pass/fail transition will negatively impact their residency match (aOR = 1.454, 95% CI: 0.515, 4.106) and specialty of choice (aOR = 3.187, 95% CI: 0.980, 10.359). 57.7% of respondents reported that the transition to a pass/fail grading system will change their study habits. CONCLUSIONS: The transition of the USMLE Step 1 to a pass/fail system has massive implications on medical students and residency programs alike. Though the majority of medical students did not prefer the USMLE Step 1 to have a pass/fail score, they must adapt their strategies to remain competitive for residency applications. Residency programs should create a composite score based off all aspects of medical students' applications in order to create a holistic and fair evaluation and ranking system.