Differential Item Functioning Analysis of United States Medical Licensing Examination Step 1 Items.

PURPOSE: Previous studies have examined and identified demographic group score differences on United States Medical Licensing Examination (USMLE) Step examinations. It is necessary to explore potential etiologies of such differences to ensure fairness of examination use. Although score differences are largely explained by preceding academic variables, one potential concern is that item-level bias may be associated with remaining group score differences. The purpose of this 2019-2020 study was to statistically identify and qualitatively review USMLE Step 1 exam questions (items) using differential item functioning (DIF) methodology. METHOD: Logistic regression DIF was used to identify and classify the effect size of DIF on Step 1 items meeting minimum sample size criteria. After using DIF to flag items statistically, subject matter expert (SME) review was used to identify potential reasons why items may have performed differently between racial and gender groups, including characteristics such as content, format, wording, context, or stimulus materials. USMLE SMEs reviewed items to identify the group difference they believed was present, if any; articulate a rationale behind the group difference; and determine whether that rationale would be considered construct relevant or construct irrelevant. RESULTS: All identified DIF rationales were relevant to the constructs being assessed and therefore did not reflect item bias. Where SME-generated rationales aligned with statistical differences (flags), they favored self-identified women on items tagged to women's health content categories and were judged to be construct relevant. CONCLUSIONS: This study did not find evidence to support the hypothesis that group-level performance differences beyond those explained by prior academic performance variables are driven by item-level bias. Health professions examination programs have an obligation to assess for group differences, and when present, investigate to what extent, if any, measurement bias plays a role.

Revisiting Retake Policy: Analyzing the Success Rates of Examinees With Multiple Attempts on the United States Medical Licensing Examination.

PURPOSE: A testing program's decision making on retake policy considers a number of factors, including fairness to examinees, examination security, examination purpose, and classification accuracy. For high-stakes licensure and certification examinations charged with protection of the public, this includes balancing fairness issues inherent in the potential for false negatives with the public protection need to minimize false positives. Since 2012, the United States Medical Licensing Examination (USMLE) has maintained a policy of ≤ 6 attempts on any examination component. The purpose of this study was to empirically examine the appropriateness of existing USMLE retake policy. METHOD: A sample of over 300,000 examinees who were administered each Step examination between 2008 and 2018 was examined to address how pass rates (per attempt and cumulatively) differed for examinees and examinee subgroups via graphic depictions of pass rates, the impact of restricting the number of attempts on examinees and examinee subgroups via logistic regression analyses, and the impact of restricting the number of attempts on classification error via classification error analyses. RESULTS: Repeaters displayed consistently lower pass rates on subsequent attempts. Regardless of Step, most passing examinees did so by their fourth attempt. Models predicting the log odds of examinees taking ≥ 5 attempts showed a significant effect of gender, race, and medical school status, although the associated sample sizes are small. Misclassification analyses showed that, as attempts increased, the percentage of false passers increased and percentage of false failers decreased. CONCLUSIONS: These considerations led to the USMLE's decision to reduce its attempt limit from 6 to 4 on any Step or Step component effective July 2021. This change balances the competing interests of fairness and examination access to the examinee population, while simultaneously maintaining a minimum standard consistent with the mission of the USMLE program to protect the public.

Examining Demographics, Prior Academic Performance, and United States Medical Licensing Examination Scores.

PURPOSE: To examine whether demographic differences exist in United States Medical Licensing Examination (USMLE) scores and the extent to which any differences are explained by students' prior academic achievement. METHOD: The authors completed hierarchical linear modeling of data for U.S. and Canadian allopathic and osteopathic medical graduates testing on USMLE Step 1 during or after 2010, and completing USMLE Step 3 by 2015. Main outcome measures were computer-based USMLE examinations: Step 1, Step 2 Clinical Knowledge, and Step 3. Test-taker characteristics included sex, self-identified race, U.S. citizenship status, English as a second language, and age at first Step 1 attempt. Covariates included composite Medical College Admission Test (MCAT) scores, undergraduate grade point average (GPA), and previous USMLE scores. RESULTS: A total of 45,154 examinees from 172 medical schools met the inclusion criteria. The sample was 67% white and 48% female; 3.7% non-U.S. citizens; and 7.4% with English as a second language. Hierarchical linear models examined demographic variables with and without covariates including MCAT scores and GPA. All Step examinations showed significant differences by gender after adding covariates, varying by Step. Racial differences were observed for each Step, attenuated by the addition of covariates. CONCLUSIONS: Demographic differences in USMLE performance were tempered by previous examination performance and undergraduate performance. Additional research is required to identify factors that contribute to demographic differences, can aid educators' identification of students who would benefit from assistance preparing for USMLE, and can assist residency program directors in assessing performance measures while meeting diversity goals.