Investigation of answer changes on the USMLE® Step 2 Clinical Knowledge examination.

BACKGROUND: Examinees often believe that changing answers will lower their scores; however, empirical studies suggest that allowing examinees to change responses may improve their performance in classroom assessments. To date, no studies have been able to examine answer changes during large scale professional credentialing or licensing examinations. METHODS: In this study, we expand the research on answer changes by analyzing responses from 27,830 examinees who completed the Step 2 Clinical Knowledge (CK) examination between August of 2015 and August of 2016. RESULTS: The results showed that although 68% of examinees changed at least one item, the overall average number of changes was small. Among the examinees who changed answers, approximately 45% increased their scores and approximately 28% decreased their scores. On average, examinees spent shortest time on the item changes from wrong to right and they were more likely to change their scores from wrong to right than right to wrong. CONCLUSIONS: Consistent with previous studies, these findings support the beneficial effects of answer changes in high-stakes medical examinations and suggest that examinees who are overly cautious about changing answers may put themselves at a disadvantage.

US Medical Student Performance on the NBME Subject Examination in Internal Medicine: Do Clerkship Sequence and Clerkship Length Matter?

BACKGROUND: Prior to graduation, US medical students are required to complete clinical clerkship rotations, most commonly in the specialty areas of family medicine, internal medicine, obstetrics and gynecology (ob/gyn), pediatrics, psychiatry, and surgery. Within a school, the sequence in which students complete these clerkships varies. In addition, the length of these rotations varies, both within a school for different clerkships and between schools for the same clerkship. OBJECTIVE: The present study investigated the effects of clerkship sequence and length on performance on the National Board of Medical Examiner's subject examination in internal medicine. PARTICIPANTS: The study sample included 16,091 students from 67 US Liaison Committee on Medical Education (LCME)-accredited medical schools who graduated in 2012 or 2013. MAIN MEASURES: Student-level measures included first-attempt internal medicine subject examination scores, first-attempt USMLE Step 1 scores, and five dichotomous variables capturing whether or not students completed rotations in family medicine, ob/gyn, pediatrics, psychiatry, and surgery prior to taking the internal medicine rotation. School-level measures included clerkship length and average Step 1 score. DESIGN: Multilevel models with students nested in schools were estimated with internal medicine subject examination scores as the dependent measure. Step 1 scores and the five dichotomous variables were treated as student-level predictors. Internal medicine clerkship length and average Step 1 score were used to predict school-to-school variation in average internal medicine subject examination scores. KEY RESULTS: Completion of rotations in surgery, pediatrics and family medicine prior to taking the internal medicine examination significantly improved scores, with the largest benefit observed for surgery (coefficient = 1.58 points; p value < 0.01); completion of rotations in ob/gyn and psychiatry were unrelated to internal medicine subject examination performance. At the school level, longer internal medicine clerkships were associated with higher scores on the internal medicine examination (coefficient = 0.23 points/week; p value < 0.01). CONCLUSIONS: The order in which students complete clinical clerkships and the length of the internal medicine clerkship are associated with their internal medicine subject examination scores. Findings may have implications for curriculum re-design.

An Examination of the Associations Among USMLE Step 3 Scores and the Likelihood of Disciplinary Action in Practice.

PURPOSE: As the last examination in the United States Medical Licensing Examination (USMLE) sequence, Step 3 provides a safeguard before physicians enter into unsupervised practice. There is, however, little validity research focusing on Step 3 scores beyond examining its associations with other educational and professional assessments thought to cover similar content. This study examines the associations between Step 3 scores and subsequent receipt of disciplinary action taken by state medical boards for problematic behavior in practice. It analyzes Step 3 total, Step 3 computer-based case simulation (CCS), and Step 3 multiple-choice question (MCQ) scores. METHOD: The final sample included 275,392 board-certified physicians who graduated from MD-granting medical schools and who passed Step 3 between 2000 and 2017. Cross-classified multilevel logistic regression models were used to examine the effects of Step 3 scores on the likelihood of receiving a disciplinary action, controlling for other USMLE scores and accounting for jurisdiction and specialty. RESULTS: Results showed that physicians with higher Step 3 total, CCS, and MCQ scores tended to have lower chances of receiving a disciplinary action, after accounting for other USMLE scores. Specifically, a 1-standard-deviation increase in Step 3 total, CCS, and MCQ score was associated with a 23%, 11%, and 17% decrease in the odds of receiving a disciplinary action, respectively. The effect of Step 2 CK score on the likelihood of receiving a disciplinary action was statistically significant, while the effect of Step 1 score became statistically nonsignificant when other Step scores were included in the analysis. CONCLUSIONS: Physicians who perform better on Step 3 are less likely to receive a disciplinary action from a state medical board for problematic behavior in practice. These findings provide some validity evidence for the use of Step 3 scores when making medical licensure decisions in the United States.

International variation in performance by clinical discipline and task on the United States medical licensing examination step 2 clinical knowledge component.

PURPOSE: To investigate country-to-country variation in performance across clinical science disciplines and tasks for examinees taking the Step 2 Clinical Knowledge (CK) component of the United States Medical Licensing Examination. METHOD: In 2012 the authors analyzed demographic characteristics, total scores, and percent-correct clinical science discipline and task scores for more than 88,500 examinees taking Step 2 CK for the first time during the 2008-2010 academic years. For each examinee and score, differences between the score and the mean performance of examinees at U.S. MD-granting medical schools were calculated, and mean differences by country of medical school were tabulated for analysis of country-to-country variation in performance by clinical discipline and task. RESULTS: Controlling for overall performance relative to U.S. examinees, results showed that international medical graduates (IMGs) performed best in Surgery and worst in Psychiatry for clinical discipline scores; for clinical tasks, IMGs performed best in Understanding Mechanisms of Disease and worst in Promoting Preventive Medicine and Health Maintenance. The pattern of results was strongest for IMGs attending schools in the Middle East and Australasia, present to a lesser degree for IMGs attending schools in Europe, and absent for IMGs attending Caribbean medical schools. CONCLUSIONS: Country-to-country differences in relative performance were present for both clinical discipline and task scores. Possible explanations include differences in learning outcomes, curriculum emphasis and clinical experience, standards of care, and culture, as well as the effects of English as a second language and relative emphasis on preparing students to take the Step 2 CK exam.

Use of multimedia on the step 1 and step 2 clinical knowledge components of USMLE: a controlled trial of the impact on item characteristics.

BACKGROUND: During 2007, multimedia-based presentations of selected clinical findings were introduced into the United States Medical Licensing Examination. This study investigated the impact of presenting cardiac auscultation findings in multimedia versus text format on item characteristics. METHOD: Content-matched versions of 43 Step 1 and 51 Step 2 Clinical Knowledge (CK) multiple-choice questions describing common pediatric and adult clinical presentations were administered in unscored sections of Step 1 and Step 2 CK. For multimedia versions, examinees used headphones to listen to the heart on a simulated chest while watching video showing associated chest and neck vein movements. Text versions described auscultation findings using standard medical terminology. RESULTS: Analyses of item responses for first-time examinees from U.S./Canadian and international medical schools indicated that multimedia items were significantly more difficult than matched text versions, were less discriminating, and required more testing time. CONCLUSIONS: Examinees can more readily interpret auscultation findings described in text using standard terminology than those same findings presented in a more authentic multimedia format. The impact on examinee performance and item characteristics is substantial.