Psychometric characteristics and response times for one-best-answer questions in relation to number and source of options.

BACKGROUND: The research reported here investigated the impact of number and source of response options on the psychometric characteristics and response times for one-best-answer MCQs. METHOD: Ninety sets of MCQs were used in two studies; numbers of options in base versions of items ranged from 11 to 25. For each set, a United States Medical Licensing Examination Step 2 item-writing committee selected the five options viewed as most appropriate. For 40 used sets, two NBME staff constructed five- and eight-option versions to maximize item discrimination. All versions of items were embedded unscored on 2003-04 Step 2 test forms. RESULTS: Versions of items with more options were harder and required more testing time; no differences in item discrimination were observed in either study, but previous versions of the items in extended matching format were more discriminating than those used in the study. CONCLUSION: Use of smaller numbers of options (and more items) results in more efficient use of testing time.

The impact of postgraduate training and timing on USMLE Step 3 performance.

PURPOSE: This study examined the extent to which differences in clinical experience, gained in postgraduate training programs, affect performance on Step 3 of the United States Medical Licensing Examination (USMLE). METHOD: Subjects in the study were 36,805 U.S. and Canadian medical school graduates who took USMLE Step 3 for the first time between November 1999 and December 2002. Regression analyses examined the relation between length and type of postgraduate training and Step 3 score after controlling for prior performance on previous USMLE examinations. RESULTS: Results indicate that postgraduate training in programs that provide exposure to a broad range of patient problems, and continued training in such areas, improves performance on Step 3. CONCLUSIONS: Study data reaffirm the validity of the USMLE Step 3 examination, and the information found in the pattern of results across specialties may be useful to residents and program directors.