Psychometric characteristics and response times for content-parallel extended-matching and one-best-answer items in relation to number of options.

BACKGROUND: This study investigated the impact of item format and number of options on the psychometric characteristics (p values and biserials) and response times for multiple-choice questions (MCQs) appearing on Step 2 of the United States Medical Licensing Examination. METHOD: In all, 192 MCQ items were used in the study. Each item was presented in two formats: in a two-item extended-matching set and as an independent item. For the extended matching format, there were two versions: a base version that included all options (10 to 26) and an 8-option version. For the independent-item format, there were three versions: a base version that included all options, and 8-option and 5-option versions created by a group of physicians that selected options without information about examinee performance. All items were embedded in unscored sections of the 2005-06 Step 2 test forms. RESULTS: Versions of items with more options were harder and required more testing time; no differences in item discrimination were observed. Mean response times for items presented in the extended-matching format were lower than for those presented as independent items, primarily because of shorter response times for the second item presented in a set. CONCLUSION: Use of the extended-matching format and smaller numbers of options per item (and more items) should result in more efficient use of testing time and greater score precision per unit of testing time.

The essential elements of undergraduate medical education in obstetrics and gynecology: a comparison of the Association of Professors of Gynecology and Obstetrics Medical Student Educational Objectives and the National Board of Medical Examiners Subject Examination.

OBJECTIVE: The objective of this study was to investigate whether the essential elements of the Association of Professors of Gynecology and Obstetrics (APGO) Medical Student Educational Objectives were adequately represented on the National Board of Medical Examiners (NBME) obstetrics and gynecology subject examination, and that the topics questioned on that examination were covered by the APGO objectives. STUDY DESIGN: The Undergraduate Medical Education Committee of APGO and the NBME staff separately reviewed the same 2 NBME obstetrics and gynecology subject examinations. The questions were mapped to the 15 essential elements of the APGO educational objectives and comparisons were made to check how well they matched. RESULTS: All the essential elements of the educational objectives were covered by the NBME subject examination. Of the questions on the examination, 99% were deemed appropriate for medical students with 70% of the questions mapping to "Priority 1" objectives. CONCLUSION: The NBME examination provides an appropriate assessment of mastery of what a medical student should learn, as represented by the APGO Medical Student Educational Objectives.

Use of multimedia on the step 1 and step 2 clinical knowledge components of USMLE: a controlled trial of the impact on item characteristics.

BACKGROUND: During 2007, multimedia-based presentations of selected clinical findings were introduced into the United States Medical Licensing Examination. This study investigated the impact of presenting cardiac auscultation findings in multimedia versus text format on item characteristics. METHOD: Content-matched versions of 43 Step 1 and 51 Step 2 Clinical Knowledge (CK) multiple-choice questions describing common pediatric and adult clinical presentations were administered in unscored sections of Step 1 and Step 2 CK. For multimedia versions, examinees used headphones to listen to the heart on a simulated chest while watching video showing associated chest and neck vein movements. Text versions described auscultation findings using standard medical terminology. RESULTS: Analyses of item responses for first-time examinees from U.S./Canadian and international medical schools indicated that multimedia items were significantly more difficult than matched text versions, were less discriminating, and required more testing time. CONCLUSIONS: Examinees can more readily interpret auscultation findings described in text using standard terminology than those same findings presented in a more authentic multimedia format. The impact on examinee performance and item characteristics is substantial.

Measurement characteristics of content-parallel single-best-answer and extended-matching questions in relation to number and source of options.

BACKGROUND: Previous research showed that extended-matching questions (EMQs) with eight options per set resulted in better score precision than EMQs with larger numbers of options or independent single-best-answer items (A-type) with five options. This study extends previous work using smaller numbers of options. METHOD: Ninety-six questions were presented in two formats on United States Medical Licensing Examination Step 2: as two-item EMQ sets and as independent A-types. Four versions of EMQs were used: five- and eight-option versions with options selected using statistics, and five- and eight-option versions with options selected by physicians. Seven A-type versions were used: three-, four-, five-, and eight-option versions with options selected using statistics, and three-, four-, and five-option versions with options selected by physicians. RESULTS: Items with more options were harder, required more time to complete, and had similar item discrimination. Option sets selected by physicians were easier, slightly more discriminating, and required less testing time. CONCLUSIONS: A-types with four or five options and EMQs with eight options make more efficient use of testing time. Provision of response statistics to content experts does not seem necessary to guide option selection.