Prospective implementation of correction for guessing in oral and maxillofacial pathology multiple-choice examinations: did student performance improve?

A standard correction for random guessing on multiple-choice examinations was implemented prospectively in an Oral and Maxillofacial Pathology course for second-year dental students. The correction was a weighted scoring formula for points awarded for correct answers, incorrect answers, and unanswered questions such that the expected gain in the multiple-choice examination score due to random guessing was zero. An equally weighted combination of four examinations using equal numbers of short-answer questions and multiple-choice questions was used for student evaluation. Scores on both types of examinations, after implementation of the correction for guessing on the multiple-choice component (academic year 2005-06), were compared with the previous year (academic year 2004-05) when correction for guessing was not used for student evaluation but was investigated retrospectively. Academically, the two classes were comparable as indicated by the grade distributions in a General Pathology course taken immediately prior to the Oral and Maxillofacial Pathology course. Agreement between scores on short-answer examinations and multiple-choice examinations was improved in the 2005-06 class compared with the 2004-05 class. Importantly, the test score means were higher on both the short-answer and multiple-choice examinations in the Oral and Maxillofacial Pathology course, and the standard deviations were significantly smaller in 2005-06 compared to 2004-05; these differences reflected an upward shift in the lower part of the grade distributions to higher grades in 2005-06. Furthermore, when students were classified by their grade in the General Pathology course, students receiving a C (numerical grade of 70-79 percent) in General Pathology had significantly improved performance in the Oral and Maxillofacial Pathology course in 2005-06, relative to 2004-05, on both short-answer and multiple-choice examinations representing an aptitude-treatment interaction. We interpret this improved performance as a response to a higher expectation imposed on the 2005-06 students by the prospective implementation of correction for guessing.

Correcting for guessing increases validity in multiple-choice examinations in an oral and maxillofacial pathology course.

A standard correction for random guessing on multiple-choice examinations was examined retrospectively in an oral and maxillofacial pathology course for second-year dental students. The correction was a weighting formula for points awarded for correct answers, incorrect answers, and unanswered questions such that the expected value of the increase in test score due to guessing was zero. We compared uncorrected and corrected scores on examinations using a multiple-choice format with scores on examinations composed of short-answer questions. The short-answer format eliminated or at least greatly reduced the potential for guessing the correct answer. Agreement of corrected multiple-choice scores with short-answer scores (intraclass correlation coefficient 0.78) was significantly (p=0.015) higher than agreement of uncorrected multiple-choice scores with short-answer scores (intraclass correlation coefficient 0.71). The higher agreement indicated increased validity for the corrected multiple-choice examination.

Fostering dental student self-assessment of knowledge by confidence scoring of multiple-choice examinations.

Creating a learning environment that fosters student acquisition of self-assessment behaviors and skills is critically important in the education and training of health professionals. Self-assessment is a vital component of competent practice and lifelong learning. This article proposes applying a version of confidence scoring of multiple-choice questions as one avenue to address this crucial educational objective for students to be able to recognize and admit what they do not know. The confidence scoring algorithm assigns one point for a correct answer, deducts fractional points for an incorrect answer, but rewards students fractional points for leaving the question unanswered in admission that they are unsure of the correct answer. The magnitude of the reward relative to the deduction is selected such that the expected gain due to random guessing, even after elimination of all but one distractor, is never greater than the reward. Curricular implementation of this confidence scoring algorithm should motivate health professions students to develop self-assessment behaviors and enable them to acquire the skills necessary to critically evaluate the extent of their current knowledge throughout their professional careers. This is a professional development competency that is emphasized in the educational standards of the Commission on Dental Accreditation (CODA).

Improving multiple-choice questions to better assess dental student knowledge: distractor utilization in oral and maxillofacial pathology course examinations.

How many incorrect response options (known as distractors) to use in multiple-choice questions has been the source of considerable debate in the assessment literature, especially relative to influence on the likelihood of students' guessing the correct answer. This study compared distractor use by second-year dental students in three successive oral and maxillofacial pathology classes that had three different examination question formats and scoring resulting in different levels of academic performance. One class was given all multiple-choice questions; the two other were given half multiple-choice questions, with and without formula scoring, and half un-cued short-answer questions. Use by at least 1 percent of the students was found to better identify functioning distractors than higher cutoffs. The average number of functioning distractors differed among the three classes and did not always correspond to differences in class scores. Increased numbers of functioning distractors were associated with higher question discrimination and greater question difficulty. Fewer functioning distractors fostered more effective student guessing and overestimation of academic achievement. Appropriate identification of functioning distractors is essential for improving examination quality and better estimating actual student knowledge through retrospective use of formula scoring, where the amount subtracted for incorrect answers is based on the harmonic mean number of functioning distractors.

Short-answer questions and formula scoring separately enhance dental student academic performance.

In this study, numerical course scores of second-year dental students in four successive classes in an oral and maxillofacial pathology course were compared. While the course content and teaching methods were essentially unchanged throughout the four years, two modest departures from the sole use of multiple-choice format questions were made in the assessment of student achievements. The modifications consisted of creating a more challenging examination procedure through the inclusion of un-cued short-answer format questions and the institution of correction-for-guessing scoring on multiple-choice examinations. Academically, the students in the four classes were comparable, as indicated by their respective numerical course score distributions in a prerequisite general pathology course in which the course content was unchanged, and all multiple-choice format questions were used to assess student academic achievements. This four-year study demonstrated that two qualitative changes in the educational environment-utilization of un-cued short-answer questions and correction for guessing scoring of multiple-choice questions-separately resulted in significant improvements in student course scores. Our results support the notion that, without any changes in curricular content or emphasis, combinations of qualitative changes in the assessment procedures alter student behavior and, as a consequence, appreciably improve their academic achievements.

Short-answer examinations improve student performance in an oral and maxillofacial pathology course.

The effect of examination question format on student performance was assessed by investigating three academically comparable second-year dental school classes in an oral and maxillofacial pathology course. One class was given examinations with all multiple-choice questions, one class was given examinations with all short-answer questions, and one class was given examinations with half multiple-choice questions and half short-answer questions. The class given examinations with half short-answer questions along with half multiple-choice questions had a significantly higher average score and grade category distribution (80-100 percent, 70-79 percent, <70 percent) than the class given examinations with all multiple-choice questions. When students in these two classes were divided into three academic ability groups based on the student's score in a prerequisite general pathology course, the class given examinations with half short-answer questions and half multiple-choice questions in the oral and maxillofacial pathology course had significantly higher scores and grade category distributions in all three ability groups. The average score and grade category distribution in the class given examinations with all short-answer questions in the oral and maxillofacial pathology course were not significantly different from the average score and grade category distribution in the class given examinations with half short-answer and half multiple-choice questions. Our interpretation of these results is that the utilization of examinations containing short-answer questions created a more challenging learning environment that motivated students to adopt more effective study regimens.