Evaluating the quality of an assessment: is the OSCE a valid and reliable assessment of clinical skill? [abstract]

ABSTRACT

In order to ensure horizontal integration, the curriculum at the Faculty of Medical Sciences, St. Augustine, includes a formalized skill lab programme in which student are taught basic clinical skills. The programme is measured at the end of year by a summative OSCE. Over the years, questions have been raised about the quality of the skills lab programme as well as the examination and its place in the curriculum. On the other hand, there is a movement towards greater use of OSCE in Phase 2. This study reports the use of a unique evaluative framework to assess the quality of the OSCE as an assessment tool. Key indicators are derived from the assessment literature and questions from a number of instrument along with new items are collated in an evaluation instrument. Key indicators include (1) construct validity, (ii) content coverage, (iii) impact upon students, (iv) fairness, (v) authenticity, (vi) reliability, (vii) usability and organization, and (viii) transparency. Evaluation data for the Year 3 OSCE (1998-2000) and OSCE for Medicine and Therapeutics are included. In line with the trend towards considering student perceptions and satisfaction with the teaching, learning and assessment provision, the focus is on the evaluation of student views, using both open and close-ended questions. However, performance scores are also considered when assessing the year 3 OSCE. Data suggest that the OSCE at the end of Phase 2 has a positive impact upon facilitating learning in Phase 2. Indeed, there is a moderate to strong relationship between performance in the OSCE and subsequent performance in Phase 2. In comparing data from recently completed OSCE in Phase 2 Medicine and Therapeutics, the possible negative impact of the high stakes nature of the examination is noted. The study recommends greater use and better structuring of this examination mode. (AU)