Evaluating the quality of an assessment: is the OSCE a valid and reliable assessment of clinical skill? [abstract]

In order to ensure horizontal integration, the curriculum at the Faculty of Medical Sciences, St. Augustine, includes a formalized skill lab programme in which student are taught basic clinical skills. The programme is measured at the end of year by a summative OSCE. Over the years, questions have been raised about the quality of the skills lab programme as well as the examination and its place in the curriculum. On the other hand, there is a movement towards greater use of OSCE in Phase 2. This study reports the use of a unique evaluative framework to assess the quality of the OSCE as an assessment tool. Key indicators are derived from the assessment literature and questions from a number of instrument along with new items are collated in an evaluation instrument. Key indicators include (1) construct validity, (ii) content coverage, (iii) impact upon students, (iv) fairness, (v) authenticity, (vi) reliability, (vii) usability and organization, and (viii) transparency. Evaluation data for the Year 3 OSCE (1998-2000) and OSCE for Medicine and Therapeutics are included. In line with the trend towards considering student perceptions and satisfaction with the teaching, learning and assessment provision, the focus is on the evaluation of student views, using both open and close-ended questions. However, performance scores are also considered when assessing the year 3 OSCE. Data suggest that the OSCE at the end of Phase 2 has a positive impact upon facilitating learning in Phase 2. Indeed, there is a moderate to strong relationship between performance in the OSCE and subsequent performance in Phase 2. In comparing data from recently completed OSCE in Phase 2 Medicine and Therapeutics, the possible negative impact of the high stakes nature of the examination is noted. The study recommends greater use and better structuring of this examination mode. (AU)

Developing and using performance indicators in evaluating quality and institutional effectiveness: the Faculty of Medical Sciences, UWI, St. Augustine [abstract]

Performance indicators are quantitative indices that describe organizational performance. They are frequently used for assessing institutional effectiveness and in performance management. In the current climate of accountability in higher education, performance indicators have an increasingly significant role in evaluation and benchmarking. In the United Kingdom (UK), the quality assurance agency has recently produced a range of performance indicators for all UK universities. Likewise, at the Office of Planning, the University of the West Indies, in its mid-term review makes use of performance indicators in assessing student progression in the key faculties at St. Augustine and Mona. However, traditional performance indicators may have limited interpretive value as evaluation indicators and benchmarks when applied to the assessment of institutional effectiveness of the Faculty of Medical Sciences. This lack of validity might relate to the unique structure of this Faculty, including the diversity in the student population, high admission scores, balanced gender ratios, a five-year multi-phased programme, a variety of assessment modes, and the facility for resits which aid academic progression. This study reports on an attempt to develop a compendium of performance indicators for use in quality management at the Faculty of Medical Sciences, St. Augustine. Data from all Phases and for all graduating classess in the medical and veterinary schools are used to evaluate the validity of a variety of access and progression indices, including value-added data and failure rates for the class of 1994 to 2000. The design of valid post-qualification indicators and a further study of non-completion are consisdered and rationalized. The study confirms limitations in the validity and reliability of common performance indicators and a lack of rigour and benchmarking status when applied to the complex programmes in the Faculty of Medical Sciences. It is argued that (1) greater attention must be payed to access indicators related to diversity, (2) progression indices should be varied and contextualized, and (3) performance indicators should include value-added indices which will allow progression rates to be compared with entry characteristics, including gender. (AU)