Improving assessment practice through cross-institutional collaboration: An exercise on the use of OSCEs.

BACKGROUND: This study was undertaken to improve assessment practice on OSCEs through collaboration across geographically dispersed medical schools in Australia. METHODS: A total of eleven OSCE stations were co-developed by four medical schools and used in summative 2011 and 2012 examinations for the assessment of clinical performance in the early clinical and exit OSCEs in each school's medical course. Partial Credit Rasch Model was used to evaluate the psychometric properties of the shared OSCE data. Evaluation of the quality assurance reports was used to determine the beneficial impact of the collaborative benchmarking exercise on learning and teaching outcomes. RESULTS: The data for each examination demonstrated sufficient fit to the Rasch model with infit mean square values ranging from 0.88 to 0.99. Person separation (1.25-1.63) indices indicated good reliability. Evaluation of perceived benefits showed that the benchmarking process was successful as it highlighted common curriculum areas requiring specific focus and provided comparable data on the quality of teaching at the participating medical schools. CONCLUSION: This research demonstrates the validity of the psychometric data and benefits of evaluating clinical competence across medical schools without the enforcement of a prescriptive national curriculum or assessment.

The Australian Medical Schools Assessment Collaboration: benchmarking the preclinical performance of medical students.

OBJECTIVES: To report the level of participation of medical schools in the Australian Medical Schools Assessment Collaboration (AMSAC); and to measure differences in student performance related to medical school characteristics and implementation methods. DESIGN: Retrospective analysis of data using the Rasch statistical model to correct for missing data and variability in item difficulty. Linear model analysis of variance was used to assess differences in student performance. SETTING AND PARTICIPANTS: 6401 preclinical students from 13 medical schools that participated in AMSAC from 2011 to 2013. MAIN OUTCOME MEASURES: Rasch estimates of preclinical basic and clinical science knowledge. RESULTS: Representation of Australian medical schools and students in AMSAC more than doubled between 2009 and 2013. In 2013 it included 12 of 19 medical schools and 68% of medical students. Graduate-entry students scored higher than students entering straight from school. Students at large schools scored higher than students at small schools. Although the significance level was high (P < 0.001), the main effect sizes were small (4.5% and 2.3%, respectively). The time allowed per multiple choice question was not significantly associated with student performance. The effect on performance of multiple assessments compared with the test items as part of a single end-of-year examination was negligible. The variables investigated explain only 12% of the total variation in student performance. CONCLUSIONS: An increasing number of medical schools are participating in AMSAC to monitor student performance in preclinical sciences against an external benchmark. Medical school characteristics account for only a small part of overall variation in student performance. Student performance was not affected by the different methods of administering test items.

Only the best: medical student selection in Australia.

Selection processes for medical schools need to be unbiased, valid, and psychometrically reliable, as well as evidence-based and transparent to all stakeholders. A range of academic and non-academic criteria are used for selection, including matriculation scores, aptitude tests and interviews. Research into selection is fraught with methodological difficulties; however, it shows positive benefits for structured selection processes. Pretest coaching and "faking good" are potential limitations of current selection procedures. Developments in medical school selection include the use of personality tests, centralised selection centres and programs to increase participation by socially disadvantaged students.