Programmatic assessment of competency-based workplace learning: when theory meets practice.

BACKGROUND: In competency-based medical education emphasis has shifted towards outcomes, capabilities, and learner-centeredness. Together with a focus on sustained evidence of professional competence this calls for new methods of teaching and assessment. Recently, medical educators advocated the use of a holistic, programmatic approach towards assessment. Besides maximum facilitation of learning it should improve the validity and reliability of measurements and documentation of competence development. We explored how, in a competency-based curriculum, current theories on programmatic assessment interacted with educational practice. METHODS: In a development study including evaluation, we investigated the implementation of a theory-based programme of assessment. Between April 2011 and May 2012 quantitative evaluation data were collected and used to guide group interviews that explored the experiences of students and clinical supervisors with the assessment programme. We coded the transcripts and emerging topics were organised into a list of lessons learned. RESULTS: The programme mainly focuses on the integration of learning and assessment by motivating and supporting students to seek and accumulate feedback. The assessment instruments were aligned to cover predefined competencies to enable aggregation of information in a structured and meaningful way. Assessments that were designed as formative learning experiences were increasingly perceived as summative by students. Peer feedback was experienced as a valuable method for formative feedback. Social interaction and external guidance seemed to be of crucial importance to scaffold self-directed learning. Aggregating data from individual assessments into a holistic portfolio judgement required expertise and extensive training and supervision of judges. CONCLUSIONS: A programme of assessment with low-stakes assessments providing simultaneously formative feedback and input for summative decisions proved not easy to implement. Careful preparation and guidance of the implementation process was crucial. Assessment for learning requires meaningful feedback with each assessment. Special attention should be paid to the quality of feedback at individual assessment moments. Comprehensive attention for faculty development and training for students is essential for the successful implementation of an assessment programme.

An admissions system to select veterinary medical students with an interest in food animals and veterinary public health.

Interest in the areas of food animals (FA) and veterinary public health (VPH) appears to be declining among prospective students of veterinary medicine. To address the expected shortage of veterinarians in these areas, the Utrecht Faculty of Veterinary Medicine has developed an admissions procedure to select undergraduates whose aptitude and interests are suited to these areas. A study using expert meetings, open interviews, and document analysis identified personal characteristics that distinguished veterinarians working in the areas of FA and VPH from their colleagues who specialized in companion animals (CA) and equine medicine (E). The outcomes were used to create a written selection tool. We validated this tool in a study among undergraduate veterinary students in their final (sixth) year before graduation. The applicability of the tool was verified in a study among first-year students who had opted to pursue either FA/VPH or CA/E. The tool revealed statistically significant differences with acceptable effect sizes between the two student groups. Because the written selection tool did not cover all of the differences between the veterinarians who specialized in FA/VPH and those who specialized in CA/E, we developed a prestructured panel interview and added it to the questionnaire. The evaluation of the written component showed that it was suitable for selecting those students who were most likely to succeed in the FA/VPH track.

Developing a classification tool based on Bloom's taxonomy to assess the cognitive level of short essay questions.

The cognitive level of short essay questions taken from assessments of two veterinary courses at the Faculty of Veterinary Medicine of Utrecht University (FVMU) was evaluated using a simplified classification tool based on the taxonomy of Bloom. Classifications were made by teaching staff members (subject matter experts, or SME) and by faculty members not involved in teaching the course (non-subject matter experts, or NSME). To compare the cognitive level assigned by raters in the SME group to that assigned by the NSME group, each test item was assigned a modal taxonomic level. The results indicate that the agreement level between a pair of raters within a group (SME or NSME) differed (34% to 77% and linear weighted Cohen's kappa coefficient 0.12 to 0.60). The agreement level on the modal taxonomic level between the SME and NSME groups for the two courses was 65% and 73%, with a linear weighted Cohen's kappa coefficient of 0.43 and 0.63 respectively. The requirement of expertise of a subject for classification is discussed. The introduction of the classification tool had a positive effect on teaching staff members' awareness of the importance of the cognitive level of assessments. Improvements to the classification tool to obtain higher agreement levels are proposed.