Shaping the right conditions in programmatic assessment: how quality of narrative information affects the quality of high-stakes decision-making.

BACKGROUND: Programmatic assessment is increasingly being implemented within competency-based health professions education. In this approach a multitude of low-stakes assessment activities are aggregated into a holistic high-stakes decision on the student's performance. High-stakes decisions need to be of high quality. Part of this quality is whether an examiner perceives saturation of information when making a holistic decision. The purpose of this study was to explore the influence of narrative information in perceiving saturation of information during the interpretative process of high-stakes decision-making. METHODS: In this mixed-method intervention study the quality of the recorded narrative information was manipulated within multiple portfolios (i.e., feedback and reflection) to investigate its influence on 1) the perception of saturation of information and 2) the examiner's interpretative approach in making a high-stakes decision. Data were collected through surveys, screen recordings of the portfolio assessments, and semi-structured interviews. Descriptive statistics and template analysis were applied to analyze the data. RESULTS: The examiners perceived less frequently saturation of information in the portfolios with low quality of narrative feedback. Additionally, they mentioned consistency of information as a factor that influenced their perception of saturation of information. Even though in general they had their idiosyncratic approach to assessing a portfolio, variations were present caused by certain triggers, such as noticeable deviations in the student's performance and quality of narrative feedback. CONCLUSION: The perception of saturation of information seemed to be influenced by the quality of the narrative feedback and, to a lesser extent, by the quality of reflection. These results emphasize the importance of high-quality narrative feedback in making robust decisions within portfolios that are expected to be more difficult to assess. Furthermore, within these "difficult" portfolios, examiners adapted their interpretative process reacting on the intervention and other triggers by means of an iterative and responsive approach.

Influence of an Exercise Program, Muscle Strength, Proprioception, and Arm Length on Veterinary Students' Bovine Pregnancy Diagnosis Accuracy.

Bovine pregnancy diagnosis (PD) by transrectal palpation (TRP) is an important skill for veterinary graduates. Factors influencing students' PD accuracy were investigated to optimize bovine PD by TRP training without increasing live animal exposure. The objective was to determine whether arm length and strength, proprioception, and exposure to a 6-week exercise training program were significantly associated with students' PD accuracy. Veterinary students (n = 128) who had previously received formal theoretical and practical training in bovine TRP and PD (live cows and TRP simulators) were assessed for PD accuracy on live cows. Prior to assessment, arm muscle strength measurement, an exercise program, and additional TRP sessions on Breed'n Betsy® simulators and live cows were offered to the students. Seventy-eight students volunteered to participate in the arm length measurement, muscle strength, and proprioception testing. Of these, 35 randomly allocated students completed a 6-week exercise program, after which muscle strength was reassessed. Each student performed PDs on six cows of which the pregnancy status, ranging from 6 weeks to 9 months pregnant or not pregnant, was predetermined by an experienced veterinarian. PD accuracy was measured as sensitivity and specificity, being defined as the proportion of pregnant or nonpregnant cows, respectively, correctly identified by the student. It is concluded that hand grip strength and participation in an exercise program are significant predictors of veterinary students' PD accuracy. Implementation of an exercise program aimed at improving grip strength in the veterinary curriculum is a novel approach to improve bovine TRP and PD training.

Electromyographic Analysis of Muscle Activation Patterns During Bovine Transrectal Palpation and the Development of the Bovine Pregnancy Diagnosis Improvement Exercise Program.

The unusual and tiring physical activity of bovine transrectal palpation (TRP) requires a novel approach to improve students' TRP and pregnancy diagnosis (PD) skills. It has been shown that students who participated in an exercise program and students who had a grip strength (GS) of more than 30 kilograms performed better in bovine PDs. Participation in the exercise program increased students' sensitivity (ability to identify pregnant cows) but did not increase total arm muscle strength. To identify which muscles are used during TRPs and to improve the exercise program, an electromyographic (EMG) analysis was used to identify muscle activation patterns and muscle activity levels during bovine TRPs. Eight subject matter experts (SMEs) each palpated two live cows and one Breed'n Betsy® rectal examination simulator while an EMG Triggered Stimulator recorded muscle activity. Muscle activation was higher for forearm muscles compared with all other examined muscle groups (p < .001); was higher during retraction of the uterus and palpation of left and right uterine horn, compared with palpation of cervix, uterine body, left ovary, and right ovary (p < .001); and showed an endurance pattern. Findings have been used to modify the previously developed exercise program in effort to improve students' TRP and PD skills. The Bovine PD Improvement Exercise Program is available to students through an online application (http://icarus.up.ac.za/vetmlp/) and aims to not only improve GS and TRP accuracy but also stamina and well-being while adding fun to busy study schedules.

Programmatic assessment: Can we provide evidence for saturation of information?

Purpose: According to the principles of programmatic assessment, a valid high-stakes assessment of the students' performance should amongst others, be based on a multiple data points, supposedly leading to saturation of information. Saturation of information is generated when a data point does not add important information to the assessor. In establishing saturation of information, institutions often set minimum requirements for the number of assessment data points to be included in the portfolio. Methods: In this study, we aimed to provide validity evidence for saturation of information by investigating the relationship between the number of data points exceeding the minimum requirements in a portfolio and the consensus between two independent assessors. Data were analyzed using a multiple logistic regression model. Results: The results showed no relation between the number of data points and the consensus. This suggests that either the consensus is predicted by other factors only, or, more likely, that assessors already reached saturation of information. This study took the first step in investigating saturation of information, further research is necessary to gain in-depth insights of this matter in relation to the complex process of decision-making.

Predictive Value of Three Different Selection Methods for Admission of Motivated and Well-Performing Veterinary Medical Students.

In search for valid and reliable selection methods that predict applicants' study motivation and academic performance during the 3-year bachelor's program at the Dutch Faculty of Veterinary Medicine (FVM), this study aimed to (1) examine the predictive value of the three FVM selection methods for study motivation and academic performance (i.e., direct admission and weighted lottery based on secondary school grade point average [GPA], and selection based on non-cognitive criteria), and (2) examine whether type and level of study motivation could be of value regarding selection of well-performing students. Data from two cohorts at the FVM (n = 186) were obtained, including mean summed scores on study motivation (using the Academic Motivation Scale [AMS] and additional items) and several academic outcome measures; among others, analyses of covariance (ANCOVA) were performed to examine differences between the three admission groups. Spearman's correlations and linear regression were applied to examine the relationship between study motivation and academic performance. Lottery-admitted students demonstrated a stronger extrinsic motivation than selected students (p < .05). Directly admitted students outperformed students from the other two admission groups on several academic outcome measures (p < .05). Only the level of motivation was related to academic performance (p < .05). According to the results, direct admission based on a high secondary school GPA in particular has predictive value for good academic performance during the 3-year bachelor's program of the veterinary course. The type of motivation seems to be of no value regarding selection of well-performing students, whereas level of motivation might be a useful criterion for this purpose.

The Development of Entrustable Professional Activities for Competency-Based Veterinary Education in Farm Animal Health.

Entrustable professional activities (EPAs) are professional tasks that can be entrusted to a student under a given level of supervision once he or she has demonstrated competence in these tasks. The EPA construct was conceived to increase transparency in objectives for clinical workplace learning and to help ensure patient safety and the quality of care. A first step in implementing EPAs in a veterinary curriculum is to identify the core EPAs of the profession. The aim of this study was to develop EPAs for farm animal health. An initial set of 36 EPAs for farm animal health was prepared by a team of six veterinarians and curriculum developers and used in a modified Delphi study. In this iterative process, the EPAs were evaluated until higher than 80% agreement was reached. Of 83 veterinarians who participated, 39 (47%) completed the Delphi procedure. After two rounds, the panel reached consensus. A small expert group further refined and reorganized the EPAs for educational purposes into seven core EPAs for farm animal health and 29 sub-EPAs. This study is an important step in optimizing competency-based training in veterinary medicine. Future steps are to implement EPAs in the curriculum and train supervisors to assess students' ability to perform EPAs with increasing levels of independence.

An instructional model for training competence in solving clinical problems.

We examined the design of a course that aims to ease the transition from pre-clinical learning into clinical work. This course is based on the premise that many of the difficulties with which students are confronted in this transition result from a lack of experience in applying knowledge in real practice situations. It is focused on the development of competence in solving clinical problems; uses an instructional model with alternating clinical practicals, demonstrations, and tutorials; and extends throughout the last pre-clinical year. We used a "proof-of-concept" approach to establish whether the core principles of the course design are feasible with regard to achieving the intended results. With the learning functions and processes as a frame of reference, retrospective analysis of the course's design features shows that this design matches the conditions from theories of the development of competence in solving clinical problems and instructional design. Three areas of uncertainty in the design are identified: the quality of the cases (information, openness), effective teaching (student and teacher roles), and adjustment to the development of competence (progress, coherence).

Am I ready for it? Students' perceptions of meaningful feedback on entrustable professional activities.

BACKGROUND: Receiving feedback while in the clinical workplace is probably the most frequently voiced desire of students. In clinical learning environments, providing and seeking performance-relevant information is often difficult for both supervisors and students. The use of entrustable professional activities (EPAs) can help to improve student assessment within competency-based education. This study aimed to illustrate what students' perceptions are of meaningful feedback viewed as conducive in preparing for performing EPA unsupervised. METHODS: In a qualitative multicentre study we explored students' perceptions on meaningful feedback related to EPAs in the clinical workplace. Focus groups were conducted in three different healthcare institutes. Based on concepts from the literature, the transcripts were coded, iteratively reduced and displayed. RESULTS: Participants' preferences regarding meaningful feedback on EPAs were quite similar, irrespective of their institution or type of clerkship. Participants explicitly mentioned that feedback on EPAs could come from a variety of sources. Feedback must come from a credible, trustworthy supervisor who knows the student well, be delivered in a safe environment and stress both strengths and points for improvement. The feedback should be provided immediately after the observed activity and include instructions for follow-up. Students would appreciate feedback that refers to their ability to act unsupervised. CONCLUSION: There is abundant literature on how feedback should be provided, and what factors influence how feedback is sought by students. This study showed that students who are training to perform an EPA unsupervised have clear ideas about how, when and from whom feedback should be delivered.

The perception of veterinary herd health management by Dutch dairy farmers and its current status in the Netherlands: a survey.

The importance of veterinary herd health management (VHHM) is increasing in both dairy farming and veterinary practice. Little is known, however, about how VHHM is perceived by farmers in terms of structure, content and satisfaction. In 2007 a questionnaire, containing questions about these three items was therefore sent to 800 Dutch dairy farmers. Farmers received two questionnaires, one for participants in VHHM and one for non-participants, allowing them to choose the appropriate one. Results were summarized and statistically analyzed. Farmers who were participating in VHHM had better farm performance. They were satisfied with the way VHHM was executed on their farm. However, there were some pressure points. Goal setting and evaluation was still not a regular part of VHHM, even though it is said to be effective in literature. Time spent on VHHM not visible to the farmer was often not charged or not clearly specified on the bill. The differences in opinions between participants and non-participants of VHHM indicated a lack of communication and/or product differentiation. Satisfaction with the way VHHM was executed on the farm had no significant influence on 305-day production. There was, however, some influence on calving interval and bulk milk somatic cell count (BMSCC).

Effect of penethamate hydriodide treatment on bacteriological cure, somatic cell count and milk production of cows and quarters with chronic subclinical Streptococcus uberis or Streptococcus dysgalactiae infection.

A randomized, controlled field trial was performed in The Netherlands to determine the therapeutic efficacy of parenteral penethamate hydriodide (Leocillin) against naturally occurring, chronic, streptococcal mastitis during lactation. Quarter milk samples were collected from subclinical cases of Streptococcus uberis or Streptococcus dysgalactiae mastitis to determine the effect of treatment on bacteriological cure and somatic cell count (SCC) at quarter level. A quarter was considered to be cured when the bacterial species, isolated prior to treatment, was not isolated from the quarter milk samples taken on days 10 and 20 post-treatment (bacteriological cure), or when a quarter milk SCC (QMSCC) was <250000 cells/ml on days 10 and 20 post-treatment (SCC cure). Longitudinal data analysis was performed to determine the effect of antibiotic therapy on SCC and milk yield at cow level. Bacteriological cure occurred in 59% of 29 treated quarters, while no cure was observed in any of the 21 untreated control quarters. Treatment resulted in a significant decrease in SCC at cow and quarter level in comparison with untreated controls. There was no significant effect of treatment on milk production. Antibacterial treatment of subclinical streptococcal infections during lactation also prevented clinical mastitis. Furthermore, the treatment may contribute to reduction of bulk milk SCC and to prevention of pathogen spread in dairy herds.