Shaping the right conditions in programmatic assessment: how quality of narrative information affects the quality of high-stakes decision-making.

BACKGROUND: Programmatic assessment is increasingly being implemented within competency-based health professions education. In this approach a multitude of low-stakes assessment activities are aggregated into a holistic high-stakes decision on the student's performance. High-stakes decisions need to be of high quality. Part of this quality is whether an examiner perceives saturation of information when making a holistic decision. The purpose of this study was to explore the influence of narrative information in perceiving saturation of information during the interpretative process of high-stakes decision-making. METHODS: In this mixed-method intervention study the quality of the recorded narrative information was manipulated within multiple portfolios (i.e., feedback and reflection) to investigate its influence on 1) the perception of saturation of information and 2) the examiner's interpretative approach in making a high-stakes decision. Data were collected through surveys, screen recordings of the portfolio assessments, and semi-structured interviews. Descriptive statistics and template analysis were applied to analyze the data. RESULTS: The examiners perceived less frequently saturation of information in the portfolios with low quality of narrative feedback. Additionally, they mentioned consistency of information as a factor that influenced their perception of saturation of information. Even though in general they had their idiosyncratic approach to assessing a portfolio, variations were present caused by certain triggers, such as noticeable deviations in the student's performance and quality of narrative feedback. CONCLUSION: The perception of saturation of information seemed to be influenced by the quality of the narrative feedback and, to a lesser extent, by the quality of reflection. These results emphasize the importance of high-quality narrative feedback in making robust decisions within portfolios that are expected to be more difficult to assess. Furthermore, within these "difficult" portfolios, examiners adapted their interpretative process reacting on the intervention and other triggers by means of an iterative and responsive approach.

Implementation of a Blended Learning Module to Teach Handling, Restraint, and Physical Examination of Cats in Undergraduate Veterinary Training.

Cats can be easily stressed in a clinical (training) setting and may show unpredictable reactions and patterns of defensive aggression. This can be a complicating factor in undergraduate veterinary training. Inexperienced veterinary students can evoke defensive feline behavior that negatively affects learning outcomes and animal welfare. As a result, restraint techniques and physical examination of cats was hardly practiced in pre-clinical training at Utrecht University. To overcome this, a new blended learning module was developed using a lecture on feline behavior; e-learning modules about feline behavior, handling, restraint, and physical examination skills; and redesigned practical sessions in which live animals and manikins were used. The aim of this study was to investigate how students' perceptions of competence and confidence changed regarding feline behavior, handling, restraint, and physical examination skills after the new module was implemented. Questionnaires were used for quantitative analysis, and focus groups were used for qualitative analysis. The results show that compared with students who followed the standard module, students who participated in the blended learning module scored higher in feeling confident with handling animals, feeling competent to perform physical examination on cats, and ability to assess whether a cat is stressed. Students with less experience with cats were more likely to show improvement in assessing a cat's stress level than students who had much experience with cats. The results demonstrate that the blended learning module improves students' learning outcomes regarding feline skills training and adds to reduction, refinement, and replacement of the use of live cats.

Electromyographic Analysis of Muscle Activation Patterns During Bovine Transrectal Palpation and the Development of the Bovine Pregnancy Diagnosis Improvement Exercise Program.

The unusual and tiring physical activity of bovine transrectal palpation (TRP) requires a novel approach to improve students' TRP and pregnancy diagnosis (PD) skills. It has been shown that students who participated in an exercise program and students who had a grip strength (GS) of more than 30 kilograms performed better in bovine PDs. Participation in the exercise program increased students' sensitivity (ability to identify pregnant cows) but did not increase total arm muscle strength. To identify which muscles are used during TRPs and to improve the exercise program, an electromyographic (EMG) analysis was used to identify muscle activation patterns and muscle activity levels during bovine TRPs. Eight subject matter experts (SMEs) each palpated two live cows and one Breed'n Betsy® rectal examination simulator while an EMG Triggered Stimulator recorded muscle activity. Muscle activation was higher for forearm muscles compared with all other examined muscle groups (p < .001); was higher during retraction of the uterus and palpation of left and right uterine horn, compared with palpation of cervix, uterine body, left ovary, and right ovary (p < .001); and showed an endurance pattern. Findings have been used to modify the previously developed exercise program in effort to improve students' TRP and PD skills. The Bovine PD Improvement Exercise Program is available to students through an online application (http://icarus.up.ac.za/vetmlp/) and aims to not only improve GS and TRP accuracy but also stamina and well-being while adding fun to busy study schedules.

Influence of an Exercise Program, Muscle Strength, Proprioception, and Arm Length on Veterinary Students' Bovine Pregnancy Diagnosis Accuracy.

Bovine pregnancy diagnosis (PD) by transrectal palpation (TRP) is an important skill for veterinary graduates. Factors influencing students' PD accuracy were investigated to optimize bovine PD by TRP training without increasing live animal exposure. The objective was to determine whether arm length and strength, proprioception, and exposure to a 6-week exercise training program were significantly associated with students' PD accuracy. Veterinary students (n = 128) who had previously received formal theoretical and practical training in bovine TRP and PD (live cows and TRP simulators) were assessed for PD accuracy on live cows. Prior to assessment, arm muscle strength measurement, an exercise program, and additional TRP sessions on Breed'n Betsy® simulators and live cows were offered to the students. Seventy-eight students volunteered to participate in the arm length measurement, muscle strength, and proprioception testing. Of these, 35 randomly allocated students completed a 6-week exercise program, after which muscle strength was reassessed. Each student performed PDs on six cows of which the pregnancy status, ranging from 6 weeks to 9 months pregnant or not pregnant, was predetermined by an experienced veterinarian. PD accuracy was measured as sensitivity and specificity, being defined as the proportion of pregnant or nonpregnant cows, respectively, correctly identified by the student. It is concluded that hand grip strength and participation in an exercise program are significant predictors of veterinary students' PD accuracy. Implementation of an exercise program aimed at improving grip strength in the veterinary curriculum is a novel approach to improve bovine TRP and PD training.

Bridging the Gap between Undergraduate Veterinary Training and Veterinary Practice with Entrustable Professional Activities.

The transition from being a veterinary student to becoming a member of the veterinary profession is known to be challenging. Despite being licensed directly after graduation, many veterinarians do not feel fully equipped to practice unsupervised when they graduate. The increasing rate of attrition from veterinary practice, and a relatively high percentage of burnout during the first years in practice, has been suggested to be related to a lack of early career support. Over the past decade, medical education has adopted the concept of entrustable professional activities (EPAs). Recently, EPAs have been proposed to restructure veterinary education to help support the transition from veterinary student to practicing veterinarian. Implementing an EPA-based approach could help to bridge the gap between school and clinical practice, potentially preventing veterinary graduates from dropping out early on from what could have been a promising and exciting professional career.

Collaborative Development of a Shared Framework for Competency-Based Veterinary Education.

Competency-based medical education is an educational innovation implemented in health professions worldwide as a means to ensure graduates meet patient and societal needs. The focus on student-centered education and programmatic outcomes offers a series of benefits to learners, institutions and society. However, efforts to establish a shared, comprehensive competency-based framework in veterinary education have lagged. This article reports on the development and outcome of a competency-based veterinary education (CBVE) framework created through multi-institutional collaboration with international input from veterinary educators and veterinary educational leaders. The CBVE Framework is designed to reflect the competencies expected of new graduates from member institutions of the Association of American Veterinary Medical Colleges (AAVMC). The CBVE Framework consists of nine domains of competence and 32 competencies, each supplemented with illustrative sub-competencies to guide veterinary schools in implementing competency-based education in their local context. The nine domains of competence are: clinical reasoning and decision-making; individual animal care and management; animal population care and management; public health; communication; collaboration; professionalism and professional identity; financial and practice management; and scholarship. Developed through diverse input to facilitate broad adoption, the CBVE Framework provides the foundation for competency-based curricula and outcomes assessment in veterinary education internationally. We believe that other groups seeking to design a collective product for broad adoption might find useful the methods used to develop the CBVE Framework, including establishing expertise diversity within a small-to-medium size working group, soliciting progressive input and feedback from stakeholders, and engaging in consensus building and critical reflection throughout the development process.

Collaborative Development of Core Entrustable Professional Activities for Veterinary Education.

Entrustable professional activities (EPAs) have been proposed as a practical framework for the implementation of competency-based education. As veterinary education moves toward a competency-based approach, core EPAs provide a context for assessment of workplace activities. This article reports on the development of eight core clinical EPAs for veterinary education created through multi-institutional collaboration, with international input from veterinary educators and veterinary educational leaders. These core EPAs are intended as minimal expectations for clinical activities that should be assessed for every graduate of Association of American Veterinary Medical Colleges member institutions. Adoption of the core EPAs and the associated Competency-Based Veterinary Education (CBVE) framework by veterinary schools is intended to promote Day One graduate competence and thereby enhance patient care and client service.

Identifying entrustable professional activities for surgical skills training in companion animal health.

BACKGROUND: Veterinary medical education is increasingly moving towards outcome-based training based on competency frameworks. A source of concern is the translation of competencies into the practice of clinical teaching, for example, surgical skills training. It is suggested that the use of entrustable professional activities (EPAs) might bridge this gap. The purpose of this study, therefore, was to identify EPAs related to surgical skills for companion animal health to enhance competency-based education. METHODS: Draft versions of EPAs related to surgical skills were established by an iterative consensus-based approach through 45-min interview sessions. These draft versions were used to explore the opinion of companion animal veterinarians, both veterinarians (specialists, residents and interns) involved in undergraduate teaching and veterinarians working in private practice involved in extramural clinical teaching, on the relevance and level of entrustment of the EPAs through a modified Delphi procedure. Mean (relevance) and median (level of entrustment) scores were calculated and textual comments were analysed to create a final framework of EPAs related to surgical skills. RESULTS AND CONCLUSION: The Delphi panel reached consensus in three rounds. Thirty-four per cent of those invited to participate in the study completed the final survey. Finally, a list of 13 EPAs related to companion animal surgical skills a student should be entrusted to perform at time of graduation was established.

Assessment Tools for Feedback and Entrustment Decisions in the Clinical Workplace: A Systematic Review.

BACKGROUND: Entrustable Professional Activities (EPAs) combine feedback and evaluation with a permission to act under a specified level of supervision and the possibility to schedule learners for clinical service. This literature review aims to identify workplace-based assessment tools that indicate progression toward unsupervised practice, suitable for entrustment decisions and feedback to learners. METHODS: A systematic search was performed in the PubMed, Embase, ERIC, and PsycINFO databases. Based on title/abstract and full text, articles were selected using predetermined inclusion and exclusion criteria. Information on workplace-based assessment tools was extracted using data coding sheets. The methodological quality of studies was assessed using the medical education research study quality instrument (MERSQI). RESULTS: The search yielded 6,371 articles (180 were evaluated in full text). In total, 80 articles were included, identifying 67 assessment tools. Only a few studies explicitly mentioned assessment tools used as a resource for entrustment decisions. Validity evidence was frequently reported, and the MERSQI score was 10.0 on average. CONCLUSIONS: Many workplace-based assessment tools were identified that potentially support learners with feedback on their development and support supervisors with providing feedback. As expected, only few articles referred to entrustment decisions. Nevertheless, the existing tools or the principals could be used for entrustment decisions, supervision level, or autonomy.

Development of core entrustable professional activities linked to a competency-based veterinary education framework.

Purpose: Despite the adoption of competency-based education in some veterinary schools over the past 15 years, only recently has a concerted effort been directed toward this in veterinary education internationally.Methods: In 2015, educational leaders from the Association of American Veterinary Medical Colleges (AAVMC) member schools came together with a strong call to action to create shared tools for clinical competency assessment.Results: This resulted in the formation of the AAVMC Competency-Based Veterinary Education (CBVE) Working Group, which then embarked on the creation of a shared competency framework and the development of eight core entrustable professional activities (EPAs) linked to this framework.Conclusions: This paper will report on the development of these EPAs and their integration with the concurrently-developed CBVE Framework.

Inter-Rater Reliability of Grading Undergraduate Portfolios in Veterinary Medical Education.

The reliability of high-stakes assessment of portfolios containing an aggregation of quantitative and qualitative data based on programmatic assessment is under debate, especially when multiple assessors are involved. In this study carried out at the Faculty of Veterinary Medicine, Utrecht University, the Netherlands, two independent assessors graded the portfolios of students in their second year of the 3-year clinical phase. The similarity of grades (i.e., equal grades) and the level of the grades were studied to estimate inter-rater reliability, taking into account the potential effects of the assessor's background (i.e., originating from a clinical or non-clinical department) and student's cohort group, gender, and chosen master track (Companion Animal Health, Equine Health, or Farm Animal/Public Health). Whereas the similarity between the two grades increased from 58% in the first year the grading system was introduced to around 80% afterwards, the grade level was lower over the next 3 years. The assessor's background had a minor effect on the proportion of similar grades, as well as on grading level. The assessor intraclass correlation was low (i.e., all assessors scored with a similar grading pattern [same range of grades]). The grades awarded to female students were higher but more often dissimilar. We conclude that the grading system was well implemented and has a high inter-rater reliability.

Programmatic assessment: Can we provide evidence for saturation of information?

Purpose: According to the principles of programmatic assessment, a valid high-stakes assessment of the students' performance should amongst others, be based on a multiple data points, supposedly leading to saturation of information. Saturation of information is generated when a data point does not add important information to the assessor. In establishing saturation of information, institutions often set minimum requirements for the number of assessment data points to be included in the portfolio. Methods: In this study, we aimed to provide validity evidence for saturation of information by investigating the relationship between the number of data points exceeding the minimum requirements in a portfolio and the consensus between two independent assessors. Data were analyzed using a multiple logistic regression model. Results: The results showed no relation between the number of data points and the consensus. This suggests that either the consensus is predicted by other factors only, or, more likely, that assessors already reached saturation of information. This study took the first step in investigating saturation of information, further research is necessary to gain in-depth insights of this matter in relation to the complex process of decision-making.

The Development of Entrustable Professional Activities for Competency-Based Veterinary Education in Farm Animal Health.

Entrustable professional activities (EPAs) are professional tasks that can be entrusted to a student under a given level of supervision once he or she has demonstrated competence in these tasks. The EPA construct was conceived to increase transparency in objectives for clinical workplace learning and to help ensure patient safety and the quality of care. A first step in implementing EPAs in a veterinary curriculum is to identify the core EPAs of the profession. The aim of this study was to develop EPAs for farm animal health. An initial set of 36 EPAs for farm animal health was prepared by a team of six veterinarians and curriculum developers and used in a modified Delphi study. In this iterative process, the EPAs were evaluated until higher than 80% agreement was reached. Of 83 veterinarians who participated, 39 (47%) completed the Delphi procedure. After two rounds, the panel reached consensus. A small expert group further refined and reorganized the EPAs for educational purposes into seven core EPAs for farm animal health and 29 sub-EPAs. This study is an important step in optimizing competency-based training in veterinary medicine. Future steps are to implement EPAs in the curriculum and train supervisors to assess students' ability to perform EPAs with increasing levels of independence.

Validity evidence for programmatic assessment in competency-based education.

INTRODUCTION: Competency-based education (CBE) is now pervasive in health professions education. A foundational principle of CBE is to assess and identify the progression of competency development in students over time. It has been argued that a programmatic approach to assessment in CBE maximizes student learning. The aim of this study is to investigate if programmatic assessment, i. e., a system of assessment, can be used within a CBE framework to track progression of student learning within and across competencies over time. METHODS: Three workplace-based assessment methods were used to measure the same seven competency domains. We performed a retrospective quantitative analysis of 327,974 assessment data points from 16,575 completed assessment forms from 962 students over 124 weeks using both descriptive (visualization) and modelling (inferential) analyses. This included multilevel random coefficient modelling and generalizability theory. RESULTS: Random coefficient modelling indicated that variance due to differences in inter-student performance was highest (40%). The reliability coefficients of scores from assessment methods ranged from 0.86 to 0.90. Method and competency variance components were in the small-to-moderate range. DISCUSSION: The current validation evidence provides cause for optimism regarding the explicit development and implementation of a program of assessment within CBE. The majority of the variance in scores appears to be student-related and reliable, supporting the psychometric properties as well as both formative and summative score applications.

When to trust our learners? Clinical teachers' perceptions of decision variables in the entrustment process.

INTRODUCTION: Clinical training programs increasingly use entrustable professional activities (EPAs) as focus of assessment. However, questions remain about which information should ground decisions to trust learners. This qualitative study aimed to identify decision variables in the workplace that clinical teachers find relevant in the elaboration of the entrustment decision processes. The findings can substantiate entrustment decision-making in the clinical workplace. METHODS: Focus groups were conducted with medical and veterinary clinical teachers, using the structured consensus method of the Nominal Group Technique to generate decision variables. A ranking was made based on a relevance score assigned by the clinical teachers to the different decision variables. Field notes, audio recordings and flip chart lists were analyzed and subsequently translated and, as a form of axial coding, merged into one list, combining the decision variables that were similar in their meaning. RESULTS: A list of 11 and 17 decision variables were acknowledged as relevant by the medical and veterinary teacher groups, respectively. The focus groups yielded 21 unique decision variables that were considered relevant to inform readiness to perform a clinical task on a designated level of supervision. The decision variables consisted of skills, generic qualities, characteristics, previous performance or other information. We were able to group the decision variables into five categories: ability, humility, integrity, reliability and adequate exposure. DISCUSSION: To entrust a learner to perform a task at a specific level of supervision, a supervisor needs information to support such a judgement. This trust cannot be credited on a single case at a single moment of assessment, but requires different variables and multiple sources of information. This study provides an overview of decision variables giving evidence to justify the multifactorial process of making an entrustment decision.

Am I ready for it? Students' perceptions of meaningful feedback on entrustable professional activities.

BACKGROUND: Receiving feedback while in the clinical workplace is probably the most frequently voiced desire of students. In clinical learning environments, providing and seeking performance-relevant information is often difficult for both supervisors and students. The use of entrustable professional activities (EPAs) can help to improve student assessment within competency-based education. This study aimed to illustrate what students' perceptions are of meaningful feedback viewed as conducive in preparing for performing EPA unsupervised. METHODS: In a qualitative multicentre study we explored students' perceptions on meaningful feedback related to EPAs in the clinical workplace. Focus groups were conducted in three different healthcare institutes. Based on concepts from the literature, the transcripts were coded, iteratively reduced and displayed. RESULTS: Participants' preferences regarding meaningful feedback on EPAs were quite similar, irrespective of their institution or type of clerkship. Participants explicitly mentioned that feedback on EPAs could come from a variety of sources. Feedback must come from a credible, trustworthy supervisor who knows the student well, be delivered in a safe environment and stress both strengths and points for improvement. The feedback should be provided immediately after the observed activity and include instructions for follow-up. Students would appreciate feedback that refers to their ability to act unsupervised. CONCLUSION: There is abundant literature on how feedback should be provided, and what factors influence how feedback is sought by students. This study showed that students who are training to perform an EPA unsupervised have clear ideas about how, when and from whom feedback should be delivered.

Students' motivation toward feedback-seeking in the clinical workplace.

In medical education, students are increasingly regarded as active seekers of feedback rather than passive recipients. Previous research showed that in the intentions of students to seek feedback, a learning and performance goal can be distinguished. In this study, we investigated the intentions (defined as level and orientation of motivation) of different performing students (low, average, and high performing students) to seek feedback in the clinical workplace using Self-Determination Theory. We conducted a quantitative study with students in their clinical clerkships and grouped them based on their performance. The level of motivation was measured by the number of Mini-CEXs each student collected. The orientation of motivation was measured by conducting the Academic Self-Regulation Questionnaire. We found that high performing students were more motivated and demonstrated higher self-determination compared to low performing students.

Feedback-giving behaviour in performance evaluations during clinical clerkships.

CONTEXT: Narrative feedback documented in performance evaluations by the teacher, i.e. the clinical supervisor, is generally accepted to be essential for workplace learning. Many studies have examined factors of influence on the usage of mini-clinical evaluation exercise (mini-CEX) instruments and provision of feedback, but little is known about how these factors influence teachers' feedback-giving behaviour. In this study, we investigated teachers' use of mini-CEX in performance evaluations to provide narrative feedback in undergraduate clinical training. METHODS: We designed an exploratory qualitative study using an interpretive approach. Focusing on the usage of mini-CEX instruments in clinical training, we conducted semi-structured interviews to explore teachers' perceptions. Between February and June 2013, we conducted interviews with 14 clinicians participated as teachers during undergraduate clinical clerkships. Informed by concepts from the literature, we coded interview transcripts and iteratively reduced and displayed data using template analysis. RESULTS: We identified three main themes of interrelated factors that influenced teachers' practice with regard to mini-CEX instruments: teacher-related factors; teacher-student interaction-related factors, and teacher-context interaction-related factors. Four issues (direct observation, relationship between teacher and student, verbal versus written feedback, formative versus summative purposes) that are pertinent to workplace-based performance evaluations were presented to clarify how different factors interact with each other and influence teachers' feedback-giving behaviour. Embedding performance observation in clinical practice and establishing trustworthy teacher-student relationships in more longitudinal clinical clerkships were considered important in creating a learning environment that supports and facilitates the feedback exchange. CONCLUSION: Teachers' feedback-giving behaviour within the clinical context results from the interaction between personal, interpersonal and contextual factors. Increasing insight into how teachers use mini-CEX instruments in daily practice may offer strategies for creating a professional learning culture in which feedback giving and seeking would be enhanced.