Key considerations in planning and designing programmatic assessment in competency-based medical education.

Programmatic assessment as a concept is still novel for many in clinical education, and there may be a disconnect between the academics who publish about programmatic assessment and the front-line clinical educators who must put theory into practice. In this paper, we clearly define programmatic assessment and present high-level guidelines about its implementation in competency-based medical education (CBME) programs. The guidelines are informed by literature and by lessons learned from established programmatic assessment approaches. We articulate five steps to consider when implementing programmatic assessment in CBME contexts: articulate the purpose of the program of assessment, determine what must be assessed, choose tools fit for purpose, consider the stakes of assessments, and define processes for interpreting assessment data. In the process, we seek to offer a helpful guide or template for front-line clinical educators. We dispel some myths about programmatic assessment to help training programs as they look to design-or redesign-programs of assessment. In particular, we highlight the notion that programmatic assessment is not 'one size fits all'; rather, it is a system of assessment that results when shared common principles are considered and applied by individual programs as they plan and design their own bespoke model of programmatic assessment for CBME in their unique context.

Clarifying essential terminology in entrustment.

With the rapid uptake of entrustable professional activties and entrustment decision-making as an approach in undergraduate and graduate education in medicine and other health professions, there is a risk of confusion in the use of new terminologies. The authors seek to clarify the use of many words related to the concept of entrustment, based on existing literature, with the aim to establish logical consistency in their use. The list of proposed definitions includes independence, autonomy, supervision, unsupervised practice, oversight, general and task-specific trustworthiness, trust, entrust(ment), entrustable professional activity, entrustment decision, entrustability, entrustment-supervision scale, retrospective and prospective entrustment-supervision scales, and entrustment-based discussion. The authors conclude that a shared understanding of the language around entrustment is critical to strengthen bridges among stages of training and practice, such as undergraduate medical education, graduate medical education, and continuing professional development. Shared language and understanding provide the foundation for consistency in interpretation and implementation across the educational continuum.

Methods Used to Assess Student Performance and Course Outcomes at the National Center of Excellence in Dairy Production Medicine Education for Veterinarians.

Between 2012 and 2014, three cohorts of senior veterinary students participated in an 8-week dairy production medicine course created by the National Center of Excellence in Dairy Production Medicine Education for Veterinarians. One goal of this course is to better prepare veterinary students to serve the increasingly complex needs of the dairy industry. In this article, we describe the assessment methods and student performance outcomes of those first three cohorts. A combination of assessment methods was used, including pre- and post-testing; instructor observations and scores on individual and group projects, including a final integrative project; and peer evaluation. Student feedback, collected via anonymous survey, provided insight into students' perceptions about the course and their learning. Performance and feedback suggest that the course was successful in preparing students for careers using skills in dairy production medicine. Pre- and post-testing was conducted for most topic modules in the course. The mean (median) pre- and post-test scores were 47% (50% ) and 83% (88%), respectively. The mean improvement in score was significant (p < .002) for all modules and cohorts. Students indicated a moderate or high degree of confidence in performing dairy production medicine skills after each module. Of students in cohorts 1, 2, and 3, respectively, 55%, 75%, and 82% felt they could provide dairy production medicine services (e.g., records analysis, problem investigation, protocol and standard operating procedure design) either alone or with some mentoring, immediately after graduation. In addition, assessment results and student feedback enabled timely course modifications during these first three cohorts.

Monitoring the Curriculum through the Student Perspective.

Student input was deliberately included as part of the curriculum implementation and assessment plan at the University of Minnesota College of Veterinary Medicine. The new curriculum included design features to encourage deeper learning such as a spiral curriculum with cross-course integration, increased open time, and more active learning. Student well-being was seen as a simultaneous need. To gather overall perceptions of workload and well-being, student volunteers from each cohort were surveyed weekly starting in 2013. Survey questions asked about out-of-class work time, level of integration, extracurricular activities, student well-being habits, paid employment, and other factors. Survey questions were combined with course data to get a full picture of week quality, total course work time, extracurricular activities, and the effects of integration. Many of our hypotheses about curricular and extracurricular impacts on week quality were disproven. Week quality was most positively affected by student factors of sleep and exercise, whereas the curricular factors of out-of-class work time, total course work time, and examination hours had the strongest negative effects. A surprising finding was that open time, in-class hours, and paid employment hours had a minimal effect on week quality. Students identified excessively heavy semesters and uneven semester workloads that resulted in early revisions to the new curriculum. Student feedback provided a view of the curriculum that was not otherwise available and resulted in early and significant impacts on the new curriculum, and they provided insight into whether planned changes had occurred and how effective various factors were in reaching the curricular goals.

Effects of an 8-Week Dairy Production Medicine Course on Veterinary Student Self-Confidence and Perceptions of Knowledge and Skills Used by Dairy Veterinarians.

The 8-week dairy production medicine course at the National Center of Excellence in Dairy Production Medicine Education for Veterinarians is designed to equip senior veterinary students with the knowledge and skills needed to serve the dairy industry. Course developers identified 59 topics of importance for dairy production medicine veterinarians. Students (N = 50) were surveyed before and after the course to determine their perceptions of (a) the importance of the 59 topics for their intended positions and (b) their knowledge and skill in those areas. We expected the course to affirm or strengthen perceptions of importance and increase confidence. Students rated 57 of the topics as moderately or very important before the course. Ratings were unchanged (56 topics) or increased (3 topics) after the course. Before the course, students believed they had a lot of knowledge and skill in just one area: animal behavior and handling. At the end of the course, students believed they had a lot of knowledge and skill in 21 areas; confidence ratings were higher for 47 of the 59 topics. Alumni were surveyed 1-2 years after graduation to determine the importance of the 59 topics to their positions, their impressions about how well the course prepared them in those areas, and whether they referred back to course materials. Feedback was used to adjust the course. The topics alumni rated as most important were similar to those students predicted would be most important. Seventy-five percent of alumni used the course website as a resource in practice.

Prior Experience, Career Intentions, and Post-Graduate Positions of Veterinary Students Who Participated in an 8-week Dairy Production Medicine Course.

Three cohorts of senior veterinary students (n = 50) from seven United States (US) colleges of veterinary medicine took an 8-week dairy production medicine course at the Dairy Center of Excellence in Production Medicine Education for Veterinarians (DCE) between 2012 and 2014. Participants completed a questionnaire before and after the course and 1 to 2 years after graduation. Objectives were to determine the prior academic training and livestock experience of course participants, to compare students' career aspirations before and after taking the course, and to identify factors associated with post-graduate position. Response rates were 58%-96%. Most students had taken undergraduate animal science courses (83%), worked (76%) and/or lived (52%) on a livestock operation, participated in youth livestock activities (63%), worked at a mixed practice (71%), taken production medicine-related elective courses (65%), taken other food animal rotations (91%), and/or done dairy externships (65%) before taking the DCE course. Students who were very likely to pursue a dairy-focused position before taking the course (36%) remained committed after the course, whereas students who were not likely initially (39%) were not further motivated by the course. Students who had worked with a dairy veterinarian were more likely to pursue a dairy-focused position than those who had not. Most course alumni accepted positions in mixed practice, with a ≥ 50% (54%) or < 50% (23%) dairy component, and post-graduate positions were consistent with students' predictions. Students who held an undergraduate degree or had worked for a dairy veterinarian were more likely to accept a dairy-focused practice position than those who did not.

Development and Implementation of a National Center of Excellence in Dairy Production Medicine Education for Veterinary Students: Description of the Effort and Lessons Learned.

The need for consortial programs to provide advanced education in food animal veterinary production medicine has been recognized and lauded for nearly three decades. This article describes one effort to create a dairy production medicine curriculum funded by a United States Department of Agriculture (USDA) Higher Education Challenge Grant. This National Center of Excellence in Dairy Production Medicine Education for Veterinarians is housed at the Dairy Education Center of the University of Minnesota and the project was a collaboration of the University of Minnesota, the University of Illinois, the University of Georgia, and Kansas State University. The article reviews the need for innovative ways to educate students who will optimally serve the dairy industry, provides a broad overview of the process of developing and delivering the eight-week dairy production medicine curriculum, and describes the challenges faced and lessons learned as a result of offering such a program.

Collaborative Development of Core Entrustable Professional Activities for Veterinary Education.

Entrustable professional activities (EPAs) have been proposed as a practical framework for the implementation of competency-based education. As veterinary education moves toward a competency-based approach, core EPAs provide a context for assessment of workplace activities. This article reports on the development of eight core clinical EPAs for veterinary education created through multi-institutional collaboration, with international input from veterinary educators and veterinary educational leaders. These core EPAs are intended as minimal expectations for clinical activities that should be assessed for every graduate of Association of American Veterinary Medical Colleges member institutions. Adoption of the core EPAs and the associated Competency-Based Veterinary Education (CBVE) framework by veterinary schools is intended to promote Day One graduate competence and thereby enhance patient care and client service.

Collaborative Development of a Shared Framework for Competency-Based Veterinary Education.

Competency-based medical education is an educational innovation implemented in health professions worldwide as a means to ensure graduates meet patient and societal needs. The focus on student-centered education and programmatic outcomes offers a series of benefits to learners, institutions and society. However, efforts to establish a shared, comprehensive competency-based framework in veterinary education have lagged. This article reports on the development and outcome of a competency-based veterinary education (CBVE) framework created through multi-institutional collaboration with international input from veterinary educators and veterinary educational leaders. The CBVE Framework is designed to reflect the competencies expected of new graduates from member institutions of the Association of American Veterinary Medical Colleges (AAVMC). The CBVE Framework consists of nine domains of competence and 32 competencies, each supplemented with illustrative sub-competencies to guide veterinary schools in implementing competency-based education in their local context. The nine domains of competence are: clinical reasoning and decision-making; individual animal care and management; animal population care and management; public health; communication; collaboration; professionalism and professional identity; financial and practice management; and scholarship. Developed through diverse input to facilitate broad adoption, the CBVE Framework provides the foundation for competency-based curricula and outcomes assessment in veterinary education internationally. We believe that other groups seeking to design a collective product for broad adoption might find useful the methods used to develop the CBVE Framework, including establishing expertise diversity within a small-to-medium size working group, soliciting progressive input and feedback from stakeholders, and engaging in consensus building and critical reflection throughout the development process.

Development of core entrustable professional activities linked to a competency-based veterinary education framework.

Purpose: Despite the adoption of competency-based education in some veterinary schools over the past 15 years, only recently has a concerted effort been directed toward this in veterinary education internationally.Methods: In 2015, educational leaders from the Association of American Veterinary Medical Colleges (AAVMC) member schools came together with a strong call to action to create shared tools for clinical competency assessment.Results: This resulted in the formation of the AAVMC Competency-Based Veterinary Education (CBVE) Working Group, which then embarked on the creation of a shared competency framework and the development of eight core entrustable professional activities (EPAs) linked to this framework.Conclusions: This paper will report on the development of these EPAs and their integration with the concurrently-developed CBVE Framework.

Curriculum Review and Revision at the University of Minnesota College of Veterinary Medicine.

Curriculum review is an essential part of ongoing curriculum development, and is a mandate of the American Veterinary Medical Association Council on Education (AVMA COE), the accrediting body of all North American schools and colleges of veterinary medicine. This article describes the steps in curriculum review undertaken by the University of Minnesota College of Veterinary Medicine (UMN CVM) in response to this mandate from the COE and to a recommendation from a recent collegiate review that was part of a larger university-level strategic planning effort. The challenges of reviewing and revising the curriculum within a short time frame were met by appointing a dedicated curriculum review board and by engaging students and faculty groups, both as focus groups and as specific faculty work sections within disciplines. Faculty voting on the process was very valuable as it permitted the curriculum review board and faculty groups to move ahead knowing there was a process in place for reassessment if most faculty did not agree with recommendations. Consistent support from the dean of the college and other administrators was vital in helping maintain momentum for curriculum review.

Closing the Loop: Using Evidence to Inform Refinements to an Admissions Process.

Substantive changes in our evaluation of applicants have been implemented in response to the findings of the various phases of outcomes assessment. This article will describe the changes that have been implemented as a result of analyzing the association of pre-veterinary academic, subjective, and behavioral-event interview scores with veterinary students' performance in the DVM program, including clinical competencies, and their performance on the North American Veterinary Licensing Examination. Pre-veterinary academic performance predicted academic performance in veterinary school. Subjective measures did not, and we have reduced the faculty workload associated with that part of the evaluation of applicants. Interview scores provide additional but small value in predicting performance on clinical competencies.

Creation of a metric for faculty effort in teaching in the veterinary curriculum.

A teaching-effort metric was generated using information from health sciences literature and self-reported data from faculty members. This metric was used to verify faculty effort based on teaching assignments, to equalize teaching between faculty members within disciplines, and to help faculty members understand the ramifications of new teaching opportunities on their overall effort.

Frequency of interactions between veterinarians and other professionals to guide interprofessional education.

The objective of this study was to determine the frequency with which veterinarians interact with individuals in other health professions and related fields of work. Four thousand members of the American Veterinary Medical Association (AVMA) were surveyed. Demographic information was collected, and participants were asked how frequently they interacted with members of various professions. General comments were solicited regarding which professions veterinarians should reach out to in order to enhance veterinary health care and which professions veterinarians could interact with to improve human health care. The overall response rate was 18.6%. Demographic information suggested that the sample was representative of the profession. Frequency of interactions ranged from never to regularly, but for no group or profession was frequent interaction the norm. For practitioners, interactions with pharmacists, both dispensing and compounding, and with drug company representatives were most common. For educators, interaction with pharmacists and drug company representatives was also high compared to other professions, and they were also more likely to interact with laboratory technicians and physicians. For those in industry, interactions with laboratory technicians, physicians, drug and food company representatives, and public health officials were most common. All other respondents described their most common interactions as being with public health officials. The veterinarians surveyed cited a need for greater interaction with physicians regarding zoonotic disease and parasitism. Veterinarians do not frequently interface with human health care professionals. Those with whom practitioners have the greatest interaction are pharmacists and representatives of drug companies. Greater interaction may benefit management of zoonotic disease and parasitism.

Relationships among common measures of student performance and scores on the North American Veterinary Licensing Examination.

OBJECTIVE: To determine relationships among several common measures of performance prior to and during veterinary school (ie, Graduate Record Examination [GRE] scores, undergraduate grade point average [UGPA], Qualifying Examination [QE] scores, overall grade point average during veterinary school [VGPA], and scores for the North American Veterinary Licensing Examination [NAVLE]). DESIGN: Longitudinal retrospective study. SAMPLE POPULATION: 192 students from the Iowa State University College of Veterinary Medicine and 152 students from the University of Minnesota College of Veterinary Medicine. PROCEDURES: Student UGPA, VGPA, and GRE score data were gathered during the normal admissions and academic processes. The QE was administered as a low-stakes examination at both institutions for the purposes of curricular assessment. Scores on the NAVLE were provided with student permission by the National Board of Veterinary Medical Examiners. Path analysis was used to explore hypothesized relationships among variables. RESULTS: GRE scores and UGPA predicted NAVLE scores indirectly through QE scores and VGPA, whereas QE scores and VGPA directly predicted NAVLE scores. The resulting models explained 58% to 62% of the variance in NAVLE scores, with QE score being the strongest predictor. CONCLUSIONS AND CLINICAL RELEVANCE: Results indicated that for veterinary school students, GRE scores, UGPA, VGPA, and QE scores could be used to predict scores on the NAVLE. This suggests that these measures could prove useful to veterinary schools when admitting students or preparing them for the NAVLE.

Comparison of student self-assessment with faculty assessment of clinical competence.

At the University of Minnesota, fourth-year veterinary students assessed their clinical competence after completion of a small-animal, internal-medicine clinical rotation using the same rotation assessment form used by supervising faculty. Grades were compared between the two groups. Students identified by faculty as low-performing were more likely to overestimate their competence in the areas of knowledge, clinical skill, and professionalism than were students identified by faculty as higher performing. This finding mirrors research results in human health professional training. Self-assessment should not be used as the primary or sole measure of clinical competence in veterinary medical training without the introduction of measures to ensure the accuracy of student self-assessment, measures that include active faculty mentoring of student self-assessment, student goal-setting and reflection, and availability of subsequent opportunities to practice additional self-assessment.

The use of a structured interview guide in veterinary college admissions.

The University of Minnesota College of Veterinary Medicine incorporated a structured behavioral interview guide in its 2004 application cycle. This article presents the challenges associated with the incorporation of the guide, the methods used to ensure appropriate training for faculty and practicing veterinarians conducting interviews, and preliminary data regarding the guide's contribution to the admissions process.