Towards a flexible and personalised development of veterinarians and veterinary nurses working in a companion animal referral care setting.

In the Netherlands, the demand for veterinarians and veterinary nurses (VNs) working within referral care is rapidly growing and currently exceeds the amount of available board-certified specialists. Simultaneously, a transparent structure to guide training and development and to assess quality of non-specialist veterinarians and VNs working in a referral setting is lacking. In response, we developed learning pathways guided by an entrustable professional activity (EPA) framework and programmatic assessment to support personalised development and competence of veterinarians and VNs working in referral settings. Between 4 and 35 EPAs varying per discipline (n = 11) were developed. To date, 20 trainees across five disciplines have been entrusted. Trainees from these learning pathways have proceeded to acquire new EPAs in addition to their already entrusted set of EPAs or progressed to specialist training during (n = 3) or after successfully completing (n = 1) the learning pathway. Due to their outcome-based approach, the learning pathways support flexible ways of development.

Programmatic Assessment of Clinical Reasoning: New Opportunities to Meet an Ongoing Challenge.

Issue: Clinical reasoning is essential to physicians' competence, yet assessment of clinical reasoning remains a significant challenge. Clinical reasoning is a complex, evolving, non-linear, context-driven, and content-specific construct which arguably cannot be assessed at one point in time or with a single method. This has posed challenges for educators for many decades, despite significant development of individual assessment methods. Evidence: Programmatic assessment is a systematic assessment approach that is gaining momentum across health professions education. Programmatic assessment, and in particular assessment for learning, is well-suited to address the challenges with clinical reasoning assessment. Several key principles of programmatic assessment are particularly well-aligned with developing a system to assess clinical reasoning: longitudinality, triangulation, use of a mix of assessment methods, proportionality, implementation of intermediate evaluations/reviews with faculty coaches, use of assessment for feedback, and increase in learners' agency. Repeated exposure and measurement are critical to develop a clinical reasoning assessment narrative, thus the assessment approach should optimally be longitudinal, providing multiple opportunities for growth and development. Triangulation provides a lens to assess the multidimensionality and contextuality of clinical reasoning and that of its different, yet related components, using a mix of different assessment methods. Proportionality ensures the richness of information on which to draw conclusions is commensurate with the stakes of the decision. Coaching facilitates the development of a feedback culture and allows to assess growth over time, while enhancing learners' agency. Implications: A programmatic assessment model of clinical reasoning that is developmentally oriented, optimizes learning though feedback and coaching, uses multiple assessment methods, and provides opportunity for meaningful triangulation of data can help address some of the challenges of clinical reasoning assessment.

Peer assessment in medical communication skills training in programmatic assessment: A qualitative study examining faculty and student perceptions.

INTRODUCTION: Current literature recommends assessment of communication skills in medical education combining different settings and multiple observers. There is still a gap in understanding about whether and how peers assessment facilitates learning in communication skills training. METHODS: We designed a qualitative study using focus group interviews and thematic analysis, in a medical course in the Netherlands. We aimed to explore medical students' and teachers' experiences, perceptions, and perspectives about challenges and facilitating factors in PACST (Peer assessment in medical communication skills training). RESULTS: Most of the participants reported that peer feedback was a valuable experience when learning communication skills. The major challenges for the quality and credibility of PACST reported by the participants are the question whether peer feedback is critical enough for learning and the difficulty of actually engaging students in the assessment process. CONCLUSION: Teachers reviewing students' peer assessments may improve the quality and their credibility and the reviewed assessments can best be used for learning purposes. We suggest to pay sufficient attention to teachers' roles in PACST, ensuring a safe and trustworthy environment and additionally helping students to internalize the value of being vulnerable during the evaluation process.

Validation of Longitudinal Progression Examinations for Prediction of APPE Readiness.

OBJECTIVE: To study curricular outcomes for the purpose of holistic improvement of the curriculum. METHODS: A single-institution retrospective cohort study evaluated 3 cohorts of Doctor of Pharmacy students entering the program through performance in Advanced Pharmacy Practice Experience (APPE) rotations. Assessment scores and pass/fail outcomes were collected from the 3 examinations to use as predictors, and the numbers of "needs improvement" (NI) and "unsatisfactory" (U) ratings from preceptors during the APPE rotations served as outcome measures. RESULTS: Pharmacy mathematics competency and Milemarker 1 (MM1) examination first-time scores, but not those from Milemarker 2 (MM2), were significantly associated with NI or U scores on required APPE rotations. Significant correlations for all examinations (pharmacy mathematics competency, MM1, and MM2) were found for the Acute Care/Inpatient APPE rotation for each cohort and the combined cohorts. Significant correlations were also found between all examinations and the APPE rotation courses Advanced Hospital and Ambulatory Care, with the exception of the 2021 cohort. Performance in the Advanced Community rotation was not associated with any of the examinations. MM1 and MM2 are both reliable measures of competence in our didactic curriculum and predictive of scoring an NI or U rating in the APPE Acute Care/Inpatient rotation. CONCLUSION: The longitudinal milestone examinations used in our institution provide a mechanism to identify students likely to struggle in required APPE rotations and target them for remediation activities.

Programmatic Assessment Prototype Design Using Integrated System Thinking And Design Thinking Framework.

Objective: To design a prototype programmatic assessment model using integration of system thinking and design thinking frameworks, and to explore barriers in the way of its implementation in medical education. METHODS: The qualitative case-study was conducted at Bahria University Medical and Dental College, Karachi from September to December 2022. Methodological approaches used were systems thinking and design thinking. Philosophical paradigm used was critical realism. Maximum variation purposive sampling technique was used for selecting faculty members and medical students. Data was collected in two sets of semi-structured interviews. The initial interviews were at the levels of empathise, define and ideate. After that low-fidelity prototype programmatic assessment design was created and presented to the participants. The second set of interviews were focussed on feedback on prototype of programmatic assessment design. Pattern matching method was used for data analysis. RESULTS: Of the 65 subjects, 20(30.7%) were faculty members; 5(25%) lecturers and 5(25%) professors each from basic and clinical sciences. The remaining 45(69.2%) were students; 5(11.1%)from each year of medical and dental streams. Initial interviews revealed that after failure, students struggle to understand the cause of failure as assessment system lacked narrative feedback mechanism and numbers or pass/fail decisions were not self-explanatory. Students lacked attitude towards continued improvement, formative assessment were not taken seriously as they carried only 10% weightage. After the programmatic assessment design was presented to the participants, faculty members voiced concerns regarding faculty training, shortage of resources and legal issues related to its implementation. Students supported the design, especially continued narrative feedback for their academic improvement. Portfolio development was considered a time-consuming task and they had reservations regarding their ability to make quality assessment portfolio. Conclusion: Implementation of programmatic assessment required faculty development and was resource-intensive, but stakeholders, especially students, were found interested in its implementation in medical education.

Operationalizing competency-based assessment: Contextualizing for cultural and gender divides.

Following current trends, educational institutions often decide to use a competency framework as an overarching structure in their assessment system. Despite the presence of a common understanding of how different examinations can contribute to the decision on attaining a particular competency, a detailed mapping of the data points appears to be a challenging area that remains to be explored. Faced with the newly emerged task of introducing the assessment of the attainment of UAE medical students against the EmiratesMEDs competency framework, Dubai Medical College for Girls (DMCG) attempted to operationalise the designed concept in the assessment system considering the cultural and gender divide. We believe that health professionals who attempt to implement contextualized competency-based assessment could benefit from being acquainted with our experience. The article offers a step-by-step guide on contextualized competency assessment operationalization, describing building the team, working with consultants and faculty development, estimating institutional assessment capacity, mapping and operationalizing the maps by using both human recourses and the software. We also offer the readers the list of enabling factors and introduce the scope of limitations in the process of developing the competency-based assessment system. We believe that following the present guide can allow educators to operationalize competency-based assessment in any context with respect to local culture and traditions.

"It Changed How I Think"-Impact of Programmatic Assessment Upon Practicing Physicians: A Qualitative Study.

Programmatic assessment is a systematic approach used to document and assess learner performance. It offers learners frequent formative feedback from a variety of contexts and uses both high- and low-stakes assessments to determine student progress. Existing research has explored learner and faculty perceptions of programmatic assessment, reporting favorable impact on faculty understanding of the importance of assessment stakes and feedback to learners while students report the ability to establish and navigate towards goals and reflect on their performance. The Cleveland Clinic Lerner College of Medicine (CCLCM) of Case Western Reserve University adopted programmatic assessment methods at its inception. With more than 18 years' experience with programmatic assessment and a portfolio-based assessment system, CCLCM is well-positioned to explore its graduates' perceptions of their programmatic assessment experiences during and after medical school. In 2020, the investigators interviewed 26 of the 339 physician graduates. Participants were purposefully sampled to represent multiple class cohorts (2009-2019), clinical specialties, and practice locations. The investigators analyzed interview transcripts using thematic analysis informed by the frameworks of self-determination theory and professional identity formation. The authors identified themes and support each with participant quotes from the interviews. Based on findings, the investigators compiled a series of recommendations for other institutions who have already or plan to incorporate elements of programmatic assessment into their curricula. The authors concluded by discussing future directions for research and additional avenues of inquiry.

'Ticked off'? Can a new outcomes-based postgraduate curriculum utilising programmatic assessment reduce assessment burden in Intensive Care Medicine?

Context: Increasing dissatisfaction with existing methods of assessment in the workplace alongside a national drive towards outcomes-based postgraduate curricula led to a recent overhaul of the way Intensive Care Medicine (ICM) trainees are assessed in the United Kingdom. Programmatic assessment methodology was utilised; the existing 'tick-box' approach using workplace-based assessment to demonstrate competencies was de-emphasised and the expertise of trainers used to assess capability relating to fewer, high-level outcomes related to distinct areas of specialist practice. Methods: A thematic analysis was undertaken investigating attitudes from 125 key stakeholders, including trainees and trainers, towards the new assessment strategy in relation to impact on assessment burden and acceptability. Results: This qualitative study suggests increased satisfaction with the transition to an outcomes-based model with capability judged by educational supervisors. However, reflecting frustration relating to current assessment in the workplace, participants felt assessment burden has been significantly reduced. The approach taken was felt to be an improved method for assessing professional practice; there was enthusiasm for this change. However, this research highlights trainee and trainer anxiety regarding how to 'pass' these expert judgement decisions of capability in the real world. Additionally, concerns relating to the impact on subgroups of trainees due to the potential influence of implicit biases on the resultant fewer but 'higher stakes' interrogative judgements became apparent. Conclusion: The move further towards a constructivist paradigm in workplace assessment in ICM reduces assessment burden yet can provoke anxiety amongst trainees and trainers requiring considered implementation. Furthermore, the perception of potential for bias in global judgements of performance requires further exploration.

Exploring the Irish general practice training community's perceptions on how an entrustable professional activities dashboard implementation could facilitate general practice training in Ireland.

BACKGROUND: The Irish General Practitioner Training (GP) Programme is currently moving to Competency-Based Medical Education (CBME), facilitated by Programmatic Assessment (PA) and Entrustable Professional Activities (EPAs). These new assessment and feedback mechanisms may provide a rich and much sought-after dataset. However, given the possible number of feedback and assessment events, and the variety of modalities used, aggregating and interpreting these can be costly and difficult. Dashboard implementations (DI) have been purposed as a solution to bridge the gap between the large datasets and the training community at all levels. AIMS: To explore the Irish GP training community's perceptions on how an EPAs DI could facilitate the delivery of GP training in Ireland. METHODS: A qualitative approach was taken, using a focus group representative of different groups in the training community. Concurrently, an EPAs DI was developed. Focus group transcripts were analysed in an iterative fashion using Template Analysis to generate themes and subthemes. RESULTS: Numerous advantages were seen in relation to the implementation of an EPAs DI around entrustment decisions, constructive alignment and summative decision-making. These advantages, however, need to be tempered with the realisation that the EPAs DI is not and should not be misinterpreted as being the learning analytic panacea for GP training. CONCLUSION: This paper outlines the perceptions from a postgraduate medical education training community on an EPAs DI, which would be applicable to other training communities considering introducing similar mechanisms.

Mobile technologies to support workplace-based assessment for entrustment decisions: Guidelines for programs and educators: AMEE Guide No. 154.

With the rise of competency-based medical education and workplace-based assessment (WBA) since the turn of the century, much has been written about methods of assessment. Direct observation and other sources of information have become standard in many clinical programs. Entrustable professional activities (EPAs) have also become a central focus of assessment in the clinical workplace. Paper and pencil (one of the earliest mobile technologies!) to document observations have become almost obsolete with the advent of digital technology. Typically, clinical supervisors are asked to document assessment ratings using forms on computers. However, accessing these forms can be cumbersome and is not easily integrated into existing clinical workflows. With a call for more frequent documentation, this practice is hardly sustainable, and mobile technology is quickly becoming indispensable. Documentation of learner performance at the point of care merges WBA with patient care and WBA increasingly uses smartphone applications for this purpose.This AMEE Guide was developed to support institutions and programs who wish to use mobile technology to implement EPA-based assessment and, more generally, any type of workplace-based assessment. It covers backgrounds of WBA, EPAs and entrustment decision-making, provides guidance for choosing or developing mobile technology, discusses challenges and describes best practices.

Teacher, Gatekeeper, or Team Member: supervisor positioning in programmatic assessment.

Competency-based assessment is undergoing an evolution with the popularisation of programmatic assessment. Fundamental to programmatic assessment are the attributes and buy-in of the people participating in the system. Our previous research revealed unspoken, yet influential, cultural and relationship dynamics that interact with programmatic assessment to influence success. Pulling at this thread, we conducted secondary analysis of focus groups and interviews (n = 44 supervisors) using the critical lens of Positioning Theory to explore how workplace supervisors experienced and perceived their positioning within programmatic assessment. We found that supervisors positioned themselves in two of three ways. First, supervisors universally positioned themselves as a Teacher, describing an inherent duty to educate students. Enactment of this position was dichotomous, with some supervisors ascribing a passive and disempowered position onto students while others empowered students by cultivating an egalitarian teaching relationship. Second, two mutually exclusive positions were described-either Gatekeeper or Team Member. Supervisors positioning themselves as Gatekeepers had a duty to protect the community and were vigilant to the detection of inadequate student performance. Programmatic assessment challenged this positioning by reorientating supervisor rights and duties which diminished their perceived authority and led to frustration and resistance. In contrast, Team Members enacted a right to make a valuable contribution to programmatic assessment and felt liberated from the burden of assessment, enabling them to assent power shifts towards students and the university. Identifying supervisor positions revealed how programmatic assessment challenged traditional structures and ideologies, impeding success, and provides insights into supporting supervisors in programmatic assessment.

Moving AACP Curriculum Quality Survey Results from Good to Great.

Although the American Association of Colleges of Pharmacy (AACP) Curriculum Quality Surveys (CQS) are required for programs to distribute and utilize as part of accreditation standards, programs face challenges in survey administration and timing, interpreting data and results, and following up on action plans. Because the CQS surveys are standardized, they can allow for greater comparison among institutions, yet interpretation of the items can vary considerably. Programs have flexibility in determining samples for administration and timing of administration (ie, number of years), but some participants (such as preceptors) can suffer from survey overload if multiple institutions administer in the same year. Determining thresholds for action and providing feedback to stakeholders on improvements made based on data triangulations can be daunting. These are a few of the elements that programs must consider when determining their own approach to the AACP CQS. Thus, the purpose of this Commentary is to describe good practices for using the AACP CQS, discuss challenges associated with the surveys, and recommend how to move the utilization of the surveys from good to great.

"A whole other competence story": exploring faculty perspectives on the process of workplace-based assessment of entrustable professional activities.

The centrality of entrustable professional activities (EPAs) in competency-based medical education (CBME) is predicated on the assumption that low-stakes, high-frequency workplace-based assessments used in a programmatic approach will result in accurate and defensible judgments of competence. While there have been conversations in the literature regarding the potential of this approach, only recently has the conversation begun to explore the actual experiences of clinical faculty in this process. The purpose of this qualitative study was to explore the process of EPA assessment for faculty in everyday practice. We conducted 18 semi-structured interviews with Anesthesia faculty at a Canadian academic center. Participants were asked to describe how they engage in EPA assessment in daily practice and the factors they considered. Interviews were audio-recorded, transcribed, and analysed using the constant comparative method of grounded theory. Participants in this study perceived two sources of tension in the EPA assessment process that influenced their scoring on official forms: the potential constraints of the assessment forms and the potential consequences of their assessment outcome. This was particularly salient in circumstances of uncertainty regarding the learner's level of competence. Ultimately, EPA assessment in CBME may be experienced as higher-stakes by faculty than officially recognized due to these tensions, suggesting a layer of discomfort and burden in the process that may potentially interfere with the goal of assessment for learning. Acknowledging and understanding the nature of this burden and identifying strategies to mitigate it are critical to achieving the assessment goals of CBME.

Perceptions of portfolio assessment in family medicine graduates: a qualitative interview study.

BACKGROUND: The use of the portfolio methodology in medical education can serve as a tool for learning, assessment, and reflection on practice. This study concentrates on perceptions of the portfolio assessment methodology among participants in the Saudi Diploma of Family Medicine program. METHODS: In this qualitative interview study, data were collected and analysed using a grounded theory approach. RESULTS: Nine codes emerged: (1) Importance of understanding the definition, objectives, and process of portfolio assessment, (2) Impact of different understandings on the part of various trainers, (3) Role of the type of assessment, (4) Workload and stress of portfolio assessment, (5) Effectiveness of the portfolio contents, (6) Role of the mentor's feedback, (7) Role in the learning process, (8) Role in practice, (9) Suggestions for portfolio improvement. Open codes were then regrouped into three axial codes: context, strategy, and outcome of portfolio assessment. CONCLUSION: This study explored a general explanation of portfolio assessment shaped by the postgraduate students. It identifies the importance of portfolio understanding in student acceptability of the portfolio assessment methodology. Thus, proper implementation is vital for the success of assessing the student by the portfolio methodology. The students perceived reflection as the most valuable part of the process, which facilitated their learning, confidence, and self-assessment. Mentor feedback is a good strategy for coping with portfolio challenges. Our findings provide some evidence of positive outcomes of portfolio assessment in practice and professional development.

The Practice of Feedback in Health Professions Education in the Hierarchical and Collectivistic Culture: a Scoping Review.

The global trend in student assessment is moving towards outcome-based assessment that requires multiple systematic data points and continuous feedback. However, in hierarchical and collectivistic cultures, one-way communication is practised more often, leading to less dialogical feedback. This scoping review explored feedback practice in Asian educational setting. Based on the 17 articles selected, the findings were categorised into four themes, i.e. inhibiting or facilitating factors of feedback, influences of cultural factors on feedback, discrepancies between students' and teachers' perceptions of feedback and impact of feedback. Hierarchical and collectivist cultural aspects, such as preference for group feedback, are pertinent to feedback practice, which likely influence the readiness for programmatic assessment.

An intentional approach to the development and implementation of meaningful assessment in advanced radiation therapy practice curricula.

Creating meaningful assessment for advanced radiation therapy practice training programs is a challenge. This is because it requires a balance of formative and summative assessments, which meet the academic and professional needs of the practitioner, as well as the requirements of local service delivery, educational and professional standards. This paper discusses educational strategies and models used to integrate assessment into theoretical and clinical curricula, allowing practitioners to demonstrate higher order cognitive knowledge, advanced level clinical performance and attitudes/values associated with advanced practice. The discussion draws upon concepts of constructive alignment and programmatic approaches to assessment, which use Bloom's taxonomy, Benner's beginner to competent model of skill development, and Miller's pyramid of clinical competence. These models are analysed with respect to an advanced practice program in adaptive radiation therapy to provide context.

Shaping the right conditions in programmatic assessment: how quality of narrative information affects the quality of high-stakes decision-making.

BACKGROUND: Programmatic assessment is increasingly being implemented within competency-based health professions education. In this approach a multitude of low-stakes assessment activities are aggregated into a holistic high-stakes decision on the student's performance. High-stakes decisions need to be of high quality. Part of this quality is whether an examiner perceives saturation of information when making a holistic decision. The purpose of this study was to explore the influence of narrative information in perceiving saturation of information during the interpretative process of high-stakes decision-making. METHODS: In this mixed-method intervention study the quality of the recorded narrative information was manipulated within multiple portfolios (i.e., feedback and reflection) to investigate its influence on 1) the perception of saturation of information and 2) the examiner's interpretative approach in making a high-stakes decision. Data were collected through surveys, screen recordings of the portfolio assessments, and semi-structured interviews. Descriptive statistics and template analysis were applied to analyze the data. RESULTS: The examiners perceived less frequently saturation of information in the portfolios with low quality of narrative feedback. Additionally, they mentioned consistency of information as a factor that influenced their perception of saturation of information. Even though in general they had their idiosyncratic approach to assessing a portfolio, variations were present caused by certain triggers, such as noticeable deviations in the student's performance and quality of narrative feedback. CONCLUSION: The perception of saturation of information seemed to be influenced by the quality of the narrative feedback and, to a lesser extent, by the quality of reflection. These results emphasize the importance of high-quality narrative feedback in making robust decisions within portfolios that are expected to be more difficult to assess. Furthermore, within these "difficult" portfolios, examiners adapted their interpretative process reacting on the intervention and other triggers by means of an iterative and responsive approach.

Curricular changes and interim posts during Covid-19: graduates' perspectives.

BACKGROUND: During the COVID-19 pandemic UK medical schools facilitated the early graduation of their final-year medical students to 'Foundation interim Year 1 (FiY1) doctors' through amendments made to curricula and final assessment. Such changes gave opportunity for evaluation. This study therefore aimed to explore 1) graduate perspective on the implementation of FiY1 and 2) how changes to course structures have affected self-reported preparedness for work. METHODS: Questionnaire surveys using Likert scale and free-text responses (n = 45), and semi-structured interviews (n = 7) were conducted with FiY1s from two UK medical schools contrasting in the amendments made to course structures. Data were analysed using quantitative methods and thematic analysis; 44% (n = 20) of respondents believed that governing health bodies had not communicated sufficiently prior to starting work. RESULTS: Graduates who had sat modified practical and written examinations reported 'legitimacy' and feeling more prepared compared to having not sat examinations (practical 100%, n = 17; written 88.3%, n = 15). Graduates from both schools agreed that carrying out assistantships as originally scheduled would have made them feel more prepared (91.1%, n = 41). CONCLUSIONS: The implementation of FiY1 was largely well received by graduates yet assistantship programmes may fulfil a similar role in normal times. Medical schools and governing bodies must ensure effective communication channels exist with students in order to better prepare them for their first posts, especially in times of crisis. Additionally, final examinations contribute to feelings of preparedness for work and instil a sense of legitimacy, a finding which is relevant to working within the current programmatic assessment structure.

An international study on the implementation of programmatic assessment: Understanding challenges and exploring solutions.

INTRODUCTION: Programmatic assessment is an approach to assessment aimed at optimizing the learning and decision function of assessment. It involves a set of key principles and ground rules that are important for its design and implementation. However, despite its intuitive appeal, its implementation remains a challenge. The purpose of this paper is to gain a better understanding of the factors that affect the implementation process of programmatic assessment and how specific implementation challenges are managed across different programs. METHODS: An explanatory multiple case (collective) approach was used for this study. We identified 6 medical programs that had implemented programmatic assessment with variation regarding health profession disciplines, level of education and geographic location. We conducted interviews with a key faculty member from each of the programs and analyzed the data using inductive thematic analysis. RESULTS: We identified two major factors in managing the challenges and complexity of the implementation process: knowledge brokers and a strategic opportunistic approach. Knowledge brokers were the people who drove and designed the implementation process acting by translating evidence into practice allowing for real-time management of the complex processes of implementation. These knowledge brokers used a 'strategic opportunistic' or agile approach to recognize new opportunities, secure leadership support, adapt to the context and take advantage of the unexpected. Engaging in an overall curriculum reform process was a critical factor for a successful implementation of programmatic assessment. DISCUSSION: The study contributes to the understanding of the intricacies of implementation processes of programmatic assessment across different institutions. Managing opportunities, adaptive planning, awareness of context, were all critical aspects of thinking strategically and opportunistically in the implementation of programmatic assessment. Future research is needed to provide a more in-depth understanding of values and beliefs that underpin the assessment culture of an organization, and how such values may affect implementation.