Programmatic assessment for learning: A programmatically designed assessment for the purpose of learning: AMEE Guide No. 174.

Programmatic assessment for learning (PAL) involves programmatically structured collection of assessment data for the purpose of learning. In this guide, we examine and provide recommendations on several aspects: First, we review the evolution that has led to the development of programmatic assessment, providing clarification of some of its terminology. Second, we outline the learning processes that guide the design of PAL, including distributed learning, interleaving, overlearning, and test-enhanced learning. Third, we review the evolving nature of validity and provide insights into validity from a program perspective. Finally, we examine opportunities, challenges, and future directions of assessment in the context of artificial intelligence.

Essential concepts for effective mixed methods research in the health professions: AMEE Guide No. 173.

Mixed-methods research involves combining quantitative and qualitative approaches and mixing and integrating at multiple stages. It is gaining more attention and interest in health professions education research and evaluation. However, many undertake mixed-methods research for the first time without a rich understanding of the differences in practice associated with mixed-methods research. This often leads to research efforts that do not result in more complex, nuanced understandings of the phenomena being studied. Mixed-methods research requires researchers to thoughtfully and often creatively weave together their projects' various qualitative and quantitative strands. This effort ideally starts at the design stage and continues through the remainder of the research effort. We have aggregated resources and organized this guide to introduce researchers to what we see as some essential concepts, practices, and scholarship in mixed methods which will be useful to those just starting on their mixed-methods journey and those already engaging in mixed-methods research. This guide begins by discussing three vital characteristics of mixed-methods research that set it apart from mono-method research: its purposes, alternative philosophical stances, and integration, long viewed as a defining characteristic. We then discuss further important considerations, such as conducting mixed-methods literature reviews, timing and priority of qualitative and quantitative strands, and research design typologies. To complement these discussions, we have provided illustrative examples of high-quality peer-reviewed mixed-methods research from health professions education and other closely associated areas (e.g. health and clinical research). This guide also includes several activities and recommended resources, such as journals, textbooks, and professional societies, that researchers can use to deepen their practice and understanding and mixed-methods research. We hope the content, resources, article examples, and activities will be the prologue to fruitful explorations of mixed-methods research, helping researchers make informed, intentional choices about future mixed-methods efforts.

Programmatic Assessment of Clinical Reasoning: New Opportunities to Meet an Ongoing Challenge.

Issue: Clinical reasoning is essential to physicians' competence, yet assessment of clinical reasoning remains a significant challenge. Clinical reasoning is a complex, evolving, non-linear, context-driven, and content-specific construct which arguably cannot be assessed at one point in time or with a single method. This has posed challenges for educators for many decades, despite significant development of individual assessment methods. Evidence: Programmatic assessment is a systematic assessment approach that is gaining momentum across health professions education. Programmatic assessment, and in particular assessment for learning, is well-suited to address the challenges with clinical reasoning assessment. Several key principles of programmatic assessment are particularly well-aligned with developing a system to assess clinical reasoning: longitudinality, triangulation, use of a mix of assessment methods, proportionality, implementation of intermediate evaluations/reviews with faculty coaches, use of assessment for feedback, and increase in learners' agency. Repeated exposure and measurement are critical to develop a clinical reasoning assessment narrative, thus the assessment approach should optimally be longitudinal, providing multiple opportunities for growth and development. Triangulation provides a lens to assess the multidimensionality and contextuality of clinical reasoning and that of its different, yet related components, using a mix of different assessment methods. Proportionality ensures the richness of information on which to draw conclusions is commensurate with the stakes of the decision. Coaching facilitates the development of a feedback culture and allows to assess growth over time, while enhancing learners' agency. Implications: A programmatic assessment model of clinical reasoning that is developmentally oriented, optimizes learning though feedback and coaching, uses multiple assessment methods, and provides opportunity for meaningful triangulation of data can help address some of the challenges of clinical reasoning assessment.

Beyond thinking fast and slow: Implications of a transtheoretical model of clinical reasoning and error on teaching, assessment, and research.

From dual process to a family of theories known collectively as situativity, both micro and macro theories of cognition inform our current understanding of clinical reasoning (CR) and error. CR is a complex process that occurs in a complex environment, and a nuanced, expansive, integrated model of these theories is necessary to fully understand how CR is performed in the present day and in the future. In this perspective, we present these individual theories along with figures and descriptive cases for purposes of comparison before exploring the implications of a transtheoretical model of these theories for teaching, assessment, and research in CR and error.

Implementation of three knowledge-oriented instructional strategies to teach clinical reasoning: Self-explanation, a concept mapping exercise, and deliberate reflection: AMEE Guide No. 150.

The teaching of clinical reasoning is essential in medical education. This guide has been written to provide educators with practical advice on the design, development, and implementation of three knowledge-oriented instructional strategies for the teaching of clinical reasoning to medical students: Self-explanation (SE), a Clinical Reasoning Mapping Exercise (CREsME), and Deliberate Reflection (DR). We first synthesize the theoretical tenets that support the use of these strategies, including knowledge organization, and development of illness scripts. We then provide a detailed description of the key components of each strategy, emphasizing the practical applications of each one by sharing specific examples. We also explore the potential for a combined application of these strategies in a longitudinal and developmental approach to teaching clinical reasoning at the undergraduate level. Finally, we discuss enablers and barriers in the implementation and integration of these teaching strategies while taking into consideration curricular needs, context, and resources. We are aware that many strategies exist and are not arguing that SE, CReSME, and DR are the most effective ones or the only ones to be adopted. Nevertheless, we selected these strategies because of overarching theoretical principles, empirical evidence supporting their use, and our own experience with them. We are hoping to provide practical advice on the implementation of these strategies to practicing educators who aim at developing an integrated approach to the teaching of clinical reasoning to medical students at different stages of their development.

Concept mapping: An aid to teaching and learning: AMEE Guide No. 157.

Concept maps are graphic representations that learners draw to depict their understanding of the meaning of a set of concepts. Concept maps can be a useful learning strategy in medical education. The purpose of this guide is to provide an understanding of the theoretical basis and instructional use of concept mapping in health professions education. The guide describes the key elements of a concept map and underscores the critical components of the implementation process, from the introduction of the activity to the use of different mapping techniques, based on purpose and context. This guide also examines the learning opportunities offered by collaborative concept mapping, including co-construction of knowledge, and provides suggestions for the use of concept mapping as an assessment for learning. Implications for the use of concept mapping as a tool for remediation are mentioned. Finally, the guide describes some of the challenges related to the implementation of this strategy.

Extending growth curves: a trajectory monitoring approach to identification and interventions in struggling medical student learners.

Given gaps in both identifying and providing targeted interventions to struggling learners, the purpose of this study is to both improve rapid identification and to improve individualized academic advising for learners using this visual representation of performance. Across three graduating classes, individual growth curves were calculated for each student on National Board of Medical Examiners customized assessments during the pre-clerkship period using their deviation from the class average at each assessment point. These deviation scores were cumulatively summed over time and were regressed onto the sequence of exams. We analyzed the difference between the regression slopes of those students placed on Academic Probation (AP) versus not, as well as differences in slopes based on the timing of when a struggling learner was placed on AP to explore learner trajectory after identification. Students on AP had an average growth slope of - 6.06 compared to + 0.89 for those not on AP. Findings also suggested that students who were placed on AP early during pre-clerkship showed significant improvement (positive changes in trajectory) compared to students identified later in the curriculum. Our findings suggest that earlier academic probation and intervention with struggling learners may have a positive effect on academic trajectory. Future research can better explore how academic trajectory monitoring and performance review can be regularly used in advising sessions with students.

An international study on the implementation of programmatic assessment: Understanding challenges and exploring solutions.

INTRODUCTION: Programmatic assessment is an approach to assessment aimed at optimizing the learning and decision function of assessment. It involves a set of key principles and ground rules that are important for its design and implementation. However, despite its intuitive appeal, its implementation remains a challenge. The purpose of this paper is to gain a better understanding of the factors that affect the implementation process of programmatic assessment and how specific implementation challenges are managed across different programs. METHODS: An explanatory multiple case (collective) approach was used for this study. We identified 6 medical programs that had implemented programmatic assessment with variation regarding health profession disciplines, level of education and geographic location. We conducted interviews with a key faculty member from each of the programs and analyzed the data using inductive thematic analysis. RESULTS: We identified two major factors in managing the challenges and complexity of the implementation process: knowledge brokers and a strategic opportunistic approach. Knowledge brokers were the people who drove and designed the implementation process acting by translating evidence into practice allowing for real-time management of the complex processes of implementation. These knowledge brokers used a 'strategic opportunistic' or agile approach to recognize new opportunities, secure leadership support, adapt to the context and take advantage of the unexpected. Engaging in an overall curriculum reform process was a critical factor for a successful implementation of programmatic assessment. DISCUSSION: The study contributes to the understanding of the intricacies of implementation processes of programmatic assessment across different institutions. Managing opportunities, adaptive planning, awareness of context, were all critical aspects of thinking strategically and opportunistically in the implementation of programmatic assessment. Future research is needed to provide a more in-depth understanding of values and beliefs that underpin the assessment culture of an organization, and how such values may affect implementation.

Factors and Interactions Influencing Direct Observation: A Literature Review Guided by Activity Theory.

PhenomenonEnsuring that future physicians are competent to practice medicine is necessary for high quality patient care and safety. The shift toward competency-based education has placed renewed emphasis on direct observation via workplace-based assessments in authentic patient care contexts. Despite this interest and multiple studies focused on improving direct observation, challenges regarding the objectivity of this assessment approach remain underexplored and unresolved. Approach: We conducted a literature review of direct observation in authentic patient contexts by systematically searching databases PubMed, Embase, Web of Science, and ERIC. Included studies comprised original research conducted in the patient care context with authentic patients, either as a live encounter or a video recording of an actual encounter, which focused on factors affecting the direct observation of undergraduate medical education (UME) or graduate medical education (GME) trainees. Because the patient care context adds factors that contribute to the cognitive load of the learner and of the clinician-observer we focused our question on such contexts, which are most useful in judgments about advancement to the next level of training or practice. We excluded articles or published abstracts not conducted in the patient care context (e.g., OSCEs) or those involving simulation, allied health professionals, or non-UME/GME trainees. We also excluded studies focused on end-of-rotation evaluations and in-training evaluation reports. We extracted key data from the studies and used Activity Theory as a lens to identify factors affecting these observations and the interactions between them. Activity Theory provides a framework to understand and analyze complex human activities, the systems in which people work, and the interactions or tensions between multiple associated factors. Findings: Nineteen articles were included in the analysis; 13 involved GME learners and 6 UME learners. Of the 19, six studies were set in the operating room and four in the Emergency department. Using Activity Theory, we discovered that while numerous studies focus on rater and tool influences, very few study the impact of social elements. These are the rules that govern how the activity happens, the environment and members of the community involved in the activity and how completion of the activity is divided up among the members of the community. Insights: Viewing direct observation via workplace-based assessment through the lens of Activity Theory may enable educators to implement curricular changes to improve direct observation of assessment. Activity Theory may allow researchers to design studies to focus on the identified underexplored interactions and influences in relation to direct observation.

Medical education in Italy: Challenges and opportunities.

Italy is a country of 60 million citizens with a high life expectancy, an increasing prevalence of chronic multi-morbidity and a public healthcare system. There are 61 medical schools and more than one thousand postgraduate programs for 50 different specialisations. In this article, we describe the Italian medical educational system and its most recent evolution towards a process of internationalization, alongside pedagogical and cultural changes. The main challenges are in the process of students' selection, which is still only based on the assessment of basic knowledge, and in the reform of the post-graduate education, which lacks an official, formal definition of the learning outcomes and the aligned methods of assessment. The opportunities come from the increasing awareness of the importance of faculty development programs. The pandemic itself acted as a catalyst of innovation, pushing toward more student-centered teaching-learning activities. Finally, an increase in international collaborations in medical education research could be effective to foster the development of medical education in the country.

Gathering Validity Evidence on an Internal Medicine Clerkship Multistep Exam to Assess Medical Student Analytic Ability.

Construct: The definition of clinical reasoning may vary among health profession educators. However, for the purpose of this paper, clinical reasoning is defined as the cognitive processes that are involved in the steps of information gathering, problem representation, generating a differential diagnosis, providing a diagnostic justification to arrive at a leading diagnosis, and formulating diagnostic and management plans. Background: Expert performance in clinical reasoning is essential for success as a physician, and has been difficult for clerkship directors to observe and quantify in a way that fosters the instruction and assessment of clinical reasoning. The purpose of this study was to gather validity evidence for the Multistep exam (MSX) format used by our medicine clerkship to assess analytical clinical reasoning abilities; we did this by examining the relationship between scores on the MSX and other external measures of clinical reasoning abilities. This analysis used dual process theory as the main theoretical framework of clinical reasoning, as well as aspects of Kane's validity framework to guide the selection of validity evidence for the investigation. We hypothesized that there would be an association between the MSX (a three-step clinical reasoning tool developed locally), and the USMLE Step 2 CS, as they share similar concepts in assessing the clinical reasoning of students. We examined the relationship between overall scores on the MSX and the Step 2 CS Integrated Clinical Encounter (ICE) score, in which the student articulates their reasoning for simulated patient cases, while controlling for examinee's internal medicine clerkship performance measures such as the NBME subject exam score and the Medicine clerkship OSCE score. Approach: A total 477 of 487 (97.9%) medical students, representing the graduating classes of 2015, 2016, 2017, who took the MSX at the end of each medicine clerkship (2012-2016), and Step 2 CS (2013-2017) were included in this study. Correlation analysis and multiple linear regression analysis were used to examine the impact of the primary explanatory variables of interest (MSX) onto the outcome variable (ICE score) when controlling for baseline variables (Medicine OSCE and NBME Medicine subject exam). Findings: The overall MSX score had a significant, positive correlation with the Step 2 CS ICE score (r = .26, P < .01). The overall MSX score was a significant predictor of Step 2 CS ICE score (ß = .19, P < .001), explaining an additional 4% of the variance of ICE beyond the NBME Medicine subject score and the Medicine OSCE score (Adjusted R2 = 13%). Conclusion: The stepwise format of the MSX provides a tool to observe clinical reasoning performance, which can be used in an assessment system to provide feedback to students on their analytical clinical reasoning. Future studies should focus on gaining additional validity evidence across different learners and multiple medical schools.

Ottawa 2020 consensus statement for programmatic assessment - 1. Agreement on the principles.

INTRODUCTION: In the Ottawa 2018 Consensus framework for good assessment, a set of criteria was presented for systems of assessment. Currently, programmatic assessment is being established in an increasing number of programmes. In this Ottawa 2020 consensus statement for programmatic assessment insights from practice and research are used to define the principles of programmatic assessment. METHODS: For fifteen programmes in health professions education affiliated with members of an expert group (n = 20), an inventory was completed for the perceived components, rationale, and importance of a programmatic assessment design. Input from attendees of a programmatic assessment workshop and symposium at the 2020 Ottawa conference was included. The outcome is discussed in concurrence with current theory and research. RESULTS AND DISCUSSION: Twelve principles are presented that are considered as important and recognisable facets of programmatic assessment. Overall these principles were used in the curriculum and assessment design, albeit with a range of approaches and rigor, suggesting that programmatic assessment is an achievable education and assessment model, embedded both in practice and research. Knowledge on and sharing how programmatic assessment is being operationalized may help support educators charting their own implementation journey of programmatic assessment in their respective programmes.

Ottawa 2020 consensus statements for programmatic assessment - 2. Implementation and practice.

INTRODUCTION: Programmatic assessment is a longitudinal, developmental approach that fosters and harnesses the learning function of assessment. Yet the implementation, a critical step to translate theory into practice, can be challenging. As part of the Ottawa 2020 consensus statement on programmatic assessment, we sought to provide descriptions of the implementation of the 12 principles of programmatic assessment and to gain insight into enablers and barriers across different institutions and contexts. METHODS: After the 2020 Ottawa conference, we surveyed 15 Health Profession Education programmes from six different countries about the implementation of the 12 principles of programmatic assessment. Survey responses were analysed using a deductive thematic analysis. RESULTS AND DISCUSSION: A wide range of implementations were reported although the principles remained, for the most part, faithful to the original enunciation and rationale. Enablers included strong leadership support, ongoing faculty development, providing students with clear expectations about assessment, simultaneous curriculum renewal and organisational commitment to change. Most barriers were related to the need for a paradigm shift in the culture of assessment. Descriptions of implementations in relation to the theoretical principles, across multiple educational contexts, coupled with explanations of enablers and barriers, provided new insights and a clearer understanding of the strategic and operational considerations in the implementation of programmatic assessment. Future research is needed to further explore how contextual and cultural factors affect implementation.

Theoretical considerations on programmatic assessment.

Introduction: Programmatic assessment (PA) is an approach to assessment aimed at optimizing learning which continues to gain educational momentum. However, the theoretical underpinnings of PA have not been clearly described. An explanation of the theoretical underpinnings of PA will allow educators to gain a better understanding of this approach and, perhaps, facilitate its use and effective implementation. The purpose of this article is twofold: first, to describe salient theoretical perspectives on PA; second to examine how theory may help educators to develop effective PA programs, helping to overcome challenges around PA.Results: We outline a number of learning theories that underpin key educational principles of PA: constructivist and social constructivist theory supporting meaning making, and longitudinality; cognitivist and cognitive development orientation scaffolding the practice of a continuous feedback process; theory of instructional design underpinning assessment as learning; self-determination theory (SDT), self-regulation learning theory (SRL), and principles of deliberate practice providing theoretical tenets for student agency and accountability.Conclusion: The construction of a plausible and coherent link between key educational principles of PA and learning theories should enable educators to pose new and important inquiries, reflect on their assessment practices and help overcome future challenges in the development and implementation of PA in their programs.

Exploring the Predictors of Post-Clerkship USMLE Step 1 Scores.

Theory: We used two theoretical frameworks for this study: a) experiential learning, whereby learners construct new knowledge based on prior experience, and learning grows out of a continuous process of reconstructing experience, and b) deliberate practice, whereby the use of testing (test-enhanced learning) promotes learning and produces better long-term retention. Hypothesis: We hypothesized that moving the USMLE Step 1 exam to follow the clerkship year would provide students with a context for basic science learning that may enhance exam performance. We also hypothesized that examination performance variables, specifically National Board of Medical Examiners (NBME) Customized Basic Science Examinations and NBME subject examinations in clinical disciplines would account for a moderate to large amount of the variance in Step 1 scores. Thus we examined predictors of USMLE Step 1 scores when taken after the core clerkship year. Method: In 2011, we revised our medical school curriculum and moved the timing of Step 1 to follow the clerkship year. We performed descriptive statistics, an ANCOVA to compare Step 1 mean scores for three graduating classes of medical students before and after the curriculum changes, and stepwise linear regression to investigate the association between independent variables and the primary outcome measure after curriculum changes. Results: 993 students took the Step 1 exam, which included graduating classes before (2012-2014, N = 491) and after (2015-2017, N = 502) the curriculum change. Step 1 scores increased significantly following curricular revision (mean 218, SD 18.2, vs. 228, SD 16.7, p < 0.01) after controlling for MCAT and undergraduate GPA. Overall, 66.4% of the variance in Step 1 scores after the clerkship year was explained by: the mean score on fourteen pre-clerkship customized NBME exams (p < 0.01, 57.0% R2); performance on the surgery NBME subject exam (p < 0.01, 3.0% R2); the pediatrics NBME subject exam (p < 0.01, 2.0% R2); the Comprehensive Basic Science Self-Assessment (p < .01, 2.0% R2) ; the internal medicine NBME subject exam (p < 0.01, 0.03% R2), pre-clerkship Integrated Clinical Skills score (p < 0.01, 0.05% R2), and the pre-matriculation MCAT (p < 0.01, 0.01% R2). Conclusion: In our institution, nearly two-thirds of the variance in performance on Step 1 taken after the clerkship year was explained mainly by pre-clerkship variables, with a smaller contribution emanating from clerkship measures. Further study is needed to uncover the specific aspects of the clerkship experience that might contribute to success on high stakes licensing exam performance.

Trust in Group Decisions: a scoping review.

BACKGROUND: Trust is a critical component of competency committees given their high-stakes decisions. Research from outside of medicine on group trust has not focused on trust in group decisions, and "group trust" has not been clearly defined. The purpose was twofold: to examine the definition of trust in the context of group decisions and to explore what factors may influence trust from the perspective of those who rely on competency committees through a proposed group trust model. METHODS: The authors conducted a literature search of four online databases, seeking articles published on trust in group settings. Reviewers extracted, coded, and analyzed key data including definitions of trust and factors pertaining to group trust. RESULTS: The authors selected 42 articles for full text review. Although reviewers found multiple general definitions of trust, they were unable to find a clear definition of group trust and propose the following: a group-directed willingness to accept vulnerability to actions of the members based on the expectation that members will perform a particular action important to the group, encompassing social exchange, collective perceptions, and interpersonal trust. Additionally, the authors propose a model encompassing individual level factors (trustor and trustee), interpersonal interactions, group level factors (structure and processes), and environmental factors. CONCLUSIONS: Higher degrees of trust at the individual and group levels have been associated with attitudinal and performance outcomes, such as quality of group decisions. Developing a deeper understanding of trust in competency committees may help these committees implement more effective and meaningful processes to make collective decisions.