How do validity experts conceptualise argumentation? It's a rhetorical question.

INTRODUCTION: Health professions education (HPE) has adopted the conceptualization of validity as an argument. However, the theoretical and practical aspects of how validity arguments should be developed, used and evaluated in HPE have not been deeply explored. Articulating the argumentation theory undergirding validity and validation can help HPE better operationalise validity as an argument. To better understand this, the authors explored how HPE validity scholars conceptualise assessment validity arguments and argumentation, seeking to understand potential consequences of these views on validation practices. METHODS: The authors used critical case sampling to identify HPE assessment validity experts in three ways: (1) participation in a prominent validity research group, (2) appearing in a bibliometric study of HPE validity publications and (3) authorship of recent HPE validity literature. Qualitative semi-structured interviews were conducted with 16 experts in HPE assessment validity from four different countries. The authors used reflexive thematic analysis to develop themes relevant to their research question. RESULTS: The authors developed three themes grounded in participants' responses: (1) In theory, HPE validity is a social and situated argument. (2) In practice, the absence of audience and evaluation stymies the social nature of HPE validity. (3) Lack of validity argumentation creates and maintains power differentials within HPE. Participants articulated that current HPE validation practices are rooted in post-positivist epistemology when they should be situated (i.e. context-dependent), audience-centric and inclusive. DISCUSSION: When discussing validity argumentation in theory, participants' descriptions reflect an interpretivist lens for evaluation that is misaligned with real-world validity practices. This misalignment likely arises from HPE's adoption of "validity as an argument" as a slogan, without integrating theoretical and practical principles of argumentation theory.

Competency-based medical education: The spark to ignite healthcare's escape fire.

High-value care is what patients deserve and what healthcare professionals should deliver. However, it is not what happens much of the time. Quality improvement master Dr. Don Berwick argued more than two decades ago that American healthcare needs an escape fire, which is a new way of seeing and acting in a crisis situation. While coined in the U.S. context, the analogy applies in other Western healthcare contexts as well. Therefore, in this paper, the authors revisit Berwick's analogy, arguing that medical education can, and should, provide the spark for such an escape fire across the globe. They assert that medical education can achieve this by fully embracing competency-based medical education (CBME) as a way to place medicine's focus on the patient. CBME targets training outcomes that prepare graduates to optimize patient care. The authors use the escape fire analogy to argue that medical educators must drop long-held approaches and tools; treat CBME implementation as an adaptive challenge rather than a technical fix; demand genuine, rich discussions and engagement about the path forward; and, above all, center the patient in all they do.

Data sharing and big data in health professions education: Ottawa consensus statement and recommendations for scholarship.

Changes in digital technology, increasing volume of data collection, and advances in methods have the potential to unleash the value of big data generated through the education of health professionals. Coupled with this potential are legitimate concerns about how data can be used or misused in ways that limit autonomy, equity, or harm stakeholders. This consensus statement is intended to address these issues by foregrounding the ethical imperatives for engaging with big data as well as the potential risks and challenges. Recognizing the wide and ever evolving scope of big data scholarship, we focus on foundational issues for framing and engaging in research. We ground our recommendations in the context of big data created through data sharing across and within the stages of the continuum of the education and training of health professionals. Ultimately, the goal of this statement is to support a culture of trust and quality for big data research to deliver on its promises for health professions education (HPE) and the health of society. Based on expert consensus and review of the literature, we report 19 recommendations in (1) framing scholarship and research through research, (2) considering unique ethical practices, (3) governance of data sharing collaborations that engage stakeholders, (4) data sharing processes best practices, (5) the importance of knowledge translation, and (6) advancing the quality of scholarship through multidisciplinary collaboration. The recommendations were modified and refined based on feedback from the 2022 Ottawa Conference attendees and subsequent public engagement. Adoption of these recommendations can help HPE scholars share data ethically and engage in high impact big data scholarship, which in turn can help the field meet the ultimate goal: high-quality education that leads to high-quality healthcare.

A realist synthesis of prospective entrustment decision making by entrustment or clinical competency committees.

INTRODUCTION: The real-world mechanisms underlying prospective entrustment decision making (PEDM) by entrustment or clinical competency committees (E/CCCs) are poorly understood. To advance understanding in this area, the authors conducted a realist synthesis of the published literature to address the following research question: In E/CCC efforts to make defensible prospective entrustment decisions (PEDs), what works, for whom, under what circumstances and why? METHODS: Realist work seeks to understand the contexts (C), mechanisms (M) and outcomes (O) that explain how and why things work (or do not). In the authors' study, contexts included individual E/CCC members, E/CCC structures and processes, and training programmes. The outcome (i.e. desired outcome) was a PED. Mechanisms were a substantial focus of the analysis and informed the core findings. To define a final corpus of 52 included papers, the authors searched four databases, screened all results from those searches and performed a full-text review of a subset of screened papers. Data extraction focused on developing context-mechanism-outcome configurations from the papers, which were used to create a theory for how PEDM leads to PEDs. RESULTS: PEDM is often driven by default (non-deliberate) decision making rather than a deliberate process of deciding whether a trainee should be entrusted or not. When defaulting, some E/CCCs find red flags that sometimes lead to being more deliberate with decision making. E/CCCs that seek to be deliberate describe PEDM that can be effortful (when data are insufficient or incongruent) or effortless (when data are robust and tell a congruent story about a trainee). Both information about trainee trustworthiness and the sufficiency of data about trainee performance influence PEDM. Several moderators influence what is considered to be sufficient data, how trustworthiness data are viewed and how PEDM is carried out. These include perceived consequences and associated risks, E/CCC member trust propensity, E/CCC member personal knowledge of and experience with trainees and E/CCC structures and processes. DISCUSSION: PEDM is rarely deliberate but should be. Data about trainee trustworthiness are foundational to making PEDs. Bias, equity and fairness are nearly absent from the papers in this synthesis, and future efforts must seek to advance understanding and practice regarding the roles of bias, equity and fairness in PEDM.

"Rater training" re-imagined for work-based assessment in medical education.

In this perspective, the authors critically examine "rater training" as it has been conceptualized and used in medical education. By "rater training," they mean the educational events intended to improve rater performance and contributions during assessment events. Historically, rater training programs have focused on modifying faculty behaviours to achieve psychometric ideals (e.g., reliability, inter-rater reliability, accuracy). The authors argue these ideals may now be poorly aligned with contemporary research informing work-based assessment, introducing a compatibility threat, with no clear direction on how to proceed. To address this issue, the authors provide a brief historical review of "rater training" and provide an analysis of the literature examining the effectiveness of rater training programs. They focus mainly on what has served to define effectiveness or improvements. They then draw on philosophical and conceptual shifts in assessment to demonstrate why the function, effectiveness aims, and structure of rater training requires reimagining. These include shifting competencies for assessors, viewing assessment as a complex cognitive task enacted in a social context, evolving views on biases, and reprioritizing which validity evidence should be most sought in medical education. The authors aim to advance the discussion on rater training by challenging implicit incompatibility issues and stimulating ways to overcome them. They propose that "rater training" (a moniker they suggest be reserved for strong psychometric aims) be augmented with "assessor readiness" programs that link to contemporary assessment science and enact the principle of compatibility between that science and ways of engaging with advances in real-world faculty-learner contexts.

Developing the Expected Entrustment Score: Accounting for Variation in Resident Assessment.

BACKGROUND: Clinical competency committees (CCCs) and residency program leaders may find it difficult to interpret workplace-based assessment (WBA) ratings knowing that contextual factors and bias play a large role. OBJECTIVE: We describe the development of an expected entrustment score for resident performance within the context of our well-developed Observable Practice Activity (OPA) WBA system. DESIGN: Observational study PARTICIPANTS: Internal medicine residents MAIN MEASURE: Entrustment KEY RESULTS: Each individual resident had observed entrustment scores with a unique relationship to the expected entrustment scores. Many residents' observed scores oscillated closely around the expected scores. However, distinct performance patterns did emerge. CONCLUSIONS: We used regression modeling and leveraged large numbers of historical WBA data points to produce an expected entrustment score that served as a guidepost for performance interpretation.

Assessing trainee performance: ensuring learner control, supporting development, and maximizing assessment moments.

In this article, the authors provide practical guidance for frontline supervisors' efforts to assess trainee performance. They focus on three areas. First, they argue the importance of promoting learner control in the assessment process, noting that providing learners agency and control can shift the stakes of assessment from high to low and promote a safe environment that facilitates learning. Second, they posit that assessment should be used to support continued development by promoting a relational partnership between trainees and supervisors. This partnership allows supervisors to reinforce desirable aspects of performance, provide real-time support for deficient areas of performance, and sequence learning with the appropriate amount of scaffolding to push trainees from competence (what they can do alone) to capability (what they are able to do with support). Finally, they advocate the importance of optimizing the use of written comments and direct observation while also recognizing that performance is interdependent in efforts to maximize assessment moments.Conclusion: Using best practices in trainee assessment can help trainees take next steps in their development in a learner-centered partnership with clinical supervisors. What is Known: • Many pediatricians are asked to assess the performance of medical students and residents they work with but few have received formal training in assessment. What is New: • This article presents evidence-based best practices for assessing trainees, including giving trainees agency in the assessment process and focusing on helping trainees take next steps in their development.

Influence of the loading protocol and platform switching in two-implant bar-retained overdentures: 3-year results from a randomized controlled equivalence clinical trial.

OBJECTIVES: To test the null hypothesis that vertical peri-implant bone level alterations (ΔIBL) are equivalent in immediately (IL) and 3-month post-placement (DL) loaded implants in mandibular implant overdentures (IODs) on two implants. MATERIALS AND METHODS: Thirty-two patients receiving two interforaminal implants, one with a platform-switched and one with a platform-matching abutment were randomly assigned to the IL or DL group (allocation ratio 1:1). All implants were primarily splinted with chairside-customized bars, converting the existing removable complete dentures to IODs. Standardized radiographs were recorded. The influence of the loading protocol (IL vs. DL), implant platform (platform switched vs. platform matching), implant site (43 vs. 33), participant age (≤65 vs. >65 years), and definition of baseline (implant placement vs. implant loading) were analyzed, applying linear regression analyses (α = 0.05). The equivalence range was [-0.4; 0.4]. RESULTS: Three participants of the IL group were lost during follow-up. The overall mean ΔIBL was -0.96 ± 0.89 mm. The ΔIBL was equivalent in terms of the implant platform and implant site but not in terms of participant age (in favor of more elderly participants) and the loading protocol. A significantly smaller ΔIBL was observed in the IL when the baseline was considered to be implant placement (p = .017), but not when it was considered to be implant loading (p = .084). CONCLUSION: Immediate loading of primary-splinted implants in two-implant bar-retained overdentures, seems beneficial relative to loading 3 months post-placement, with respect to ΔIBL. The ΔIBL were equivalent in terms of platform switching.

How argumentation theory can inform assessment validity: A critical review.

INTRODUCTION: Many health professions education (HPE) scholars frame assessment validity as a form of argumentation in which interpretations and uses of assessment scores must be supported by evidence. However, what are purported to be validity arguments are often merely clusters of evidence without a guiding framework to evaluate, prioritise, or debate their merits. Argumentation theory is a field of study dedicated to understanding the production, analysis, and evaluation of arguments (spoken or written). The aim of this study is to describe argumentation theory, articulating the unique insights it can offer to HPE assessment, and presenting how different argumentation orientations can help reconceptualize the nature of validity in generative ways. METHODS: The authors followed a five-step critical review process consisting of iterative cycles of focusing, searching, appraising, sampling, and analysing the argumentation theory literature. The authors generated and synthesised a corpus of manuscripts on argumentation orientations deemed to be most applicable to HPE. RESULTS: We selected two argumentation orientations that we considered particularly constructive for informing HPE assessment validity: New rhetoric and informal logic. In new rhetoric, the goal of argumentation is to persuade, with a focus on an audience's values and standards. Informal logic centres on identifying, structuring, and evaluating arguments in real-world settings, with a variety of normative standards used to evaluate argument validity. DISCUSSION: Both new rhetoric and informal logic provide philosophical, theoretical, or practical groundings that can advance HPE validity argumentation. New rhetoric's foregrounding of audience aligns with HPE's social imperative to be accountable to specific stakeholders such as the public and learners. Informal logic provides tools for identifying and structuring validity arguments for analysis and evaluation.

Making prospective entrustment decisions: Knowing limits, seeking help and defaulting.

INTRODUCTION: Many studies focused on entrustment have not considered prospective entrustment decisions, where clinical competency and entrustment committees determine how much supervision trainees will require in the future for tasks occurring in not completely known contexts. The authors sought to explore factors that influence and determine prospective entrustment decisions made by members of such committees in graduate medical education (GME) and undergraduate medical education (UME). METHODS: The authors conducted a constructivist grounded theory study with 23 faculty participants from GME and UME clinical competency and entrustment committees in the United States between October 2020 and March 2021. Interviews sought to explore factors and considerations participants weigh in making prospective entrustment decisions about trainees. Data collection and analysis occurred in an iterative fashion, ensuring constant comparison. Theoretical sampling was used to confirm, disconfirm and elaborate on the evolving results. RESULTS: Trainees' ability to know limits and seek help is the foundation of participants' prospective entrustment decision making. Most participants, however, describe a presumption of trainee readiness to progress and describe commonly making default prospective entrustment decisions unless 'red flags' in performance are present. Although participants desire sufficient and trusted data about trainee performance to inform decisions, they often lack it. Finally, the perceived permanence and consequences of prospective entrustment decisions influence how participants weigh other factors contributing to prospective entrustment decisions. CONCLUSION: Trainees' ability to know limits and seek help appears to be the foundation of prospective entrustment decision making. Training programmes should strive to collect and employ robust data supporting and questioning the presence of these attributes.

Entrustable professional activities versus competencies and skills: Exploring why different concepts are often conflated.

Despite explanations in the literature, a returning question in the use of entrustable professional activities (EPAs) is how to distinguish them from competencies and skills. In this article, we attempt to analyze the causes of the frequent confusion and conflation of EPAs with competencies and skills, and argue why the distinction is important for education, qualification and patient safety. 'Tracheotomy', 'lumbar puncture', 'interprofessional collaboration' for example are colloquially called 'skills', but its is a person's ability to perform these activities that is the actual skill; the EPA is simply the activity itself. We identify two possible causes for the confusion. One is a tendency to frame all educational objectives as EPAs. Many objectives of medical training can be conceptualized as EPAs, if 'the ability to do X' is the corresponding competency; but that does not work for all. We offer ways to deal with objectives of training that are not usefully conceptualized as EPAs. A more fundamental cause relates to entrustment decisions. The permission to contribute to health care reflects entrustment. Entrustment decisions are the links or pivots between a person's readiness for the task and the actual task execution. However, if entrustment decisions do not lead to increased autonomy in the practice of health care, but only serve to decide upon the advancement to a next stage of training, EPAs can become the tick boxes learners feel they need to collect to 'pass'. Gradually, then, EPAs can loose their original meaning of units of practice for which one becomes qualified.

Development of Resident-Sensitive Quality Measures for Inpatient General Internal Medicine.

BACKGROUND: Graduate medical education (GME) training has long-lasting effects on patient care quality. Despite this, few GME programs use clinical care measures as part of resident assessment. Furthermore, there is no gold standard to identify clinical care measures that are reflective of resident care. Resident-sensitive quality measures (RSQMs), defined as "measures that are meaningful in patient care and are most likely attributable to resident care," have been developed using consensus methodology and piloted in pediatric emergency medicine. However, this approach has not been tested in internal medicine (IM). OBJECTIVE: To develop RSQMs for a general internal medicine (GIM) inpatient residency rotation using previously described consensus methods. DESIGN: The authors used two consensus methods, nominal group technique (NGT) and a subsequent Delphi method, to generate RSQMs for a GIM inpatient rotation. RSQMs were generated for specific clinical conditions found on a GIM inpatient rotation, as well as for general care on a GIM ward. PARTICIPANTS: NGT participants included nine IM and medicine-pediatrics (MP) residents and six IM and MP faculty members. The Delphi group included seven IM and MP residents and seven IM and MP faculty members. MAIN MEASURES: The number and description of RSQMs generated during this process. KEY RESULTS: Consensus methods resulted in 89 RSQMs with the following breakdown by condition: GIM general care-21, diabetes mellitus-16, hyperkalemia-14, COPD-13, hypertension-11, pneumonia-10, and hypokalemia-4. All RSQMs were process measures, with 48% relating to documentation and 51% relating to orders. Fifty-eight percent of RSQMs were related to the primary admitting diagnosis, while 42% could also be related to chronic comorbidities that require management during an admission. CONCLUSIONS: Consensus methods resulted in 89 RSQMs for a GIM inpatient service. While all RSQMs were process measures, they may still hold value in learner assessment, formative feedback, and program evaluation.

Identifying Strategies for the Management of Interruptions for Novice Triage Nurses Using an Online Modified Delphi Method.

PURPOSE: To use the Delphi Method to identify strategies used by triage nurses to effectively manage interruptions. DESIGN: This study was based on the concepts of Benner's Novice to Expert Model. An online, modified Delphi approach was used to engage triage, education, and operational management experts in generating consensus recommendations on successful strategies to address triage interruptions in the emergency department. METHODS AND ANALYSIS: A panel of nine triage, education, and operational management experts were selected based on their publication and presentation history. This panel participated in three Delphi rounds, providing individual responses during each round. All responses were entered into a RedCap database, which allowed research team members to synthesize the results and return summaries to the participants. Final consensus was reached among this panel regarding recommendations for successful strategies to address triage interruptions that can be encompassed in a training module. The experts were then asked to identify the best instructional modality for teaching each of the interruption management strategies. FINDINGS: Eight strategies to mitigate the impact of interruptions were identified: (a) ensure nurses understand impact of interruptions; (b) ensure nurses understand consequences of interruptions on cognitive demands of healthcare workers that could influence behavior and lead to errors; (c) apologize to current patient before tending to interruption and give expectation of when you will return; (d) triage the interruption and decide to (i) ignore interruption, (ii) acknowledge, but delay servicing, interruption, or (iii) acknowledge and service interruption, delaying completion of interrupted task; (e) identify urgent communication as anything clinically significant that impacts the patient immediately or requires immediate intervention; (f) use focused questions to clarify whether interruption can wait; (g) redirect nonpriority interruptions; and (h) finish safety-critical task or tasks near completion before tending to an interruption. The Delphi participants recommended the best teaching modality was simulation for six of the strategies. CONCLUSIONS: Participants agreed that there are strategies that can be taught to novice triage nurses to mitigate the impact of interruptions. The experts in operations management, emergency nursing, and education agree that creating simulations to teach each of these strategies is an effective way to educate nurses. CLINICAL RELEVANCE: Interruptions impact the quality of care provided to patients. Training nurses to prevent interruptions and mitigate the impact of interruptions when they occur has the potential to improve patient outcomes.

If we could turn back time: Imagining time-variable, competency-based medical education in the context of COVID-19.

The COVID-19 pandemic has exposed a paradox in historical models of medical education: organizations responsible for applying consistent standards for progression have needed to adapt to training environments marked by inconsistency and change. Although some institutions have maintained their traditional requirements, others have accelerated their programs to rush nearly graduated trainees to the front lines. One interpretation of the unplanned shortening of the duration of training programs during a crisis is that standards have been lowered. But it is also possible that these trainees were examined according to the same standards as usual and were judged to have already met them. This paper discusses the impacts of the COVID-19 pandemic on the current workforce, provides an analysis of how competency-based medical education (CBME) in the context of the pandemic might have mitigated wide-scale disruption, and identifies structural barriers to achieving an ideal state. The paper further calls upon universities, health centres, governments, certifying bodies, regulatory authorities, and health care professionals to work collectively on a truly time-variable model of CBME. The pandemic has made clear that time variability in medical education already exists and should be adopted widely and formally. If our systems today had used a framework of outcome competencies, sequenced progression, tailored learning, focused instruction, and programmatic assessment, we may have been even more nimble in changing our systems to care for our patients with COVID-19.

Describing Overarching Curricular Goals for Individualized Education.

Phenomenon: The phenomenon of individualized education, an essential component of competency-based medical education, addresses individual learner needs while working toward standardized learning outcomes. One challenge with broadly implementing individualized education is the lack of a pragmatic operational definition. To formalize expectations for individualized education, the Accreditation Council of Graduate Medical Education in 2013 began requiring six months of individualized curriculum (IC) during pediatric residency; however, there is not a national standard of formal curricular goals for the IC as an educational entity. Examining and describing the overarching curricular goals of IC could provide a framework for discourse about and further study of individualized education in medicine across disciplines and the continuum of medical education. Thus, we aimed to describe the phenomenon of individualized education through the lens of the goals of the IC in pediatric residency in the United States. Approach: In 2017, a purposeful sample of Pediatric Residency leaders were recruited to represent a diverse sample of program sizes, regions of the country, and importance of the IC to the program leadership. They completed an online survey with open-ended questions describing formal and implicit goals of their program's IC. The authors analyzed responses initially using conventional content analysis, then investigated whether the themes for program goals aligned with any existing educational theory. The concepts and language aligned with the principles of self-determination theory (SDT); therefore, the IC goals were subsequently grouped using the SDT domains of relatedness, autonomy, and competence. A focus group with a subset of survey respondents was conducted for member checking and elaboration of concepts. Findings: Program leaders from a diverse sample of 36 programs participated in the survey and a subset of 11 programs participated in the focus group. The common goals across all programs are listed in parentheses and organized by domains of SDT: 1)Relatedness goals (engage in mentorship, select a career) cultivate resident's professional identity based on their desired future career path; 2)autonomy goals (create a learning plan, practice accountability) help residents plan their path; and fulfillment of their plans lead to 3)competence goals (develop targeted clinical skills and knowledge, ensure comprehensive exposure, address learning gaps), ensuring they have a comprehensive skill set for their chosen identity. Insights: This study provides a framework to describe the phenomenon of individualized education through the lens of IC curricular goals in pediatric residency. The goals for IC that emerged from this study serve as a pragmatic framework for implementation of individualized education. They provide a common language and structure to promote more rigorous and collaborative study of individualized education across programs, disciplines, and settings in medicine. They may function as a roadmap for learners to navigate educational activities and for programs to help shape the experiences of their learners and examine outcomes of individualized education in their programs. The framework can also help individual pediatric residency programs structure improvements to their IC. Disciplines beyond pediatrics may also use this framework to better structure elective experiences to capitalize on the benefits of individual education.

Becoming a deliberately developmental organization: Using competency based assessment data for organizational development.

Medical education is situated within health care and educational organizations that frequently lag in their use of data to learn, develop, and improve performance. How might we leverage competency-based medical education (CBME) assessment data at the individual, program, and system levels, with the goal of redefining CBME from an initiative that supports the development of physicians to one that also fosters the development of the faculty, administrators, and programs within our organizations? In this paper we review the Deliberately Developmental Organization (DDO) framework proposed by Robert Kegan and Lisa Lahey, a theoretical framework that explains how organizations can foster the development of their people. We then describe the DDO's conceptual alignment with CBME and outline how CBME assessment data could be used to spur the transformation of health care and educational organizations into digitally integrated DDOs. A DDO-oriented use of CBME assessment data will require intentional investment into both the digitalization of assessment data and the development of the people within our organizations. By reframing CBME in this light, we hope that educational and health care leaders will see their investments in CBME as an opportunity to spur the evolution of a developmental culture.

On the validity of summative entrustment decisions.

Health care revolves around trust. Patients are often in a position that gives them no other choice than to trust the people taking care of them. Educational programs thus have the responsibility to develop physicians who can be trusted to deliver safe and effective care, ultimately making a final decision to entrust trainees to graduate to unsupervised practice. Such entrustment decisions deserve to be scrutinized for their validity. This end-of-training entrustment decision is arguably the most important one, although earlier entrustment decisions, for smaller units of professional practice, should also be scrutinized for their validity. Validity of entrustment decisions implies a defensible argument that can be analyzed in components that together support the decision. According to Kane, building a validity argument is a process designed to support inferences of scoring, generalization across observations, extrapolation to new instances, and implications of the decision. A lack of validity can be caused by inadequate evidence in terms of, according to Messick, content, response process, internal structure (coherence) and relationship to other variables, and in misinterpreted consequences. These two leading frameworks (Kane and Messick) in educational and psychological testing can be well applied to summative entrustment decision-making. The authors elaborate the types of questions that need to be answered to arrive at defensible, well-argued summative decisions regarding performance to provide a grounding for high-quality safe patient care.

Capturing outcomes of competency-based medical education: The call and the challenge.

There is an urgent need to capture the outcomes of the ongoing global implementation of competency-based medical education (CBME). However, the measurement of downstream outcomes following educational innovations, such as CBME is fraught with challenges stemming from the complexities of medical training, the breadth and variability of inputs, and the difficulties attributing outcomes to specific educational elements. In this article, we present a logic model for CBME to conceptualize an impact pathway relating to CBME and facilitate outcomes evaluation. We further identify six strategies to mitigate the challenges of outcomes measurement: (1) clearly identify the outcome of interest, (2) distinguish between outputs and outcomes, (3) carefully consider attribution versus contribution, (4) connect outcomes to the fidelity and integrity of implementation, (5) pay attention to unanticipated outcomes, and (6) embrace methodological pluralism. Embracing these challenges, we argue that careful and thoughtful evaluation strategies will move us forward in answering the all-important question: Are the desired outcomes of CBME being achieved?

Competency-based time-variable training internationally: Ensuring practical next steps in the wake of the COVID-19 pandemic.

Competency-based medical education has been advocated as the future of medical education for nearly a half-century. Inherent to this is the promise that advancement and transitions in training would be defined by readiness to practice rather than by time. Of the logistical problems facing competency-based, time-variable (CBTV) training, enacting time variability may be the largest hurdle to clear. Although it is true that an 'all or nothing' approach to CBTV training would require massive overhauls of both medical education and health care systems, the authors propose that training institutions should gradually evolve within their current environments to incrementally move toward the best version of CBTV training for learners, supervisors, and patients. In support of this evolution, the authors seek to demonstrate the feasibility of advancing toward the goal of realistic CBTV training by detailing examples of successful CBTV training and describing key features of initial steps toward CBTV training implementation.

Outcomes of competency-based medical education: A taxonomy for shared language.

As the global transformation of postgraduate medical training continues, there are persistent calls for program evaluation efforts to understand the impact and outcomes of competency-based medical education (CBME) implementation. The measurement of a complex educational intervention such as CBME is challenging because of the multifaceted nature of activities and outcomes. What is needed, therefore, is an organizational taxonomy to both conceptualize and categorize multiple outcomes. In this manuscript we propose a taxonomy that builds on preceding works to organize CBME outcomes across three domains: focus (educational, clinical), level (micro, meso, macro), and timeline (training, transition to practice, practice). We also provide examples of how to conceptualize outcomes of educational interventions across medical specialties using this taxonomy. By proposing a shared language for outcomes of CBME, we hope that this taxonomy will help organize ongoing evaluation work and catalyze those seeking to engage in the evaluation effort to help understand the impact and outcomes of CBME.