A realist evaluation of prospective entrustment decisions in paediatric residency clinical competency committees.

PURPOSE: Making entrustment decisions (granting more responsibility, advancement and graduation) are important actions in medical training that pose risks to trainees and patients if not done well. A previous realist synthesis of the existing literature revealed that clinical competency committees (CCCs) do not typically make deliberate entrustment decisions, instead defaulting to the promotion and graduation of trainees in the absence of red flags. This study sought further understanding of these areas through empirical data. METHODS: The authors conducted a realist inquiry to better understand how CCC prospective entrustment decision-making is carried out in paediatric residency programs. They conducted four CCC meeting observations and 18 interviews with CCC members at eight sites in an effort to confirm, disconfirm, and elaborate an existing theory that was based on a literature synthesis. RESULTS: The literature-based theory held up well against the empiric data collected in this study. Therefore, the authors did not modify that theory and instead developed three new demi-regularities (recurring patterns in data when conducting realist work) that add detail and nuance to their previous understanding of this model. These new demi-regularities focus on (i) expounding on how deliberate actions of CCCs focus more on resident development than on resident entrustment; (ii) elucidating that effortful work is not only about reconciling a paucity of data or incongruent data but also working hard to 'do the right thing' for residents; and (iii) describing how programs consider bias, equity and fairness, with a wide range of intentionality from being reactive to being proactive. CONCLUSION: This study offers evidence of deliberate CCC efforts to support resident development. Moving forward, a similar focus should be more consistently placed on equitable entrustment and advancement decisions to balance both of these foundational goals.

What Are We Made For? Mobilizing Medical Education Research for Impact.

ABSTRACT: During the past several decades, medical education research has advanced in many ways. However, the field has struggled somewhat with translating knowledge into practice. The field has tremendous potential to generate insights that may improve educational outcomes, enhance teaching experiences, reduce costs, promote equity, and inform policy. However, the gap between research and practice requires attention and reflection. In this commentary, the authors reflect on ways that medical education researchers can balance relevance and rigor, while discussing a potential path forward. First, medical education research can learn from implementation science, which focuses on adopting and sustaining best practices in real-world settings. Second, gaining a deeper understanding of the complex and dynamic ways that medical education contexts may influence the uptake of research findings into practice would facilitate the translation and mobilization of knowledge into practical settings. Third, moving from unilateral knowledge translation to participatory knowledge mobilization and engaging diverse stakeholders as active participants in the research process can also enhance impact and influence research findings. Overall, for medical education research to effect meaningful change, it must transition from producing generalizable findings to generating context-specific insights and embracing participatory knowledge mobilization. This shift will involve rethinking traditional research approaches and fostering collaboration with knowledge users to cocreate and implement innovative solutions tailored to their unique settings.

Competency-based medical education: Connecting training outcomes to patient care.

Competency-based medical education (CBME) is a patient-centered and learner-focused approach to education where curricula are delivered in a manner tailored to the individuals' learning needs, and assessment focuses on ensuring trainees achieve requisite and clearly specified learning outcomes. Despite calls to focus assessment on what matters for patients. In this article, the authors explore one aspect of this next era: the use of electronic health record clinical performance indicators, such as Resident-Sensitive Quality Measures (RSQMs) and TRainee Attributable and Automatable Care Evaluations in Real-time (TRACERs), for learner assessment. They elaborate on both the promise and the potential limitations of using such measures in a program of learner assessment.

Reimagining Initial Certification as a Process That Begins During Residency to Support Continuous Development Across Training and Practice.

ABSTRACT: In the United States, initial board certification remains focused on a high-stakes knowledge examination after completion of training. A more contemporary view supports a program of assessment that includes multiple types and sources of data with an emphasis on direct workplace observation to get the best picture of an individual's performance. In this article, the authors reimagine initial certification as a continuous assessment for learning that begins in residency, focuses on both knowledge acquisition and its application, and interdigitates intentionally with the first cycle of maintenance of certification to advance learning and smooth the transition from training to practice. A more expanded view of competence, as a 3-layered construct (canonical, contextual, and personalized), supports this proposal. Canonical competence (context-independent knowledge)-best assessed through examinations of knowledge-is most heavily weighted and assessed during medical school but remains critical throughout one's career. Contextual competence (context-dependent knowledge) is best assessed in the workplace and is key during residency and fellowship as trainees navigate a myriad of clinical work environments. Personalized competence, representing the totality of one's unique areas of expertise, is best demonstrated during the practice years when deliberate practice experience supports the growth of personalized expertise and discoveries that advance the field. Shifting initial board certification from relying on an anxiety-provoking, high-stakes, often single moment in time standardized examination to a nuanced approach that is part of an individual trainee's program of assessment offers a more just and robust decision about readiness for unsupervised practice. Such a model would also sow the seeds of meaningful individualization of learning needs that begins in training, continues through practice, and lays the foundation for improving the quality of care for patients within a given practice as well as shifting the current contiguous phases of training and practice into a true continuum.

Level of Supervision for the Entrustable Professional Activities Common to General Pediatrics and the Subspecialties Decreases from Residency to Fellowship.

OBJECTIVES: To compare level of supervision (LOS) ratings of graduating pediatric residents with their assessments as fellows for the five Entrustable Professional Activities (EPAs) common to general pediatrics and the subspecialties and to determine if the difference between ratings from residency to fellowship is less for the QI and Practice Management EPAs, since the skills needed to perform these may be less context-dependent. METHODS: We compared ratings of graduating residents with their assessments as fellows using LOS data from two sequential EPA studies. RESULTS: There were 65 ratings from 41 residents at the first fellow assessment. At graduation, most residents needed little to no supervision for all EPAs with 94% (61/65) of ratings level four or five. In contrast, only 5/65 (8%) of the first fellow assessments were level four or five. The ratings difference for the QI and Practice Management EPAs was similar to the others. CONCLUSIONS: LOS ratings for the EPAs common to generalists and subspecialists reset as residents become fellows. There was no evidence that the QI and Practice Management EPAs are less context-dependent. This study provides additional validity evidence for using these LOS scales to assess trainees in pediatric residency and fellowship.

Program Director Attitudes Toward Acceptance of Pediatric Dentistry Entrustable Professional Activities (EPAs).

Purpose: To acquire comments on pediatric dentistry entrustable professional activities (EPAs) from pediatric dentistry residency program directors (PDs). Methods: An electronic survey invited PDs to evaluate 16 previously developed EPAs on whether they were critical to patient safety, resident education, or both. PDs were asked to evaluate a fully developed EPA to assess structure and clarity and describe barriers to EPA. Descriptive statistics were completed. Results: Forty-one of 103 PDs completed the entire survey. Eighty-five percent (36 of 42) of PDs believed EPAs are critical to pediatric dentistry education, and 81 percent (34 of 42) believed EPAs are critical to patient safety. Eighty-one percent of PDs would likely use EPAs when available. Seventy-five percent (31 of 41) of PDs reported that they have had a resident who would have benefited from a longer duration of training. Conclusions: The majority of pediatric dentistry residency program director participants surveyed reported that entrustable professional activities are critical to patient safety and resident education. EPAs may be a valuable option for assessing residents' readiness for graduation.

How do validity experts conceptualise argumentation? It's a rhetorical question.

INTRODUCTION: Health professions education (HPE) has adopted the conceptualization of validity as an argument. However, the theoretical and practical aspects of how validity arguments should be developed, used and evaluated in HPE have not been deeply explored. Articulating the argumentation theory undergirding validity and validation can help HPE better operationalise validity as an argument. To better understand this, the authors explored how HPE validity scholars conceptualise assessment validity arguments and argumentation, seeking to understand potential consequences of these views on validation practices. METHODS: The authors used critical case sampling to identify HPE assessment validity experts in three ways: (1) participation in a prominent validity research group, (2) appearing in a bibliometric study of HPE validity publications and (3) authorship of recent HPE validity literature. Qualitative semi-structured interviews were conducted with 16 experts in HPE assessment validity from four different countries. The authors used reflexive thematic analysis to develop themes relevant to their research question. RESULTS: The authors developed three themes grounded in participants' responses: (1) In theory, HPE validity is a social and situated argument. (2) In practice, the absence of audience and evaluation stymies the social nature of HPE validity. (3) Lack of validity argumentation creates and maintains power differentials within HPE. Participants articulated that current HPE validation practices are rooted in post-positivist epistemology when they should be situated (i.e. context-dependent), audience-centric and inclusive. DISCUSSION: When discussing validity argumentation in theory, participants' descriptions reflect an interpretivist lens for evaluation that is misaligned with real-world validity practices. This misalignment likely arises from HPE's adoption of "validity as an argument" as a slogan, without integrating theoretical and practical principles of argumentation theory.

The Next Era of Assessment: Building a Trustworthy Assessment System.

Assessment in medical education has evolved through a sequence of eras each centering on distinct views and values. These eras include measurement (e.g., knowledge exams, objective structured clinical examinations), then judgments (e.g., workplace-based assessments, entrustable professional activities), and most recently systems or programmatic assessment, where over time multiple types and sources of data are collected and combined by competency committees to ensure individual learners are ready to progress to the next stage in their training. Significantly less attention has been paid to the social context of assessment, which has led to an overall erosion of trust in assessment by a variety of stakeholders including learners and frontline assessors. To meaningfully move forward, the authors assert that the reestablishment of trust should be foundational to the next era of assessment. In our actions and interventions, it is imperative that medical education leaders address and build trust in assessment at a systems level. To that end, the authors first review tenets on the social contextualization of assessment and its linkage to trust and discuss consequences should the current state of low trust continue. The authors then posit that trusting and trustworthy relationships can exist at individual as well as organizational and systems levels. Finally, the authors propose a framework to build trust at multiple levels in a future assessment system; one that invites and supports professional and human growth and has the potential to position assessment as a fundamental component of renegotiating the social contract between medical education and the health of the public.

Using Theory-Informed Group Processes to Make TIMELESS Promotion Decisions.

ABSTRACT: Competency-based medical education (CBME) depends on effective programs of assessment to achieve the desired outcomes and goals of training. Residency programs must be able to defend clinical competency committee (CCC) group decisions about learner readiness for practice, including decisions about time-variable resident promotion and graduation. In this article, the authors describe why CCC group decision-making processes should be supported by theory and review 3 theories they used in designing their group processes: social decision scheme theory, functional theory, and wisdom of crowds. They describe how these theories were applied in a competency-based, time-variable training pilot-Transitioning in Internal Medicine Education Leveraging Entrustment Scores Synthesis (TIMELESS) at the University of Cincinnati internal medicine residency program in 2020-2022-to increase the defensibility of their CCC group decision-making. This work serves as an example of how use of theory can bolster validity arguments supporting group decisions about resident readiness for practice.

Competency-based medical education: The spark to ignite healthcare's escape fire.

High-value care is what patients deserve and what healthcare professionals should deliver. However, it is not what happens much of the time. Quality improvement master Dr. Don Berwick argued more than two decades ago that American healthcare needs an escape fire, which is a new way of seeing and acting in a crisis situation. While coined in the U.S. context, the analogy applies in other Western healthcare contexts as well. Therefore, in this paper, the authors revisit Berwick's analogy, arguing that medical education can, and should, provide the spark for such an escape fire across the globe. They assert that medical education can achieve this by fully embracing competency-based medical education (CBME) as a way to place medicine's focus on the patient. CBME targets training outcomes that prepare graduates to optimize patient care. The authors use the escape fire analogy to argue that medical educators must drop long-held approaches and tools; treat CBME implementation as an adaptive challenge rather than a technical fix; demand genuine, rich discussions and engagement about the path forward; and, above all, center the patient in all they do.

Lessons From Organic Chemistry: The Case for Considering Both High Standards and Equity in Assessment.

ABSTRACT: In this commentary, the authors explore the tension of balancing high performance standards in medical education with the acceptability of those standards to stakeholders (e.g., learners and patients). The authors then offer a lens through which this tension might be considered and ways forward that focus on both patient outcomes and learner needs.In examining this phenomenon, the authors argue that high performance standards are often necessary. Societal accountability is key to medical education, with the public demanding that training programs prepare physicians to provide high-quality care. Medical schools and residency programs, therefore, require rigorous standards to ensure graduates are ready to care for patients. At the same time, learners' experience is important to consider. Making sure that performance standards are acceptable to stakeholders supports the validity of assessment decisions.Equity should also be central to program evaluation and validity arguments when considering performance standards. Currently, learners across the continuum are variably prepared for the next phase in training and often face inequities in resource availability to meet high passing standards, which may lead to learner attrition. Many students who face these inequities come from underrepresented or disadvantaged backgrounds and are essential to ensuring a diverse medical workforce to meet the needs of patients and society. When these students struggle, it contributes to the leaky pipeline of more socioeconomically and racially diverse applicants.The authors posit that 4 key factors can balance the tension between high performance standards and stakeholder acceptability: standards that are acceptable and defensible, progression that is time variable, requisite support structures that are uniquely tailored for each learner, and assessment systems that are equitably designed.

The Next Era of Assessment: Can Ensuring High-Quality, Equitable Patient Care Be the Defining Characteristic?

ABSTRACT: Previous eras of assessment in medical education have been defined by how assessment is done, from knowledge exams popularized in the 1960s to the emergence of work-based assessment in the 1990s to current efforts to integrate multiple types and sources of performance data through programmatic assessment. Each of these eras was a response to why assessment was performed (e.g., assessing medical knowledge with exams; assessing communication, professionalism, and systems competencies with work-based assessment). Despite the evolution of assessment eras, current evidence highlights the graduation of trainees with foundational gaps in the ability to provide high-quality care to patients presenting with common problems, and training program leaders report they graduate trainees they would not trust to care for themselves or their loved ones. In this article, the authors argue that the next era of assessment should be defined by why assessment is done: to ensure high-quality, equitable care. Assessment should place focus on demanding graduates possess the knowledge, skills, attitudes, and adaptive expertise to meet the needs of all patients and ensuring that graduates are able to do this in an equitable fashion. The authors explore 2 patient-focused assessment approaches that could help realize the promise of this envisioned era: entrustable professional activities (EPAs) and resident sensitive quality measures (RSQMs)/TRainee Attributable and Automatable Care Evaluations in Real-time (TRACERs). These examples illustrate how the envisioned next era of assessment can leverage existing and new data to provide precision education assessment that focuses on providing formative and summative feedback to trainees in a manner that seeks to ensure their learning outcomes prepare them to ensure high-quality, equitable patient outcomes.

Finding Medicine's Moneyball: How Lessons From Major League Baseball Can Advance Assessment in Precision Education.

ABSTRACT: Precision education (PE) leverages longitudinal data and analytics to tailor educational interventions to improve patient, learner, and system-level outcomes. At present, few programs in medical education can accomplish this goal as they must develop new data streams transformed by analytics to drive trainee learning and program improvement. Other professions, such as Major League Baseball (MLB), have already developed extremely sophisticated approaches to gathering large volumes of precise data points to inform assessment of individual performance.In this perspective, the authors argue that medical education-whose entry into precision assessment is fairly nascent-can look to MLB to learn the possibilities and pitfalls of precision assessment strategies. They describe 3 epochs of player assessment in MLB: observation, analytics (sabermetrics), and technology (Statcast). The longest tenured approach, observation, relies on scouting and expert opinion. Sabermetrics brought new approaches to analyzing existing data in a way that better predicted which players would help the team win. Statcast created precise, granular data about highly attributable elements of player performance while helping to account for nonplayer factors that confound assessment such as weather, ballpark dimensions, and the performance of other players. Medical education is progressing through similar epochs marked by workplace-based assessment, learning analytics, and novel measurement technologies. The authors explore how medical education can leverage intersectional concepts of MLB player and medical trainee assessment to inform present and future directions of PE.

A realist synthesis of prospective entrustment decision making by entrustment or clinical competency committees.

INTRODUCTION: The real-world mechanisms underlying prospective entrustment decision making (PEDM) by entrustment or clinical competency committees (E/CCCs) are poorly understood. To advance understanding in this area, the authors conducted a realist synthesis of the published literature to address the following research question: In E/CCC efforts to make defensible prospective entrustment decisions (PEDs), what works, for whom, under what circumstances and why? METHODS: Realist work seeks to understand the contexts (C), mechanisms (M) and outcomes (O) that explain how and why things work (or do not). In the authors' study, contexts included individual E/CCC members, E/CCC structures and processes, and training programmes. The outcome (i.e. desired outcome) was a PED. Mechanisms were a substantial focus of the analysis and informed the core findings. To define a final corpus of 52 included papers, the authors searched four databases, screened all results from those searches and performed a full-text review of a subset of screened papers. Data extraction focused on developing context-mechanism-outcome configurations from the papers, which were used to create a theory for how PEDM leads to PEDs. RESULTS: PEDM is often driven by default (non-deliberate) decision making rather than a deliberate process of deciding whether a trainee should be entrusted or not. When defaulting, some E/CCCs find red flags that sometimes lead to being more deliberate with decision making. E/CCCs that seek to be deliberate describe PEDM that can be effortful (when data are insufficient or incongruent) or effortless (when data are robust and tell a congruent story about a trainee). Both information about trainee trustworthiness and the sufficiency of data about trainee performance influence PEDM. Several moderators influence what is considered to be sufficient data, how trustworthiness data are viewed and how PEDM is carried out. These include perceived consequences and associated risks, E/CCC member trust propensity, E/CCC member personal knowledge of and experience with trainees and E/CCC structures and processes. DISCUSSION: PEDM is rarely deliberate but should be. Data about trainee trustworthiness are foundational to making PEDs. Bias, equity and fairness are nearly absent from the papers in this synthesis, and future efforts must seek to advance understanding and practice regarding the roles of bias, equity and fairness in PEDM.

AI-Enabled Medical Education: Threads of Change, Promising Futures, and Risky Realities Across Four Potential Future Worlds.

BACKGROUND: The rapid trajectory of artificial intelligence (AI) development and advancement is quickly outpacing society's ability to determine its future role. As AI continues to transform various aspects of our lives, one critical question arises for medical education: what will be the nature of education, teaching, and learning in a future world where the acquisition, retention, and application of knowledge in the traditional sense are fundamentally altered by AI? OBJECTIVE: The purpose of this perspective is to plan for the intersection of health care and medical education in the future. METHODS: We used GPT-4 and scenario-based strategic planning techniques to craft 4 hypothetical future worlds influenced by AI's integration into health care and medical education. This method, used by organizations such as Shell and the Accreditation Council for Graduate Medical Education, assesses readiness for alternative futures and effectively manages uncertainty, risk, and opportunity. The detailed scenarios provide insights into potential environments the medical profession may face and lay the foundation for hypothesis generation and idea-building regarding responsible AI implementation. RESULTS: The following 4 worlds were created using OpenAI's GPT model: AI Harmony, AI conflict, The world of Ecological Balance, and Existential Risk. Risks include disinformation and misinformation, loss of privacy, widening inequity, erosion of human autonomy, and ethical dilemmas. Benefits involve improved efficiency, personalized interventions, enhanced collaboration, early detection, and accelerated research. CONCLUSIONS: To ensure responsible AI use, the authors suggest focusing on 3 key areas: developing a robust ethical framework, fostering interdisciplinary collaboration, and investing in education and training. A strong ethical framework emphasizes patient safety, privacy, and autonomy while promoting equity and inclusivity. Interdisciplinary collaboration encourages cooperation among various experts in developing and implementing AI technologies, ensuring that they address the complex needs and challenges in health care and medical education. Investing in education and training prepares professionals and trainees with necessary skills and knowledge to effectively use and critically evaluate AI technologies. The integration of AI in health care and medical education presents a critical juncture between transformative advancements and significant risks. By working together to address both immediate and long-term risks and consequences, we can ensure that AI integration leads to a more equitable, sustainable, and prosperous future for both health care and medical education. As we engage with AI technologies, our collective actions will ultimately determine the state of the future of health care and medical education to harness AI's power while ensuring the safety and well-being of humanity.