In EPAs we trust, is quality and safety a must? A cross-specialty analysis of entrustable professional activity guides.

PURPOSE: The inclusion of quality improvement (QI) and patient safety (PS) into CanMEDS reflects an expectation that graduating physicians are competent in these areas upon training completion. To ensure that Canadian postgraduate specialty training achieves this, the translation of QI/PS competencies into training standards as part of the implementation of competency-based medical education requires special attention. METHODS: We conducted a cross-specialty, multi-method analysis to examine how QI/PS was incorporated into the EPA Guides across 11 postgraduate specialties in Canada. RESULTS: We identify cross-specialty variability in how QI/PS is incorporated, positioned, and emphasized in EPAs and milestones. QI/PS was primarily referenced alongside clinical activities rather than as a sole competency or discrete activity. Patterns were characterized in how QI/PS became incorporated into milestones through repetition and customization. QI/PS was also decoupled, conceptualized, and emphasized differently across specialties. CONCLUSIONS: Variability in the inclusion of QI/PS in EPAs and milestones has important implications considering the visibility and influence of EPA Guides in practice. As specialties revisit and revise EPA Guides, there is a need to balance the standardization of foundational QI/PS concepts to foster shared understanding while simultaneously ensuring context-sensitive applications across specialties. Beyond QI/PS, this study illuminates the challenges and opportunities that lie in bridging theoretical frameworks with practical implementation in medical education, prompting broader consideration of how intrinsic roles and emergent areas are effectively incorporated into competency-based medical education.

Assessing the utility of a novel entrustment-supervision assessment tool.

BACKGROUND: Work-based assessments (WBAs) are increasingly used to inform decisions about trainee progression. Unfortunately, WBAs often fail to discriminate between trainees of differing abilities and have poor reliability. Entrustment-supervision scales may improve WBA performance, but there is a paucity of literature directly comparing them to traditional WBA tools. METHODS: The Ottawa Emergency Department Shift Observation Tool (O-EDShOT) is a previously published WBA tool employing an entrustment-supervision scale with strong validity evidence. This pre-/post-implementation study compares the performance of the O-EDShOT with that of a traditional WBA tool using norm-based anchors. All assessments completed in 12-month periods before and after implementing the O-EDShOT were collected, and generalisability analysis was conducted with year of training, trainees within year and forms within trainee as nested factors. Secondary analysis included assessor as a factor. RESULTS: A total of 3908 and 3679 assessments were completed by 99 and 116 assessors, for 152 and 138 trainees in the pre- and post-implementation phases respectively. The O-EDShOT generated a wider range of awarded scores than the traditional WBA, and mean scores increased more with increasing level of training (0.32 vs. 0.14 points per year, p = 0.01). A significantly greater proportion of overall score variability was attributable to trainees using the O-EDShOT (59%) compared with the traditional tool (21%, p < 0.001). Assessors contributed less to overall score variability for the O-EDShOT than for the traditional WBA (16% vs. 37%). Moreover, the O-EDShOT required fewer completed assessments than the traditional tool (27 vs. 51) for a reliability of 0.8. CONCLUSION: The O-EDShOT outperformed a traditional norm-referenced WBA in discriminating between trainees and required fewer assessments to generate a reliable estimate of trainee performance. More broadly, this study adds to the body of literature suggesting that entrustment-supervision scales generate more useful and reliable assessments in a variety of clinical settings.

The voices of medical education scholarship: Describing the published landscape.

INTRODUCTION: The voices of authors who publish medical education literature have a powerful impact on the field's discourses. Researchers have identified a lack of author diversity, which suggests potential epistemic injustice. This study investigates author characteristics to provide an evidence-based starting point for communal discussion with the intent to move medical education towards a future that holds space for, and values, diverse ways of knowing. METHOD: The authors conducted a bibliometric analysis of all articles published in 24 medical education journals published between 2000 and 2020 to identify author characteristics, with an emphasis on author gender and geographic location and their intersection. Article metadata was downloaded from Web of Science. Genderize.io was used to predict author gender. RESULTS: The journals published 37 263 articles authored by 62 708 unique authors. Males were more prevalent across all authorship positions (n = 62 828; 55.7%) than females (n = 49 975; 44.3%). Authors listed affiliations in 146 countries of which 95 were classified as Global South. Few articles were written by multinational teams (n = 3765; 16.2%). Global South authors accounted for 12 007 (11.4%) author positions of which 3594 (3.8%) were female. DISCUSSION: This study provides an evidence-based starting point to discuss the imbalance of author voices in medical education, especially when considering the intersection of gender and geographical location, which further suggests epistemic injustice in medical education. If the field values a diversity of perspectives, there is considerable opportunity for improvement by engaging the community in discussions about what knowledge matters in medical education, the role of journals in promoting diversity, how to best use this baseline data and how to continue studying epistemic injustice in medical education.

Stress and conflict from tacit culture forges professional identity in newly graduated independent physicians.

To transition successfully into independent practice, newly graduated independent physicians (new "attendings") undergo a process of professional identity formation (PIF) as a clinician within a new community of practice (CoP). PIF is crafted by socialization within a CoP including transfer of tacit knowledge. While certain tacit knowledge is critical for professional identity, we understand little how it shapes PIF. We set out to describe the tacit knowledge acquired by new attendings within a CoP and how it contributes to PIF. Informed by constructivist grounded theory, we interviewed 23 new attendings about the tacit knowledge they acquired in early practice. Data collection and analysis occurred iteratively. We identified themes using constant comparative analysis and generated a theory that underwent member checking and feedback. Implicit standards from group culture imparted high expectations on new attendings and led to internal stress. New attendings also encountered a tacit code of conduct as behavioral elements of group culture. These elements created external conflict between new attendings and group members such as departmental colleagues, consulting physicians, and other health professionals. Depending on the support they received, new attendings responded to the stress and conflict in three ways: they doubted, adjusted, or avoided. These strategies molded their professional identity, and moved them towards or away from the CoP as they navigated their transition and PIF. We describe a novel theory of how tacit group culture shaped new attending physicians' professional identity in a new community of practice. Internal stress and external conflict occurred due to high expectations and tacit culture elements. New attendings' doubt, adjust, or avoid responses, shaped by support they received, in turn crafted their professional identity. Education leaders should prepare graduating trainees to navigate aspects of transition to independent practice successfully.

Delineating the field of medical education: Bibliometric research approach(es).

BACKGROUND: The field of medical education remains poorly delineated such that there is no broad consensus of articles or journals that comprise 'the field'. This lack of consensus indicates a missed opportunity for researchers to generate insights about the field that could facilitate conducting bibliometric studies and other research designs (e.g., systematic reviews) and also enable individuals to identify themselves as 'medical education researchers'. Other fields have utilised bibliometric field delineation, which is the assigning of articles or journals to a certain field in an effort to define that field. PROCESS: In this Research Approach, three bibliometric field delineation approaches-information retrieval, core journals, and journal co-citation-are introduced. For each approach, the authors describe attempts to apply it in medical education and identify related strengths and weaknesses. Based on co-citation, the authors propose the Medical Education Journal List 24 (MEJ-24), as a starting point for delineating medical education and invite the community to collaborate on improving and potentially expanding this list. PEARLS: As a research approach, field delineation is complicated, and there is no clear best way to delineate the field of medical education. However, recent advances in information science provide potentially fruitful approaches to deal with the field's complexity. When considering these approaches, researchers should consider collaborating with bibliometricians. Bibliometric approaches rely on available metadata for articles and journals, which necessitates that researchers examine the metadata prior to analysis to understand its strengths and weaknesses, and to assess how this might affect data interpretation. While using bibliometric approaches for field delineation is valuable, it is important to remember that these techniques are only as good as the research team's interpretation of the data, which suggests that an expanded approach is needed to better delineate medical education, an approach that includes active discussion within the medical education community.

Implementation of competence committees during the transition to CBME in Canada: A national fidelity-focused evaluation.

PURPOSE: This study evaluated the fidelity of competence committee (CC) implementation in Canadian postgraduate specialist training programs during the transition to competency-based medical education (CBME). METHODS: A national survey of CC chairs was distributed to all CBME training programs in November 2019. Survey questions were derived from guiding documents published by the Royal College of Physicians and Surgeons of Canada reflecting intended processes and design. RESULTS: Response rate was 39% (113/293) with representation from all eligible disciplines. Committee size ranged from 3 to 20 members, 42% of programs included external members, and 20% included a resident representative. Most programs (72%) reported that a primary review and synthesis of resident assessment data occurs prior to the meeting, with some data reviewed collectively during meetings. When determining entrustable professional activity (EPA) achievement, most programs followed the national specialty guidelines closely with some exceptions (53%). Documented concerns about professionalism, EPA narrative comments, and EPA entrustment scores were most highly weighted when determining resident progress decisions. CONCLUSIONS: Heterogeneity in CC implementation likely reflects local adaptations, but may also explain some of the variable challenges faced by programs during the transition to CBME. Our results offer educational leaders important fidelity data that can help inform the larger evaluation and transformation of CBME.

Ready, set, go! Evaluating readiness to implement competency-based medical education.

PURPOSE: Organizational readiness is critical for successful implementation of an innovation. We evaluated program readiness to implement Competence by Design (CBD), a model of Competency-Based Medical Education (CBME), among Canadian postgraduate training programs. METHODS: A survey of program directors was distributed 1 month prior to CBD implementation in 2019. Questions were informed by the R = MC2 framework of organizational readiness and addressed: program motivation, general capacity for change, and innovation-specific capacity. An overall readiness score was calculated. An ANOVA was conducted to compare overall readiness between disciplines. RESULTS: Survey response rate was 42% (n = 79). The mean overall readiness score was 74% (30-98%). There was no difference in scores between disciplines. The majority of respondents agreed that successful implementation of CBD was a priority (74%), and that their leadership (94%) and faculty and residents (87%) were supportive of change. Fewer perceived that CBD was a move in the right direction (58%) and that implementation was a manageable change (53%). Curriculum mapping, competence committees and programmatic assessment activities were completed by >90% of programs, while <50% had engaged off-service disciplines. CONCLUSION: Our study highlights important areas where programs excelled in their preparation for CBD, as well as common challenges that serve as targets for future intervention to improve program readiness for CBD implementation.

"Glocalization" in medical education: A framework underlying implementing CBME in a local context.

BACKGROUND/PURPOSE: The implementation of competency-based medical education is a social construction process within a local and cultural context. However, little is known about the process of adaptation to different systems, known as "glocalization". We analyzed the documents in the development of a milestone project from adapting global standards into a local context and identified a framework underlying this process. METHODS: Taiwan Society of Emergency Medicine (TSEM) had developed learning milestones based on the ACGME's version through series of consensus methods including committee work, nominal group technique (NGT), and a modified Delphi method. We applied qualitative content analysis to characterize the evolution of the three versions of TSEM and the original ACGME milestones documents and to explore the meaning behind the differences revealed by the glocalization process. RESULTS: We found 48 differences between ACGME and TSEM milestones. Among these differences, one was made by committee work, 44 came from NGT, and 3 were from the modified Delphi process. Two themes and seven sub-themes emerged from the coding process to explain the contextualization process of the milestones. CONCLUSION: We identified a framework that incorporates local expression and local needs into the process called glocalization through which global models of competency-based standards could be optimally implemented in a local context with different systems and cultures.

Does the public feel prepared to be substitute decision-makers? Gaps in preparedness and support for a high school curriculum: a national survey.

BACKGROUND: Substitute decision-makers (SDMs) make decisions on behalf of patients who do not have capacity, in line with previously expressed wishes, values and beliefs. However, miscommunications and poor awareness of previous wishes often lead to inappropriate care. Increasing public preparedness to communicate on behalf of loved ones may improve care in patients requiring an SDM. METHODS: We conducted an online survey in January 2019 with a representative sample of the Canadian population. The primary outcome was self-reported preparedness to be an SDM. The secondary outcome was support for a high school curriculum on the role of SDMs. The effect of socio-demographics, known enablers and barriers to acting as an SDM, and attitudes towards a high school curriculum were assessed using multivariate analysis. RESULTS: Of 1,000 participants, 53.1% felt prepared to be an SDM, and 75.4% stated they understood their loved one's values. However, only 55.6% reported having had a meaningful conversation with their loved one about values and wishes, and only 61.7% reported understanding the SDM role. Engagement in advance care planning for oneself was low (23.1%). Age, experience, training and comfort with communication were associated with preparedness in our multivariate analysis. A high school curriculum was supported by 61.1% of respondents, with 28.3% neutral and 10.6% against it. INTERPRETATION: There is a gap between perceived and actual preparedness to be an SDM. Many report understanding their loved one's values yet have not asked them about wishes in illness or end of life. The majority of respondents support high school education to improve preparedness.

Capturing outcomes of competency-based medical education: The call and the challenge.

There is an urgent need to capture the outcomes of the ongoing global implementation of competency-based medical education (CBME). However, the measurement of downstream outcomes following educational innovations, such as CBME is fraught with challenges stemming from the complexities of medical training, the breadth and variability of inputs, and the difficulties attributing outcomes to specific educational elements. In this article, we present a logic model for CBME to conceptualize an impact pathway relating to CBME and facilitate outcomes evaluation. We further identify six strategies to mitigate the challenges of outcomes measurement: (1) clearly identify the outcome of interest, (2) distinguish between outputs and outcomes, (3) carefully consider attribution versus contribution, (4) connect outcomes to the fidelity and integrity of implementation, (5) pay attention to unanticipated outcomes, and (6) embrace methodological pluralism. Embracing these challenges, we argue that careful and thoughtful evaluation strategies will move us forward in answering the all-important question: Are the desired outcomes of CBME being achieved?

Outcomes of competency-based medical education: A taxonomy for shared language.

As the global transformation of postgraduate medical training continues, there are persistent calls for program evaluation efforts to understand the impact and outcomes of competency-based medical education (CBME) implementation. The measurement of a complex educational intervention such as CBME is challenging because of the multifaceted nature of activities and outcomes. What is needed, therefore, is an organizational taxonomy to both conceptualize and categorize multiple outcomes. In this manuscript we propose a taxonomy that builds on preceding works to organize CBME outcomes across three domains: focus (educational, clinical), level (micro, meso, macro), and timeline (training, transition to practice, practice). We also provide examples of how to conceptualize outcomes of educational interventions across medical specialties using this taxonomy. By proposing a shared language for outcomes of CBME, we hope that this taxonomy will help organize ongoing evaluation work and catalyze those seeking to engage in the evaluation effort to help understand the impact and outcomes of CBME.

Questioning medical competence: Should the Covid-19 crisis affect the goals of medical education?

The COVID-19 pandemic has disrupted many societal institutions, including health care and education. Although the pandemic's impact was initially assumed to be temporary, there is growing conviction that medical education might change more permanently. The International Competency-based Medical Education (ICBME) collaborators, scholars devoted to improving physician training, deliberated how the pandemic raises questions about medical competence. We formulated 12 broad-reaching issues for discussion, grouped into micro-, meso-, and macro-level questions. At the individual micro level, we ask questions about adaptability, coping with uncertainty, and the value and limitations of clinical courage. At the institutional meso level, we question whether curricula could include more than core entrustable professional activities (EPAs) and focus on individualized, dynamic, and adaptable portfolios of EPAs that, at any moment, reflect current competence and preparedness for disasters. At the regulatory and societal macro level, should conditions for licensing be reconsidered? Should rules of liability be adapted to match the need for rapid redeployment? We do not propose a blueprint for the future of medical training but rather aim to provoke discussions needed to build a workforce that is competent to cope with future health care crises.

Training disrupted: Practical tips for supporting competency-based medical education during the COVID-19 pandemic.

The COVID-19 pandemic has disrupted healthcare systems around the world, impacting how we deliver medical education. The normal day-to-day routines have been altered for a number of reasons, including changes to scheduled training rotations, physical distancing requirements, trainee redeployment, and heightened level of concern. Medical educators will likely need to adapt their programs to maximize learning, maintain effective care delivery, and ensure competent graduates. Along with a continued focus on learner/faculty wellness, medical educators will have to optimize existing training experiences, adapt those that are no longer viable, employ new technologies, and be flexible when assessing competencies. These practical tips offer guidance on how to adapt medical education programs within the constraints of the pandemic landscape, stressing the need for communication, innovation, collaboration, flexibility, and planning within the era of competency-based medical education.

A "fit for purpose" framework for medical education accreditation system design.

BACKGROUND: Accreditation is a key feature of many medical education systems, helping to ensure that programs teach and assess learners according to applicable standards, provide optimal learning environments, and produce professionals who are competent to practise in challenging and evolving health care systems. Although most medical education accreditation systems apply similar standards domains and process elements, there can be substantial variation among accreditation systems at the level of design and implementation. A discussion group at the 2013 World Summit on Outcomes-Based Accreditation examined best practices in health professional education accreditation systems and identified that the literature examining the effectiveness of different approaches to accreditation is scant. Although some frameworks for accreditation design do exist, they are often specific to one phase of the medical education continuum. MAIN TEXT: This paper attempts to define a framework for the operational design of medical education accreditation that articulates design options as well as their contextual and practical implications. It assumes there is no single set of best practices in accreditation system development but, rather, an underlying set of design decisions. A "fit for purpose" approach aims to ensure that a system, policy, or program is designed and operationalized in a manner best suited to local needs and contexts. This approach is aligned with emerging models for education and international development that espouse decentralization. CONCLUSION: The framework highlights that, rather than a single best practice, variation among accreditation systems is appropriate provided that is it tailored to the needs of local contexts. Our framework is intended to provide guidance to administrators, policy-makers, and educators regarding different approaches to medical education accreditation and their applicability and appropriateness in local contexts.

The role of accreditation in 21st century health professions education: report of an International Consensus Group.

BACKGROUND: Accreditation is considered an essential ingredient for an effective system of health professions education (HPE) globally. While accreditation systems exist in various forms worldwide, there has been little written about the contemporary enterprise of accreditation and even less about its role in improving health care outcomes. We set out to 1) identify a global, contemporary definition of accreditation in the health professions, 2) describe the relationship of educational accreditation to health care outcomes, 3) identify important questions and recurring issues in twenty-first century HPE accreditation, and 4) propose a framework of essential ingredients in present-day HPE accreditation. METHODS: We identified health professions accreditation leaders via a literature search and a Google search of HPE institutions, as well as by accessing the networks of other leaders. These leaders were invited to join an international consensus consortium to advance the scholarship and thinking about HPE accreditation. We describe the consensus findings from the International Health Professions Accreditation Outcomes Consortium (IHPAOC). RESULTS: We define accreditation as the process of formal evaluation of an educational program, institution, or system against defined standards by an external body for the purposes of quality assurance and enhancement. In the context of HPE, accreditation is distinct from other forms of program evaluation or research. Accreditation can enhance health care outcomes because of its ability to influence and standardize the quality of training programs, continuously enhance curriculum to align with population needs, and improve learning environments. We describe ten fundamental and recurring elements of accreditation systems commonly found in HPE and provide an overview of five emerging developments in accreditation in the health professions based on the consensus findings. CONCLUSIONS: Accreditation has taken on greater importance in contemporary HPE. These consensus findings provide frameworks of core elements of accreditation systems and both recurring and emerging design issues. HPE scholars, educators, and leaders can build on these frameworks to advance research, development, and operation of high-quality accreditation systems worldwide.

Resident duty hours: Families' knowledge and perceptions in the paediatric intensive care unit.

BACKGROUND AND OBJECTIVES: Resident duty hours remain a controversial topic in the literature. Competing interests include patient safety, resident education, and resident well-being. No studies, however, have sought family members' perspectives on duty hours in the paediatric context. This study aimed to explore family members' knowledge of trainee duty hours, and their perspectives on the balance between shift duration and hand-off frequency. METHODS: We surveyed family members of patients admitted ≥ 24 hours in the paediatric intensive care unit at an academic center. We simultaneously collected daily logs of hours worked by trainees. Descriptive statistics were used to analyze survey responses and trainee duty hours. RESULTS: One-hundred and one family members responded (75%). Respondents demonstrated knowledge of trainees working long duty hours but reported lower averages than the trainee logs (55 versus 66 hours per week and 16 versus 24 hours per shift). Elements related to both potential trainee fatigue and hand-offs raised concern in more than half of respondents. When asked to choose between a familiar trainee working a prolonged shift, or an unfamiliar trainee at the start of their shift, respondents were divided (52% versus 48%, respectively). CONCLUSIONS: Family members of critically ill paediatric patients are aware that trainees provide patient care while working long duty hours with minimal sleep. Despite this awareness, long shifts retain value with some families, possibly due to continuity. Changes to duty hours and hand-off frequency may pose an unrealized harm on family-centered care, as well as patient-provider relationships, and further study is warranted.

Entrustment Checkpoint: Clinical Supervisors' Perceptions of the Emergency Department Oral Case Presentation.

Phenomenon: The oral case presentation represents a unique method of communication and forms the foundation for trainee-supervisor interactions in the clinical setting. Recently, entrustment has been highlighted as an essential element of trainee-supervisor interactions. Despite the growing body of knowledge concerning entrustment in medical education, how supervisors conceptualize the oral case presentation as a contributor to entrustment decision making during clinical supervision remains unknown. Given their widespread use, oral case presentations may represent a potential tool for future frameworks of workplace-based assessment. This study sought to explore what factors influence supervisors' expectations of oral case presentation content and how the oral case presentation may contribute to entrustment decision making. Approach: Using qualitative methodology, semistructured interviews were conducted from a purposive sample of attending emergency medicine physicians at an academic medical center from 2015 to 2016. Thematic analysis of the semistructured interview transcripts was conducted by 2 investigators using line-by-line coding and constant comparative analysis. Key themes were identified through consensus. Theoretical sampling occurred until thematic saturation was reached. Findings: Twenty-one attending physicians were interviewed. Four factors were found to influence supervisor expectations pertaining to oral case presentation content (trainee level, trainee familiarity, clinical context, and clinical task). Further, the oral case presentation was found to serve as a means of indirect observation and an entrustment check point informing future decisions relating to trainee supervision. Insights: The oral case presentation represents a core activity within the trainee-supervisor relationship in which entrustment plays a central role. Given the numerous factors influencing oral case presentation content, we caution supervisors against relying solely on the oral case presentation as an entrustment check point, as this may lead to inaccurate judgments of trainee competence. We recommend that the oral case presentation be used in conjunction with other means of direct and indirect observation to assist with entrustment decisions relating to trainee supervision.

Supervisor-trainee continuity and the quality of work-based assessments.

CONTEXT: Work-based assessments (WBAs) represent an increasingly important means of reporting expert judgements of trainee competence in clinical practice. However, the quality of WBAs completed by clinical supervisors is of concern. The episodic and fragmented interaction that often occurs between supervisors and trainees has been proposed as a barrier to the completion of high-quality WBAs. OBJECTIVES: The primary purpose of this study was to determine the effect of supervisor-trainee continuity on the quality of assessments documented on daily encounter cards (DECs), a common form of WBA. The relationship between trainee performance and DEC quality was also examined. METHODS: Daily encounter cards representing three differing degrees of supervisor-trainee continuity (low, intermediate, high) were scored by two raters using the Completed Clinical Evaluation Report Rating (CCERR), a previously published nine-item quantitative measure of DEC quality. An analysis of variance (anova) was performed to compare mean CCERR scores among the three groups. Linear regression analysis was conducted to examine the relationship between resident performance and DEC quality. RESULTS: Differences in mean CCERR scores were observed between the three continuity groups (p = 0.02); however, the magnitude of the absolute differences was small (partial eta-squared = 0.03) and not educationally meaningful. Linear regression analysis demonstrated a significant inverse relationship between resident performance and CCERR score (p < 0.001, r2  = 0.18). This inverse relationship was observed in both groups representing on-service residents (p = 0.001, r2  = 0.25; p = 0.04, r2  = 0.19), but not in the Off-service group (p = 0.62, r2  = 0.05). CONCLUSIONS: Supervisor-trainee continuity did not have an educationally meaningful influence on the quality of assessments documented on DECs. However, resident performance was found to affect assessor behaviours in the On-service group, whereas DEC quality remained poor regardless of performance in the Off-service group. The findings suggest that greater attention should be given to determining ways of improving the quality of assessments reported for off-service residents, as well as for those residents demonstrating appropriate clinical competence progression.

Implementing competency-based medical education: Moving forward.

For more than 60 years, competency-based education has been proposed as an approach to education in many disciplines. In medical education, interest in CBME has grown dramatically in the last decade. This editorial introduces a series of papers that resulted from summits held in 2013 and 2016 by the International CBME Collaborators, a scholarly network whose members are interested in developing competency-based approaches to preparing the next generation of health professionals. An overview of the papers is given, as well as a summary of landmarks in the conceptual evolution and implementation of CBME. This series follows on a first collection of papers published by the International CBME Collaborators in Medical Teacher in 2010.