Competence By Design: a transformational national model of time-variable competency-based postgraduate medical education.

Postgraduate medical education is an essential societal enterprise that prepares highly skilled physicians for the health workforce. In recent years, PGME systems have been criticized worldwide for problems with variable graduate abilities, concerns about patient safety, and issues with teaching and assessment methods. In response, competency based medical education approaches, with an emphasis on graduate outcomes, have been proposed as the direction for 21st century health profession education. However, there are few published models of large-scale implementation of these approaches. We describe the rationale and design for a national, time-variable competency-based multi-specialty system for postgraduate medical education called Competence by Design. Fourteen innovations were bundled to create this new system, using the Van Melle Core Components of competency based medical education as the basis for the transformation. The successful execution of this transformational training system shows competency based medical education can be implemented at scale. The lessons learned in the early implementation of Competence by Design can inform competency based medical education innovation efforts across professions worldwide.

In EPAs we trust, is quality and safety a must? A cross-specialty analysis of entrustable professional activity guides.

PURPOSE: The inclusion of quality improvement (QI) and patient safety (PS) into CanMEDS reflects an expectation that graduating physicians are competent in these areas upon training completion. To ensure that Canadian postgraduate specialty training achieves this, the translation of QI/PS competencies into training standards as part of the implementation of competency-based medical education requires special attention. METHODS: We conducted a cross-specialty, multi-method analysis to examine how QI/PS was incorporated into the EPA Guides across 11 postgraduate specialties in Canada. RESULTS: We identify cross-specialty variability in how QI/PS is incorporated, positioned, and emphasized in EPAs and milestones. QI/PS was primarily referenced alongside clinical activities rather than as a sole competency or discrete activity. Patterns were characterized in how QI/PS became incorporated into milestones through repetition and customization. QI/PS was also decoupled, conceptualized, and emphasized differently across specialties. CONCLUSIONS: Variability in the inclusion of QI/PS in EPAs and milestones has important implications considering the visibility and influence of EPA Guides in practice. As specialties revisit and revise EPA Guides, there is a need to balance the standardization of foundational QI/PS concepts to foster shared understanding while simultaneously ensuring context-sensitive applications across specialties. Beyond QI/PS, this study illuminates the challenges and opportunities that lie in bridging theoretical frameworks with practical implementation in medical education, prompting broader consideration of how intrinsic roles and emergent areas are effectively incorporated into competency-based medical education.

Competency-Based Medical Education at Scale: A Road Map for Transforming National Systems of Postgraduate Medical Education.

In the past decade, the Canadian system of postgraduate medical education has been transformed with the implementation of a new approach to competency based medical education called Competence by Design. The Royal College of Physicians and Surgeons of Canada (Royal College) developed an approach to time-variable competency based medical education and adapted that design for medical, surgical, and diagnostic disciplines. New educational standards and entrustable professional activities consistent with this approach were co-created with 67 specialties and subspecialties, and implementation was scaled up across 17 universities and over 1000 postgraduate training programs. Partner engagement, systematic design of workshops to create discipline specific competency-based standards of education, and agile adaptation were all key ingredients for success. This paper describes the strategies applied by the Royal College, lessons learned regarding transformative change in the complex system of postgraduate medical education, and the current status of the Competence by Design initiative. The approach taken and lessons learned by the Royal College may be useful for other educators who are planning a transformation to CBME or any other major educational reform.

Implementing Competence Committees on a National Scale: Design and Lessons Learned.

Competence committees (CCs) are a recent innovation to improve assessment decision-making in health professions education. CCs enable a group of trained, dedicated educators to review a portfolio of observations about a learner's progress toward competence and make systematic assessment decisions. CCs are aligned with competency based medical education (CBME) and programmatic assessment. While there is an emerging literature on CCs, little has been published on their system-wide implementation. National-scale implementation of CCs is complex, owing to the culture change that underlies this shift in assessment paradigm and the logistics and skills needed to enable it. We present the Royal College of Physicians and Surgeons of Canada's experience implementing a national CC model, the challenges the Royal College faced, and some strategies to address them. With large scale CC implementation, managing the tension between standardization and flexibility is a fundamental issue that needs to be anticipated and addressed, with careful consideration of individual program needs, resources, and engagement of invested groups. If implementation is to take place in a wide variety of contexts, an approach that uses multiple engagement and communication strategies to allow for local adaptations is needed. Large-scale implementation of CCs, like any transformative initiative, does not occur at a single point but is an evolutionary process requiring both upfront resources and ongoing support. As such, it is important to consider embedding a plan for program evaluation at the outset. We hope these shared lessons will be of value to other educators who are considering a large-scale CBME CC implementation.

Evaluating Competence by Design as a Large System Change Initiative: Readiness, Fidelity, and Outcomes.

Program evaluation is an essential, but often neglected, activity in any transformational educational change. Competence by Design was a large-scale change initiative to implement a competency-based time-variable educational system in Canadian postgraduate medical education. A program evaluation strategy was an integral part of the build and implementation plan for CBD from the beginning, providing insights into implementation progress, challenges, unexpected outcomes, and impact. The Competence by Design program evaluation strategy was built upon a logic model and three pillars of evaluation: readiness to implement, fidelity and integrity of implementation, and outcomes of implementation. The program evaluation strategy harvested from both internally driven studies and those performed by partners and invested others. A dashboard for the program evaluation strategy was created to transparently display a real-time view of Competence by Design implementation and facilitate continuous adaptation and improvement. The findings of the program evaluation for Competence by Design drove changes to all aspects of the Competence by Design implementation, aided engagement of partners, supported change management, and deepened our understanding of the journey required for transformational educational change in a complex national postgraduate medical education system. The program evaluation strategy for Competence by Design provides a framework for program evaluation for any large-scale change in health professions education.

Design and Implementation of a National Program of Assessment Model - Integrating Entrustable Professional Activity Assessments in Canadian Specialist Postgraduate Medical Education.

Traditional approaches to assessment in health professions education systems, which have generally focused on the summative function of assessment through the development and episodic use of individual high-stakes examinations, may no longer be appropriate in an era of competency based medical education. Contemporary assessment programs should not only ensure collection of high-quality performance data to support robust decision-making on learners' achievement and competence development but also facilitate the provision of meaningful feedback to learners to support reflective practice and performance improvement. Programmatic assessment is a specific approach to designing assessment systems through the intentional selection and combination of a variety of assessment methods and activities embedded within an educational framework to simultaneously optimize the decision-making and learning function of assessment. It is a core component of competency based medical education and is aligned with the goals of promoting assessment for learning and coaching learners to achieve predefined levels of competence. In Canada, postgraduate specialist medical education has undergone a transformative change to a competency based model centred around entrustable professional activities (EPAs). In this paper, we describe and reflect on the large scale, national implementation of a program of assessment model designed to guide learning and ensure that robust data is collected to support defensible decisions about EPA achievement and progress through training. Reflecting on the design and implications of this assessment system may help others who want to incorporate a competency based approach in their own country.

Coaching in Competence by Design: A New Model of Coaching in the Moment and Coaching Over Time to Support Large Scale Implementation.

Coaching is an increasingly popular means to provide individualized, learner-centered, developmental guidance to trainees in competency based medical education (CBME) curricula. Aligned with CBME's core components, coaching can assist in leveraging the full potential of this educational approach. With its focus on growth and improvement, coaching helps trainees develop clinical acumen and self-regulated learning skills. Developing a shared mental model for coaching in the medical education context is crucial to facilitate integration and subsequent evaluation of success. This paper describes the Royal College of Physicians and Surgeons of Canada's coaching model, one that is theory based, evidence informed, principle driven and iteratively and developed by a multidisciplinary team. The coaching model was specifically designed, fit for purpose to the postgraduate medical education (PGME) context and implemented as part of Competence by Design (CBD), a new competency based PGME program. This coaching model differentiates two coaching roles, which reflect different contexts in which postgraduate trainees learn and develop skills. Both roles are supported by the RX-OCR process: developing Relationship/Rapport, setting eXpectations, Observing, a Coaching conversation, and Recording/Reflecting. The CBD Coaching Model and its associated RX-OCR faculty development tool support the implementation of coaching in CBME. Coaching in the moment and coaching over time offer important mechanisms by which CBD brings value to trainees. For sustained change to occur and for learners and coaches to experience the model's intended benefits, ongoing professional development efforts are needed. Early post implementation reflections and lessons learned are provided.

Enabling Implementation of Competency Based Medical Education through an Outcomes-Focused Accreditation System.

Competency based medical education is being adopted around the world. Accreditation plays a vital role as an enabler in the adoption and implementation of competency based medical education, but little has been published about how the design of an accreditation system facilitates this transformation. The Canadian postgraduate medical education environment has recently transitioned to an outcomes-based accreditation system in parallel with the adoption of competency based medical education. Using the Canadian example, we characterize four features of an accreditation system that can facilitate the implementation of competency based medical education: theoretical underpinning, quality focus, accreditation standards, and accreditation processes. Alignment of the underlying educational theories within the accreditation system and educational paradigm drives change in a consistent and desired direction. An accreditation system that prioritizes quality improvement over quality assurance promotes educational system development and progressive change. Accreditation standards that achieve the difficult balance of being sufficiently detailed yet flexible foster a high fidelity of implementation without stifling innovation. Finally, accreditation processes that recognize the change process, encourage program development, and are not overly punitive all enable the implementation of competency based medical education. We also discuss the ways in which accreditation can simultaneously hinder the implementation of this approach. As education bodies adopt competency based medical education, particular attention should be paid to the role that accreditation plays in successful implementation.

Strategies to Enable Transformation in Medical Education: Faculty and Trainee Development in Competence By Design.

Transformative changes in health professions education need to incorporate effective faculty development, but few very large-scale faculty development designs have been described. The Royal College of Physicians and Surgeons of Canada's Competence by Design project was launched to transform the delivery of postgraduate medical education in Canada using a competency-based model. In this paper we outline the goals, principles, and rationale of the Royal College's national strategy for faculty and resident development initiatives to support the implementation of Competence by Design. We describe the activities and resources for both faculty and trainees that facilitated the redesign of training programs for each specialty and subspecialty at the national level, as well as supporting the implementation of the redesign at the local level. This undertaking was not without its challenges: we thus reflect on those challenges, enablers, and the lessons learned, and discuss a continuous quality improvement approach that was taken to iteratively inform the implementation process moving forward.

The Next Era of Assessment: Building a Trustworthy Assessment System.

Assessment in medical education has evolved through a sequence of eras each centering on distinct views and values. These eras include measurement (e.g., knowledge exams, objective structured clinical examinations), then judgments (e.g., workplace-based assessments, entrustable professional activities), and most recently systems or programmatic assessment, where over time multiple types and sources of data are collected and combined by competency committees to ensure individual learners are ready to progress to the next stage in their training. Significantly less attention has been paid to the social context of assessment, which has led to an overall erosion of trust in assessment by a variety of stakeholders including learners and frontline assessors. To meaningfully move forward, the authors assert that the reestablishment of trust should be foundational to the next era of assessment. In our actions and interventions, it is imperative that medical education leaders address and build trust in assessment at a systems level. To that end, the authors first review tenets on the social contextualization of assessment and its linkage to trust and discuss consequences should the current state of low trust continue. The authors then posit that trusting and trustworthy relationships can exist at individual as well as organizational and systems levels. Finally, the authors propose a framework to build trust at multiple levels in a future assessment system; one that invites and supports professional and human growth and has the potential to position assessment as a fundamental component of renegotiating the social contract between medical education and the health of the public.

Expanding Interdisciplinarity: A Bibliometric Study of Medical Education Using the Medical Education Journal List-24 (MEJ-24).

Introduction: Interdisciplinary research, which integrates input (e.g., data, techniques, theories) from two or more disciplines, is critical for solving wicked problems. Medical education research is assumed to be interdisciplinary. However, researchers have questioned this assumption. The present study, a conceptual replication, clarifies the nature of medical education interdisciplinarity by analyzing the citations of medical education journal articles. Method: The authors retrieved the cited references of all articles in 22 medical education journals between 2001-2020 from Web of Science (WoS). We then identified the WoS classifications for the journals of each cited reference. Results: We analyzed 31,283 articles referencing 723,683 publications. We identified 493,973 (68.3%) of those cited references in 6,618 journals representing 242 categories, which represents 94% of all WoS categories. Close to half of all citations were categorized as "education, scientific disciplines" and "healthcare sciences and services". Over the study period, the number of references consistently increased as did the representation of categories to include a diversity of topics such as business, management, and linguistics. Discussion: Our study aligns with previous research, suggesting that medical education research could be described as inwardly focused. However, the observed growth of categories and their increasing diversity over time indicates that medical education displays increasing interdisciplinarity. Now visible, the field can raise awareness of and promote interdisciplinarity, if desired, by seeking and highlighting opportunities for future growth.

Assessing the utility of a novel entrustment-supervision assessment tool.

BACKGROUND: Work-based assessments (WBAs) are increasingly used to inform decisions about trainee progression. Unfortunately, WBAs often fail to discriminate between trainees of differing abilities and have poor reliability. Entrustment-supervision scales may improve WBA performance, but there is a paucity of literature directly comparing them to traditional WBA tools. METHODS: The Ottawa Emergency Department Shift Observation Tool (O-EDShOT) is a previously published WBA tool employing an entrustment-supervision scale with strong validity evidence. This pre-/post-implementation study compares the performance of the O-EDShOT with that of a traditional WBA tool using norm-based anchors. All assessments completed in 12-month periods before and after implementing the O-EDShOT were collected, and generalisability analysis was conducted with year of training, trainees within year and forms within trainee as nested factors. Secondary analysis included assessor as a factor. RESULTS: A total of 3908 and 3679 assessments were completed by 99 and 116 assessors, for 152 and 138 trainees in the pre- and post-implementation phases respectively. The O-EDShOT generated a wider range of awarded scores than the traditional WBA, and mean scores increased more with increasing level of training (0.32 vs. 0.14 points per year, p = 0.01). A significantly greater proportion of overall score variability was attributable to trainees using the O-EDShOT (59%) compared with the traditional tool (21%, p < 0.001). Assessors contributed less to overall score variability for the O-EDShOT than for the traditional WBA (16% vs. 37%). Moreover, the O-EDShOT required fewer completed assessments than the traditional tool (27 vs. 51) for a reliability of 0.8. CONCLUSION: The O-EDShOT outperformed a traditional norm-referenced WBA in discriminating between trainees and required fewer assessments to generate a reliable estimate of trainee performance. More broadly, this study adds to the body of literature suggesting that entrustment-supervision scales generate more useful and reliable assessments in a variety of clinical settings.

The voices of medical education scholarship: Describing the published landscape.

INTRODUCTION: The voices of authors who publish medical education literature have a powerful impact on the field's discourses. Researchers have identified a lack of author diversity, which suggests potential epistemic injustice. This study investigates author characteristics to provide an evidence-based starting point for communal discussion with the intent to move medical education towards a future that holds space for, and values, diverse ways of knowing. METHOD: The authors conducted a bibliometric analysis of all articles published in 24 medical education journals published between 2000 and 2020 to identify author characteristics, with an emphasis on author gender and geographic location and their intersection. Article metadata was downloaded from Web of Science. Genderize.io was used to predict author gender. RESULTS: The journals published 37 263 articles authored by 62 708 unique authors. Males were more prevalent across all authorship positions (n = 62 828; 55.7%) than females (n = 49 975; 44.3%). Authors listed affiliations in 146 countries of which 95 were classified as Global South. Few articles were written by multinational teams (n = 3765; 16.2%). Global South authors accounted for 12 007 (11.4%) author positions of which 3594 (3.8%) were female. DISCUSSION: This study provides an evidence-based starting point to discuss the imbalance of author voices in medical education, especially when considering the intersection of gender and geographical location, which further suggests epistemic injustice in medical education. If the field values a diversity of perspectives, there is considerable opportunity for improvement by engaging the community in discussions about what knowledge matters in medical education, the role of journals in promoting diversity, how to best use this baseline data and how to continue studying epistemic injustice in medical education.

Stress and conflict from tacit culture forges professional identity in newly graduated independent physicians.

To transition successfully into independent practice, newly graduated independent physicians (new "attendings") undergo a process of professional identity formation (PIF) as a clinician within a new community of practice (CoP). PIF is crafted by socialization within a CoP including transfer of tacit knowledge. While certain tacit knowledge is critical for professional identity, we understand little how it shapes PIF. We set out to describe the tacit knowledge acquired by new attendings within a CoP and how it contributes to PIF. Informed by constructivist grounded theory, we interviewed 23 new attendings about the tacit knowledge they acquired in early practice. Data collection and analysis occurred iteratively. We identified themes using constant comparative analysis and generated a theory that underwent member checking and feedback. Implicit standards from group culture imparted high expectations on new attendings and led to internal stress. New attendings also encountered a tacit code of conduct as behavioral elements of group culture. These elements created external conflict between new attendings and group members such as departmental colleagues, consulting physicians, and other health professionals. Depending on the support they received, new attendings responded to the stress and conflict in three ways: they doubted, adjusted, or avoided. These strategies molded their professional identity, and moved them towards or away from the CoP as they navigated their transition and PIF. We describe a novel theory of how tacit group culture shaped new attending physicians' professional identity in a new community of practice. Internal stress and external conflict occurred due to high expectations and tacit culture elements. New attendings' doubt, adjust, or avoid responses, shaped by support they received, in turn crafted their professional identity. Education leaders should prepare graduating trainees to navigate aspects of transition to independent practice successfully.

Ready, set, go! Evaluating readiness to implement competency-based medical education.

PURPOSE: Organizational readiness is critical for successful implementation of an innovation. We evaluated program readiness to implement Competence by Design (CBD), a model of Competency-Based Medical Education (CBME), among Canadian postgraduate training programs. METHODS: A survey of program directors was distributed 1 month prior to CBD implementation in 2019. Questions were informed by the R = MC2 framework of organizational readiness and addressed: program motivation, general capacity for change, and innovation-specific capacity. An overall readiness score was calculated. An ANOVA was conducted to compare overall readiness between disciplines. RESULTS: Survey response rate was 42% (n = 79). The mean overall readiness score was 74% (30-98%). There was no difference in scores between disciplines. The majority of respondents agreed that successful implementation of CBD was a priority (74%), and that their leadership (94%) and faculty and residents (87%) were supportive of change. Fewer perceived that CBD was a move in the right direction (58%) and that implementation was a manageable change (53%). Curriculum mapping, competence committees and programmatic assessment activities were completed by >90% of programs, while <50% had engaged off-service disciplines. CONCLUSION: Our study highlights important areas where programs excelled in their preparation for CBD, as well as common challenges that serve as targets for future intervention to improve program readiness for CBD implementation.

Does direct observation influence the quality of workplace-based assessment documentation?

Background: A key component of competency-based medical education (CBME) is direct observation of trainees. Direct observation has been emphasized as integral to workplace-based assessment (WBA) yet previously identified challenges may limit its successful implementation. Given these challenges, it is imperative to fully understand the value of direct observation within a CBME program of assessment. Specifically, it is not known whether the quality of WBA documentation is influenced by observation type (direct or indirect). Methods: The objective of this study was to determine the influence of observation type (direct or indirect) on quality of entrustable professional activity (EPA) assessment documentation within a CBME program. EPA assessments were scored by four raters using the Quality of Assessment for Learning (QuAL) instrument, a previously published three-item quantitative measure of the quality of written comments associated with a single clinical performance score. An analysis of variance was performed to compare mean QuAL scores among the direct and indirect observation groups. The reliability of the QuAL instrument for EPA assessments was calculated using a generalizability analysis. Results: A total of 244 EPA assessments (122 direct observation, 122 indirect observation) were rated for quality using the QuAL instrument. No difference in mean QuAL score was identified between the direct and indirect observation groups (p = 0.17). The reliability of the QuAL instrument for EPA assessments was 0.84. Conclusions: Observation type (direct or indirect) did not influence the quality of EPA assessment documentation. This finding raises the question of how direct and indirect observation truly differ and the implications for meta-raters such as competence committees responsible for making judgments related to trainee promotion.

Implementation of competence committees during the transition to CBME in Canada: A national fidelity-focused evaluation.

PURPOSE: This study evaluated the fidelity of competence committee (CC) implementation in Canadian postgraduate specialist training programs during the transition to competency-based medical education (CBME). METHODS: A national survey of CC chairs was distributed to all CBME training programs in November 2019. Survey questions were derived from guiding documents published by the Royal College of Physicians and Surgeons of Canada reflecting intended processes and design. RESULTS: Response rate was 39% (113/293) with representation from all eligible disciplines. Committee size ranged from 3 to 20 members, 42% of programs included external members, and 20% included a resident representative. Most programs (72%) reported that a primary review and synthesis of resident assessment data occurs prior to the meeting, with some data reviewed collectively during meetings. When determining entrustable professional activity (EPA) achievement, most programs followed the national specialty guidelines closely with some exceptions (53%). Documented concerns about professionalism, EPA narrative comments, and EPA entrustment scores were most highly weighted when determining resident progress decisions. CONCLUSIONS: Heterogeneity in CC implementation likely reflects local adaptations, but may also explain some of the variable challenges faced by programs during the transition to CBME. Our results offer educational leaders important fidelity data that can help inform the larger evaluation and transformation of CBME.

Delineating the field of medical education: Bibliometric research approach(es).

BACKGROUND: The field of medical education remains poorly delineated such that there is no broad consensus of articles or journals that comprise 'the field'. This lack of consensus indicates a missed opportunity for researchers to generate insights about the field that could facilitate conducting bibliometric studies and other research designs (e.g., systematic reviews) and also enable individuals to identify themselves as 'medical education researchers'. Other fields have utilised bibliometric field delineation, which is the assigning of articles or journals to a certain field in an effort to define that field. PROCESS: In this Research Approach, three bibliometric field delineation approaches-information retrieval, core journals, and journal co-citation-are introduced. For each approach, the authors describe attempts to apply it in medical education and identify related strengths and weaknesses. Based on co-citation, the authors propose the Medical Education Journal List 24 (MEJ-24), as a starting point for delineating medical education and invite the community to collaborate on improving and potentially expanding this list. PEARLS: As a research approach, field delineation is complicated, and there is no clear best way to delineate the field of medical education. However, recent advances in information science provide potentially fruitful approaches to deal with the field's complexity. When considering these approaches, researchers should consider collaborating with bibliometricians. Bibliometric approaches rely on available metadata for articles and journals, which necessitates that researchers examine the metadata prior to analysis to understand its strengths and weaknesses, and to assess how this might affect data interpretation. While using bibliometric approaches for field delineation is valuable, it is important to remember that these techniques are only as good as the research team's interpretation of the data, which suggests that an expanded approach is needed to better delineate medical education, an approach that includes active discussion within the medical education community.