Competency-based medical education across the continuum: How well aligned are medical school EPAs to residency milestones?

INTRODUCTION: Competency-based medical education (CBME) provides a framework for describing learner progression throughout training. However, specific approaches to CBME implementation vary widely across educational settings. Alignment between various methods used across the continuum is critical to support transitions and assess learner performance. The purpose of this study was to investigate alignment between CBME frameworks used in undergraduate medical education (UME) and graduate medical education (GME) settings using the US context as a model. METHOD: The authors analyzed content from the core entrustable professional activities for entering residency (Core EPAs; UME model) and residency milestones (GME model). From that analysis, they performed a series of cross-walk activities to investigate alignment between frameworks. After independent review, authors discussed findings until consensus was reached. RESULTS: Some alignment was found for activities associated with history taking, physical examination, differential diagnosis, patient safety, and interprofessional care; however, there were far more examples of misalignment. CONCLUSIONS: These findings highlight challenges creating alignment of assessment frameworks across the continuum of training. The importance of these findings includes implications for assessment and persistence of the educational gap across UME and GME. The authors provide four next steps to improve upon the continuum of education.

The Internal Medicine Reporting Milestones: Cross-sectional Description of Initial Implementation in U.S. Residency Programs.

BACKGROUND: High-quality assessment of resident performance is needed to guide individual residents' development and ensure their preparedness to provide patient care. To facilitate this aim, reporting milestones are now required across all internal medicine (IM) residency programs. OBJECTIVE: To describe initial milestone ratings for the population of IM residents by IM residency programs. DESIGN: Cross-sectional study. SETTING: IM residency programs. PARTICIPANTS: All IM residents whose residency program directors submitted milestone data at the end of the 2013-2014 academic year. MEASUREMENTS: Ratings addressed 6 competencies and 22 subcompetencies. A rating of "not assessable" indicated insufficient information to evaluate the given subcompetency. Descriptive statistics were calculated to describe ratings across competencies and training years. RESULTS: Data were available for all 21 774 U.S. IM residents from all 383 programs. Overall, 2889 residents (1621 in postgraduate year 1 [PGY-1], 902 in PGY-2, and 366 in PGY-3) had at least 1 subcompetency rated as not assessable. Summaries of average ratings by competency and training year showed higher ratings for PGY-3 residents in all competencies. Overall ratings for each of the 6 individual competencies showed that fewer than 1% of third-year residents were rated as "unsatisfactory" or "conditional on improvement." However, when subcompetency milestone ratings were used, 861 residents (12.8%) who successfully completed training had at least 1 competency with all corresponding subcompetencies graded below the threshold of "readiness for unsupervised practice." LIMITATION: Data were derived from a point in time in the first reporting period in which milestones were used. CONCLUSION: The initial milestone-based evaluations of IM residents nationally suggest that documenting developmental progression of competency is possible over training years. Subcompetencies may identify areas in which residents might benefit from additional feedback and experience. Future work is needed to explore how milestones are used to support residents' development and enhance residency curricula. PRIMARY FUNDING SOURCE: None.

Overarching challenges to the implementation of competency-based medical education.

Medical education is under increasing pressure to more effectively prepare physicians to meet the needs of patients and populations. With its emphasis on individual, programmatic, and institutional outcomes, competency-based medical education (CBME) has the potential to realign medical education with this societal expectation. Implementing CBME, however, comes with significant challenges. This manuscript describes four overarching challenges that must be confronted by medical educators worldwide in the implementation of CBME: (1) the need to align all regulatory stakeholders in order to facilitate the optimization of training programs and learning environments so that they support competency-based progression; (2) the purposeful integration of efforts to redesign both medical education and the delivery of clinical care; (3) the need to establish expected outcomes for individuals, programs, training institutions, and health care systems so that performance can be measured; and (4) the need to establish a culture of mutual accountability for the achievement of these defined outcomes. In overcoming these challenges, medical educators, leaders, and policy-makers will need to seek collaborative approaches to common problems and to learn from innovators who have already successfully made the transition to CBME.

Correlations Between Ratings on the Resident Annual Evaluation Summary and the Internal Medicine Milestones and Association With ABIM Certification Examination Scores Among US Internal Medicine Residents, 2013-2014.

Importance: US internal medicine residency programs are now required to rate residents using milestones. Evidence of validity of milestone ratings is needed. Objective: To compare ratings of internal medicine residents using the pre-2015 resident annual evaluation summary (RAES), a nondevelopmental rating scale, with developmental milestone ratings. Design, Setting, and Participants: Cross-sectional study of US internal medicine residency programs in the 2013-2014 academic year, including 21 284 internal medicine residents (7048 postgraduate-year 1 [PGY-1], 7233 PGY-2, and 7003 PGY-3). Exposures: Program director ratings on the RAES and milestone ratings. Main Outcomes and Measures: Correlations of RAES and milestone ratings by training year; correlations of medical knowledge ratings with American Board of Internal Medicine (ABIM) certification examination scores; rating of unprofessional behavior using the 2 systems. Results: Corresponding RAES ratings and milestone ratings showed progressively higher correlations across training years, ranging among competencies from 0.31 (95% CI, 0.29 to 0.33) to 0.35 (95% CI, 0.33 to 0.37) for PGY-1 residents to 0.43 (95% CI, 0.41 to 0.45) to 0.52 (95% CI, 0.50 to 0.54) for PGY-3 residents (all P values <.05). Linear regression showed ratings differed more between PGY-1 and PGY-3 years using milestone ratings than the RAES (all P values <.001). Of the 6260 residents who attempted the certification examination, the 618 who failed had lower ratings using both systems for medical knowledge than did those who passed (RAES difference, -0.9; 95% CI, -1.0 to -0.8; P < .001; milestone medical knowledge 1 difference, -0.3; 95% CI, -0.3 to -0.3; P < .001; and medical knowledge 2 difference, -0.2; 95% CI, -0.3 to -0.2; P < .001). Of the 26 PGY-3 residents with milestone ratings indicating deficiencies on either of the 2 medical knowledge subcompetencies, 12 failed the certification examination. Correlation of RAES ratings for professionalism with residents' lowest professionalism milestone ratings was 0.44 (95% CI, 0.43 to 0.45; P < .001). Conclusions and Relevance: Among US internal medicine residents in the 2013-2014 academic year, milestone-based ratings correlated with RAES ratings but with a greater difference across training years. Both rating systems for medical knowledge correlated with ABIM certification examination scores. Milestone ratings may better detect problems with professionalism. These preliminary findings may inform establishment of the validity of milestone-based assessment.

How faculty members experience workplace-based assessment rater training: a qualitative study.

CONTEXT: Direct observation of clinical skills is a common approach in workplace-based assessment (WBA). Despite widespread use of the mini-clinical evaluation exercise (mini-CEX), faculty development efforts are typically required to improve assessment quality. Little consensus exists regarding the most effective training methods, and few studies explore faculty members' reactions to rater training. OBJECTIVES: This study was conducted to qualitatively explore the experiences of faculty staff with two rater training approaches - performance dimension training (PDT) and a modified approach to frame of reference training (FoRT) - to elucidate how such faculty development can be optimally designed. METHODS: In a qualitative study of a multifaceted intervention using complex intervention principles, 45 out-patient resident faculty preceptors from 26 US internal medicine residency programmes participated in a rater training faculty development programme. All participants were interviewed individually and in focus groups during and after the programme to elicit how the training influenced their approach to assessment. A constructivist grounded theory approach was used to analyse the data. RESULTS: Many participants perceived that rater training positively influenced their approach to direct observation and feedback, their ability to use entrustment as the standard for assessment, and their own clinical skills. However, barriers to implementation and change included: (i) a preference for holistic assessment over frameworks; (ii) challenges in defining competence; (iii) difficulty in changing one's approach to assessment, and (iv) concerns about institutional culture and buy-in. CONCLUSIONS: Rater training using PDT and a modified approach to FoRT can provide faculty staff with assessment skills that are congruent with principles of criterion-referenced assessment and entrustment, and foundational principles of competency-based education, while providing them with opportunities to reflect on their own clinical skills. However, multiple challenges to incorporating new forms of training exist. Ongoing efforts to improve WBA are needed to address institutional and cultural contexts, and systems of care delivery.

Understanding trust as an essential element of trainee supervision and learning in the workplace.

Clinical supervision requires that supervisors make decisions about how much independence to allow their trainees for patient care tasks. The simultaneous goals of ensuring quality patient care and affording trainees appropriate and progressively greater responsibility require that the supervising physician trusts the trainee. Trust allows the trainee to experience increasing levels of participation and responsibility in the workplace in a way that builds competence for future practice. The factors influencing a supervisor's trust in a trainee are related to the supervisor, trainee, the supervisor-trainee relationship, task, and context. This literature-based overview of these five factors informs design principles for clinical education that support the granting of entrustment. Entrustable professional activities offer promise as an example of a novel supervision and assessment strategy based on trust. Informed by the design principles offered here, entrustment can support supervisors' accountability for the outcomes of training by maintaining focus on future patient care outcomes.

Opening the black box of clinical skills assessment via observation: a conceptual model.

OBJECTIVES: This study was intended to develop a conceptual framework of the factors impacting on faculty members' judgements and ratings of resident doctors (residents) after direct observation with patients. METHODS: In 2009, 44 general internal medicine faculty members responsible for out-patient resident teaching in 16 internal medicine residency programmes in a large urban area in the eastern USA watched four videotaped scenarios and two live scenarios of standardised residents engaged in clinical encounters with standardised patients. After each, faculty members rated the resident using a mini-clinical evaluation exercise and were individually interviewed using a semi-structured interview. Interviews were videotaped, transcribed and analysed using grounded theory methods. RESULTS: Four primary themes that provide insights into the variability of faculty assessments of residents' performance were identified: (i) the frames of reference used by faculty members when translating observations into judgements and ratings are variable; (ii) high levels of inference are used during the direct observation process; (iii) the methods by which judgements are synthesised into numerical ratings are variable, and (iv) factors external to resident performance influence ratings. From these themes, a conceptual model was developed to describe the process of observation, interpretation, synthesis and rating. CONCLUSIONS: It is likely that multiple factors account for the variability in faculty ratings of residents. Understanding these factors informs potential new approaches to faculty development to improve the accuracy, reliability and utility of clinical skills assessment.

Competency-based education and training in internal medicine.

Recent efforts to improve medical education include adopting a new framework based on 6 broad competencies defined by the Accreditation Council for Graduate Medical Education. In this article, the Alliance for Academic Internal Medicine Education Redesign Task Force II examines the advantages and challenges of a competency-based educational framework for medical residents. Efforts to refine specific competencies by developing detailed milestones are described, and examples of training program initiatives using a competency-based approach are presented. Meeting the challenges of a competency-based framework and supporting these educational innovations require a robust faculty development program. Challenges to competency-based education include teaching and evaluating the competencies related to practice-based learning and improvement and systems-based practice, as well as implementing a flexible time frame to achieve competencies. However, the Alliance for Academic Internal Medicine Education Redesign Task Force II does not favor reducing internal medicine training to less than 36 months as part of competency-based education. Rather, the 36-month time frame should allow for remediation to address deficiencies in achieving competencies and for diverse enrichment experiences in such areas as quality of care and practice improvement for residents who have demonstrated skills in all required competencies.

The importance of faculty development in the transition to competency-based medical education.

Medical teachers trained in conventional educational systems need faculty development to prepare them to function effectively in a competency-based medical education (CBME) system. Faculty development can provide knowledge about CBME, training in new teaching techniques in different domains of medical practice, and new strategies for providing the authentic and regular assessment that is an essential aspect of CBME. A systems-wide approach as well as efforts to provide training in CBME to individual teachers in both the undergraduate and postgraduate systems will be important. The wide implementation of CBME will be challenging and slow, and will meet with resistance, but various strategies can be used address these challenges. Faculty development is fundamental to the effectiveness of those strategies.

Identifying the policy implications of competency-based education.

At their 2009 consensus conference, the International CBME Collaborators proposed a number of central tenets of CBME in order to advance the field of medical education. Although the proposed conceptualization of CBME offers several advantages and opportunities, including a greater emphasis on outcomes, a mechanism for the promotion of learner-centred curricula, and the potential to move away from time-based training and credentialing in medicine, it is also associated with several significant barriers to adoption. This paper examines the concepts of CBME through a broad educational policy lens, identifying considerations for medical education leaders, health care institutions, and policy-makers at both the meso (program, institutional) and macro (health care system, inter-jurisdictional, and international) levels. Through this analysis, it is clear that CBME is associated with a number of complex challenges and questions, and cannot be considered in isolation from the complex systems in which it functions. Much more work is needed to engage stakeholders in dialogue, to debate the issues, and to identify possible solutions.

Competency-based medical education in postgraduate medical education.

With the introduction of Tomorrow's Doctors in 1993, medical education began the transition from a time- and process-based system to a competency-based training framework. Implementing competency-based training in postgraduate medical education poses many challenges but ultimately requires a demonstration that the learner is truly competent to progress in training or to the next phase of a professional career. Making this transition requires change at virtually all levels of postgraduate training. Key components of this change include the development of valid and reliable assessment tools such as work-based assessment using direct observation, frequent formative feedback, and learner self-directed assessment; active involvement of the learner in the educational process; and intensive faculty development that addresses curricular design and the assessment of competency.

Competency-based medical education: theory to practice.

Although competency-based medical education (CBME) has attracted renewed interest in recent years among educators and policy-makers in the health care professions, there is little agreement on many aspects of this paradigm. We convened a unique partnership - the International CBME Collaborators - to examine conceptual issues and current debates in CBME. We engaged in a multi-stage group process and held a consensus conference with the aim of reviewing the scholarly literature of competency-based medical education, identifying controversies in need of clarification, proposing definitions and concepts that could be useful to educators across many jurisdictions, and exploring future directions for this approach to preparing health professionals. In this paper, we describe the evolution of CBME from the outcomes movement in the 20th century to a renewed approach that, focused on accountability and curricular outcomes and organized around competencies, promotes greater learner-centredness and de-emphasizes time-based curricular design. In this paradigm, competence and related terms are redefined to emphasize their multi-dimensional, dynamic, developmental, and contextual nature. CBME therefore has significant implications for the planning of medical curricula and will have an important impact in reshaping the enterprise of medical education. We elaborate on this emerging CBME approach and its related concepts, and invite medical educators everywhere to enter into further dialogue about the promise and the potential perils of competency-based medical curricula for the 21st century.

Commitment to Change and Challenges to Implementing Changes After Workplace-Based Assessment Rater Training.

PURPOSE: Faculty development for clinical faculty who assess trainees is necessary to improve assessment quality and impor tant for competency-based education. Little is known about what faculty plan to do differently after training. This study explored the changes faculty intended to make after workplace-based assessment rater training, their ability to implement change, predictors of change, and barriers encountered. METHOD: In 2012, 45 outpatient internal medicine faculty preceptors (who supervised residents) from 26 institutions participated in rater training. They completed a commitment to change form listing up to five commitments and ranked (on a 1-5 scale) their motivation for and anticipated difficulty implementing each change. Three months later, participants were interviewed about their ability to implement change and barriers encountered. The authors used logistic regression to examine predictors of change. RESULTS: Of 191 total commitments, the most common commitments focused on what faculty would change about their own teaching (57%) and increasing direct observation (31%). Of the 183 commitments for which follow-up data were available, 39% were fully implemented, 40% were partially implemented, and 20% were not implemented. Lack of time/competing priorities was the most commonly cited barrier. Higher initial motivation (odds ratio [OR] 2.02; 95% confidence interval [CI] 1.14, 3.57) predicted change. As anticipated difficulty increased, implementation became less likely (OR 0.67; 95% CI 0.49, 0.93). CONCLUSIONS: While higher baseline motivation predicted change, multiple system-level barriers undermined ability to implement change. Rater-training faculty development programs should address how faculty motivation and organizational barriers interact and influence ability to change.

Ensuring Resident Competence: A Narrative Review of the Literature on Group Decision Making to Inform the Work of Clinical Competency Committees.

Background The expectation for graduate medical education programs to ensure that trainees are progressing toward competence for unsupervised practice prompted requirements for a committee to make decisions regarding residents' progress, termed a clinical competency committee (CCC). The literature on the composition of these committees and how they share information and render decisions can inform the work of CCCs by highlighting vulnerabilities and best practices. Objective We conducted a narrative review of the literature on group decision making that can help characterize the work of CCCs, including how they are populated and how they use information. Methods English language studies of group decision making in medical education, psychology, and organizational behavior were used. Results The results highlighted 2 major themes. Group member composition showcased the value placed on the complementarity of members' experience and lessons they had learned about performance review through their teaching and committee work. Group processes revealed strengths and limitations in groups' understanding of their work, leader role, and information-sharing procedures. Time pressure was a threat to the quality of group work. Conclusions Implications of the findings include the risks for committees that arise with homogeneous membership, limitations to available resident performance information, and processes that arise through experience rather than deriving from a well-articulated purpose of their work. Recommendations are presented to maximize the effectiveness of CCC processes, including their membership and access to, and interpretation of, information to yield evidence-based, well-reasoned judgments.