Is Competency-Based Medical Education being implemented as intended? Early lessons learned from Physical Medicine and Rehabilitation.

Background: As competency-based medical education (CBME) curricula are introduced in residency programs across Canada, systematic evaluation efforts are needed to ensure fidelity of implementation. This study evaluated early outcomes of CBME implementation in one Canadian Physical Medicine and Rehabilitation program that was an early adopter of CBME, with an aim to inform continuous quality improvement initiatives and CBME implementation nationwide. Methods: Using Rapid Evaluation methodology, informed by the CBME Core Components Framework, the intended outcomes of CBME were compared to actual outcomes. Results: Results suggested that a culture of feedback and coaching already existed in this program prior to CBME implementation, yet faculty felt that CBME added a framework to support feedback. The small program size was valuable in fostering strong relationships and individualized learning. However, participants expressed concerns about CBME fostering a reductionist approach to the development of competence. Challenges existed with direct observation, clear expectations for off-service training experiences, and tracking trainee progress. There was trepidation surrounding national curricular change, yet the institution-wide approach to CBME implementation created shared experiences and a community of practice. Conclusions: Program evaluation can help understand gaps between planned versus enacted implementation of CBME, and foster adaptations to improve the fidelity of implementation.

Contexte: À mesure que les programmes d'approche par compétences (APC) en formation médicale sont introduits dans les programmes de résidence au Canada, des efforts d'évaluation systématiques sont nécessaires pour assurer la fidélité de la mise en œuvre. Cette étude a évalué les premiers résultats de la mise en œuvre de l'APC en formation médicale dans un programme canadien de médecine physique et réadaptation, qui a été un des premiers à adopter l'APC, dans le but d'orienter les initiatives d'amélioration continue de la qualité et de la mise en œuvre de l'APC à l'échelle nationale. Méthodes: En utilisant une méthodologie d'évaluation rapide, fondée sur le cadre des composantes de base de l'APC en formation médicale, les résultats escomptés de l'APC ont été comparés aux résultats réels. Résultats: Les résultats suggèrent qu'une culture de la rétroaction et de l'encadrement existait déjà dans ce programme avant la mise en œuvre de l'APC, mais le corps professoral a estimé que l'APC en formation médicale a ajouté un cadre pour soutenir cette rétroaction. La petite taille du programme a permis de favoriser des relations solides et un apprentissage individualisé. Cependant, les participants ont exprimé des inquiétudes quant au fait que l'APC favorise une approche réductionniste du développement des compétences. L'observation directe, les attentes claires en matière d'expériences de formation hors de l'environnement clinique et le suivi des progrès des résidents posent problème. Le changement de programme national a suscité des inquiétudes, mais l'approche institutionnelle de la mise en œuvre de l'APC a permis de partager des expériences et de créer une communauté de pratique. Conclusions: L'évaluation des programmes peut aider à comprendre les écarts entre la mise en œuvre planifiée et effective de l'APC en formation médicale, et de favoriser les adaptations pour améliorer le respect des conditions de mise en œuvre.

Is time really of the essence? Timeliness of narrative feedback in ophthalmology CBME assessments.

PURPOSE: Competency-based medical education relies on a strong program of assessment, and quality comments play a vital role in ensuring its success. The goal of this study is to determine the effect of the timeliness of assessment completion on the quality of the feedback. MATERIALS AND METHODS: Using the Quality of Assessment for Learning (QuAL) score 2478 assessments were evaluated. The assessments included those completed between July 2017 and December 2020 for 18 ophthalmology residents. Spearman correlation, Mann-Whitney U and Kruskal-Wallis tests were used to assess variations in QuAL scores based on the timeliness of assessment completion. RESULTS: The timeliness of assessment completion ranged from 0 to 299 d with the mean time for completion being 3 d. As the delay increased, the QuAL score decreased. Feedback provided 4, 5, and 14 d post-encounter demonstrated statistically significant differences in the QuAL score. Additionally, there was a significant difference in the timeliness of feedback when there is no written comment. CONCLUSIONS: This study demonstrates that the timeliness of assessment completion might have an effect on the quality of written feedback. Written feedback should be completed within 14 d of the encounter to optimize quantity and quality.

Application of the R2C2 Model to In-the-Moment Feedback and Coaching.

PURPOSE: The R2C2 (relationship, reaction, content, coaching) model is an iterative, evidence-based, theory-informed approach to feedback and coaching that enables preceptors and learners to build relationships, explore reactions and reflections, confirm content, and coach for change and cocreate an action plan. This study explored application of the R2C2 model for in-the-moment feedback conversations between preceptors and learners and the factors that influence its use. METHOD: A qualitative study using framework analysis through the lens of experiential learning was undertaken with 15 trained preceptor-learner dyads. Data were collected during feedback sessions and follow-up interviews between March 2021 and July 2022. The research team familiarized themselves with the data, used a coding template to document examples of the model's application, reviewed the initial framework and revised the coding template, indexed and summarized the data, created a summary document, examined the transcripts for alignment with each model phase, and identified illustrative quotations and overarching themes. RESULTS: Fifteen dyads were recruited from 8 disciplines (11 preceptors were paired with a single resident [n = 9] or a single medical student [n = 2]; 2 preceptors each had 2 residents). All dyads were able to apply the R2C2 phases of building relationships, exploring reactions and reflections, and confirming content. Many struggled with the coaching components, specifically in creating an action plan and follow-up arrangements. Preceptor skill in applying the model, time available for feedback conversations, and the nature of the relationship impacted how the model was applied. CONCLUSIONS: The R2C2 model can be adapted to contexts where in-the-moment feedback conversations occur shortly after a clinical encounter. Experiential learning approaches applying the R2C2 model are critical. Skillful application of the model requires that learners and preceptors go beyond confirming an area of change and deliberately engage in coaching and cocreating an action plan.

Using a rapid-cycle approach to evaluate implementation of competency-based medical education in ophthalmology.

OBJECTIVE: As competency-based medical education is being implemented across Canada, there is an increasing need to evaluate the progress to date, including identification of strengths and weaknesses, to inform program development. Ophthalmology is preparing for a national launch in coming years. The purpose of this study was to describe key stakeholders' lived experiences in the competency-based medical education foundation-of-discipline stage in one ophthalmology department. DESIGN: Using a case-study approach, a qualitative rapid-cycle evaluation was conducted during the 2018-2019 academic year. PARTICIPANTS: Residents, faculty, academic advisors, competence committee members, the program director, the program administrator, and the educational consultant were invited to participate in the program evaluation. METHODS: The rapid-cycle evaluation consisted of 2 evaluation cycles, with the first round of interviews and focus groups occurring in October 2018 and the second round in March 2019. Recommendations were implemented in November 2019 and June 2019. All data were analyzed thematically using NVivo. RESULTS: Three main themes emerged across all data sets: developing a shared understanding (e.g., role expectations and changes to assessment), refining assessment processes and tools (e.g., the need for streamlining and clarification), and feedback (e.g., perceived benefits and value of narrative comments). CONCLUSIONS: Exploring lived experiences in this study resulted in positive and immediate improvements to the residency program. The recommendations and approach will be useful to other Canadian departments and institutions as they prepare for Competence by Design.

An adaptation-focused evaluation of Canada's first competency-based medical education implementation in radiology.

OBJECTIVES: Systematic program evaluation of the Queen's University diagnostic radiology residency program following transition to a competency-based medical education (CBME) curriculum. METHODS: Rapid Evaluation methodology and the Core Components Framework were utilized to measure CBME implementation. A combination of interviews and focus groups were held with program leaders (n = 6), faculty (n = 10), both CBME stream and traditional stream residents (n = 6), and program staff (n = 2). Interviews and focus groups were transcribed and analyzed abductively. Study team met with program leaders to review common themes and plan potential adaptations. RESULTS: Strengths of CBME implementation included more frequent and timely feedback as well as the role of the Academic Advisor. However, frontline faculty felt insufficiently supported with regards to the theory and practical implementation of the new curriculum and found assessment tools unintuitive. The circumstances surrounding the curricular implementation also resulted in some negative sentiment. Additional faculty and resident education workshops were identified as areas for improvement as well as changes to assessment tools for increased clarity. Residents overall viewed the changes favorably, with traditional stream residents indicating that they also had a desire for increased feedback. CONCLUSIONS: Rapid Evaluation is an effective method for program assessment following curricular change in diagnostic radiology. A departmental champion driving enthusiasm for change from within may be valuable. Adequate resident and faculty education is key to maximize change and smooth the transition. Advances in knowledge: This study provides insights for other radiology training programs transitioning to a CBME framework and provides a structure for programmatic assessment.