Lessons learned and new strategies for success: Evaluating the Implementation of Competency-Based Medical Education in Queen's Pediatrics.

Objectives: In 2017, Queen's University launched Competency-Based Medical Education (CBME) across 29 programs simultaneously. Two years post-implementation, we asked key stakeholders (faculty, residents, and program leaders) within the Pediatrics program for their perspectives on and experiences with CBME so far. Methods: Program leadership explicitly described the intended outcomes of implementing CBME. Focus groups and interviews were conducted with all stakeholders to describe the enacted implementation. The intended versus enacted implementations were compared to provide insight into needed adaptations for program improvement. Results: Overall, stakeholders saw value in the concept of CBME. Residents felt they received more specific feedback and monthly Competence Committee (CC) meetings and Academic Advisors were helpful. Conversely, all stakeholders noted the increased expectations had led to a feeling of assessment fatigue. Faculty noted that direct observation and not knowing a resident's previous performance information was challenging. Residents wanted to see faculty initiate assessments and improved transparency around progress and promotion decisions. Discussion: The results provided insight into how well the intended outcomes had been achieved as well as areas for improvement. Proposed adaptations included a need for increased direct observation and exploration of faculty accessing residents' previous performance information. Education was provided on the performance expectations of residents and how progress and promotion decisions are made. As well, "flex blocks" were created to help residents customize their training experience to meet their learning needs. The results of this study can be used to inform and guide implementation and adaptations in other programs and institutions.

An adaptation-focused evaluation of Canada's first competency-based medical education implementation in radiology.

OBJECTIVES: Systematic program evaluation of the Queen's University diagnostic radiology residency program following transition to a competency-based medical education (CBME) curriculum. METHODS: Rapid Evaluation methodology and the Core Components Framework were utilized to measure CBME implementation. A combination of interviews and focus groups were held with program leaders (n = 6), faculty (n = 10), both CBME stream and traditional stream residents (n = 6), and program staff (n = 2). Interviews and focus groups were transcribed and analyzed abductively. Study team met with program leaders to review common themes and plan potential adaptations. RESULTS: Strengths of CBME implementation included more frequent and timely feedback as well as the role of the Academic Advisor. However, frontline faculty felt insufficiently supported with regards to the theory and practical implementation of the new curriculum and found assessment tools unintuitive. The circumstances surrounding the curricular implementation also resulted in some negative sentiment. Additional faculty and resident education workshops were identified as areas for improvement as well as changes to assessment tools for increased clarity. Residents overall viewed the changes favorably, with traditional stream residents indicating that they also had a desire for increased feedback. CONCLUSIONS: Rapid Evaluation is an effective method for program assessment following curricular change in diagnostic radiology. A departmental champion driving enthusiasm for change from within may be valuable. Adequate resident and faculty education is key to maximize change and smooth the transition. Advances in knowledge: This study provides insights for other radiology training programs transitioning to a CBME framework and provides a structure for programmatic assessment.

Re-examining the value proposition for Competency-Based Medical Education.

The adoption of competency-based medical education (CBME) by Canadian postgraduate training programs has created a storm of excitement and controversy. Implementing the system-wide Competency by Design (CBD) project initiated by the Royal College of Physicians & Surgeons of Canada (RCPSC), is an ambitious transformative change challenge. Not surprisingly, tensions have arisen across the country around the theoretical underpinnings of CBME and the practicalities of implementation, resulting in calls for evidence justifying its value. Assumptions have been made on both sides of the argument contributing to an atmosphere of unhealthy protection of the status quo, premature conclusions of CBME's worth, and an oversimplification of risks and costs to participants. We feel that a renewed effort to find a shared vision of medical education and the true value proposition of CBME is required to recreate a growth-oriented mindset. Also, the aspirational assertion of a direct link between CBME and improved patient outcomes requires deferral until further implementation and study has occurred. However, we perceive more concrete and immediate value of CBME arises from the societal contract physicians have, the connection to maintaining self-regulation, and the potential customization of training for learners.

L'adoption de la formation médicale axée sur les compétences (FMAC) dans les programmes canadiens d'études postdoctorales a suscité une tempête d'enthousiasme et de controverse. La mise en œuvre à l'échelle du système du projet Compétence par conception (CPC), lancé par le Collège royal des médecins et chirurgiens du Canada (CRMCC), pose le défi d'un changement ambitieux et transformateur. Il n'est pas surprenant que des tensions soient apparues dans tout le pays autour des fondements théoriques de la FMAC et des aspects pratiques de sa mise en œuvre, donnant lieu à des demandes de preuves pour démontrer sa valeur.1 Détracteurs et partisans ont avancé des suppositions, contribuant à un climat malsain de protection du statu quo, à des conclusions prématurées sur la valeur de la FMAC et à une simplification exagérée des risques et des coûts pour les participants.Nous estimons qu'un effort pour retrouver une vision commune de l'éducation médicale et une proposition sérieuse quant à la valeur de la FMAC sont de mise afin de restaurer une attitude orientée vers l'avancement. De plus, il conviendrait de s'abstenir d'affirmer l'existence d'un lien direct entre la FMAC et l'amélioration des résultats pour les patients en attendant qu'une mise en oeuvre plus étendue et que de la recherche plus approfondie aient eu lieu. Cependant, on peut observer dans la FMAC une valeur concrète et immédiate découlant de la présence d'un engagement des médecins envers la société, de son orientation vers le maintien de l'autorégulation et de la personnalisation potentielle de la formation pour les apprenants.

Living in a World of Change: Bridging the Gap From Competency-Based Medical Education Theory to Practice in Canada.

Within graduate medical education, many educators are experiencing a climate of significant change. One transformation, competency-based medical education (CBME), is occurring simultaneously across much of the world, and implementation will require navigating numerous tensions and paradoxes. Successful transformation requires many types of power and is most likely to happen when the medical education community of professionals is engaged in designing, experimenting, acting, and sensemaking together.In this complex climate, the craft of change facilitators and community leaders is needed more than ever. National top-down policies and structures, while important, are not sufficient. The operationalization of new advances is best done when local leaders are afforded room to shape their local context. An evidence-based approach to thinking about the transformative change associated with CBME needs to be adopted. In this age of entrustment, 3 priorities are paramount: (1) engage, entrust, and empower professionals with increasing shared ownership of the innovation; (2) better prepare education professionals in leadership and transformational change techniques in the complex system of medical education; and (3) leverage the wider community of practice to maximize local CBME customization. These recommendations, although based largely on the Canadian experience, are intended to inform CBME transformation in any context.

Simulation in Canadian postgraduate emergency medicine training - a national survey.

OBJECTIVES: Simulation-based education (SBE) is an important training strategy in emergency medicine (EM) postgraduate programs. This study sought to characterize the use of simulation in FRCPC-EM residency programs across Canada. METHODS: A national survey was administered to residents and knowledgeable program representatives (PRs) at all Canadian FRCPC-EM programs. Survey question themes included simulation program characteristics, the frequency of resident participation, the location and administration of SBE, institutional barriers, interprofessional involvement, content, assessment strategies, and attitudes about SBE. RESULTS: Resident and PR response rates were 63% (203/321) and 100% (16/16), respectively. Residents reported a median of 20 (range 0-150) hours of annual simulation training, with 52% of residents indicating that the time dedicated to simulation training met their needs. PRs reported the frequency of SBE sessions ranging from weekly to every 6 months, with 15 (94%) programs having an established simulation curriculum. Two (13%) of the programs used simulation for resident assessment, although 15 (94%) of PRs indicated that they would be comfortable with simulation-based assessment. The most common PR-identified barriers to administering simulation were a lack of protected faculty time (75%) and a lack of faculty experience with simulation (56%). Interprofessional involvement in simulation was strongly valued by both residents and PRs. CONCLUSIONS: SBE is frequently used by Canadian FRCPC-EM residency programs. However, there exists considerable variability in the structure, frequency, and timing of simulation-based activities. As programs transition to competency-based medical education, national organizations and collaborations should consider the variability in how SBE is administered.

How we developed a comprehensive resuscitation-based simulation curriculum in emergency medicine.

Over the past decade, simulation-based education has emerged as a new and exciting adjunct to traditional bedside teaching and learning. Simulation-based education seems particularly relevant to emergency medicine training where residents have to master a very broad skill set, and may not have sufficient real clinical opportunities to achieve competence in each and every skill. In 2006, the Emergency Medicine program at Queen's University set out to enhance our core curriculum by developing and implementing a series of simulation-based teaching sessions with a focus on resuscitative care. The sessions were developed in such as way as to satisfy the four conditions associated with optimum learning and improvement of performance; appropriate difficulty of skill, repetitive practice, motivation, and immediate feedback. The content of the sessions was determined with consideration of the national training requirements set out by the Royal College of Physicians & Surgeons of Canada. Sessions were introduced in a stepwise fashion, starting with a cardiac resuscitation series based on the AHA ACLS guidelines, and leading up to a more advanced resuscitation series as staff became more adept at teaching with simulation, and as residents became more comfortable with this style of learning. The result is a longitudinal resuscitation curriculum that begins with fundamental skills of resuscitation and crisis resource management (CRM) in the first 2 years of residency and progresses through increasingly complex resuscitation cases where senior residents are expected to play a leadership role. This paper documents how we developed, implemented, and evaluated this resuscitation-based simulation curriculum for Emergency Medicine postgraduate trainees, with discussion of some of the challenges encountered.

Queen's simulation assessment tool: development and validation of an assessment tool for resuscitation objective structured clinical examination stations in emergency medicine.

INTRODUCTION: The use of high-fidelity simulation is emerging as an effective approach to competency-based assessment in medical education. We aimed to develop and validate a modifiable anchored global assessment scoring tool for simulation-based Objective Structured Clinical Examinations (OSCEs) of resuscitation competence in postgraduate emergency medicine (EM) trainees. METHODS: The Queen's Simulation Assessment Tool was developed using a modified Delphi technique with a panel of EM physicians. Ten standardized resuscitation OSCE scenarios were administered to EM trainees, and their video-recorded performances were scored by 3 independent and blinded EM attending physicians using the Queen's Simulation Assessment Tool. Correlational analyses and analysis of variance were applied to measure the discriminatory capabilities and interrater reliability of each scenario. A fully crossed generalizability study was conducted for each examination. RESULTS: Emergency medicine postgraduate trainees at Queen's University (n = 19-25 per station) participated in the study over 3 years. Interrater reliability showed acceptable levels of agreement for each scenario (mean Spearman ρ = 0.75 [0.63-0.87]; mean interclass correlation coefficient, 0.69 [0.58-0.87]). Discriminatory validity was strong, with senior residents outperforming junior residents in all but 1 of the 10 scenarios. Generalizability studies found the trainee and trainee by scenario interactions as the largest contributors to variance, with G coefficients ranging from 0.67 to 0.84. Resident trainees reported comfort being assessed in the simulation environment (3.8/5) and found the simulation-based examination valuable to their learning (4.6/5). CONCLUSIONS: This study describes the development and validation of a novel modifiable anchored global assessment scoring tool for simulation-based OSCE assessment of resuscitation competence in postgraduate EM trainees.

Development and evaluation of a simulation-based resuscitation scenario assessment tool for emergency medicine residents.

OBJECTIVE: We sought to develop and validate a three-station simulation-based Objective Structured Clinical Examination (OSCE) tool to assess emergency medicine resident competency in resuscitation scenarios. METHODS: An expert panel of emergency physicians developed three scenarios for use with high-fidelity mannequins. For each scenario, a corresponding assessment tool was developed with an essential actions (EA) checklist and a global assessment score (GAS). The scenarios were (1) unstable ventricular tachycardia, (2) respiratory failure, and (3) ST elevation myocardial infarction. Emergency medicine residents were videotaped completing the OSCE, and three clinician experts independently evaluated the videotapes using the assessment tool. RESULTS: Twenty-one residents completed the OSCE (nine residents in the College of Family Physicians of Canada-Emergency Medicine [CCFP-EM] program, six junior residents in the Fellow of the Royal College of Physicians of Canada-Emergency Medicine [FRCP-EM] program, six senior residents in the FRCP-EM). Interrater reliability for the EA scores was good but varied between scenarios (Spearman rho = [1] 0.68, [2] 0.81, [3] 0.41). Interrater reliability for the GAS was also good, with less variability (rho = [1] 0.64, [2] 0.56, [3] 0.62). When comparing GAS scores, senior FRCP residents outperformed CCFP-EM residents in all scenarios and junior residents in two of three scenarios (p < 0.001 to 0.01). Based on EA scores, senior FRCP residents outperformed CCFP-EM residents, but junior residents outperformed senior FRCP residents in scenario 1 and CCFP-EM residents in all scenarios (p = 0.006 to 0.04). CONCLUSION: This study outlines the creation of a high-fidelity simulation assessment tool for trainees in emergency medicine. A single-point GAS demonstrated stronger relational validity and more consistent reliability in comparison with an EA checklist. This preliminary work will provide a foundation for ongoing future development of simulation-based assessment tools.

Performance criteria for emergency medicine residents: a job analysis.

OBJECTIVE: A major role of admission interviews is to assess a candidate's suitability for a residency program. Structured interviews have greater reliability and validity than do unstructured ones. The development of content for a structured interview is typically based on the dimensions of performance that are perceived as important to succeed in a particular line of work. A formal job analysis is normally conducted to determine these dimensions. The dimensions essential to succeed as an emergency medicine (EM) resident have not yet been studied. We aimed to analyze the work of EM residents to determine these essential dimensions. METHODS: The "critical incident technique" was used to generate scenarios of poor and excellent resident performance. Two reviewers independently read each scenario and labelled the performance dimensions that were reflected in each. All labels assigned to a particular scenario were pooled and reviewed again until a consensus was reached. RESULTS: Five faculty members (25% of our total faculty) comprised the subject experts. Fifty-one incidents were generated and 50 different labels were applied. Eleven dimensions of performance applied to at least 5 incidents. "Professionalism" was the most valued performance dimension, represented in 56% of the incidents, followed by "self-confidence" (22%), "experience" (20%) and "knowledge" (20%). CONCLUSION: "Professionalism," "self-confidence," "experience" and "knowledge" were identified as the performance dimensions essential to succeed as an EM resident based on our formal job analysis using the critical incident technique. Performing a formal job analysis may assist training program directors with developing admission interviews.

Simulation in interprofessional education for patient-centred collaborative care.

AIM: This paper is a report of preliminary evaluations of an interprofessional education through simulation project by focusing on learner and teacher reactions to the pilot modules. BACKGROUND: Approaches to interprofessional education vary widely. Studies indicate, however, that active, experiential learning facilitate it. Patient simulators require learners to incorporate knowing, being and doing in action. A theoretically based competency framework was developed to guide interprofessional education using simulation. The framework includes a typology of shared, complementary and profession-specific competencies. Each competency type is associated with an intraprofessional, multiprofessional, or interprofessional teaching modality and with the professional composition of learner groups. METHOD: The project is guided by an action research approach in which ongoing evaluation generates knowledge to modify and further develop it. Preliminary evaluations of the first pilot module, cardiac resuscitation rounds, among 101 nursing students, 42 medical students and 70 junior medical residents were conducted in 2005-2007 using a questionnaire with rating scales and open-ended questions. Another 20 medical students, 7 junior residents and 45 nursing students completed a questionnaire based on the Interdisciplinary Education Perception scale. FINDINGS: Simulation-based learning provided students with interprofessional activities they saw as relevant for their future as practitioners. They embraced both the interprofessional and simulation components enthusiastically. Attitudinal scores and responses were consistently positive among both medical and nursing students. CONCLUSION: Interprofessional education through simulation offers a promising approach to preparing future healthcare professionals for the collaborative models of healthcare delivery being developed internationally.

Interprofessional resuscitation rounds: a teamwork approach to ACLS education.

PURPOSE: We developed and implemented a series of interprofessional resuscitation rounds targeting fourth year nursing and medical students, and junior residents from a variety of specialty programs. METHODS: Each two hour session was conducted in our patient simulation lab, and was held weekly during the academic year. Students were given specific instruction on the roles and responsibilities of resuscitation team members, and then teams of five worked through pre-defined Advanced Cardiac Life Support (ACLS) scenarios on a high fidelity patient simulator. At the end of each session students completed an anonymous evaluation of the program via a standardized questionnaire using Likert rating scales. RESULTS: A total of 222 evaluations (101 nursing students, 42 medical students, and 79 junior residents) were submitted from October 2005 to April 2006. Mean scores reflected a strong consensus that these rounds were valuable for their training, provided a vehicle for understanding team roles in resuscitation, and that these rounds should be mandatory for all medical and nursing trainees. Participants also expressed a desire for additional interprofessional training. CONCLUSION: Despite challenges inherent in teaching a diverse group of learners, these interprofessional resuscitation rounds were rated highly by nursing and medical trainees as valuable learning experiences.