Competence by Design: The Role of High-Stakes Examinations in a Competence Based Medical Education System.

Competency based medical education is developed utilizing a program of assessment that ideally supports learners to reflect on their knowledge and skills, allows them to exercise a growth mindset that prepares them for coaching and eventual lifelong learning, and can support important progression and certification decisions. Examinations can serve as an important anchor to that program of assessment, particularly when considering their strength as an independent, third-party assessment with evidence that they can predict future physician performance and patient outcomes. This paper describes the aims of the Royal College of Physicians and Surgeons of Canada's ("the Royal College") certification examinations, their future role, and how they relate to the Competence by Design model, particularly as the culture of workplace assessment and the evidence for validity evolves. For example, high-stakes examinations are stressful to candidates and focus learners on exam preparation rather than clinical learning opportunities, particularly when they should be developing greater autonomy. In response, the Royal College moved the written examination earlier in training and created an exam quality review, by a specialist uninvolved in development, to review the exam for clarity and relevance. While learners are likely to continue to focus on the examination as an important hurdle to overcome, they will be preparing earlier in training, allowing them the opportunity to be more present and refine their knowledge when discussing clinical cases with supervisors in the Transition to Practice phase. The quality review process better aligns the exam to clinical practice and can improve the educational impact of the examination preparation process.

Is there a role for a learner education handover as part of the Medical Council of Canada assessment and licensing process?

The transition from undergraduate medical education (UGME) to postgraduate medical education (PGME) is a time of vulnerability for medical schools, postgraduate residency programs, and most importantly, traineesThere is a disconnect between the UGME and PGME experience. Student information shared by UGME is primarily summative of knowledge and skills; PGME programs are unaware of specific learner accommodation requirements, tailored supervisory needs, or potential professionalism concerns identified during UGMEThis lack of integration between UGME and PGME increases potential risk to learners, postgrad programs and patientsBetter linkages and communication along the education continuum could optimize learning and reduce inefficiency and riskThe Medical Council of Canada (MCC) has asked if there is a role for a learner handover (LH) within their licensing processes; however the intended purpose of an LH must first be determinedA Canadian-based LH referred to as a Learner Education Handover (LEH) model including disclosure of student learning/disability accommodation needs, general health concerns, EDI/religious requirements, professionalism concerns, and recommendations for special focus in residency of specific areas of medical knowledge/skill is described.Findings from beta and pilot testing support the value and feasibility of the LEH model. Fundamental principles are outlined: LEH occurs post-residency matchLEH should be forward facing; focused on ongoing or recurring learner issues and needsLearners must be included in the processImplementation would require participation by all Canadian medical schools and all learnersImplementation challenges include: Ensuring learner safety following information disclosureEngaging UGME DeansProtection of information ensuring a 'need-to-know' status is maintainedIncorporating the LEH into the licensing activity could enable the MCC to support a system that proactively responds to learner needs, optimizes physician performance and promotes safe, high quality patient care.

La transition de la formation médicale prédoctorale (FMPrD) vers la formation médicale postdoctorale (FMPoD) est une période de vulnérabilité pour les facultés de médecine, les programmes de résidence et, surtout, les apprenants.Un gouffre sépare l'expérience de la FMPrD et celle de la FMPoD. L'information sur les étudiants partagée par les programmes de FMPrD consiste principalement en une évaluation sommative de leurs connaissances et habiletés; les programmes de FMPoD ne sont pas renseignés sur les besoins d'accommodement spécifiques et les besoins de supervision sur mesure des apprenants ou sur d'éventuelles préoccupations en lien avec la conduite professionnelle relevés pendant la formation de premier cycle.Ce manque d'intégration entre la FMPrC et la FMPoD augmente les risques pour les apprenants, les programmes de formation postdoctorale et les patients.Des liens plus solides et une meilleure communication tout au long du continuum éducatif pourraient optimiser l'apprentissage et réduire l'inefficacité et les risques.Le Conseil médical du Canada (CMC) a posé la question à savoir s'il y aurait une place pour le transfert d'information sur les apprenants dans le cadre de ses processus d'octroi de licences; toutefois, il faut d'abord déterminer l'objectif visé par le transfert d'information.Nous décrivons un modèle canadien de transfert d'information sur les apprenants, appelé modèle de transfert pour la formation des apprenants (TFA), qui comprend la divulgation des besoins de formation des apprenants et les mesures d'accommodement nécessaires selon leurs difficultés, des préoccupations générales en matière de santé, des exigences en matière d'équité/diversité/inclusion et de religion, des préoccupations en matière de professionnalisme et des recommandations concernant l'accent à mettre sur des domaines spécifiques des connaissances et d'habiletés pendant la résidence.Les résultats des tests bêta et des essais pilotes confirment la valeur et la faisabilité du modèle TFA.Le modèle est fondé sur les principes fondamentaux suivants : Le transfert d'information sur les apprenants a lieu après le jumelage de résidenceLe TFA doit être orienté vers l'avenir; il est axé sur les problèmes et les besoins permanents ou récurrents des apprenantsLes apprenants doivent participer au processusLa mise en œuvre du modèle exigerait la participation de toutes les facultés de médecine et de tous les apprenants au CanadaLa mise en œuvre comprend les défis suivants : Assurer la sécurité des apprenants après la divulgation de l'informationMobiliser les vice-deans des programmes de FMPrCAssurer la protection de l'information en respectant le principe du besoin de savoirL'intégration du transfert d'information sur les apprenants à l'activité d'octroi de licences pourrait permettre au CMC de soutenir un système qui répond de manière proactive aux besoins des apprenants, optimise le rendement des médecins et favorise la sécurité et la qualité des soins aux patients.

Effective accreditation in postgraduate medical education: from process to outcomes and back.

BACKGROUND: The accreditation of medical educational programs is thought to be important in supporting program improvement, ensuring the quality of the education, and promoting diversity, equity, and population health. It has long been recognized that accreditation systems will need to shift their focus from processes to outcomes, particularly those related to the end goals of medical education: the creation of broadly competent, confident professionals and the improvement of health for individuals and populations. An international group of experts in accreditation convened in 2013 to discuss this shift. MAIN TEXT: Participants unequivocally supported the inclusion of more outcomes-based criteria in medical education accreditation, specifically those related to the societal accountability of the institutions in which the education occurs. Meaningful and feasible outcome metrics, however, are hard to identify. They are regionally variable, often temporally remote from the educational program, difficult to measure, and susceptible to confounding factors. The group identified the importance of health outcomes of the clinical milieu in which education takes place in influencing outcomes of its graduates. The ability to link clinical data with individual practice over time is becoming feasible with large repositories of assessment data linked to patient outcomes. This was seen as a key opportunity to provide more continuous oversight and monitoring of program impact. The discussants identified several risks that might arise should outcomes measures completely replace process issues. Some outcomes can be measured only by proxy process elements, and some learner experience issues may best be measured by such process elements: in brief, the "how" still matters. CONCLUSIONS: Accrediting bodies are beginning to view the use of practice outcome measures as an important step toward better continuous educational quality improvement. The use of outcomes will present challenges in data collection, aggregation, and interpretation. Large datasets that capture clinical outcomes, experience of care, and health system performance may enable the assessment of multiple dimensions of program quality, assure the public that the social contract is being upheld, and allow identification of exemplary programs such that all may improve. There remains a need to retain some focus on process, particularly those related to the learner experience.

The current utility and future use of the medical student performance record: A survey of perceptions across Canada.

INTRODUCTION: The MSPR is a Canada wide tool that provides aggregate information on MD students' performance during training and used widely as part of PG admissions. This survey study elicits the perceptions of PG admissions stakeholders on the current use and future utility of the MSPR in Canada. METHODS: PG admissions stakeholders across the faculties of medicine were convenience sampled for a 15-question online survey in the fall of 2018. Participants were asked how and when the MSPR is incorporated into the admissions process and perceptions and recommendations for improvement. Data are summarized descriptively and thematically. RESULTS: Responses came from 164 participants across the 17 faculties of medicine. The MSPR was widely used (92%), most commonly in the file review process (52%) for professionalism issues. The majority of responses indicated that MSPRs were not fair for all MD students (60%) and required revision (74%) with greater emphasis required on transparency, professionalism, and narrative comments. DISCUSSION: The results indicate that though MSPRs are widely used in PG admissions their perceived value is limited to a few specific sources of information and to specific parts of the admissions process. There are significant concerns from PG stakeholders on the utility of MSPRs and future changes should align with the needs of these stakeholders while balancing the concerns of students and undergraduate programs.

INTRODUCTION: Le DREM est un outil pancanadien qui procure des renseignements regroupés sur le rendement des étudiants en médecine lors de la formation et il est largement utilisé dans le cadre des admissions post-doctorales. Cette étude par sondage révèle les perceptions des parties prenantes dans les admissions aux études médicales post-doctorales sur l'utilisation actuelle et l'utilité future du DREM au Canada. MÉTHODES: Les parties prenantes impliquées dans les admissions aux études médicales post-doctorales de l'ensemble des facultés de médecine ont été échantillonnés par convenance pour un sondage en ligne de 15 questions à l'automne 2018. Nous avons demandé aux participants comment et quand le DREM est intégré dans le processus d'admission et leurs perceptions et recommandations pour des améliorations. Les données sont résumées de manière descriptive par thèmes. RÉSULTATS: Les réponses proviennent de 164 participants provenant des 17 facultés de médecine. Le DREM a été largement utilisé (92 %) plus communément dans le processus d'examen du dossier (52 %) pour des questions de professionnalisme. La plupart des réponses indiquaient que les DREM n'étaient pas justes pour tous les étudiants en médecine (60 %) et nécessitaient une révision (74 %) avec une attention particulière sur la transparence, le professionnalisme et les commentaires narratifs. DISCUSSION: Les résultats indiquent que bien que les DREM soient largement utilisés dans l'admission aux études médicales post-doctorales, leur valeur perçue est limitée à quelques sources particulières de renseignements et à des parties précises du processus d'admission. Il existe des préoccupations importantes des parties prenantes aux études post-doctorales sur l'utilité des DREM et les changements futurs devraient correspondre aux besoins de ces parties prenantes, tout en équilibrant les préoccupations des étudiants et des programmes de premier cycle.

Design Thinking-Informed Simulation: An Innovative Framework to Test, Evaluate, and Modify New Clinical Infrastructure.

INTRODUCTION: Designing new healthcare facilities is complex and transitions to new clinical environments carry high risks, as unanticipated problems may arise resulting in inefficient care and patient harm. Design thinking, a human-centered design method, represents a unique framework to support the planning, testing, and evaluation of new clinical spaces throughout all phases of construction. Healthcare simulation has been used to test new clinical spaces, yet most report using simulation only in the late design stages. Moreover, healthcare design models have potentially underused human factors approaches calling for human-centered design. We applied a multimodal simulation-based approach underpinned by the principles of design thinking throughout the planning and construction stages of a newly renovated academic emergency department. METHODS: A multidisciplinary team developed and integrated 3 simulation strategies (table-top, mock-up, and in situ simulation) into the 5-step process of design thinking. Through end-user engagement, we identified potential challenges, prototyped solutions through table-top and mock-up simulations, and iteratively tested these solutions through in situ simulation within the actual clinical space. RESULTS: The team used end-user engagement and feedback to brainstorm and implement effective solutions to problems encountered before opening the new emergency department. The iterative steps and targeted use of simulation resulted in redesigning departmental processes and actual clinical space while mitigating anticipated safety threats and departmental deficiencies. CONCLUSIONS: Design thinking coupled with multimodal simulation across all phases of construction enhanced the design and testing of new clinical infrastructure. Applying this approach early, thoroughly, and efficiently will help healthcare organizations plan changes to clinical spaces.

Living in a World of Change: Bridging the Gap From Competency-Based Medical Education Theory to Practice in Canada.

Within graduate medical education, many educators are experiencing a climate of significant change. One transformation, competency-based medical education (CBME), is occurring simultaneously across much of the world, and implementation will require navigating numerous tensions and paradoxes. Successful transformation requires many types of power and is most likely to happen when the medical education community of professionals is engaged in designing, experimenting, acting, and sensemaking together.In this complex climate, the craft of change facilitators and community leaders is needed more than ever. National top-down policies and structures, while important, are not sufficient. The operationalization of new advances is best done when local leaders are afforded room to shape their local context. An evidence-based approach to thinking about the transformative change associated with CBME needs to be adopted. In this age of entrustment, 3 priorities are paramount: (1) engage, entrust, and empower professionals with increasing shared ownership of the innovation; (2) better prepare education professionals in leadership and transformational change techniques in the complex system of medical education; and (3) leverage the wider community of practice to maximize local CBME customization. These recommendations, although based largely on the Canadian experience, are intended to inform CBME transformation in any context.

The competency-based medical education evolution of Canadian emergency medicine specialist training.

Canadian specialist emergency medicine (EM) residency training is undergoing the most significant transformation in its history. This article describes the rationale, process, and redesign of EM competency-based medical education. The rationale for this evolution in residency education includes 1) improved public trust by increasing transparency of the quality and rigour of residency education, 2) improved fiscal accountability to government and institutions regarding specialist EM training, 3) improved assessment systems to replace poor functioning end-of-rotation assessment reports and overemphasis on high-stakes, end-of-training examinations, and 4) and tailored learning for residents to address individualized needs. A working group with geographic and stakeholder representation convened over a 2-year period. A consensus process for decision-making was used. Four key design features of the new residency education design include 1) specialty EM-specific outcomes to be achieved in residency; 2) designation of four progressive stages of training, linked to required learning experiences and entrustable professional activities to be achieved at each stage; 3) tailored learning that provides residency programs and learner flexibility to adapt to local resources and learner needs; and 4) programmatic assessment that emphasizes systematic, longitudinal assessments from multiple sources, and sampling sentinel abilities. Required future study includes a program evaluation of this complex education intervention to ensure that intended outcomes are achieved and unintended outcomes are identified.

It's a Marathon, Not a Sprint: Rapid Evaluation of Competency-Based Medical Education Program Implementation.

PURPOSE: Despite the broad endorsement of competency-based medical education (CBME), myriad difficulties have arisen in program implementation. The authors sought to evaluate the fidelity of implementation and identify early outcomes of CBME implementation using Rapid Evaluation to facilitate transformative change. METHOD: Case-study methodology was used to explore the lived experience of implementing CBME in the emergency medicine postgraduate program at Queen's University, Canada, using iterative cycles of Rapid Evaluation in 2017-2018. After the intended implementation was explicitly described, stakeholder focus groups and interviews were conducted at 3 and 9 months post-implementation to evaluate the fidelity of implementation and early outcomes. Analyses were abductive, using the CBME core components framework and data-driven approaches to understand stakeholders' experiences. RESULTS: In comparing planned with enacted implementation, important themes emerged with resultant opportunities for adaption. For example, lack of a shared mental model resulted in frontline difficulty with assessment and feedback and a concern that the granularity of competency-focused assessment may result in "missing the forest for the trees," prompting the return of global assessment. Resident engagement in personal learning plans was not uniformly adopted, and learning experiences tailored to residents' needs were slow to follow. CONCLUSIONS: Rapid Evaluation provided critical insights into the successes and challenges of operationalizing CBME. Implementing the practical components of CBME was perceived as a sprint, while realizing the principles of CBME and changing culture in postgraduate training was a marathon requiring sustained effort in the form of frequent evaluation and continuous faculty and resident development.

Know thy platypus: The insidiousness of cognitive bias.

Cognitive biases are omnipresent in medical practice. The danger of such biases rests largely on the fact that they are difficult to recognize and, by their nature, are positively reinforcing. This paper describes a real experience in which I discovered a classic example of cognitive bias to which I had fallen victim and the lengths I went to in order to preserve the bias. The paper concludes with some cautionary advice and mitigation strategies for educators and practitioners.

Transition to practice: Evaluating the need for formal training in supervision and assessment among senior emergency medicine residents and new to practice emergency physicians.

OBJECTIVES: Emergency medicine residents may be transitioning to practice with minimal training on how to supervise and assess trainees. Our study sought to examine: 1) physician comfort with supervision and assessment, 2) what the current training gaps are within these competencies, and 3) what barriers or enablers might exist in implementing curricular improvements. METHODS: Qualitative data were collected in two phases through individual interviews from September 2016 to November 2017, at the University of Toronto and McMaster University after receiving ethics approval from both sites. Eligible participants were final year emergency medicine residents, residents pursuing an enhanced skills program in emergency medicine, and attendings within their first 3 years of practice. A semi-structured interview guide was developed and refined after phase one, to reflect content identified in the first set of interviews. All interviews were recorded, transcribed, coded, and collapsed into themes. Data analysis was guided by constructivist grounded theory. RESULTS: A thematic analysis revealed five themes: 1) Supervision and assessment skills were acquired passively through modelling, 2) the training available in these areas is variably used, creating a diversity of comfort levels, 3) competing priorities in the emergency department represent significant barriers to improving supervision and assessment; 4) providing negative feedback is difficult and often avoided; and 5) competence by design will act as an impetus for formal curriculum development in these areas. CONCLUSIONS: As programs transition to competence by design, there will be a need for formal training in supervision and assessment, with a focus on negative feedback, to achieve a standardized level of competence among emergency physicians.

Emergency medicine resident wellness: Lessons learned from a national survey.

OBJECTIVES: Emergency medicine (EM) residents face many challenges during residency. Given the negative effects of residency training and the paucity of information on EM resident wellness experiences, we conducted a national survey to characterize the current landscape of Canadian EM resident wellness. METHODS: A cross-sectional study of Canadian EM residents was done using an online survey created by a Canadian Association of Emergency Physicians Resident Section working group on wellness. Surveys were sent to chief residents in Canadian EM residency programs accredited by either the Royal College of Physicians and Surgeons of Canada (RCPSC) or the College of Family Physicians of Canada (CFPC) in English and French. RESULTS: Thirty-one EM programs were contacted (14 RCPSC and 17 CFPC), and 216 (42%) responses were collected. A multitude of negative wellness impacts were noted, including falling asleep while driving and motor vehicle collisions post-night or during a 24-hour call shift. Moreover, experiences included verbal, physical, and sexual harassment, and reports of low mood and suicidal ideation. Wellness supports were not always accessed after negative incidents. Residents reported deficits in formal wellness instruction, with support for formal EM program wellness time. CONCLUSIONS: Canadian EM residents face a multitude of psychosocial and physical wellness challenges, while supports may not be adequate. Opportunities exist to further investigate resident wellness with validated tools, engage stakeholders, and advance the EM resident wellness agenda.

Back from basics: integration of science and practice in medical education.

CONTEXT: In 1988, the Edinburgh Declaration challenged medical teachers, curriculum designers and leaders to make an organised effort to change medical education for the better. Among a series of recommendations was a call to integrate training in science and clinical practice across a breadth of clinical contexts. The aim was to create physicians who could serve the needs of all people and provide care in a multitude of contexts. In the years since, in the numerous efforts towards integration, new models of curricula have been proposed and implemented with varying levels of success. SCOPE OF REVIEW: In this paper, we examine the evolution of curricular integration since the Edinburgh Declaration, and discuss theoretical advances and practical solutions. In doing so, we draw on recent consensus reports on the state of medical education, emblematic initiatives reported in the literature, and developments in education theory pertinent to the role of integrated curricula. CONCLUSIONS: Interest in integration persists despite 30 years of efforts to respond to the Edinburgh Declaration. We argue, however, that a critical shift has taken place with respect to the conception of integration, whereby empirical models support a view of integration as pertaining to both cognitive activity and curricular structure. In addition, we describe a broader definition of 'basic science' relevant to clinical practice that encompasses social and behavioural sciences, as well as knowledge derived from biomedical science.

Admission factors associated with international medical graduate certification success: a collaborative retrospective review of postgraduate medical education programs in Ontario.

BACKGROUND: The failure rate on certification examinations of The College of Family Physicians of Canada (CFPC) and the Royal College of Physicians and Surgeons of Canada (RCPSC) is significantly higher for international medical graduates than for Canadian medical school graduates. The purpose of the current study was to generate evidence that supports or refutes the validity of hypotheses proposed to explain the lower success rates. METHODS: We conducted retrospective analyses of admissions and certification data to determine the factors associated with success of international medical graduate residents on the certification examinations. International medical graduates who entered an Ontario residency program between 2005 and 2012 and had written a certification examination by the time of the analysis (2015) were included in the study. Data available at the time of admission for each resident, including demographic characteristics, previous experiences and previous professional experiences, were collected from each of the 6 Ontario medical schools and matched with certification examination results provided by The CFPC and the RCPSC. We developed logistic regression models to determine the association of each factor with success on the examinations. RESULTS: Data for 900 residents were analyzed. The models revealed resident age to be strongly associated with performance across all examinations. Fluency in English, female sex and the Human Development Index value associated with the country of medical school training had differential associations across the examinations. INTERPRETATION: The findings should contribute to an improved understanding of certification success by international medical graduates, help residency programs identify at-risk residents and underpin the development of specific educational and remedial interventions. In considering the results, it should be kept in mind that some variables are not amenable to changes in selection criteria.

CAEP 2016 Academic Symposium: How to have an impact as an emergency medicine educator and scholar.

BACKGROUND: In a time of major medical education transformation, emergency medicine (EM) needs to nurture education scholars who will influence EM education practice. However, the essential ingredients to ensure a career with impact in EM education are not clear. OBJECTIVE: To describe how to prepare EM educators for a high-impact career. METHODS: The Canadian Association of Emergency Physicians (CAEP) Academic Section commissioned an "Education Impact" working group (IWG) to guide the creation of consensus recommendations from the EM community. EM educators from across Canada were initially recruited from the networks of the IWG members, and additional educators were recruited via snowball sampling. "High impact educators" were nominated by this network. The high impact educators were then interviewed using a structured question guide. These interviews were transcribed and coded for themes using qualitative methods. The process continued until no new themes were identified. Proposed themes and recommendations were presented to the EM community at the CAEP 2016 Academic Symposium. Feedback was then incorporated into a final set of recommendations. RESULTS: Fifty-five (71%) of 77 of identified Canadian EM educators participated, and 170 names of high impact educators were submitted and ranked by frequency. The IWG achieved sufficiency of themes after nine interviews. Five recommendations were made: 1) EM educators can pursue a high impact career by leveraging either traditional or innovative career pathways; 2) EM educators starting their education careers should have multiple senior mentors; 3) Early-career EM educators should immerse themselves in their area of interest and cultivate a community of practice, not limited to EM; 4) Every academic EM department and EM teaching site should have access to an EM educator with protected time and recognition for their EM education scholarship; and 5) Educators at all stages should continuously compile an impact portfolio. CONCLUSIONS: We describe a unique set of recommendations to develop educators who will influence EM, derived from a consensus from the EM community. EM leaders, educators, and aspiring educational scholars should consider how to implement this guide towards enhancing our specialty's educational mission.