Design and Implementation of a National Program of Assessment Model - Integrating Entrustable Professional Activity Assessments in Canadian Specialist Postgraduate Medical Education.

Traditional approaches to assessment in health professions education systems, which have generally focused on the summative function of assessment through the development and episodic use of individual high-stakes examinations, may no longer be appropriate in an era of competency based medical education. Contemporary assessment programs should not only ensure collection of high-quality performance data to support robust decision-making on learners' achievement and competence development but also facilitate the provision of meaningful feedback to learners to support reflective practice and performance improvement. Programmatic assessment is a specific approach to designing assessment systems through the intentional selection and combination of a variety of assessment methods and activities embedded within an educational framework to simultaneously optimize the decision-making and learning function of assessment. It is a core component of competency based medical education and is aligned with the goals of promoting assessment for learning and coaching learners to achieve predefined levels of competence. In Canada, postgraduate specialist medical education has undergone a transformative change to a competency based model centred around entrustable professional activities (EPAs). In this paper, we describe and reflect on the large scale, national implementation of a program of assessment model designed to guide learning and ensure that robust data is collected to support defensible decisions about EPA achievement and progress through training. Reflecting on the design and implications of this assessment system may help others who want to incorporate a competency based approach in their own country.

The Next Era of Assessment: Building a Trustworthy Assessment System.

Assessment in medical education has evolved through a sequence of eras each centering on distinct views and values. These eras include measurement (e.g., knowledge exams, objective structured clinical examinations), then judgments (e.g., workplace-based assessments, entrustable professional activities), and most recently systems or programmatic assessment, where over time multiple types and sources of data are collected and combined by competency committees to ensure individual learners are ready to progress to the next stage in their training. Significantly less attention has been paid to the social context of assessment, which has led to an overall erosion of trust in assessment by a variety of stakeholders including learners and frontline assessors. To meaningfully move forward, the authors assert that the reestablishment of trust should be foundational to the next era of assessment. In our actions and interventions, it is imperative that medical education leaders address and build trust in assessment at a systems level. To that end, the authors first review tenets on the social contextualization of assessment and its linkage to trust and discuss consequences should the current state of low trust continue. The authors then posit that trusting and trustworthy relationships can exist at individual as well as organizational and systems levels. Finally, the authors propose a framework to build trust at multiple levels in a future assessment system; one that invites and supports professional and human growth and has the potential to position assessment as a fundamental component of renegotiating the social contract between medical education and the health of the public.

Assessing the utility of a novel entrustment-supervision assessment tool.

BACKGROUND: Work-based assessments (WBAs) are increasingly used to inform decisions about trainee progression. Unfortunately, WBAs often fail to discriminate between trainees of differing abilities and have poor reliability. Entrustment-supervision scales may improve WBA performance, but there is a paucity of literature directly comparing them to traditional WBA tools. METHODS: The Ottawa Emergency Department Shift Observation Tool (O-EDShOT) is a previously published WBA tool employing an entrustment-supervision scale with strong validity evidence. This pre-/post-implementation study compares the performance of the O-EDShOT with that of a traditional WBA tool using norm-based anchors. All assessments completed in 12-month periods before and after implementing the O-EDShOT were collected, and generalisability analysis was conducted with year of training, trainees within year and forms within trainee as nested factors. Secondary analysis included assessor as a factor. RESULTS: A total of 3908 and 3679 assessments were completed by 99 and 116 assessors, for 152 and 138 trainees in the pre- and post-implementation phases respectively. The O-EDShOT generated a wider range of awarded scores than the traditional WBA, and mean scores increased more with increasing level of training (0.32 vs. 0.14 points per year, p = 0.01). A significantly greater proportion of overall score variability was attributable to trainees using the O-EDShOT (59%) compared with the traditional tool (21%, p < 0.001). Assessors contributed less to overall score variability for the O-EDShOT than for the traditional WBA (16% vs. 37%). Moreover, the O-EDShOT required fewer completed assessments than the traditional tool (27 vs. 51) for a reliability of 0.8. CONCLUSION: The O-EDShOT outperformed a traditional norm-referenced WBA in discriminating between trainees and required fewer assessments to generate a reliable estimate of trainee performance. More broadly, this study adds to the body of literature suggesting that entrustment-supervision scales generate more useful and reliable assessments in a variety of clinical settings.

The voices of medical education scholarship: Describing the published landscape.

INTRODUCTION: The voices of authors who publish medical education literature have a powerful impact on the field's discourses. Researchers have identified a lack of author diversity, which suggests potential epistemic injustice. This study investigates author characteristics to provide an evidence-based starting point for communal discussion with the intent to move medical education towards a future that holds space for, and values, diverse ways of knowing. METHOD: The authors conducted a bibliometric analysis of all articles published in 24 medical education journals published between 2000 and 2020 to identify author characteristics, with an emphasis on author gender and geographic location and their intersection. Article metadata was downloaded from Web of Science. Genderize.io was used to predict author gender. RESULTS: The journals published 37 263 articles authored by 62 708 unique authors. Males were more prevalent across all authorship positions (n = 62 828; 55.7%) than females (n = 49 975; 44.3%). Authors listed affiliations in 146 countries of which 95 were classified as Global South. Few articles were written by multinational teams (n = 3765; 16.2%). Global South authors accounted for 12 007 (11.4%) author positions of which 3594 (3.8%) were female. DISCUSSION: This study provides an evidence-based starting point to discuss the imbalance of author voices in medical education, especially when considering the intersection of gender and geographical location, which further suggests epistemic injustice in medical education. If the field values a diversity of perspectives, there is considerable opportunity for improvement by engaging the community in discussions about what knowledge matters in medical education, the role of journals in promoting diversity, how to best use this baseline data and how to continue studying epistemic injustice in medical education.

Does direct observation influence the quality of workplace-based assessment documentation?

Background: A key component of competency-based medical education (CBME) is direct observation of trainees. Direct observation has been emphasized as integral to workplace-based assessment (WBA) yet previously identified challenges may limit its successful implementation. Given these challenges, it is imperative to fully understand the value of direct observation within a CBME program of assessment. Specifically, it is not known whether the quality of WBA documentation is influenced by observation type (direct or indirect). Methods: The objective of this study was to determine the influence of observation type (direct or indirect) on quality of entrustable professional activity (EPA) assessment documentation within a CBME program. EPA assessments were scored by four raters using the Quality of Assessment for Learning (QuAL) instrument, a previously published three-item quantitative measure of the quality of written comments associated with a single clinical performance score. An analysis of variance was performed to compare mean QuAL scores among the direct and indirect observation groups. The reliability of the QuAL instrument for EPA assessments was calculated using a generalizability analysis. Results: A total of 244 EPA assessments (122 direct observation, 122 indirect observation) were rated for quality using the QuAL instrument. No difference in mean QuAL score was identified between the direct and indirect observation groups (p = 0.17). The reliability of the QuAL instrument for EPA assessments was 0.84. Conclusions: Observation type (direct or indirect) did not influence the quality of EPA assessment documentation. This finding raises the question of how direct and indirect observation truly differ and the implications for meta-raters such as competence committees responsible for making judgments related to trainee promotion.

Competency-Based Assessment in Psychiatric Education: A Systems Approach.

Medical education programs are failing to meet the health needs of patients and communities. Misalignments exist on multiple levels, including content (what trainees learn), pedagogy (how trainees learn), and culture (why trainees learn). To address these challenges effectively, competency-based assessment (CBA) for psychiatric medical education must simultaneously produce life-long learners who can self-regulate their own growth and trustworthy processes that determine and accelerate readiness for independent practice. The key to effectively doing so is situating assessment within a carefully designed system with several, critical, interacting components: workplace-based assessment, ongoing faculty development, learning analytics, longitudinal coaching, and fit-for-purpose clinical competency committees.

Advancing Workplace-Based Assessment in Psychiatric Education: Key Design and Implementation Issues.

With the adoption of competency-based medical education, assessment has shifted from traditional classroom domains of knows and knows how to the workplace domain of doing. This workplace-based assessment has 2 purposes; assessment of learning (summative feedback) and the assessment for learning (formative feedback). What the trainee does becomes the basis for identifying growth edges and determining readiness for advancement and ultimately independent practice. High-quality workplace-based assessment programs require thoughtful choices about the framework of assessment, the tools themselves, the platforms used, and the contexts in which the assessments take place, with an emphasis on direct observation.

Barriers and Enablers to Direct Observation of Trainees' Clinical Performance: A Qualitative Study Using the Theoretical Domains Framework.

PURPOSE: Direct observation is essential to assess and provide feedback to medical trainees. However, calls for its increased use in medical training persist as learners report that direct observation occurs infrequently. This study applied a theory-driven approach to systematically investigate barriers and enablers to direct observation in residency training. METHOD: From September 2016 to July 2017, semistructured interviews of faculty and residents at The Ottawa Hospital were conducted and analyzed. An interview guide based on the theoretical domains framework (TDF) was used to capture 14 domains that may influence direct observation. Interview transcripts were independently coded using direct content analysis, and specific beliefs were generated by grouping similar responses. Relevant domains were identified based on the frequencies of beliefs reported, presence of conflicting beliefs, and perceived influence on direct observation practices. RESULTS: Twenty-five interviews (12 residents, 13 faculty) were conducted, representing 10 specialties. Ten TDF domains were identified as influencing direct observation: knowledge, skills, beliefs about consequences, social/professional role and identity, intention, goals, memory/attention/decision processes, environmental context and resources, social influences, and behavioral regulation. Discord between faculty and resident intentions, coupled with social expectations that residents should be responsible for ensuring that observations occur, was identified as a key barrier. Additionally, competing demands identified across multiple TDF domains emerged as a pervasive theme. CONCLUSIONS: This study identified key barriers and enablers to direct observation. These influencing factors provide a basis for the development of potential strategies aimed at embedding direct observation as a routine pedagogical practice in residency training.

Evolving concepts of assessment in a competency-based world.

Competency-based medical education (CBME) is an approach to the design of educational systems or curricula that focuses on graduate abilities or competencies. It has been adopted in many jurisdictions, and in recent years an explosion of publications has examined its implementation and provided a critique of the approach. Assessment in a CBME context is often based on observations or judgments about an individual's level of expertise; it emphasizes frequent, direct observation of performance along with constructive and timely feedback to ensure that learners, including clinicians, have the expertise they need to perform entrusted tasks. This paper explores recent developments since the publication in 2010 of Holmboe and colleagues' description of CBME assessment. Seven themes regarding assessment that arose at the second invitational summit on CBME, held in 2013, are described: competency frameworks, the reconceptualization of validity, qualitative methods, milestones, feedback, assessment processes, and assessment across the medical education continuum. Medical educators interested in CBME, or assessment more generally, should consider the implications for their practice of the review of these emerging concepts.

Core principles of assessment in competency-based medical education.

The meaningful assessment of competence is critical for the implementation of effective competency-based medical education (CBME). Timely ongoing assessments are needed along with comprehensive periodic reviews to ensure that trainees continue to progress. New approaches are needed to optimize the use of multiple assessors and assessments; to synthesize the data collected from multiple assessors and multiple types of assessments; to develop faculty competence in assessment; and to ensure that relationships between the givers and receivers of feedback are appropriate. This paper describes the core principles of assessment for learning and assessment of learning. It addresses several ways to ensure the effectiveness of assessment programs, including using the right combination of assessment methods and conducting careful assessor selection and training. It provides a reconceptualization of the role of psychometrics and articulates the importance of a group process in determining trainees' progress. In addition, it notes that, to reach its potential as a driver in trainee development, quality care, and patient safety, CBME requires effective information management and documentation as well as ongoing consideration of ways to improve the assessment system.

CAEP 2016 Academic Symposium: How to have an impact as an emergency medicine educator and scholar.

BACKGROUND: In a time of major medical education transformation, emergency medicine (EM) needs to nurture education scholars who will influence EM education practice. However, the essential ingredients to ensure a career with impact in EM education are not clear. OBJECTIVE: To describe how to prepare EM educators for a high-impact career. METHODS: The Canadian Association of Emergency Physicians (CAEP) Academic Section commissioned an "Education Impact" working group (IWG) to guide the creation of consensus recommendations from the EM community. EM educators from across Canada were initially recruited from the networks of the IWG members, and additional educators were recruited via snowball sampling. "High impact educators" were nominated by this network. The high impact educators were then interviewed using a structured question guide. These interviews were transcribed and coded for themes using qualitative methods. The process continued until no new themes were identified. Proposed themes and recommendations were presented to the EM community at the CAEP 2016 Academic Symposium. Feedback was then incorporated into a final set of recommendations. RESULTS: Fifty-five (71%) of 77 of identified Canadian EM educators participated, and 170 names of high impact educators were submitted and ranked by frequency. The IWG achieved sufficiency of themes after nine interviews. Five recommendations were made: 1) EM educators can pursue a high impact career by leveraging either traditional or innovative career pathways; 2) EM educators starting their education careers should have multiple senior mentors; 3) Early-career EM educators should immerse themselves in their area of interest and cultivate a community of practice, not limited to EM; 4) Every academic EM department and EM teaching site should have access to an EM educator with protected time and recognition for their EM education scholarship; and 5) Educators at all stages should continuously compile an impact portfolio. CONCLUSIONS: We describe a unique set of recommendations to develop educators who will influence EM, derived from a consensus from the EM community. EM leaders, educators, and aspiring educational scholars should consider how to implement this guide towards enhancing our specialty's educational mission.

Implementation of a structured hospital-wide morbidity and mortality rounds model.

IMPORTANCE: There is a paucity of literature on the quality and effectiveness of institutional morbidity & mortality (M&M) rounds processes. OBJECTIVE: We sought to implement and evaluate the effectiveness of a hospital-wide structured M&M rounds model at improving the quality of M&M rounds across multiple specialties. DESIGN, SETTING, PARTICIPANTS: We conducted a prospective interventional study involving 24 clinical groups (1584 physicians) at a tertiary care teaching hospital from January 2013 to June 2015. INTERVENTION: We implemented the published Ottowa M&M Model (OM3): appropriate case selection, cognitive/system issues analyses, interprofessional participation, dissemination of lessons and effector mechanisms. MAIN OUTCOMES AND MEASURES: We created an OM3 scoring index reflecting these elements to measure the quality of M&M rounds. Secondary outcomes include explicit discussions of cognitive/system issues and resultant action items. RESULTS: OM3 scores for all participating groups improved significantly from a median of 12.0/24 (95% CI 10 to 14) to 20.0/24 (95% CI 18 to 21). An increased frequency of in-rounds discussion around cognitive biases (pre 154/417 (37%), post 256/466 (55%); p<0.05) and system issues (pre 175/417 (42%), post 259/466 (62%); p<0.05) were reported by participants via online surveys postintervention, while in-person surveys throughout the intervention period demonstrated even higher frequencies (cognitive biases 1222/1437 (85%); system issues 1250/1437 (87%)). We found 45 action items resulting directly from M&M rounds postintervention, compared with none preintervention. CONCLUSIONS AND RELEVANCE: Implementation of a structured model enhanced the quality of M&M rounds with demonstrable policy improvements hospital wide. The OM3 can be feasibly implemented at other hospitals to effectively improve quality of M&M rounds across different specialties.

Criteria for social media-based scholarship in health professions education.

BACKGROUND: Social media are increasingly used in health professions education. How can innovations and research that incorporate social media applications be adjudicated as scholarship? OBJECTIVE: To define the criteria for social media-based scholarship in health professions education. METHOD: In 2014 the International Conference on Residency Education hosted a consensus conference of health professions educators with expertise in social media. An expert working group drafted consensus statements based on a literature review. Draft consensus statements were posted on an open interactive online platform 2 weeks prior to the conference. In-person and virtual (via Twitter) participants modified, added or deleted draft consensus statements in an iterative fashion during a facilitated 2 h session. Final consensus statements were unanimously endorsed. RESULTS: A review of the literature demonstrated no existing criteria for social media-based scholarship. The consensus of 52 health professions educators from 20 organisations in four countries defined four key features of social media-based scholarship. It must (1) be original; (2) advance the field of health professions education by building on theory, research or best practice; (3) be archived and disseminated; and (4) provide the health professions education community with the ability to comment on and provide feedback in a transparent fashion that informs wider discussion. CONCLUSIONS: Not all social media activities meet the standard of education scholarship. This paper clarifies the criteria, championing social media-based scholarship as a legitimate academic activity in health professions education.

Emergency medicine point-of-care ultrasonography: a national needs assessment of competencies for general and expert practice.

INTRODUCTION: Emergency medicine point-of-care ultrasonography (EM-PoCUS) is a core competency for residents in the Royal College of Physicians and Surgeons of Canada and College of Family Physicians of Canada emergency medicine (EM) training programs. Although EM-PoCUS fellowships are currently offered in Canada, there is little consensus regarding what training should be included in a Canadian EM-PoCUS fellowship curriculum or how this contrasts with the training received in an EM residency.Objectives To conduct a systematic needs assessment of major stakeholders to define the essential elements necessary for a Canadian EM-PoCUS fellowship training curriculum. METHODS: We carried out a national survey of experts in EM-PoCUS, EM residency program directors, and EM residents. Respondents were asked to identify competencies deemed either nonessential to EM practice, essential for general EM practice, essential for advanced EM practice, or essential for EM-PoCUS fellowship trained (''expert'') practice. RESULTS: The response rate was 81% (351 of 435). PoCUS was deemed essential to general EM practice for basic cardiac, aortic, trauma, and procedural imaging. PoCUS was deemed essential to advanced EM practice in undifferentiated symptomatology, advanced chest pathologies, and advanced procedural applications. Expert-level PoCUS competencies were identified for administrative, pediatric, and advanced gynecologic applications. Eighty-seven percent of respondents indicated that there was a need for EM-PoCUS fellowships, with an ideal length of 6 months. CONCLUSION: This is the first needs assessment of major stakeholders in Canada to identify competencies for expert training in EM-PoCUS. The competencies should form the basis for EM-PoCUS fellowship programs in Canada.

When guidelines don't guide: the effect of patient context on management decisions based on clinical practice guidelines.

PURPOSE: This study examines the influence of patient social context on physicians' adherence to clinical practice guidelines (CPGs). METHOD: Expert emergency medicine (EM) physicians and novice physicians (EM residents) were surveyed using an Internet-based program between January and July of 2013. Participants were presented clinical cases and were asked to indicate if they would order or prescribe a specified test or treatment. Cases were chosen from four domains where CPGs exist, and were constructed to include or exclude a "context variable" (CV). Both expert and novice physicians' CPG adherence rate in the CV condition was compared with that in the no CV condition. The CPG adherence rates in CV and no CV conditions were also compared between expert and novice EM physicians. RESULTS: Expert EM physicians (n = 28) were less likely to adhere to CPGs in the CV condition compared with the no CV condition (56% versus 80%, respectively; odds ratio [OR] = 0.32, 95% confidence interval [CI]: 0.17-0.53, P < .001). Experts were less likely to adhere to CPGs in the CV condition when compared with novice physicians (n = 28) (56% versus 67%; OR = 0.62, 95% CI: 0.39-1.0, P = .039). Expert and novice EM physicians did not differ in their adherence to CPGs in the no CV condition. CONCLUSIONS: Participants were sensitive to both the best clinical evidence of benefit, as recommended by CPGs, and patient context when determining how care should be managed.

Education scholarship in emergency medicine part 1: innovating and improving teaching and learning.

As emergency medicine (EM) education evolves, a more advanced understanding of education scholarship is required. This article is the first in a series of three articles that reports the recommendations of the 2013 education scholarship consensus conference of the Academic Section of the Canadian Association of Emergency Physicians. Adopting the Canadian Association for Medical Education's definition, education scholarship (including both research and innovation) is defined. A rationale for why education scholarship should be a priority for EM is discussed.

Education scholarship in emergency medicine part 3: a "how-to" guide.

Successful emergency medicine (EM) education scholarship requires a systematic approach that includes searching the (grey) literature, mobilizing resources, adopting frameworks to focus the innovation, integrating a component of program evaluation, and disseminating the innovation via traditional and emerging avenues. This paper provides direction for EM teachers and educators looking to transform their education innovation into scholarship. Recommendations on producing EM education scholarship from the 2013 consensus conference of the Academic Section of the Canadian Association of Emergency Physicians are presented.

Education scholarship in emergency medicine part 2: supporting and developing scholars.

Emergency medicine (EM) is defined, in part, by clinical excellence across an immense breadth of content and the provision of exemplary bedside teaching to a wide variety of learners. The specialty is also well-suited to a number of emerging areas of education scholarship, particularly in relation to team-based learning, clinical reasoning, acute care response, and simulation-based teaching. The success of EM education scholarship will be predicated on systematic, collective attention to providing the infrastructure for this to occur. Specifically, as a new generation of emergency physicians prepares for education careers, academic organizations need to develop means not only to identify potential scholars but also to mentor, support, and encourage their careers. This paper summarizes the supporting literature and presents related recommendations from a 2013 consensus conference on EM education scholarship led by the Academic Section of the Canadian Association of Emergency Physicians.

Enhancing the quality of morbidity and mortality rounds: the Ottawa M&M model.

OBJECTIVES: The objective of this study was to determine the feasibility and acceptability of a structured morbidity and mortality (M&M) rounds model through an innovative educational intervention. METHODS: The authors engaged the Departments of Emergency Medicine (EM) and Trauma Services at a tertiary care teaching hospital. A needs assessment was performed; the Ottawa M&M rounds model was developed, implemented, and then evaluated as a four-part intervention. This consisted of: 1) physician training on case selection and analysis, 2) engaging interprofessional members, 3) disseminating lessons learned, and 4) creating an administrative pathway for acting on issues identified through the M&M rounds. The measures of intervention feasibility included the proportion of sessions adherent to the new model and M&M rounds attendance. Pre- and postintervention surveys of presenters and attendees were used to determine intervention acceptability. M&M presentation content was reviewed to determine the most frequently adopted components of the model. RESULTS: Nine of 14 (64.3%) sessions were adherent to three of four components of the Ottawa M&M Model. Of those M&M attendees who responded to the survey (796 of 912, 87.2%), improvements were found in M&M rounds attendance as well as perceived effect on clinical practice at both individual and departmental levels. Thirty-seven case presentations were analyzed and improvements postintervention were found in appropriate case selection and recognition of cognitive and system issues. CONCLUSIONS: The Ottawa M&M Model was a feasible intervention that was perceived to be effective by both presenters and attendees. The authors believe that this could be readily applied to any hospital department seeking to enhance quality of care and patient safety.