Motivations for Entrustable Professional Activity Assessment: Gaps Between Curriculum Theory and Resident Reality.

Background Previous research demonstrates mixed reactions from residents toward competency-based medical education (CBME), and entrustable professional activities (EPAs) specifically. However, understanding what motivates residents to obtain EPAs may be vital to the longevity of CBME, given the emphasis on assessment for learning under this paradigm. Objective This study explored resident perspectives across 3 domains: motivation for obtaining EPAs, perceived importance of EPAs, and overall thoughts on CBME curriculum. Methods This was a sequential exploratory mixed-methods study involving 2 phases of data collection. Phase 1 was semi-structured interviews with residents enrolled in CBME at one Canadian institution from November 2019 to July 2020. Analyses included thematic and manifest content analysis. Phase 2 was an electronic close-ended survey to capture residents' primary motivation for requesting EPAs and importance of EPAs for learning. Survey data were analyzed descriptively. Results Of 120 eligible residents, 25 (21%) and 107 (89%) participated in the interview and survey, respectively. Program requirement was the dominant motivation for obtaining EPAs. There was variability in perceived importance of EPAs on learning. Increased resident workload, gaming the system to maximize EPA scores, and lack of shared ownership from preceptors were cited as critiques of the curriculum. Survey responses corroborated interview findings. Conclusions Although many residents recognize the value of EPAs, the majority are not intrinsically motivated to seek out assessment under the current CBME framework.

Competence by Design: The Role of High-Stakes Examinations in a Competence Based Medical Education System.

Competency based medical education is developed utilizing a program of assessment that ideally supports learners to reflect on their knowledge and skills, allows them to exercise a growth mindset that prepares them for coaching and eventual lifelong learning, and can support important progression and certification decisions. Examinations can serve as an important anchor to that program of assessment, particularly when considering their strength as an independent, third-party assessment with evidence that they can predict future physician performance and patient outcomes. This paper describes the aims of the Royal College of Physicians and Surgeons of Canada's ("the Royal College") certification examinations, their future role, and how they relate to the Competence by Design model, particularly as the culture of workplace assessment and the evidence for validity evolves. For example, high-stakes examinations are stressful to candidates and focus learners on exam preparation rather than clinical learning opportunities, particularly when they should be developing greater autonomy. In response, the Royal College moved the written examination earlier in training and created an exam quality review, by a specialist uninvolved in development, to review the exam for clarity and relevance. While learners are likely to continue to focus on the examination as an important hurdle to overcome, they will be preparing earlier in training, allowing them the opportunity to be more present and refine their knowledge when discussing clinical cases with supervisors in the Transition to Practice phase. The quality review process better aligns the exam to clinical practice and can improve the educational impact of the examination preparation process.

Implicit versus explicit first impressions in performance-based assessment: will raters overcome their first impressions when learner performance changes?

First impressions can influence rater-based judgments but their contribution to rater bias is unclear. Research suggests raters can overcome first impressions in experimental exam contexts with explicit first impressions, but these findings may not generalize to a workplace context with implicit first impressions. The study had two aims. First, to assess if first impressions affect raters' judgments when workplace performance changes. Second, whether explicitly stating these impressions affects subsequent ratings compared to implicitly-formed first impressions. Physician raters viewed six videos where learner performance either changed (Strong to Weak or Weak to Strong) or remained consistent. Raters were assigned two groups. Group one (n = 23, Explicit) made a first impression global rating (FIGR), then scored learners using the Mini-CEX. Group two (n = 22, Implicit) scored learners at the end of the video solely with the Mini-CEX. For the Explicit group, in the Strong to Weak condition, the FIGR (M = 5.94) was higher than the Mini-CEX Global rating (GR) (M = 3.02, p < .001). In the Weak to Strong condition, the FIGR (M = 2.44) was lower than the Mini-CEX GR (M = 3.96 p < .001). There was no difference between the FIGR and the Mini-CEX GR in the consistent condition (M = 6.61, M = 6.65 respectively, p = .84). There were no statistically significant differences in any of the conditions when comparing both groups' Mini-CEX GR. Therefore, raters adjusted their judgments based on the learners' performances. Furthermore, raters who made their first impressions explicit showed similar rater bias to raters who followed a more naturalistic process.

The development and evaluation of a provider-focused educational intervention about alcohol use disorder in patients with cirrhosis.

Background: Alcohol use disorder (AUD) is a leading cause of cirrhosis. Insufficient clinician knowledge and comfort managing AUD impacts access to treatment. Using Kern's Framework for Curriculum Development, we aimed to (i) develop and evaluate the effect of an "AUD in cirrhosis" educational intervention on clinicians' knowledge, attitudes, comfort, preparedness, and intention (practice habits) to integrate AUD management into their practice, and (ii) assess clinicians' motivation using Self Determination Theory. Methods: Kern's approach was used for curriculum development. Pilot session feedback informed a three-part flipped-classroom series conducted by interdisciplinary clinicians in hepatology, psychiatry, primary care, and addiction psychology. Participants watched a video followed by a live session focused on (a) withdrawal, (b) screening and brief intervention, and (c) prescribing pharmacotherapy. Questionnaires assessing knowledge and practice habits were adapted from the literature. Attitudes were evaluated using the Short Alcohol and Alcohol Problems Perception Questionnaire (SAAPPQ). Self Determination Theory informed motivation questions. Results: Paired sample t-tests on pre-post questionnaires (n = 229 clinicians; 95 completed questionnaires) revealed significant improvements in preparedness and comfort screening, providing a brief intervention, prescribing pharmacotherapy, and SAAPPQ domains. No significant changes were observed in the intention to prescribe pharmacotherapy. Effect size analysis showed medium to large effects across most topic areas. Conclusions: The developed sessions improved knowledge, attitudes, and practice habits of clinicians caring for this patient population. Given the rise in AUD and significant consequences in cirrhosis, this data offers promise that interactive education may improve practice habits of clinicians interfacing with this patient population.

"Building Your Neurology Acumen": A Flipped Classroom Approach for Internal Medicine Residents.

Globally, internal medicine (IM) residents often feel they lack the knowledge and skills to approach patients presenting with neurologic issues. We conducted a multiple method needs assessment to determine the feasibility of a novel neurology flipped classroom (FC) curriculum for internal medicine residents. Our primary findings include participants: (1) finding neurology a useful rotation; (2) feeling uncomfortable with the neurological examination; and (3) endorsing flipped classroom as a potential alternative but with significant barriers. Our findings elucidate upon the various extrinsic/intrinsic motivators for resident education and illustrate the need to re-examine the way in which neurology is being taught to off-service residents.

Clinicians' perspectives and perceived barriers to caring for patients with alcohol use disorder and cirrhosis.

BACKGROUND: Alcohol use disorder (AUD) is one of the leading etiologies for liver cirrhosis and liver transplantation. Few individuals with AUD receive guideline-based care in the form of screening, brief intervention, referral to treatment, or prescription of pharmacotherapy for relapse prevention. We interviewed clinicians across Alberta to assess the current experience and perceived barriers to managing AUD in people who have cirrhosis. The aim of this paper is to summarize these findings to inform the development of an educational intervention. METHODS: We used a qualitative descriptive approach to explore the experiences of clinicians who care for patients with cirrhosis and AUD in Alberta. We conducted semi-structured interviews directed by an interview guide. Interviews were recorded and transcribed verbatim. We used an inductive thematic analysis approach whereby transcripts were coded, with codes grouped into larger categories, then themes. RESULTS: Sixteen clinicians participated in this study. Many participants acknowledged that they do not use a standardized approach to screening, brief intervention, and referral to treatment. Through thematic analysis we identified four themes surrounding barriers to managing AUD in patients with cirrhosis: (i) Practicing within knowledge constraints, (ii) Navigating limited resources and system challenges, (iii) Balancing the complexity of cirrhosis and AUD, and (iv) Acknowledging the influence of provider perceptions on care. CONCLUSION: This article presents the perspectives of clinicians who care for people who have AUD and cirrhosis. Significant barriers exist, including limited knowledge and resources, systemic challenges, and patient complexity. The information gathered will be used to develop an educational intervention that will delve deeper into these issues in order to have the greatest impact on clinicians who routinely interface with this patient population.

Critical care ultrasound training: a survey exploring the "education gap" between potential and reality in Canada.

BACKGROUND: Critical care ultrasound (CCUS) is now a core competency for Canadian critical care medicine (CCM) physicians, but little is known about what education is delivered, how competence is assessed, and what challenges exist. We evaluated the Canadian CCUS education landscape and compared it against published recommendations. METHODS: A 23-item survey was developed and incorporated a literature review, national recommendations, and expert input. It was sent in the spring of 2019 to all 13 Canadian Adult CCM training programs via their respective program directors. Three months were allowed for data collection and descriptive statistics were compiled. RESULTS: Eleven of 13 (85%) programs responded, of which only 7/11 (64%) followed national recommendations. Curricula differed, as did how education was delivered: 8/11 (72%) used hands-on training; 7/11 (64%) used educational rounds; 5/11 (45%) used image interpretation sessions, and 5/11 (45%) used scan-based feedback. All 11 employed academic half-days, but only 7/11 (64%) used experience gained during clinical service. Only 2/11 (18%) delivered multiday courses, and 2/11 (18%) had mandatory ultrasound rotations. Most programs had only 1 or 2 local CCUS expert-champions, and only 4/11 (36%) assessed learner competency. Common barriers included educators receiving insufficient time and/or support. CONCLUSIONS: Our national survey is the first in Canada to explore CCUS education in critical care. It suggests that while CCUS education is rapidly developing, gaps persist. These include variation in curriculum and delivery, insufficient access to experts, and support for educators.

Learner Phenotypes in Competency-Based Medical Education.

With the launch of competency-based medical education internationally, the practical realities of implementation have failed to live up to many of the proposed theoretical benefits. Specifically, as educators we have observed a number of assessment challenges that seem directly related to identified learner phenotypes. This commentary seeks to describe these specific learner phenotypes, along with actionable recommendations for programs and their competence committees in order to overcome the associated obstacles in assessment. We describe strategies related to both the individual and program level, which can be utilized for both short-term adjustments and long-term programmatic transformation.

Development of a critical care ultrasound curriculum using a mixed-methods needs assessment and engagement of frontline healthcare professionals.

PURPOSE: Experts recommend that critical care medicine (CCM) practitioners should be adept at critical care ultrasound (CCUS). Published surveys highlight that many institutions have no deliberate strategy, no formalized curriculum, and insufficient engagement of CCM faculty and trainees. Consequently, proficiency is non-uniform. Accordingly, we performed a needs assessment to develop an inter-professional standardized CCUS curriculum as a foundation towards universal basic fluency. METHODS: Mixed-methods study of CCM trainees, attendings, and nurse practitioners working across five academic and community medical-surgical intensive care units in Edmonton, Alberta. We used qualitative focus groups followed by quantitative surveys to explore, refine, and integrate results into a curriculum framework. RESULTS: Focus groups with 19 inter-professional practitioners identified major themes including perceived benefits, learning limitations, priorities, perceived risks, characteristics of effective instruction, ensuring long-term success, and achieving competency. Sub-themes highlighted rapid attrition of skill following one- to two-day workshops, lack of skilled faculty, lack of longitudinal training, and the need for site-based mentorship. Thirty-five practitioners (35/70: 50%) completed the survey. Prior training included workshops (16/35; 46%) and self-teaching (11/35; 31%). Eleven percent (4/35) described concerns about potential errors in CCUS performance. The survey helped to refine resources, content, delivery, and assessment. Integration of qualitative and quantitative findings produced a comprehensive curriculum framework. CONCLUSION: Building on published recommendations, our needs assessment identified additional priorities for a CCUS curriculum framework. Specifically, there is a perceived loss of skills following short workshops and insufficient strategies to sustain learning. Addressing these deficits could narrow the gap between national recommendations and frontline needs.

RéSUMé: OBJECTIF: Les experts recommandent que les intensivistes soient habiles en échographie aux soins intensifs. Les sondages publiés révèlent que de nombreux établissements ne possèdent pas de stratégie réfléchie ni de curriculum formalisé en échographie, tout en souffrant d'un manque d'implication du corps professoral et des résidents. Les aptitudes ne sont donc pas uniformes. C'est pourquoi nous avons réalisé une évaluation des besoins afin de mettre au point un curriculum interprofessionnel standardisé en échographie aux soins intensifs qui servira de fondation vers une maîtrise des aptitudes de bases universelles. MéTHODE: Nous avons réalisé une étude en méthodes mixtes auprès de résidents, de patrons et d'infirmiers et infirmières praticiens en soins intensifs travaillant dans cinq unités de soins intensifs médico-chirurgicaux universitaires et communautaires à Edmonton, en Alberta. Nous avons formé des groupes de réflexion qualitatifs puis avons réalisé des sondages quantitatifs afin d'explorer, d'approfondir et d'intégrer nos résultats dans un cadre de curriculum. RéSULTATS: Des groupes de réflexion composés de 19 praticiens interprofessionnels ont identifié des thèmes majeurs, notamment les avantages perçus, les obstacles à l'apprentissage, les priorités, les risques perçus, les caractéristiques d'un enseignement efficace, l'assurance d'un succès à long terme et l'acquisition des compétences. Les sous-thèmes ont mis en lumière une attrition rapide des compétences après des ateliers d'un ou deux jours, le manque d'enseignants compétents, le manque de formation longitudinale, et le besoin de mentorat sur le lieu de travail. Trente-cinq praticiens (35/70; 50 %) ont répondu au sondage. Les formations antérieures incluaient des ateliers (16/35; 46 %) et de l'auto-apprentissage (11/35; 31 %). Onze pourcent (4/35) des répondants ont fait part d'inquiétudes concernant les erreurs potentielles dans la performance de l'échographie aux soins intensifs. Le sondage a permis de raffiner les ressources, le contenu, la livraison et l'évaluation. L'intégration des résultats qualitatifs et quantitatifs a permis de réaliser un cadre exhaustif de curriculum. CONCLUSION: En nous appuyant sur les recommandations publiées, notre évaluation des besoins a identifié des priorités supplémentaires pour encadrer une formation d'échographie destinée aux soins intensifs. Plus spécifiquement, il existe une perte perçue des compétences à la suite d'ateliers courts et il manque de stratégies pour soutenir l'apprentissage. En palliant ces manques, il pourrait être possible de réduire le fossé entre les recommandations nationales et les besoins en première ligne.

Effect of Detailed OSCE Score Reporting on Learning and Anxiety in Medical School.

INTRODUCTION: There is growing literature on increasing feedback from Objective Structured Clinical Examinations (OSCEs) and one approach is a score report. The purpose of this study was to implement and evaluate a score report for a second and fourth-year medical school OSCE. METHODS: We developed an electronic OSCE score report that displayed comments and performance by domain within and across stations (checklist items and rating scales were tagged to each domain). Our initial pilot released the score report after pass/fail decisions but subsequent iterations released the score report the same day as the exam. Our evaluation approach included both student surveys and focus groups. RESULTS: Students felt the OSCE score report was accurate, identified strengths and weaknesses, and would likely cause them to take future action, with second-year students more likely to act on the report than fourth year students. The thematic analysis revealed barriers and enablers to utilizing feedback as well as the power of the score report to reduce anxiety. CONCLUSIONS: Our OSCE score report was simple to develop and implement the same day as an OSCE with an overall positive response from students with respect to accuracy and ability to use the information for future learning.

The Canadian Medical Student Ultrasound Curriculum: A Statement From the Canadian Ultrasound Consensus for Undergraduate Medical Education Group.

OBJECTIVES: This study sought to establish by expert review a consensus-based, focused ultrasound curriculum, consisting of a foundational set of focused ultrasound skills that all Canadian medical students would be expected to attain at the end of the medical school program. METHODS: An expert panel of 21 point-of-care ultrasound and educational leaders representing 15 of 17 (88%) Canadian medical schools was formed and participated in a modified Delphi consensus method. Experts anonymously rated 195 curricular elements on their appropriateness to include in a medical school curriculum using a 5-point Likert scale. The group defined consensus as 70% or more experts agreeing to include or exclude an element. We determined a priori that no more than 3 rounds of voting would be performed. RESULTS: Of the 195 curricular elements considered in the first round of voting, the group reached consensus to include 78 and exclude 24. In the second round, consensus was reached to include 4 and exclude 63 elements. In our final round, with 1 additional item added to the survey, the group reached consensus to include an additional 3 and exclude 8 elements. A total of 85 curricular elements reached consensus to be included, with 95 to be excluded. Sixteen elements did not reach consensus to be included or excluded. CONCLUSIONS: By expert opinion-based consensus, the Canadian Ultrasound Consensus for Undergraduate Medical Education Group recommends that 85 curricular elements be considered for inclusion for teaching in the Canadian medical school focused ultrasound curricula.

Impact of tablet-scoring and immediate score sheet review on validity and educational impact in an internal medicine residency Objective Structured Clinical Exam (OSCE).

Introduction: The Objective Structured Clinical Examination (OSCE) is used globally for formative and summative purposes. The objective of this study was to examine the impact of tablet-scoring on sources of validity evidence for an Internal Medicine residency OSCE. Methods: We compared paper-scored OSCEs from 2014 to tablet-scored OSCEs in 2015 for missing data, amount of comments, and time to pass/fail decision. We then examined in 2016 the impact on learning of showing residents their score sheets and asking them to write an action plan immediately after an OSCE. Results: Tablet-scoring significantly reduced stations with missing data from 1.8 to 0.2%, and stations without comments from 42 to 28% with an increase in word count per comment. Time to official results reduced from 3 weeks to 12 h with tablet-scoring. Residents who wrote a learning plan after reviewing their OSCE score sheets were more likely (with medium to large effect sizes) to pursue further studying and/or change their behavior (e.g. history taking or physical examination) in the clinical environment. Conclusions: OSCE tablet-scoring improved many sources of validity evidence, especially educational impact with timeliness of feedback supporting a change in behavior, a hard to achieve goal of educational interventions.

Internal medicine residents' achievement goals and efficacy, emotions, and assessments.

BACKGROUND: Achievement goal theory is consistently associated with specific cognitions, emotions, and behaviours that support learning in many domains, but has not been examined in postgraduate medical education. The purpose of this research was to examine internal medicine residents' achievement goals, and how these relate to their sense of self-efficacy, epistemic emotions, and valuing of formative compared to summative assessments. These outcomes will be important as programs transition more to competency based education that is characterized by ongoing formative assessments. METHODS: Using a correlational design, we distributed a self-report questionnaire containing 49 items measuring achievement goals, self-efficacy, emotions, and response to assessments to internal medicine residents. We used Pearson correlations to examine associations between all variables. RESULTS: Mastery-approach goals were positively associated with self-efficacy and curiosity and negatively correlated with frustration and anxiety. Mastery-approach goals were associated with a greater value for feedback derived from annual ACP exams, end-of-rotation written exams, and annual OSCEs. Performance-approach goals were only associated with valuing ACP exams. CONCLUSION: Mastery-approach goals were associated with self-efficacy and epistemic emotions among residents, two constructs that facilitate autonomous learning. Residents with mastery-approach goals also appeared to value a wider range of types of assessment data. This profile will likely be beneficial for learners in a competency-based environment that involves high levels of formative feedback.

Beyond hands-on and hands-off: supervisory approaches and entrustment on the inpatient ward.

CONTEXT: The concept of entrustment has garnered significant attention in medical specialties, despite variability in supervision styles and entrustment decisions. There is a need to further study the enactment of supervision on inpatient wards to inform competency-based assessment design. METHODS: Attending physicians, while supervising on clinical teaching inpatient wards, were invited to describe a recent moment of enacting supervision with an internal medicine resident. Constructivist grounded theory guided data collection and analysis. Interview transcripts were analysed in iterative cycles to inform data collection. Constant comparison was used to build a theory of supervision from the identified themes. RESULTS: In 2016-2017, 23 supervisors from two Canadian universities with supervision reputations ranging from very involved to less involved participated in one or two interviews (total: 28). Supervisors were not easily dichotomised into styles based on behaviour because all used similar oversight strategies. Supervisors described adjusting between 'hands-on' (e.g. detail oriented) and 'hands-off' (e.g. less visible on ward) styles depending on the context. All also contended with the competing roles of clinical teacher and care provider. Supervisors made a distinction between the terms `entrust' and `trust', and did not grant complete entrustment to senior residents. CONCLUSIONS: We propose that a supervisor's perceived responsibility for the ward underlies adjustments between 'hands-on' (i.e. personal ward responsibility) and 'hands-off' (i.e. shared ward responsibility) styles. Our approaches to clinical supervision model combines this responsibility tension with the tension between patient care and teaching to illustrate four supervisory approaches, each with unique priorities influencing entrustment. Given the fluidity in supervision, documenting changes in oversight strategies, rather than absolute levels of entrustment, may be more informative for assessment purposes. Research is needed to determine if there is sufficient association between the supervision provided, the entrustment decision made and the supervisor's trust in a trainee to use these as proxies in assessing a trainee's competence.

Twelve tips for developing an OSCE that measures what you want.

The Objective Structured Clinical Examination (OSCE) is used globally for both high and low stakes assessment. Despite its extensive use, very few published articles provide a set of best practices for developing an OSCE, and of those that do, none apply a modern understanding of validity. This article provides 12 tips for developing an OSCE guided by Kane's validity framework to ensure the OSCE is assessing what it purports to measure. The 12 tips are presented in the order they would be operationalized during OSCE development.

Lifelong learning along the education and career continuum: meta-analysis of studies in health professions.

INTRODUCTION: Lifelong learning is an integral part of health professionals' maintenance of competence. Several studies have examined the orientation toward lifelong learning at various stages of the education and career continuum; however, none has looked at changes throughout training and practice. The objective of the present study was to determine if there are differences between groups defined by their places on the education and career continuum. METHODS: The authors performed a group-level meta-analysis on studies that used the 14-item Jefferson Scale of Physician Lifelong Learning or its variants. Eleven published articles, which reported on studies with post-secondary health professions students, residents, and practicing health professionals met the inclusion criteria. In total, there were 12 independent data sets, with four data sets per group. RESULTS: In total, over seven thousand students, residents, and practicing health professionals responded to the Jefferson Scale (N=7.269). Individual study means tendency to be high, suggesting a high orientation toward lifelong learning among the trainees (students and residents) and practicing health professionals. Meta-analysis results indicated that the orientation toward lifelong learning tended to increase gradually along the education and career continuum. Significant differences in the group means were found between the trainees and practicing health professionals. CONCLUSION: In the reviewed studies, the orientation toward lifelong learning among students, residents, and practicing professionals was high. Nonetheless, although based on separate cohorts, it appears that the orientation toward lifelong learning continues to develop even after the completion of formal training.

The effect on reliability and sensitivity to level of training of combining analytic and holistic rating scales for assessing communication skills in an internal medicine resident OSCE.

OBJECTIVE: Although previous research has compared checklists to rating scales for assessing communication, the purpose of this study was to compare the effect on reliability and sensitivity to level of training of an analytic, a holistic, and a combined analytic-holistic rating scale in assessing communication skills. METHODS: The University of Alberta Internal Medicine Residency runs OSCEs for postgraduate year (PGY) 1 and 2 residents and another for PGY-4 residents. Communication stations were scored with an analytic scale (empathy, non-verbal skills, verbal skills, and coherence subscales) and a holistic scale. Authors analyzed reliability of individual and combined scales using generalizability theory and evaluated each scale's sensitivity to level of training. RESULTS: For analytic, holistic, and combined scales, 12, 12, and 11 stations respectively yielded a Phi of 0.8 for the PGY-1,2 cohort, and 16, 16, and 14 stations yielded a Phi of 0.8 for the PGY-4 cohort. PGY-4 residents scored higher on the combined scale, the analytic rating scale, and the non-verbal and coherence subscales. CONCLUSION: A combined analytic-holistic rating scale increased score reliability and was sensitive to level of training. PRACTICE IMPLICATIONS: Given increased validity evidence, OSCE developers should consider combining analytic and holistic scales when assessing communication skills.