Implicit versus explicit first impressions in performance-based assessment: will raters overcome their first impressions when learner performance changes?

First impressions can influence rater-based judgments but their contribution to rater bias is unclear. Research suggests raters can overcome first impressions in experimental exam contexts with explicit first impressions, but these findings may not generalize to a workplace context with implicit first impressions. The study had two aims. First, to assess if first impressions affect raters' judgments when workplace performance changes. Second, whether explicitly stating these impressions affects subsequent ratings compared to implicitly-formed first impressions. Physician raters viewed six videos where learner performance either changed (Strong to Weak or Weak to Strong) or remained consistent. Raters were assigned two groups. Group one (n = 23, Explicit) made a first impression global rating (FIGR), then scored learners using the Mini-CEX. Group two (n = 22, Implicit) scored learners at the end of the video solely with the Mini-CEX. For the Explicit group, in the Strong to Weak condition, the FIGR (M = 5.94) was higher than the Mini-CEX Global rating (GR) (M = 3.02, p < .001). In the Weak to Strong condition, the FIGR (M = 2.44) was lower than the Mini-CEX GR (M = 3.96 p < .001). There was no difference between the FIGR and the Mini-CEX GR in the consistent condition (M = 6.61, M = 6.65 respectively, p = .84). There were no statistically significant differences in any of the conditions when comparing both groups' Mini-CEX GR. Therefore, raters adjusted their judgments based on the learners' performances. Furthermore, raters who made their first impressions explicit showed similar rater bias to raters who followed a more naturalistic process.

"Building Your Neurology Acumen": A Flipped Classroom Approach for Internal Medicine Residents.

Globally, internal medicine (IM) residents often feel they lack the knowledge and skills to approach patients presenting with neurologic issues. We conducted a multiple method needs assessment to determine the feasibility of a novel neurology flipped classroom (FC) curriculum for internal medicine residents. Our primary findings include participants: (1) finding neurology a useful rotation; (2) feeling uncomfortable with the neurological examination; and (3) endorsing flipped classroom as a potential alternative but with significant barriers. Our findings elucidate upon the various extrinsic/intrinsic motivators for resident education and illustrate the need to re-examine the way in which neurology is being taught to off-service residents.

Critical care ultrasound training: a survey exploring the "education gap" between potential and reality in Canada.

BACKGROUND: Critical care ultrasound (CCUS) is now a core competency for Canadian critical care medicine (CCM) physicians, but little is known about what education is delivered, how competence is assessed, and what challenges exist. We evaluated the Canadian CCUS education landscape and compared it against published recommendations. METHODS: A 23-item survey was developed and incorporated a literature review, national recommendations, and expert input. It was sent in the spring of 2019 to all 13 Canadian Adult CCM training programs via their respective program directors. Three months were allowed for data collection and descriptive statistics were compiled. RESULTS: Eleven of 13 (85%) programs responded, of which only 7/11 (64%) followed national recommendations. Curricula differed, as did how education was delivered: 8/11 (72%) used hands-on training; 7/11 (64%) used educational rounds; 5/11 (45%) used image interpretation sessions, and 5/11 (45%) used scan-based feedback. All 11 employed academic half-days, but only 7/11 (64%) used experience gained during clinical service. Only 2/11 (18%) delivered multiday courses, and 2/11 (18%) had mandatory ultrasound rotations. Most programs had only 1 or 2 local CCUS expert-champions, and only 4/11 (36%) assessed learner competency. Common barriers included educators receiving insufficient time and/or support. CONCLUSIONS: Our national survey is the first in Canada to explore CCUS education in critical care. It suggests that while CCUS education is rapidly developing, gaps persist. These include variation in curriculum and delivery, insufficient access to experts, and support for educators.

Learner Phenotypes in Competency-Based Medical Education.

With the launch of competency-based medical education internationally, the practical realities of implementation have failed to live up to many of the proposed theoretical benefits. Specifically, as educators we have observed a number of assessment challenges that seem directly related to identified learner phenotypes. This commentary seeks to describe these specific learner phenotypes, along with actionable recommendations for programs and their competence committees in order to overcome the associated obstacles in assessment. We describe strategies related to both the individual and program level, which can be utilized for both short-term adjustments and long-term programmatic transformation.

Development of a critical care ultrasound curriculum using a mixed-methods needs assessment and engagement of frontline healthcare professionals.

PURPOSE: Experts recommend that critical care medicine (CCM) practitioners should be adept at critical care ultrasound (CCUS). Published surveys highlight that many institutions have no deliberate strategy, no formalized curriculum, and insufficient engagement of CCM faculty and trainees. Consequently, proficiency is non-uniform. Accordingly, we performed a needs assessment to develop an inter-professional standardized CCUS curriculum as a foundation towards universal basic fluency. METHODS: Mixed-methods study of CCM trainees, attendings, and nurse practitioners working across five academic and community medical-surgical intensive care units in Edmonton, Alberta. We used qualitative focus groups followed by quantitative surveys to explore, refine, and integrate results into a curriculum framework. RESULTS: Focus groups with 19 inter-professional practitioners identified major themes including perceived benefits, learning limitations, priorities, perceived risks, characteristics of effective instruction, ensuring long-term success, and achieving competency. Sub-themes highlighted rapid attrition of skill following one- to two-day workshops, lack of skilled faculty, lack of longitudinal training, and the need for site-based mentorship. Thirty-five practitioners (35/70: 50%) completed the survey. Prior training included workshops (16/35; 46%) and self-teaching (11/35; 31%). Eleven percent (4/35) described concerns about potential errors in CCUS performance. The survey helped to refine resources, content, delivery, and assessment. Integration of qualitative and quantitative findings produced a comprehensive curriculum framework. CONCLUSION: Building on published recommendations, our needs assessment identified additional priorities for a CCUS curriculum framework. Specifically, there is a perceived loss of skills following short workshops and insufficient strategies to sustain learning. Addressing these deficits could narrow the gap between national recommendations and frontline needs.

RéSUMé: OBJECTIF: Les experts recommandent que les intensivistes soient habiles en échographie aux soins intensifs. Les sondages publiés révèlent que de nombreux établissements ne possèdent pas de stratégie réfléchie ni de curriculum formalisé en échographie, tout en souffrant d'un manque d'implication du corps professoral et des résidents. Les aptitudes ne sont donc pas uniformes. C'est pourquoi nous avons réalisé une évaluation des besoins afin de mettre au point un curriculum interprofessionnel standardisé en échographie aux soins intensifs qui servira de fondation vers une maîtrise des aptitudes de bases universelles. MéTHODE: Nous avons réalisé une étude en méthodes mixtes auprès de résidents, de patrons et d'infirmiers et infirmières praticiens en soins intensifs travaillant dans cinq unités de soins intensifs médico-chirurgicaux universitaires et communautaires à Edmonton, en Alberta. Nous avons formé des groupes de réflexion qualitatifs puis avons réalisé des sondages quantitatifs afin d'explorer, d'approfondir et d'intégrer nos résultats dans un cadre de curriculum. RéSULTATS: Des groupes de réflexion composés de 19 praticiens interprofessionnels ont identifié des thèmes majeurs, notamment les avantages perçus, les obstacles à l'apprentissage, les priorités, les risques perçus, les caractéristiques d'un enseignement efficace, l'assurance d'un succès à long terme et l'acquisition des compétences. Les sous-thèmes ont mis en lumière une attrition rapide des compétences après des ateliers d'un ou deux jours, le manque d'enseignants compétents, le manque de formation longitudinale, et le besoin de mentorat sur le lieu de travail. Trente-cinq praticiens (35/70; 50 %) ont répondu au sondage. Les formations antérieures incluaient des ateliers (16/35; 46 %) et de l'auto-apprentissage (11/35; 31 %). Onze pourcent (4/35) des répondants ont fait part d'inquiétudes concernant les erreurs potentielles dans la performance de l'échographie aux soins intensifs. Le sondage a permis de raffiner les ressources, le contenu, la livraison et l'évaluation. L'intégration des résultats qualitatifs et quantitatifs a permis de réaliser un cadre exhaustif de curriculum. CONCLUSION: En nous appuyant sur les recommandations publiées, notre évaluation des besoins a identifié des priorités supplémentaires pour encadrer une formation d'échographie destinée aux soins intensifs. Plus spécifiquement, il existe une perte perçue des compétences à la suite d'ateliers courts et il manque de stratégies pour soutenir l'apprentissage. En palliant ces manques, il pourrait être possible de réduire le fossé entre les recommandations nationales et les besoins en première ligne.

Effect of Detailed OSCE Score Reporting on Learning and Anxiety in Medical School.

INTRODUCTION: There is growing literature on increasing feedback from Objective Structured Clinical Examinations (OSCEs) and one approach is a score report. The purpose of this study was to implement and evaluate a score report for a second and fourth-year medical school OSCE. METHODS: We developed an electronic OSCE score report that displayed comments and performance by domain within and across stations (checklist items and rating scales were tagged to each domain). Our initial pilot released the score report after pass/fail decisions but subsequent iterations released the score report the same day as the exam. Our evaluation approach included both student surveys and focus groups. RESULTS: Students felt the OSCE score report was accurate, identified strengths and weaknesses, and would likely cause them to take future action, with second-year students more likely to act on the report than fourth year students. The thematic analysis revealed barriers and enablers to utilizing feedback as well as the power of the score report to reduce anxiety. CONCLUSIONS: Our OSCE score report was simple to develop and implement the same day as an OSCE with an overall positive response from students with respect to accuracy and ability to use the information for future learning.

Impact of tablet-scoring and immediate score sheet review on validity and educational impact in an internal medicine residency Objective Structured Clinical Exam (OSCE).

Introduction: The Objective Structured Clinical Examination (OSCE) is used globally for formative and summative purposes. The objective of this study was to examine the impact of tablet-scoring on sources of validity evidence for an Internal Medicine residency OSCE. Methods: We compared paper-scored OSCEs from 2014 to tablet-scored OSCEs in 2015 for missing data, amount of comments, and time to pass/fail decision. We then examined in 2016 the impact on learning of showing residents their score sheets and asking them to write an action plan immediately after an OSCE. Results: Tablet-scoring significantly reduced stations with missing data from 1.8 to 0.2%, and stations without comments from 42 to 28% with an increase in word count per comment. Time to official results reduced from 3 weeks to 12 h with tablet-scoring. Residents who wrote a learning plan after reviewing their OSCE score sheets were more likely (with medium to large effect sizes) to pursue further studying and/or change their behavior (e.g. history taking or physical examination) in the clinical environment. Conclusions: OSCE tablet-scoring improved many sources of validity evidence, especially educational impact with timeliness of feedback supporting a change in behavior, a hard to achieve goal of educational interventions.

Effect of clinically discriminating, evidence-based checklist items on the reliability of scores from an Internal Medicine residency OSCE.

Objective structured clinical examinations (OSCEs) are used worldwide for summative examinations but often lack acceptable reliability. Research has shown that reliability of scores increases if OSCE checklists for medical students include only clinically relevant items. Also, checklists are often missing evidence-based items that high-achieving learners are more likely to use. The purpose of this study was to determine if limiting checklist items to clinically discriminating items and/or adding missing evidence-based items improved score reliability in an Internal Medicine residency OSCE. Six internists reviewed the traditional checklists of four OSCE stations classifying items as clinically discriminating or non-discriminating. Two independent reviewers augmented checklists with missing evidence-based items. We used generalizability theory to calculate overall reliability of faculty observer checklist scores from 45 first and second-year residents and predict how many 10-item stations would be required to reach a Phi coefficient of 0.8. Removing clinically non-discriminating items from the traditional checklist did not affect the number of stations (15) required to reach a Phi of 0.8 with 10 items. Focusing the checklist on only evidence-based clinically discriminating items increased test score reliability, needing 11 stations instead of 15 to reach 0.8; adding missing evidence-based clinically discriminating items to the traditional checklist modestly improved reliability (needing 14 instead of 15 stations). Checklists composed of evidence-based clinically discriminating items improved the reliability of checklist scores and reduced the number of stations needed for acceptable reliability. Educators should give preference to evidence-based items over non-evidence-based items when developing OSCE checklists.

Impact of personal goals on the internal medicine R4 subspecialty match: a Q methodology study.

BACKGROUND: There has been a decline in interest in general internal medicine that has resulted in a discrepancy between internal medicine residents' choice in the R4 subspecialty match and societal need. Few studies have focused on the relative importance of personal goals and their impact on residents' choice. The purpose of this study was to assess if internal medicine residents can be grouped based on their personal goals and how each group prioritizes these goals compared to each other. A secondary objective was to explore whether we could predict a resident's desired subspecialty choice based on their constellation of personal goals. METHODS: We used Q methodology to examine how postgraduate year 1-3 internal medicine residents could be grouped based on their rankings of 36 statements (derived from our previous qualitative study). Using each groups' defining and distinguishing statements, we predicted their subspecialties of interest. We also collected the residents' first choice in the subspecialty match and used a kappa test to compare our predicted subspecialty group to the residents' self-reported first choice. RESULTS: Fifty-nine internal medicine residents at the University of Alberta participated between 2009 and 2010 with 46 Q sorts suitable for analysis. The residents loaded onto four factors (groups) based on how they ranked statements. Our prediction of each groups' desired subspecialties with their defining and/or distinguishing statements are as follows: group 1 - general internal medicine (variety in practice); group 2 - gastroenterology, nephrology, and respirology (higher income); group 3 - cardiology and critical care (procedural, willing to entertain longer training); group 4 - rest of subspecialties (non-procedural, focused practice, and valuing more time for personal life). There was moderate agreement (kappa = 0.57) between our predicted desired subspecialty group and residents' self-reported first choice (p < 0.001). CONCLUSION: This study suggests that most residents fall into four groups based on a constellation of personal goals when choosing an internal medicine subspecialty. The key goals that define and/or distinguish between these groups are breadth of practice, lifestyle, desire to do procedures, length of training, and future income potential. Using these groups, we were able to predict residents' first subspecialty group with moderate success.

Determinants of internal medicine residents' choice in the Canadian R4 fellowship match: a qualitative study.

BACKGROUND: There is currently a discrepancy between Internal Medicine residents' decisions in the Canadian subspecialty fellowship match (known as the R4 match) and societal need. Some studies have been published examining factors that influence career choices. However, these were either demographic factors or factors pre-determined by the authors' opinion as possibly being important to incorporate into a survey. METHODS: A qualitative study was undertaken to identify factors that determine the residents choice in the subspecialty (R4) fellowship match using focus group discussions involving third and fourth year internal medicine residents RESULTS: Based on content analysis of the discussion data, we identified five themes:1) Practice environment including acuity of practice, ability to do procedures, lifestyle, job prospects and income 2) Exposure in rotations and to role models 3) Interest in subspecialty's patient population and common diseases 4) Prestige and respect of subspecialty 5) Fellowship training environment including fellowship program resources and length of training CONCLUSIONS: There are a variety of factors that contribute to Internal Medicine residents' fellowship choice in Canada, many of which have been identified in previous survey studies. However, we found additional factors such as the resources available in a fellowship program, the prestige and respect of a subspecialty/career, and the recent trend towards a two-year General Internal Medicine fellowship in our country.