The Ottawa resident observation form for nurses (O-RON): evaluation of an assessment tool's psychometric properties in different specialties.

BACKGROUND: Workplace-based assessment (WBA) used in post-graduate medical education relies on physician supervisors' feedback. However, in a training environment where supervisors are unavailable to assess certain aspects of a resident's performance, nurses are well-positioned to do so. The Ottawa Resident Observation Form for Nurses (O-RON) was developed to capture nurses' assessment of trainee performance and results have demonstrated strong evidence for validity in Orthopedic Surgery. However, different clinical settings may impact a tool's performance. This project studied the use of the O-RON in three different specialties at the University of Ottawa. METHODS: O-RON forms were distributed on Internal Medicine, General Surgery, and Obstetrical wards at the University of Ottawa over nine months. Validity evidence related to quantitative data was collected. Exit interviews with nurse managers were performed and content was thematically analyzed. RESULTS: 179 O-RONs were completed on 30 residents. With four forms per resident, the ORON's reliability was 0.82. Global judgement response and frequency of concerns was correlated (r = 0.627, P < 0.001). CONCLUSIONS: Consistent with the original study, the findings demonstrated strong evidence for validity. However, the number of forms collected was less than expected. Exit interviews identified factors impacting form completion, which included clinical workloads and interprofessional dynamics.

Implicit versus explicit first impressions in performance-based assessment: will raters overcome their first impressions when learner performance changes?

First impressions can influence rater-based judgments but their contribution to rater bias is unclear. Research suggests raters can overcome first impressions in experimental exam contexts with explicit first impressions, but these findings may not generalize to a workplace context with implicit first impressions. The study had two aims. First, to assess if first impressions affect raters' judgments when workplace performance changes. Second, whether explicitly stating these impressions affects subsequent ratings compared to implicitly-formed first impressions. Physician raters viewed six videos where learner performance either changed (Strong to Weak or Weak to Strong) or remained consistent. Raters were assigned two groups. Group one (n = 23, Explicit) made a first impression global rating (FIGR), then scored learners using the Mini-CEX. Group two (n = 22, Implicit) scored learners at the end of the video solely with the Mini-CEX. For the Explicit group, in the Strong to Weak condition, the FIGR (M = 5.94) was higher than the Mini-CEX Global rating (GR) (M = 3.02, p < .001). In the Weak to Strong condition, the FIGR (M = 2.44) was lower than the Mini-CEX GR (M = 3.96 p < .001). There was no difference between the FIGR and the Mini-CEX GR in the consistent condition (M = 6.61, M = 6.65 respectively, p = .84). There were no statistically significant differences in any of the conditions when comparing both groups' Mini-CEX GR. Therefore, raters adjusted their judgments based on the learners' performances. Furthermore, raters who made their first impressions explicit showed similar rater bias to raters who followed a more naturalistic process.

Implementation of the Canadian Cardiovascular Society guidelines for perioperative risk assessment and management: an interrupted time series study.

PURPOSE: The Canadian Cardiovascular Society (CCS) guidelines for patients undergoing non-cardiac surgery address the lack of standardized management for patients at risk of perioperative cardiovascular complications. Our interdisciplinary group evaluated the implementation of these guidelines. METHODS: We used an interrupted time series design to evaluate the effect of implementation of the CCS guidelines, using routinely collected hospital data. The study population consisted of elective, non-cardiac surgery patients who were: i) inpatients following surgery and ii) age ≥ 65 or age 45-64 yr with a Revised Cardiac Risk Index ≥ 1. Outcomes included adherence to troponin I (TnI) monitoring (primary) and adherence to appropriate consultant care for patients with elevated TnI (secondary). Exploratory outcomes included cost measures and clinical outcomes such as length of stay. RESULTS: We included 1,421 patients (706 pre- and 715 post-implementation). We observed a 67% absolute increase (95% confidence interval, 55 to 80; P < 0.001) in adherence to TnI testing following the implementation of the guidelines. In patients who had elevated TnI following guideline implementation (n = 64), the majority (85%) received appropriate follow-up care in the form of a general medicine or cardiology consult, all received at least one electrocardiogram, and half received at least one advanced cardiac test (e.g., cardiac perfusion scan, or percutaneous intervention). CONCLUSIONS: Our study showed the ability to implement and adhere to the CCS guidelines. Large-scale multicentre evaluations of CCS guideline implementation are needed to gain a better understanding of potential effects on clinically relevant outcomes.

RéSUMé: OBJECTIF: Les lignes directrices de la Société canadienne de cardiologie (SCC) concernant les patients subissant une chirurgie non cardiaque ont été conçues pour pallier l'absence de standardisation dans la prise en charge des patients à risque de complications cardiovasculaires périopératoires. Notre groupe interdisciplinaire a évalué la mise en œuvre de ces lignes directrices. MéTHODE: Nous avons utilisé une méthodologie de série chronologique interrompue pour évaluer l'effet de la mise en œuvre des lignes directrices de la SCC, à l'aide des données hospitalières habituellement recueillies. La population à l'étude se composait de patients de chirurgies non cardiaques non urgentes qui étaient : i) hospitalisés après leur chirurgie et ii) âgés de ≥ 65 ans ou de 45 à 64 ans avec un Indice de risque cardiaque révisé ≥ 1. Les critères d'évaluation comprenaient l'observance du monitorage de la troponine I (TnI) (critère d'évaluation primaire) et l'observance des soins spécialisés appropriés aux patients présentant un taux élevé de TnI (critère secondaire). Les critères exploratoires comprenaient des mesures de coûts et des résultats cliniques tels que la durée de séjour. RéSULTATS: Nous avons inclus 1421 patients (706 avant et 715 après la mise en œuvre). Nous avons observé une augmentation absolue de 67 % (intervalle de confiance de 95 %, 55 à 80; P < 0,001) de l'observance des tests de la TnI suite à la mise en œuvre des lignes directrices. Parmi les patients présentant un taux élevé de TnI suite à la mise en œuvre des lignes directrices (n = 64), la majorité (85%) a reçu des soins de suivi appropriés sous la forme d'une consultation en médecine générale ou en cardiologie; tous ont subi au moins un électrocardiogramme, et la moitié ont passé au moins un examen cardiaque subséquent (p. ex., évaluation de la perfusion myocardique par scintigraphie ou cathétérisme percutané). CONCLUSION: Notre étude a montré qu'il est possible de mettre en œuvre et d'adhérer aux nouvelles lignes directrices de la SCC. Des évaluations multicentriques à grande échelle portant sur la mise en œuvre des lignes directrices de la SCC sont nécessaires pour mieux comprendre ses effets potentiels sur les devenirs cliniquement pertinents.

Avoid reinventing the wheel: implementation of the Ottawa Clinic Assessment Tool (OCAT) in Internal Medicine.

BACKGROUND: Workplace based assessment (WBA) is crucial to competency-based education. The majority of healthcare is delivered in the ambulatory setting making the ability to run an entire clinic a crucial core competency for Internal Medicine (IM) trainees. Current WBA tools used in IM do not allow a thorough assessment of this skill. Further, most tools are not aligned with the way clinical assessors conceptualize performances. To address this, many tools aligned with entrustment decisions have recently been published. The Ottawa Clinic Assessment Tool (OCAT) is an entrustment-aligned tool that allows for such an assessment but was developed in the surgical setting and it is not known if it can perform well in an entirely different context. The aim of this study was to implement the OCAT in an IM program and collect psychometric data in this different setting. Using one tool across multiple contexts may reduce the need for tool development and ensure that tools used have proper psychometric data to support them. METHODS: Psychometrics characteristics were determined. Descriptive statistics and effect sizes were calculated. Scores were compared between levels of training (juniors (PGY1), seniors (PGY2s and PGY3s) & fellows (PGY4s and PGY5s)) using a one-way ANOVA. Safety for independent practice was analyzed with a dichotomous score. Variance components were generated and used to estimate the reliability of the OCAT. RESULTS: Three hundred ninety OCATs were completed over 52 weeks by 86 physicians assessing 44 residents. The range of ratings varied from 2 (I had to talk them through) to 5 (I did not need to be there) for most items. Mean scores differed significantly by training level (p < .001) with juniors having lower ratings (M = 3.80 (out of 5), SD = 0.49) than seniors (M = 4.22, SD = - 0.47) who had lower ratings than fellows (4.70, SD = 0.36). Trainees deemed safe to run the clinic independently had significantly higher mean scores than those deemed not safe (p < .001). The generalizability coefficient that corresponds to internal consistency is 0.92. CONCLUSIONS: This study's psychometric data demonstrates that we can reliably use the OCAT in IM. We support assessing existing tools within different contexts rather than continuous developing discipline-specific instruments.

Point of care ultrasound training for internal medicine: a Canadian multi-centre learner needs assessment study.

BACKGROUND: Significant gaps currently exist in the Canadian internal medicine point-of-care ultrasound (POCUS) curriculum. From a learner's perspective, it remains unknown what key POCUS skills should be prioritized. This needs assessment study seeks to establish educational priorities for POCUS for internal medicine residents at five Canadian residency training programs. METHODS: All internal medicine trainees [postgraduate year (PGY) 1-5] from five internal medicine residency training programs in Canada (n = 598) were invited to complete an online survey on 15 diagnostic POCUS applications, 9 bedside procedures, and 18 POCUS knowledge items. For POCUS applications and procedures, participants were asked how applicable they are to patient care in internal medicine and the participants' reported skills in those domains. Self-reported knowledge and skills were rated on a 5-point Likert scale, where 1 = very poor and 5 = very good. Applicability was rated, where 1 = not at all applicable and 5 = very applicable. RESULTS: A total of 253 of 598 residents (42%) participated in our study. Data from one centre (n = 15) was removed because of low response rate (15%) and significant baseline differences between those trainees and the remaining participants. Of the remaining analyzable data from four training programs (n = 238), participants reported highest applicability to internal medicine for the following applications and procedures: identifying ascites/free fluid [mean applicability score of 4.9 ± standard deviation (SD) 0.4]; gross left ventricular function (mean 4.8 ± SD 0.5) and pericardial effusion (mean 4.7 ± SD 0.5); thoracentesis (mean score 4.9 ± SD 0.3), central line insertion (mean 4.9 ± SD 0.3), and paracentesis (mean 4.9 ± SD 0.3), respectively. Overall reported knowledge/skills was low, with skill gaps being the highest for identifying deep vein thrombosis (mean gap 2.7 ± SD 1.1), right ventricular strain (mean 2.7 ± SD 1.1), and gross left ventricular function (mean 2.7 ± SD 1.0). CONCLUSIONS: Many POCUS applications and procedures were felt to be applicable to the practice of internal medicine. Significant skill gaps exist in the four Canadian training programs included in the study. POCUS curriculum development efforts should target training based on these perceived skill gaps.

Using the Entrustable Professional Activities Framework in the Assessment of Procedural Skills.

BACKGROUND: The entrustable professional activity (EPA) framework has been identified as a useful approach to assessment in competency-based education. To apply an EPA framework for assessment, essential skills necessary for entrustment to occur must first be identified. OBJECTIVE: Using an EPA framework, our study sought to (1) define the essential skills required for entrustment for 7 bedside procedures expected of graduates of Canadian internal medicine (IM) residency programs, and (2) develop rubrics for the assessment of these procedural skills. METHODS: An initial list of essential skills was defined for each procedural EPA by focus groups of experts at 4 academic centers using the nominal group technique. These lists were subsequently vetted by representatives from all Canadian IM training programs through a web-based survey. Consensus (more than 80% agreement) about inclusion of each item was sought using a modified Delphi exercise. Qualitative survey data were analyzed using a framework approach to inform final assessment rubrics for each procedure. RESULTS: Initial lists of essential skills for procedural EPAs ranged from 10 to 24 items. A total of 111 experts completed the national survey. After 2 iterations, consensus was reached on all items. Following qualitative analysis, final rubrics were created, which included 6 to 10 items per procedure. CONCLUSIONS: These EPA-based assessment rubrics represent a national consensus by Canadian IM clinician educators. They provide a practical guide for the assessment of procedural skills in a competency-based education model, and a robust foundation for future research on their implementation and evaluation.