Evaluating the Construct Validity of Competencies: A Retrospective Analysis.

Background: A competency-based framework focuses on alignment between professional standards and assessment design. This alignment implies improved measurement validity, yet it has not been established that competence in one context predicts performance in another context. High-stakes competence assessments offer insights into the relationship between assessment design and competencies. Methods/Analyses: The internationally educated nurses competency assessment program (IENCAP) was developed at Touchstone Institute in collaboration with the College of Nurses of Ontario (CNO) and includes a 12-station OSCE. Each station evaluated the same 10 competencies. We submitted competency scores to a multi-trait multi-method matrix analysis to evaluate the convergent and discriminant validity of competencies. Results/Observations: All correlations were significant and positive; however, we did not find evidence of convergent or discriminant validity. Correlations were higher between different competencies evaluated within the same station (mean correlation = 0.60) compared to identical competencies evaluated across different stations (mean correlation = 0.19). Discussion: The results do not provide evidence of construct validity for competencies. While competency-based approaches emphasize various generalized knowledge, skills, and attitudes, these findings indicate that the clinical context is a major determinant of performance. Conclusion: The context-dependent nature of competencies requires multiple assessments in varied contexts. Performance on a single competency cannot be determined in a single occasion. Supplementary Information: The online version contains supplementary material available at 10.1007/s40670-023-01794-z.

Assessment Practices in Continuing Professional Development Activities in Health Professions: A Scoping Review.

CONTEXT: In continuing professional development (CPD), educators face the need to develop and implement innovative assessment strategies to adhere to accreditation standards and support lifelong learning. However, little is known about the development and validation of these assessment practices. We aimed to document the breadth and depth of what is known about the development and implementation of assessment practices within CPD activities. METHODS: We conducted a scoping review using the framework proposed by Arksey and O'Malley (2005) and updated in 2020. We examined five databases and identified 1733 abstracts. Two team members screened titles and abstracts for inclusion/exclusion. After data extraction, we conducted a descriptive analysis of quantitative data and a thematic analysis of qualitative data. RESULTS: A total of 130 studies were retained for the full review. Most reported assessments are written assessments (n = 100), such as multiple-choice items (n = 79). In 99 studies, authors developed an assessment for research purpose rather than for the CPD activity itself. The assessment validation process was detailed in 105 articles. In most cases, the authors examined the content with experts (n = 57) or pilot-tested the assessment (n = 50). We identified three themes: 1-satisfaction with assessment choices; 2-difficulties experienced during the administration of the assessment; and 3-complexity of the validation process. CONCLUSION: Building on the adage "assessment drives learning," it is imperative that the CPD practices contribute to the intended learning and limit the unintended negative consequences of assessment. Our results suggest that validation processes must be considered and adapted within CPD contexts.

Exploring gender influences in the quality of workplace-based assessments.

INTRODUCTION: Workplace-based assessments are an important tool for trainee feedback and as a means of reporting expert judgments of trainee competence in the workplace. However, the literature has demonstrated that gender bias can exist within these assessments. We aimed to determine whether gender differences in the quality of workplace-based assessment data exist in our residency training program. METHODS: This study was conducted at the University of Ottawa in the Department of Emergency Medicine. Four end-of-shift workplace-based assessments completed by men faculty and four completed by women faculty were randomly selected for each resident during the 2018-2019 academic year. Two blinded raters scored each workplace-based assessment using the Completed Clinical Evaluation Report Rating (CCERR), a published nine-item quantitative measure of workplace-based assessment quality. A 2 × 2 mixed measures analysis of variance (ANOVA) of resident gender and faculty gender was conducted, with mean CCERR score as the dependent variable. The ANOVA was repeated with mean workplace-based assessment rating as the dependent variable. RESULTS: A total of 363 workplace-based assessments were analyzed for 46 residents. There were no significant effects of faculty or resident gender on the quality of workplace-based assessments (p = 0.30). There was no difference in mean workplace-based assessment ratings between women and men residents (p = 0.92), and no interaction between resident and faculty gender (p = 0.62). Mean CCERR score was 25.8, SD = 4.2, indicating average quality assessments. CONCLUSIONS: We did not find faculty or resident gender differences in the quality of workplace-based assessments completed in our training program. While the literature has previously demonstrated gender bias in trainee assessments, our results are not surprising as assessment culture varies by institution and program. Our study cautions against generalizing gender bias across contexts, and offers an approach that educators can use to evaluate whether gender bias in the quality of trainee assessments exists within their program.

RéSUMé: INTRODUCTION: Les évaluations sur le lieu de travail constituent un outil important pour le retour d'information des stagiaires et comme moyen de rapporter les jugements d'experts sur les compétences des stagiaires sur le lieu de travail. Cependant, la littérature a démontré que des préjugés sexistes peuvent exister dans ces évaluations. Nous avons cherché à déterminer s'il existe des différences entre les sexes dans la qualité des données d'évaluation sur le lieu de travail dans notre programme de formation en résidence. MéTHODES: Cette étude a été menée à l'Université d'Ottawa dans le département de médecine d'urgence. Quatre évaluations en fin de poste de travail complétées par des professeurs hommes et 4 complétées par des professeurs femmes ont été sélectionnées au hasard pour chaque résident au cours de l'année universitaire 2018-2019. Deux évaluateurs en aveugle ont noté chaque évaluation sur le lieu de travail à l'aide du Completed Clinical Evaluation Report Rating (CCERR), une mesure quantitative publiée en neuf points de la qualité de l'évaluation sur le lieu de travail. Une analyse de variance (ANOVA) à mesures mixtes 2 × 2 du sexe des résidents et du sexe des enseignants a été réalisée, avec le score CCERR moyen comme variable dépendante. L'ANOVA a été répétée en prenant comme variable dépendante la note moyenne de l'évaluation sur le lieu de travail. RéSULTATS: Au total, 363 évaluations sur le lieu de travail ont été analysées pour 46 résidents. Il n'y avait aucun effet significatif du sexe du corps professoral ou du résident sur la qualité des évaluations en milieu de travail (p = 0,30). Il n'y avait pas de différence dans les évaluations moyennes sur le lieu de travail entre les femmes et les hommes résidents (p = 0,92), et pas d'interaction entre le sexe du résident et celui de la faculté (p = 0,62). Le score moyen du CCERR était de 25,8, SD = 4,2, ce qui indique des évaluations de qualité moyenne. CONCLUSIONS: Nous n'avons pas constaté de différences entre les sexes au sein du corps professoral ou des résidents en ce qui concerne la qualité des évaluations en milieu de travail effectuées dans le cadre de notre programme de formation. Bien que la littérature ait déjà démontré l'existence de préjugés sexistes dans les évaluations des stagiaires, nos résultats ne sont pas surprenants car la culture de l'évaluation varie selon les établissements et les programmes. Notre étude met en garde contre la généralisation des préjugés sexistes dans tous les contextes et propose une approche que les éducateurs peuvent utiliser pour évaluer s'il existe des préjugés sexistes dans la qualité des évaluations des stagiaires au sein de leur programme.

Is COVID-19 perceived as a threat to equal career opportunities amongst Swiss medical students? A cross-sectional survey study from Bern and Geneva.

Introduction: Students frequently rely on part-time jobs to earn a living wage. We sought to evaluate the sociodemographic status of Swiss medical students and their perception regarding equal career opportunities in view of impaired part-time job opportunities under the COVID-19 pandemic. Methods: We conducted an anonymous online survey among Swiss medical students from Bern and Geneva over a period of 4 months between December 2020 and April 2021. We evaluated sociodemographic data, current living situation, part-time job occupation as well as other sources of income to fund living expenses, and, by means of a five-point Likert scale (1=strongly disagree and 5=strongly agree), whether COVID-19 was perceived as impeding equal career opportunities. Results: Of 968 participants, corresponding to around 13.8% of all medical students in Switzerland, 81.3% had part-time jobs. Amongst the employed, 54.8% worked to afford living expenses and 28.9% reported a negative financial impact due to reduced part-time jobs under the pandemic. The loss of part-time jobs was perceived to make medical studies a privilege for students with higher socioeconomic status (4.11±1.0), whose opportunity to study is independent of a regular income. A governmental backup plan was considered crucial to support affected students (4.22±0.91). Discussion: COVID-19 and its sequelae are perceived as a threat for Swiss medical students and lead to a disadvantage for those with lower socioeconomic status. Nationwide measures should be established to foster equal career opportunities.

Analyzing interprofessional teamwork in the operating room: An exploratory observational study using conventional and alternative approaches.

Intraoperative teamwork is vital for patient safety. Conventional tools for studying intraoperative teamwork typically rely on behaviorally anchored rating scales applied at the individual or team level, while others capture narrative information across several units of analysis. This prospective observational study characterizes teamwork using two conventional tools (Operating Theatre Team Non-Technical Skills Assessment Tool [NOTECHS]; Team Emergency Assessment Measure [TEAM]), and one alternative approach (modified-Systems Engineering Initiative for Patient Safety [SEIPS] model). We aimed to explore the advantages and disadvantages of each for providing feedback to improve teamwork practice. Fifty consecutive surgical cases at a Canadian academic hospital were recorded with the OR Black Box®, analyzed by trained raters, and summarized descriptively. Teamwork performance was consistently high within and across cases rated with NOTECHS and TEAMS. For cases analyzed with the modified-SEIPS tool, both optimal and suboptimal teamwork behaviors were identified, and team resilience was frequently observed. NOTECHS and TEAM provided summative assessments and overall pattern descriptions, while SEIPS facilitated a deeper understanding of teamwork processes. As healthcare organizations continue to prioritize teamwork improvement, SEIPS may provide valuable insights regarding teamwork behavior and the broader context influencing performance. This may ultimately enhance the development and effectiveness of multi-level teamwork interventions.

The Importance of Sociocultural Factors in Moderating the Applicability of Test-Enhanced Learning to Continuing Professional Development.

ABSTRACT: Many studies have shown that repeated testing promotes learning, an effect known as test-enhanced learning. Although test-enhanced learning has been well-established within health care trainees, there are mixed findings of whether testing facilitates learning within continuing professional development (CPD) contexts. This piece argues that for testing to support learning in CPD, there is need to understand the various social, cultural, and structural factors that influence clinicians' willingness to participate in regular testing activities. The solution to this problem is conceptual in nature, whereby CPD programs must consider how learning cultures shape practicing clinicians' perspectives toward testing.

Development and Validation of the Transition Readiness Assessment Instrument in Type 1 Diabetes "On TRAck".

OBJECTIVES: Transition to adult care is challenging for youth with type 1 diabetes (T1D) and their caregivers. We have developed the diabetes-specific "On TRAck" transition readiness scale, and in this study we assess its reliability and validity compared with TRANSITION-Q, a generic transition readiness questionnaire. METHODS: We systematically created 3 versions of On TRAck: adolescent, parent and health-care provider (HCP) versions (for case managers and physicians). Among adolescents 13 to 18 years of age with T1D at a single academic centre, we conducted an exploratory factor analysis and assessed interrater agreement, internal consistency and relationship with age; recent glycated hemoglobin (A1C); and recent diabetic ketoacidosis (DKA) with On TRAck and TRANSITION-Q. RESULTS: One hundred fifteen adolescents (aged 15.8±1.6 years and diabetes duration 6.7±4.1 years), their caregivers and diabetes HCPs participated. The final 24-item adolescent and parent scales (with 3 subscales: "Self-efficacy," "Autonomy" and "Support & maturity") and the 3-item HCP version had a Cronbach's alpha of 0.86 to 0.93. Adolescent scores correlated with parents (r=0.64), case managers (r=0.39) and physicians (r=0.28). Mean adolescent score was 190.3±27.1 points out of 240. Adolescent scores were 3.4 points higher per year of age (p=0.03) and 4.4 points higher for every 1% lower A1C (p=0.01), but were not associated with DKA. TRANSITION-Q was associated with age. On TRAck HCP scores were associated with adolescent's age, A1C and DKA. CONCLUSIONS: On TRAck represents a new psychometrically comprehensive diabetes-specific scale that can be used in adolescent diabetes clinics for measuring transition readiness. It is a multidimensional instrument with ease of use and high reliability scores.

Mental Disorder Symptoms and the Relationship with Resilience among Paramedics in a Single Canadian Site.

There is growing recognition in research and policy of a mental health crisis among Canada's paramedics; however, despite this, epidemiological surveillance of the problem is in its infancy. Just weeks before the emergence of the COVID-19 pandemic, we surveyed paramedics from a single, large, urban paramedic service in Ontario, Canada to assess for symptom clusters consistent with post-traumatic stress disorder (PTSD), major depressive disorder, and generalized anxiety disorder and to identify potential risk factors for each. In total, we received 589 completed surveys (97% completion rate) and found that 11% screened positive for PTSD, 15% screened positive for major depressive disorder, and 15% screened positive for generalized anxiety disorder, with one in four active-duty paramedics screening positive for any of the three as recently as February 2020. In adjusted analyses, the risk of a positive screen varied as a function of employment classification, gender, self-reported resilience, and previous experience as a member of the service's peer support team. Our findings support the position that paramedics screen positive for mental disorders at high rates-a problem likely to have worsened since the onset of the COVID-19 pandemic. We echo the calls of researchers and policymakers for urgent action to support paramedic mental health in Canada.

Role Identity, Dissonance, and Distress among Paramedics.

Role identity theory describes the purpose and meaning in life that comes, in part, from occupying social roles. While robustly linked to health and wellbeing, this may become unideal when an individual is unable to fulfill the perceived requirements of an especially salient role in the manner that they believe they should. Amid high rates of mental illness among public safety personnel, we interviewed a purposely selected sample of 21 paramedics from a single service in Ontario, Canada, to explore incongruence between an espoused and able-to-enact paramedic role identity. Situated in an interpretivist epistemology and using successive rounds of thematic analysis, we developed a framework for role identity dissonance wherein chronic, identity-relevant disruptive events cause emotional and psychological distress. While some participants were able to recalibrate their sense of self and understanding of the role, for others, this dissonance was irreconcilable, contributing to disability and lost time from work. In addition to contributing a novel perspective on paramedic mental health and wellbeing, our work also offers a modest contribution to the theory in using the paramedic context as an example to consider identity disruption through chronic workplace stress.

The McMaster Advanced Communication Competencies Model for Psychiatry (MACC Model).

OBJECTIVE: Communication is a core competency for all physicians in training. In Canada, the importance of communication during residency is recognized through the CanMEDS framework. Although literature exists around teaching communication skills to residents, research in psychiatry residents is lacking. The purpose of this study was to explore how faculty members conceptualize the development of communication skills in psychiatry residents and develop a model reflecting this. METHODS: The authors used a constructivist grounded theory approach. Purposive sampling was used to select 14 faculty educators who regularly supervise psychiatry residents in a single university-based residency training program. Semi-structured interviews were conducted to explore educators' perceptions of how communication skills develop during residency. Constant comparative analysis occurred concurrently with data collection until thematic theoretical sufficiency was reached and relationships between themes determined. RESULTS: Five themes underlie the McMaster Advanced Communication Competencies model, describing the progressive development of communication abilities in residents. Three themes identify foundational sets of abilities including: refining common foundational relational abilities, developing foundational specific psychiatric communication abilities, and learning to reflect upon and manage internal reactions. These foundational pillars then allow residents to develop a personalized art of flexible psychiatric interviewing, and skillfully partner with patients in co-creating care plans. CONCLUSIONS: This research describes a preliminary communication competency model for teaching and assessing psychiatry residents. It defines the core communication abilities required for residents to progress to independent practice. Future research could explore and test the model's validity and transferability.

Classic test and generalizability theories are both useful for cross-cultural adaptation of an allergic rhinitis health-related quality of life questionnaire.

BACKGROUND: An instrument to assess Allergic Rhinitis (AR) Health-Related Quality of Life (HRQL) in adult patients was developed in Spain. No validated instrument is currently cross-culturally adapted for use in daily practice to assess HRQL in AR patients in Colombia. PURPOSE: The aim of this study was to evaluate the measurement performance of an AR-HRQL specific questionnaire, ESPRINT-15 (Cuestionario ESPañol de Calidad de Vida en RINiTis), in Colombian adult patients with AR using the Classic Test Theory (CTT) and the Generalizability theory (G-theory) frameworks. METHODS: We conducted the cross-cultural adaptation in 2 stages. In stage 1, we evaluated comprehensibility, acceptability, and feasibility of ESPRINT-15 in healthy adults and adult patients with AR. In stage 2, we examined both reliability and validity of ESPRINT-15 scores using CTT and overall reliability applying the G-theory in adult patients with AR. RESULTS: For feasibility and acceptability, all items showed a higher than 95% level of understanding, and modifications in the original questionnaire were unnecessary. Reliability and validity using CTT showed a high internal consistency (Cronbach's alpha and Mc McDonald's omega = 0.95) and test-retest reliability (scores from 0.70 to 0.76). The overall reliability score using G-theory was 0.75, and G-coefficients scores associated with internal consistency and test-retest reliability measures were 0.96 and 0.61, respectively. Validity using exploratory factor analysis (EFA) identified 2 factors instead of the original ESPRINT-15 4 domains. However, confirmatory factor analysis (CFA) showed good fit regarding the original model. CONCLUSIONS: The proposed cross-cultural adaptation of ESPRINT-15 showed good reliability and validity measures. Additionally, it was easy to use and administer. ESPRINT-15 can be used clinically and for research in Colombian adults' patients with AR. CTT and the G-theory can be used in epidemiological studies to adapt AR-HRQL questionnaires cross-culturally in adult patients with AR.

A pre-clerkship simulation-based procedural skills curriculum: decreasing anxiety and improving confidence of procedural skill performance.

INTRODUCTION: Pre-clerkship procedural skills training is not yet a standard across Canadian medical school curricula, resulting in limited exposure to procedures upon entering clerkship. While simulated skills training has been documented in the literature to improve performance in technical ability, anxiety and confidence have yet to be investigated despite their documented impact on performance and learning. This study therefore aims to evaluate the effect of pre-clerkship procedural skills training on medical student anxiety and confidence. METHODS: A procedural skills training program was designed based on an evidence-based near-peer, flipped classroom model of education. Ninety-two second-year medical students volunteered for the study. Fifty-six were randomized to the training group, and 36 were randomized to the control group. Students in the training group attended seven procedural skills tutorials over seven months. The control group represented the average medical school student without standardized procedural training. Student anxiety and confidence were assessed at the beginning and end of the program using the State Trait Anxiety Inventory and Confidence Questionnaires. RESULTS: Students who participated in the procedural skills program demonstrated greater reductions in their state anxiety and greater improvements in confidence compared to the control group. CONCLUSION: Longitudinal procedural skills training in the simulation setting has demonstrated improvements in anxiety and confidence among pre-clerkship medical students. These added benefits to training have the potential to ease medical students' transition into clerkship, while also contributing to a safer and more effective clinical experience. Therefore, future integration of standardized pre-clerkship procedural skills training within medical school curricula should be considered.

INTRODUCTION: Les cursus de pré-externat des facultés de médecine canadiennes ne prévoient pas systématiquement de formation sur les habiletés techniques avant, si bien qu'à leur arrivée à l'externat, les étudiants ont une exposition limitée aux gestes techniques. Tandis que l'amélioration des aptitudes techniques par la simulation est bien documentée, il n'y a aucune étude sur l'effet de ces simulations sur l'anxiété et la confiance malgré leur impact avéré sur la performance et l'apprentissage. Cette étude vise donc à évaluer l'effet d'une formation sur les habiletés techniques au pré-externat sur l'anxiété et la confiance des étudiants en médecine. MÉTHODES: Une formation sur les habiletés techniques a été conçue sur la base d'un modèle d'enseignement fondé sur les données probantes, par les quasi-pairs suivant une approche de classe inversée. Quatre-vingt-douze étudiants en deuxième année de médecine ont participé à l'étude sur une base volontaire. Cinquante-six d'entre eux ont été répartis au hasard dans le groupe qui devait recevoir la formation, et 36 dans le groupe contrôle. Les étudiants du groupe qui a reçu la formation ont assisté à sept tutoriels étalés sur une période de sept mois. Le groupe témoin représentait l'étudiant en médecine moyen sans formation normalisée axée sur les habiletés techniques. L'anxiété et la confiance des étudiants ont été évaluées au début et à la fin du programme à l'aide du questionnaire State Trait Anxiety Inventory (questionnaire sur l'anxiété chronique et réactionnelle) et d'un questionnaire sur la confiance. RÉSULTATS: Chez les étudiants qui ont participé au programme de formation sur les habiletés techniques, la baisse de l'anxiété et l'amélioration de la confiance en soi ont été plus importantes que chez les étudiants du groupe contrôle. CONCLUSION: La formation longitudinale axée sur les habiletés techniques en contexte de simulation a eu des effets positifs en ce qui concerne l'anxiété et la confiance chez les étudiants en médecine au pré-externat. La formation offre l'avantage supplémentaire de faciliter la transition des étudiants en médecine vers l'externat, tout en contribuant à rendre l'expérience clinique plus sûre et plus efficace. Il serait donc intéressant pour les facultés de médecine d'intégrer dans le cursus une formation normalisée axée sur les habiletés techniques au pré-externat.

Leading from Behind: An Educational Intervention to Address Faculty and Learner Preparedness for Competence By Design in Psychiatry.

PURPOSE: Residency training programs across Canada are beginning to implement the Royal College of Physicians and Surgeons of Canada's new Competence By Design (CBD) framework in medical education. The objective of the current research was to assess faculty members' and learners' understanding of, and preparedness for, the national shift to CBD in psychiatry before and after an educational intervention. METHODS: The current research implemented a pre-test/post-test design to investigate faculty members' and learners' perceptions and attitudes towards competency-based medical education (CBME) and CBD before and after a one-hour educational session delivered by an expert on CBME. RESULTS: Of the 104 session attendees, 83 (79.8%) completed the pre-survey and 80 (76.9%) completed the post-survey. Both groups reported a moderate level of baseline knowledge of CBME and CBD. Knowledge of CBME improved significantly for both faculty members (p = 0.03) and learners (p < 0.01) after the education session; however, only learners showed a significant increase in knowledge of the CBD framework following the education session (p < 0.01). Further, only learners demonstrated a significant increase in perceived preparedness for CBD following the session (p = 0.02). CONCLUSION: Overall, a brief, one-hour education session was at least somewhat effective at improving knowledge and preparedness for psychiatry's transition to CBD. In order to facilitate the transition to CBD and to assist in the rollout of future policy changes, psychiatry departments should provide both faculty members and learners with educational sessions and resources prior to the policy implementation.

"When in Doubt, Ask the Patient": A Quantitative, Patient-Oriented Approach to Formative Assessment of CanMEDS Roles.

Introduction: Since the introduction of competency-based frameworks into postgraduate medical curricula, educators have struggled to implement robust assessment tools that document the progression of necessary skills. The global movement towards competency-based medical education demands validated assessment tools. Our objective was to provide validity evidence for the Ottawa CanMEDS Competency Assessment Tool (OCCAT), designed to assess clinical performance in the communicator, professional, and health advocate CanMEDS roles. Methods: We developed the OCCAT, a 29-item questionnaire informed by specialty-specific Entrustable Professional Activities and consultation with stakeholders, including patients. Our sample included nine neonatal-perinatal medicine and maternal fetal medicine fellows rotating through antenatal high-risk clinics at the Ottawa Hospital. Following 70 unique encounters, the OCCAT was completed by patients and learners. Generalizability theory was used to determine overall reliability of scores. Differences in self and patient ratings were assessed using analyses of variance. Results: Generalizability analysis demonstrated that both questionnaires produced reliable scores (G-coefficient > 0.9). Self-scores were significantly lower than patient scores across all competencies, F(1, 6) = 13.9, p = .007. Variability analysis demonstrated that trainee scores varied across all competencies, suggesting both groups were able to recognize competencies as distinct and discriminate favorable behaviors belonging to each. Discussion: Our findings lend support to the movement to integrate self-assessment and patient feedback in formal evaluations for the purpose of enriched learner experiences and improved patient outcomes. We anticipate that the OCCAT will facilitate bridging to competency-based medical education.

One Room Schoolhouse: A Novel Intervention for Inspired Academic Half-Day Learning in Distributed Campus Settings.

INTRODUCTION: Some studies on academic half days (AHDs) suggest that learning in this context is associated with a lack of educational engagement. This challenge may be amplified in distributed campus settings, where geographical disadvantages demand reliance on videoconferencing or considerable time spent travelling to in-person learning events. Concerns about the educational effectiveness of AHDs by learners within our distributed campus setting led to the development and evaluation of the One Room Schoolhouse (ORS), a unique, evidence-informed, community-based curriculum that partially replaced the AHD sessions delivered at the main campus. It was hypothesized that creating an AHD experience that was clinically reflective of the community in which residents practiced and where residents were given the autonomy to implement novel pedagogical elements would result in better test scores and improved learner satisfaction among ORS learners. METHODS: The ORS was implemented at McMaster University's Waterloo Regional Campus in 2017. Residents across training cohorts (N = 9) engaged in co-learning based on scenarios co-developed from clinical experiences within the region. The learning approach relied on multiple, evidence-informed pedagogical strategies. A multi-method approach was used to evaluate the ORS curriculum. Between-subject analyses of variance were used to compare scores on practice exams (COPE and PRITE), in-training assessment reports (ITARs), and objective structured clinical exams (OSCEs) between learners who took part in the ORS and learners at the main campus. A semi-structured focus group probing residents' experiences with the ORS was analyzed using interpretive description. RESULTS: ORS learners significantly outperformed learners at the main campus on the November OSCE (p = .02), but not on the COPE, PRITE, ITARs, or September OSCE (p's < .05). Qualitative themes suggested advantages of the ORS in inspiring learning, engaging learners, and improving self-confidence in knowledge acquisition. These findings are aligned with the broader literature on learner agency, social development, and communities of practice. CONCLUSION: While the quantitative data only showed a significant difference between the 2 curricula on 1 measure (ie, the November OSCE), the qualitative findings offered an opportunity for educators to reimagine what medical education might consist of beyond the confines of a "traditional" AHD. Creating opportunities to enhance personal agency when acquiring knowledge, inspiring engagement about patient-related problems, and incorporating interdisciplinary learning through community engagement were critical pedagogical elements that were attributed to the success of the ORS.

Breaking bad news to cancer patients in times of COVID-19.

Breaking bad news is a mandatory provision in the professional life of nearly every physician. One of its most frequent occasions is the diagnosis of malignancy. Responding to the recipients' emotions is a critical issue in the delivery of unsettling information, and has an impact on the patient's trust in the treating physician, adjustment to illness and ultimately treatment. Since the World Health Organization (WHO) declared COVID-19 a pandemic on March 11, 2020, several measures of social distancing and isolation have been introduced to our clinical setting. In the wake of these restrictions, it is important to reexamine existing communication guidelines to determine their applicability to face-to-face counseling in the context of social distancing, as well as to new communication technologies, such as telemedicine. We address these issues and discuss strategies to convey bad news the most empathetic and comprehensible way possible.

Comparing the Ottawa Emergency Department Shift Observation Tool (O-EDShOT) to the traditional daily encounter card: measuring the quality of documented assessments.

OBJECTIVES: The Ottawa Emergency Department Shift Observation Tool (O-EDShOT) is a workplace-based assessment designed to assess a trainee's performance across an entire shift. It was developed in response to validity concerns with traditional end-of-shift workplace-based assessments, such as the daily encounter card. The O-EDShOT previously demonstrated strong psychometric characteristics; however, it remains unknown whether the O-EDShOT facilitates measurable improvements in the quality of documented assessments compared to daily encounter cards. METHODS: Three randomly selected daily encounter cards and three O-EDShOTs completed by 24 faculty were scored by two raters using the Completed Clinical Evaluation Report Rating (CCERR), a previously published 9-item quantitative measure of the quality of a completed workplace-based assessment. Automated-CCERR (A-CCERR) scores, which do not require raters, were also calculated. Paired sample t tests were conducted to compare the quality of assessments between O-EDShOTs and DECs as measured by the CCERR and A-CCERR. RESULTS: CCERR scores were significantly higher for O-EDShOTs (mean(SD) = 25.6(2.6)) compared to daily encounter cards (21.5(3.9); t(23) = 5.2, p < 0.001, d = 1.1). A-CCERR scores were also significantly higher for O-EDShOTs (mean(SD) = 18.5(1.6)) than for daily encounter cards (15.5(1.2); t(24) = 8.4, p < 0.001). CCERR items 1, 4 and 9 were rated significantly higher for O-EDShOTs compared to daily encounter cards. CONCLUSIONS: The O-EDShOT yields higher quality documented assessments when compared to the traditional end-of-shift daily encounter card. Our results provide additional validity evidence for the O-EDShOT as an assessment tool for capturing trainee on-shift performance that can be used as a stimulus for actionable feedback and as a source for high-quality workplace-based assessment data to inform decisions about emergency medicine trainee progress and promotion.

RéSUMé: OBJECTIFS: L'outil d'observation des quarts de travail des services d'urgence d'Ottawa (O-EDShOT) est une évaluation en milieu de travail conçue pour évaluer la performance d'un stagiaire pendant tout un quart de travail. Il a été développé en réponse à des problèmes de validité avec les évaluations traditionnelles en milieu de travail de fin de quart de travail, comme la fiche de rencontre quotidienne (DEC). Le O-EDShOT avait préalablement démontré de fortes caractéristiques psychométriques; cependant, on ignore toujours si l'O-EDShOT facilite des améliorations mesurables de la qualité des évaluations documentées par rapport aux fiches de rencontre quotidiennes. MéTHODES: Trois fiches de rencontre quotidiennes sélectionnées au hasard et trois O-EDShOT complétés par 24 membres du corps professoral ont été marqués par deux évaluateurs à l'aide de Completed Clinical Evaluation Report Rating (CCERR), une mesure quantitative en 9 points publiée précédemment de la qualité d'une évaluation en milieu de travail réalisée. Les scores du CCERR automatisé (A-CCERR), qui ne nécessitent pas d'évaluateur, ont également été calculés. Des tests t d'échantillons appariés ont été effectués pour comparer la qualité des évaluations entre les O-EDShOT et les DEC, telle que mesurée par le CCERR et l'A-CCERR. RéSULTATS: Les scores CCERR étaient significativement plus élevés pour les O-EDShOT (moyenne (ET) = 25,6 (2,6)) par rapport aux fiches de rencontre quotidiennes (21,5 (3,9) ; t (23) = 5,2, p < 0,001, d = 1,1). Les scores A-CCERR étaient également significativement plus élevés pour les O-EDShOT (moyenne (ET) = 18,5 (1,6)) que pour les fiches de rencontre quotidiennes (15,5 (1,2) ; t (24) = 8,4, p < 0,001). Les points 1, 4 et 9 du CCERR ont été notés beaucoup plus haut pour les O-EDShOT que pour les fiches de rencontre quotidiennes. CONCLUSIONS: L'O-EDShOT produit des évaluations documentées de meilleure qualité par rapport à la traditionnelle fiche de rencontre quotidienne de fin de quart de travail. Nos résultats fournissent des preuves de validité supplémentaires pour l'O-EDShOT en tant qu'outil d'évaluation pour saisir les performances des stagiaires en poste qui peuvent être utilisés comme stimulus pour une rétroaction exploitable, et comme source de données d'évaluation de haute qualité en milieu de travail pour notifier les décisions sur la progression et promotion des stagiaires en médecine d'urgence.

Validity of entrustment scales within anesthesiology residency training.

INTRODUCTION: Competency-based medical education requires robust assessment in authentic clinical environments. Using work-based assessments, entrustment scales have emerged as a means of describing a trainee's ability to perform competently. Nevertheless, psychometric properties of entrustment-based assessment are relatively unknown, particularly in anesthesiology. This study assessed the generalizability and extrapolation evidence for entrustment scales within a program of assessment during anesthesiology training. METHODS: Entrustment scores were collected during the first seven blocks of training for three resident cohorts. Entrustment scores were assessed during daily evaluations using a Clinical Case Assessment Tool (CCAT) within the preoperative, intraoperative, and postoperative setting. The reliability of the entrustment scale was estimated using generalizability theory. Spearman's correlations measured the relationship between median entrustment scores and percentiles scores on the Anesthesia Knowledge Test (AKT)-1 and AKT-6, mean Objective Structured Clinical Examination (OSCE) scores, and rankings of performance by the Clinical Competence Committee (CCC). RESULTS: Analyses were derived from 2,309 CCATs from 35 residents. The reliability or generalizability (G) coefficient of the entrustment scale was 0.73 (95% confidence interval [CI], 0.70 to 0.76), and the internal consistency was 0.86 (95% CI, 0.84 to 0.88). Intraoperative entrustment scores significantly correlated with the AKT-6 (rho = 0.51, P = 0.01), mean OSCE (rho = 0.45, P = 0.04), and CCC performance rankings (rho = 0.52, P = 0.006). CONCLUSION: As part of an assessment program, entrustment scales used early during anesthesiology training showed evidence of validity. Intraoperative entrustment scores had good reliability and showed acceptable internal consistency. Interpreting entrustment scores in this setting may constitute a valuable adjunct complementing traditional summative evaluations.

RéSUMé: INTRODUCTION: La formation médicale fondée sur les compétences nécessite une évaluation rigoureuse dans des environnements cliniques authentiques. Se fondant sur des évaluations basées sur le travail, les échelles de confiance sont apparues comme une méthode pour décrire la capacité d'un résident à performer de façon compétente. Toutefois, les propriétés psychométriques de l'évaluation basée sur la confiance sont relativement peu connues, particulièrement en anesthésiologie. Cette étude a évalué les données de généralisabilité et d'extrapolation des échelles de confiance dans le cadre d'un programme d'évaluation pendant la formation en anesthésiologie. MéTHODE: Les notes sur les échelles de confiance ont été colligées pendant les sept premiers blocs de formation de trois cohortes de résidents. Les notes sur les échelles de confiance ont été évaluées pendant les évaluations quotidiennes à l'aide d'un Outil d'évaluation des compétences cliniques (CCAT ­ Clinical Case Assessment Tool) dans les cadres préopératoire, peropératoire et postopératoire. La fiabilité de l'échelle de confiance a été estimée à l'aide de la théorie de la généralisabilité. Les corrélations de Spearman ont mesuré la relation entre des notes moyennes sur l'échelle de confiance et les scores de percentile aux examens de connaissances en anesthésiologie AKT-1 et AKT-6, les scores moyens à l'Examen clinique objectif structuré (ECOS), et les classements de performance par le Comité de compétences cliniques (CCC). RéSULTATS: Les analyses ont été dérivées à partir de 2309 résultats CCAT de 35 résidents. Le coefficient de fiabilité ou de généralisabilité (G) de l'échelle de confiance était de 0,73 (intervalle de confiance [IC] 95 %, 0,70 à 0,76), et la cohérence interne était de 0,86 (IC 95 %, 0,84 à 0,88). Les scores de confiance peropératoires étaient corrélés de manière significative aux scores sur l'AKT-6 (rho = 0,51, P = 0,01), aux scores moyens à l'ECOS (rho = 0,45, P = 0,04) et aux classements de performance du CCC (rho = 0,52, P = 0,006). CONCLUSION: Dans le cadre d'un programme d'évaluation, la validité des échelles de confiance utilisées en début de la formation en anesthésiologie a été éprouvée. Les scores de confiance peropératoires ont démontré une bonne fiabilité et une cohérence interne acceptable. L'interprétation des scores de confiance dans ce cadre pourrait constituer un ajout précieux qui complèterait les évaluations sommatives.

Identifying essential procedural skills in Canadian undergraduate medical education.

INTRODUCTION: We conducted a national survey to characterize current Canadian procedural skills training in Undergraduate Medical Education (UGME). The goals were to identify the most important procedures students should know upon graduation and assess clinician-educator perceptions regarding implementation of a pre-clerkship procedural program. METHODS: We distributed the survey to physician-educators across Canada's 17 medical schools. Respondents were directed to an individualized survey that collected demographic data, physician-educator responses on essential procedural skills, as well as physician-educator opinions on the value of a pre-clerkship procedural training program. RESULTS: The response rate for this survey was 21% (42 out of 201 distributed surveys were completed). The top 10 most important procedures identified by physician-educators included IV Access, Airway Management, Local anesthesia/field block, Casting, Spontaneous Vaginal Delivery, Testing for STIs, Phlebotomy, Suturing of Lacerations, Nasogastric Tube Insertion, and Venipuncture. Physician-educators supported a pre-clerkship procedural program. CONCLUSIONS: Identifying the most crucial procedural skills is the first step in implementing a competency-based procedural skills training program for Canadian medical students. With the list of essential skills, and the support for physician-educators in developing a pre-clerkship procedural skills curriculum, hopefully there can be future development of formalized curricula.

CONTEXTE: Nous avons mené un sondage à l'échelle nationale pour caractériser les compétences procédurales canadiennes dans la formation médicale de premier cycle. L'objectif était de reconnaître les plus importantes procédures que les étudiants devaient connaître à la fin de leur formation et d'évaluer les perceptions des cliniciens éducateurs au sujet de la mise en œuvre de leur programme procédural avant les stages. MÉTHODES: Nous avons distribué le sondage à des médecins éducateurs dans les 17 écoles de médecine du Canada. Les répondants ont été dirigés vers un sondage individualisé qui recueillait les données démographiques, les réponses des médecins éducateurs sur les compétences procédurales essentielles, ainsi que les opinions des médecins éducateurs sur la valeur du programme de formation procédurale avant les stages. RÉSULTATS: Le taux de réponse à ce sondage a été de 21 % (42 des 201 sondages distribués ont été remplis). Les dix plus importantes procédures recensées par les médecins éducateurs comprenaient l'accès IV, l'assistance respiratoire, le bloc anesthésie locale/champ, le moulage de plâtre, l'accouchement spontané par voie vaginale, les tests d'ITS, la phlébotomie, la suture des lacérations, l'insertion d'une sonde nasogastrique et la ponction veineuse. Les médecins éducateurs soutenaient un programme procédural avant les stages. CONCLUSIONS: Établir les compétences procédurales les plus essentielles représente la première étape dans la mise en œuvre d'un programme de formation dans les compétences procédurales fondé sur les compétences pour les étudiants canadiens en médecine. Avec la liste de compétences essentielles et le soutien des médecins éducateurs dans le développement d'un programme de compétences procédurales avant les stages, nous espérons qu'un programme structuré sera élaboré.

Assessing the process and outcome of the development of practice guidelines and recommendations: PANELVIEW instrument development.

BACKGROUND: Guideline recommendations may be affected by flaws in the process, inappropriate panel member selection or conduct, conflicts of interest and other factors. To our knowledge, no validated tool exists to evaluate guideline development from the perspective of those directly involved in the process. Our objective was to develop and validate a universal tool, the PANELVIEW instrument, to assess guideline processes, methods and outcomes from the perspective of the participating guideline panellists and group members. METHODS: We performed a systematic literature search and surveys of guideline groups (identified through contacting international organizations and convenience sampling of working panels) to inform item generation. Subsequent groups of guideline methodologists and panellists reviewed items for face validity and missing items. We used surveys, interviews and expert review for item reduction and phrasing. For reliability assessment and feedback, we tested the PANELVIEW tool in 8 international guideline groups. RESULTS: We surveyed 62 members from 13 guideline panels, contacted 19 organizations and reviewed 20 source documents to generate items. Fifty-three additional key informants provided feedback about phrasing of the items and response options. We reduced the number of items from 95 to 34 across domains that included administration, training, conflict of interest, group dynamics, chairing, evidence synthesis, formulating recommendations and publication. The tool takes about 10 minutes to complete and showed acceptable measurement properties. INTERPRETATION: The PANELVIEW instrument fills a gap by enabling guideline organizations to involve clinicians, patients and other participants in evaluating their guideline processes. The tool can inform quality improvement of existing or new guideline programs, focusing on insight into and transparency of the guideline development process, methods and outcomes.