Using Natural Language Processing to Evaluate the Quality of Supervisor Narrative Comments in Competency-Based Medical Education.

PURPOSE: Learner development and promotion rely heavily on narrative assessment comments, but narrative assessment quality is rarely evaluated in medical education. Educators have developed tools such as the Quality of Assessment for Learning (QuAL) tool to evaluate the quality of narrative assessment comments; however, scoring the comments generated in medical education assessment programs is time intensive. The authors developed a natural language processing (NLP) model for applying the QuAL score to narrative supervisor comments. METHOD: Samples of 2,500 Entrustable Professional Activities assessments were randomly extracted and deidentified from the McMaster (1,250 comments) and Saskatchewan (1,250 comments) emergency medicine (EM) residency training programs during the 2019-2020 academic year. Comments were rated using the QuAL score by 25 EM faculty members and 25 EM residents. The results were used to develop and test an NLP model to predict the overall QuAL score and QuAL subscores. RESULTS: All 50 raters completed the rating exercise. Approximately 50% of the comments had perfect agreement on the QuAL score, with the remaining resolved by the study authors. Creating a meaningful suggestion for improvement was the key differentiator between high- and moderate-quality feedback. The overall QuAL model predicted the exact human-rated score or 1 point above or below it in 87% of instances. Overall model performance was excellent, especially regarding the subtasks on suggestions for improvement and the link between resident performance and improvement suggestions, which achieved 85% and 82% balanced accuracies, respectively. CONCLUSIONS: This model could save considerable time for programs that want to rate the quality of supervisor comments, with the potential to automatically score a large volume of comments. This model could be used to provide faculty with real-time feedback or as a tool to quantify and track the quality of assessment comments at faculty, rotation, program, or institution levels.

Examining enablers and barriers to entrustable professional activity acquisition using the theoretical domains framework: A qualitative framework analysis study.

Background: Without a clear understanding of the factors contributing to the effective acquisition of high-quality entrustable professional activity (EPA) assessments, trainees, supervising faculty, and training programs may lack appropriate strategies for successful EPA implementation and utilization. The purpose of this study was to identify barriers and facilitators to acquiring high-quality EPA assessments in Canadian emergency medicine (EM) training programs. Methods: We conducted a qualitative framework analysis study utilizing the Theoretical Domains Framework (TDF). Semistructured interviews of EM resident and faculty participants underwent audio recording, deidentification, and line-by-line coding by two authors, being coded to extract themes and subthemes across the domains of the TDF. Results: From 14 interviews (eight faculty and six residents) we identified, within the 14 TDF domains, major themes and subthemes for barriers and facilitators to EPA acquisition for both faculty and residents. The two most cited domains (and their frequencies) among residents and faculty were environmental context and resources (56) and behavioral regulation (48). Example strategies to improving EPA acquisition include orienting residents to the competency-based medical education (CBME) paradigm, recalibrating expectations relating to "low ratings" on EPAs, engaging in continuous faculty development to ensure familiarity and fluency with EPAs, and implementing longitudinal coaching programs between residents and faculty to encourage repetitive longitudinal interactions and high-quality specific feedback. Conclusions: We identified key strategies to support residents, faculty, programs, and institutions in overcoming barriers and improving EPA assessment processes. This is an important step toward ensuring the successful implementation of CBME and the effective operationalization of EPAs within EM training programs.

Can ED chest pain patients with intermediate HEART scores be managed as outpatients?

PURPOSE: Current guidelines recommend hospital admission for patients who present to the emergency department (ED) with chest pain and are scored as intermediate risk for adverse outcomes based on the HEART score. While hospital admission for these patients allows for timely investigation and treatment, it is a resource-intensive process. This study examines whether intermediate HEART score patients can be safely managed on an outpatient basis through rapid access chest pain clinics. METHODS: This retrospective observational study included all ED chest pain patients referred to rapid access clinics from January 2018 to April 2020 in Regina and Saskatoon, Saskatchewan. ED physician HEART scores were used in lieu of reviewer HEART scores when available. The primary outcome was the rate of major adverse coronary events (MACE), a composite measure of death, acute coronary syndrome, stroke, coronary angiography, and revascularization at 6 weeks in intermediate-risk patients. Secondary outcomes were the type of MACE, rate of MACE before rapid access clinic appointment and the most predictive component of the HEART score. RESULTS: There were 1989 ED referrals, of which 817 were for intermediate-risk patients. 9.3% of intermediate-risk patients had a MACE at 6 weeks. MACE occurred before rapid access clinic follow-up in 1.1% of intermediate-risk patients, with coronary angiography being the most common MACE. Excluding coronary angiography, the risk of MACE before rapid access clinic follow-up was 0.7% in intermediate-risk patients. Components of the HEART score most predictive of MACE were troponin (OR 11.0, 95% CI: 3.7-32.3) and history (5.3, 95% CI: 2.4-11.8). CONCLUSION: This study demonstrates that rapid access clinics are likely a safe alternative to admission for intermediate-risk chest pain patients and could reduce costly inpatient admissions for chest pain. With angiography excluded, MACE rates were well below the American College of Emergency Physicians cited 2% threshold.

RéSUMé: OBJECTIF: Les directives actuelles recommandent l'admission à l'hôpital des patients qui se présentent aux urgences avec une douleur thoracique et qui sont classés comme présentant un risque intermédiaire d'effets indésirables selon le score HEART. Bien que l'hospitalisation de ces patients permette une investigation et un traitement en temps opportun, il s'agit d'un processus exigeant en ressources. Cette étude examine si les patients ayant un score HEART intermédiaire peuvent être pris en charge en toute sécurité en ambulatoire par des cliniques d'accès rapide aux douleurs thoraciques. MéTHODES: Cette étude observationnelle rétrospective a inclus tous les patients souffrant de douleurs thoraciques aux urgences orientés vers des cliniques d'accès rapide de janvier 2018 à avril 2020 à Regina et Saskatoon, en Saskatchewan. Les scores HEART des médecins des urgences ont été utilisés à la place des scores HEART des examinateurs lorsqu'ils étaient disponibles. Le principal résultat était le taux d'événements coronariens indésirables majeurs (MACE), une mesure composite du décès, du syndrome coronarien aigu, de l'accident vasculaire cérébral, de l'angiographie coronaire et de la revascularisation à 6 semaines chez les patients à risque intermédiaire. Les résultats secondaires étaient le type de MACE, le taux de MACE avant un rendez-vous à la clinique d'accès rapide et la composante la plus prédictive du score HEART. RéSULTATS: Il y a eu 1989 orientations vers les urgences, dont 817 pour des patients à risque intermédiaire. 9,3 % des patients à risque intermédiaire ont subi un MACE à 6 semaines. Un MACE est survenu avant le suivi clinique d'accès rapide chez 1,1 % des patients à risque intermédiaire, la coronarographie étant le MACE le plus fréquent. À l'exclusion de l'angiographie coronarienne, le risque de MACE avant le suivi clinique d'accès rapide était de 0,7 % chez les patients à risque intermédiaire. Les composants du score HEART les plus prédictifs de MACE étaient la troponine (OR 11,0, IC 95 % : 3,7-32,3) et les antécédents (5,3, IC 95 % : 2,4-11,8). CONCLUSION: Cette étude démontre que les cliniques d'accès rapide sont probablement une alternative sûre à l'admission pour les patients souffrant de douleurs thoraciques à risque intermédiaire et pourraient réduire les admissions coûteuses de patients hospitalisés pour des douleurs thoraciques. En excluant l'angiographie, les taux de MACE étaient bien inférieurs au seuil de 2 % cité par l'American College of Emergency Physicians.

Educator's blueprint: A how-to guide for creating a high-quality infographic.

Infographics are a valuable tool for increasing knowledge translation and dissemination. They can be used to simplify complex topics and supplement the written text of a study. This Educator's Blueprint paper will provide 10 strategies for creating high-quality infographics. These strategies include selecting appropriate content, defining the target audience, considering the format, selecting the software, using consistent font and color schemes, increasing image utilization, ensuring a consistent flow of ideas, avoiding copyright and HIPAA violations, getting feedback from others, and utilizing effective dissemination strategies. These strategies will help guide educators to increase their ability to create more effective infographics.

Developing a dashboard to meet the needs of residents in a competency-based training program: A design-based research project.

BACKGROUND: Canadian specialty programs are implementing Competence By Design, a competency-based medical education (CBME) program which requires frequent assessments of entrustable professional activities. To be used for learning, the large amount of assessment data needs to be interpreted by residents, but little work has been done to determine how visualizing and interacting with this data can be supported. Within the University of Saskatchewan emergency medicine residency program, we sought to determine how our residents' CBME assessment data should be presented to support their learning and to develop a dashboard that meets our residents' needs. METHODS: We utilized a design-based research process to identify and address resident needs surrounding the presentation of their assessment data. Data was collected within the emergency medicine residency program at the University of Saskatchewan via four resident focus groups held over 10 months. Focus group discussions were analyzed using a grounded theory approach to identify resident needs. This guided the development of a dashboard which contained elements (data, analytics, and visualizations) that support their interpretation of the data. The identified needs are described using quotes from the focus groups as well as visualizations of the dashboard elements. RESULTS: Resident needs were classified under three themes: (1) Provide guidance through the assessment program, (2) Present workplace-based assessment data, and (3) Present other assessment data. Seventeen dashboard elements were designed to address these needs. CONCLUSIONS: Our design-based research process identified resident needs and developed dashboard elements to meet them. This work will inform the creation and evolution of CBME assessment dashboards designed to support resident learning.

CONTEXTE: Les programmes canadiens de spécialité sont à implanter la compétence par conception (CPC), un programme d'éducation médicale par compétences qui nécessite des évaluations fréquentes des activités professionnelles confiables. Pour servir aux fins d'apprentissage, la grande quantité de données d'évaluation doit être interprétée par les résidents, mais peu de travaux ont été réalisés pour déterminer comment la visualisation et l'interaction avec ces données peuvent être soutenues. Dans le cadre du programme de résidence en médecine d'urgence de l'Université de Saskatchewan, nous avons cherché à déterminer comment les données d'évaluation de la CPC de nos résidents devraient être présentées pour soutenir leur apprentissage et pour développer un tableau de bord qui réponde aux besoins de nos résidents. MÉTHODOLOGIE: Nous avons utilisé un processus de recherche orientée par la conception pour cerner les besoins des résidents en lien avec la présentation de leurs données d'évaluation. Les données ont été recueillies au cours du programme de résidence en médecine d'urgence de l'Université de Saskatchewan grâce à quatre groupes de discussion de résidents qui se sont tenus sur une période de 10 mois. Les groupes de discussion ont été analysés en utilisant l'approche de la théorisation ancrée (Grounded Theory) pour cerner les besoins des résidents, pour guider le développement d'un tableau de bord contenant des éléments (données, analyses et visualisations) qui soutiennent leur interprétation de leurs propres données. Les besoins identifiés sont décrits à l'aide de citations des groupes de discussion ainsi que de visualisations des éléments du tableau de bord. RÉSULTATS: Les besoins des résidents ont été classés sous trois thèmes : 1. être guidés quant au programme d'évaluation, 2. présenter des données d'évaluation en milieu de travail, et 3. présenter d'autres données d'évaluation. Dix-sept éléments du tableau de bord ont été conçus pour répondre à ces besoins. CONCLUSIONS: Notre méthode de recherche orientée par conception a permis de cerner les besoins des résidents et d'élaborer les éléments d'un tableau de bord pour y répondre. Ce travail servira de base à la création et à l'évolution des tableaux de bord d'évaluation en CPC conçus pour soutenir l'apprentissage des résidents.

CAEP 2019 Academic Symposium: Got competence? Best practices in trainee progress decisions.

BACKGROUND: Competence committees play a key role in a competency-based system of assessment. These committees are tasked with reviewing and synthesizing clinical performance data to make judgments regarding residents' competence. Canadian emergency medicine (EM) postgraduate training programs recently implemented competence committees; however, a paucity of literature guides their work. OBJECTIVE: The objective of this study was to develop consensus-based recommendations to optimize the function and decisions of competence committees in Canadian EM training programs. METHODS: Semi-structured interviews of EM competence committee chairs were conducted and analyzed. The interview guide was informed by a literature review of competence committee structure, processes, and best practices. Inductive thematic analysis of interview transcripts was conducted to identify emerging themes. Preliminary recommendations, based on themes, were drafted and presented at the 2019 CAEP Academic Symposium on Education. Through a live presentation and survey poll, symposium attendees representing the national EM community participated in a facilitated discussion of the recommendations. The authors incorporated this feedback and identified consensus among symposium attendees on a final set of nine high-yield recommendations. CONCLUSION: The Canadian EM community used a structured process to develop nine best practice recommendations for competence committees addressing: committee membership, meeting processes, decision outcomes, use of high-quality performance data, and ongoing quality improvement. These recommendations can inform the structure and processes of competence committees in Canadian EM training programs.

Developing a dashboard to meet Competence Committee needs: a design-based research project.

BACKGROUND: Competency-based programs are being adopted in medical education around the world. Competence Committees must visualize learner assessment data effectively to support their decision-making. Dashboards play an integral role in decision support systems in other fields. Design-based research allows the simultaneous development and study of educational environments. METHODS: We utilized a design-based research process within the emergency medicine residency program at the University of Saskatchewan to identify the data, analytics, and visualizations needed by its Competence Committee, and developed a dashboard incorporating these elements. Narrative data were collected from two focus groups, five interviews, and the observation of two Competence Committee meetings. Data were qualitatively analyzed to develop a thematic framework outlining the needs of the Competence Committee and to inform the development of the dashboard. RESULTS: The qualitative analysis identified four Competence Committee needs (Explore Workplace-Based Assessment Data, Explore Other Assessment Data, Understand the Data in Context, and Ensure the Security of the Data). These needs were described with narratives and represented through visualizations of the dashboard elements. CONCLUSIONS: This work addresses the practical challenges of supporting data-driven decision making by Competence Committees and will inform the development of dashboards for programs, institutions, and learner management systems.

CONTEXTE: Les programmes fondés sur la compétence sont adoptés dans la formation médicale à travers le monde. Les comités des compétences doivent visualiser efficacement les données d'évaluation des apprenants pour soutenir leurs prises de décision. Les tableaux de bord jouent un rôle essentiel dans les systèmes d'aide à la décision dans d'autres disciplines. La recherche orientée par la conception permet le développement et l'étude simultanés des environnements éducatifs. MÉTHODES: Nous avons utilisé un processus de recherche orienté par la conception au sein du programme de résidence en médecine d'urgence à l'Université de la Saskatchewan pour déterminer les données, les analyses et les visuels dont a besoin son comité des compétences, et avons développé un tableau de bord intégrant ces éléments. Les données narratives ont été recueillies auprès de deux groupes de discussion, lors de cinq entrevues et par l'observation de deux réunions du comité des compétences. Les données ont été analysées de manière qualitative pour élaborer un cadre thématique soulignant les besoins du comité des compétences et orienter le développement du tableau de bord. RÉSULTATS: L'analyse qualitative a dégagé quatre besoins du comité des compétences (explorer les données d'évaluation en milieu de travail, explorer d'autres données d'évaluation, comprendre les données dans leur contexte et s'assurer la sécurité des données). Ces besoins ont étédécrits avec des récits et représentés par des visuelsdes éléments du tableau de bord. CONCLUSIONS: Le présent travail aborde les difficultés pratiques de soutenir une prise de décision fondée sur des données par les comités des compétences et oriente le développement des tableaux de bord pour les programmes, les établissements et les systèmes de gestion des apprenants.

A transparent and defensible process for applicant selection within a Canadian emergency medicine residency program.

OBJECTIVES: The Canadian Resident Matching Service (CaRMS) selection process has come under scrutiny due to the increasing number of unmatched medical graduates. In response, we outline our residency program's selection process including how we have incorporated best practices and novel techniques. METHODS: We selected file reviewers and interviewers to mitigate gender bias and increase diversity. Four residents and two attending physicians rated each file using a standardized, cloud-based file review template to allow simultaneous rating. We interviewed applicants using four standardized stations with two or three interviewers per station. We used heat maps to review rating discrepancies and eliminated rating variance using Z-scores. The number of person-hours that we required to conduct our selection process was quantified and the process outcomes were described statistically and graphically. RESULTS: We received between 75 and 90 CaRMS applications during each application cycle between 2017 and 2019. Our overall process required 320 person-hours annually, excluding attendance at the social events and administrative assistant duties. Our preliminary interview and rank lists were developed using weighted Z-scores and modified through an organized discussion informed by heat mapped data. The difference between the Z-scores of applicants surrounding the interview invitation threshold was 0.18-0.3 standard deviations. Interview performance significantly impacted the final rank list. CONCLUSIONS: We describe a rigorous resident selection process for our emergency medicine training program which incorporated simultaneous cloud-based rating, Z-scores, and heat maps. This standardized approach could inform other programs looking to adopt a rigorous selection process while providing applicants guidance and reassurance of a fair assessment.

The development of entrustable professional activities reference cards to support the implementation of Competence by Design in emergency medicine.

We designed two practical, user-friendly, low-cost, aesthetically pleasing resources, with the goal of introducing residents and observers to a new Competence by Design assessment system based on entrustable professional activities. They included a set of rotation- and stage-specific entrustable professional activities reference cards for bedside use by residents and observers and a curriculum board to organize the entrustable professional activities reference cards by stages of training based on our program's curriculum map. A survey of 14 emergency medicine residents evaluated the utilization and helpfulness of these resources. They had a positive impact on our program's transition to Competence by Design and could be successfully incorporated into other residency programs to support the introduction of entrustable professional activities-based Competence by Design assessment systems.

Le groupe a conçu deux documents à la fois pratiques, conviviaux, peu coûteux et agréables à l'œil, dans le but de présenter aux résidents et aux observateurs un nouveau système d'évaluation du modèle d'acquisition des compétences par conception, fondé sur des activités professionnelles confiables. Les documents comprenaient un ensemble de fiches de référence illustrant des activités professionnelles confiables liées aux étapes de formation et aux stages cliniques, à utiliser au chevet par les résidents et les observateurs ainsi qu'un tableau cartonné représentatif du curriculum visant à répartir les fiches de référence liées aux activités professionnelles confiables selon les étapes de formation fondées sur la carte du programme. Quatorze résidents en médecine d'urgence ont évalué l'utilisation et l'utilité de ces documents. L'enquête a révélé que ces derniers avaient facilité la transition vers le modèle d'acquisition des compétences par conception, et se prêteraient bien à d'autres programmes de résidence afin d'aider à la mise en œuvre de systèmes d'évaluation de la nouvelle approche de formation, fondés sur des activités professionnelles confiables.

The effect of an infographic promotion on research dissemination and readership: A randomized controlled trial.

OBJECTIVE: Journals use social media to increase the awareness of their publications. Infographics show research findings in a concise and visually appealing manner, well suited for dissemination on social media platforms. We hypothesized that infographic abstracts promoted on social media would increase the dissemination and online readership of the parent research articles. METHODS: Twenty-four articles were chosen from the six issues of CJEM published between July 2016 and June 2017 and randomized to infographic or control groups. All articles were disseminated through the journal's social media accounts (Twitter and Facebook). Control articles were promoted using a screen capture image of each article's abstract on the journal's social media accounts. Infographic articles were promoted similarly using a visual infographic. Infographics were also published and promoted on the CanadiEM.org's website and social media channels. Abstract views, full-text views, and the change in Altmetric score were compared between groups using unpaired two-tailed t-tests. RESULTS: There were no significant differences in the groups at baseline. Abstract views (mean, 95% CI) were higher in the infographics (379, 287-471) than the control group (176, 136-215, p<0.001). Mean change in Altmetric scores was higher in the infographics (26, 18-34) than in the control group (3, 2-4, p<0.0001). There was no difference in full-text views between the infographics (50, 0-101) and control groups (25, 18-32). CONCLUSION: The promotion of CJEM articles using infographics on social media and the CanadiEM.org website increased Altmetric scores and abstract views. Infographics may have a role in increasing awareness of medical literature.

The impact of social media promotion with infographics and podcasts on research dissemination and readership.

OBJECTIVE: In 2015 and 2016, the Canadian Journal of Emergency Medicine (CJEM) Social Media (SoMe) Team collaborated with established medical websites to promote CJEM articles using podcasts and infographics while tracking dissemination and readership. METHODS: CJEM publications in the "Original Research" and "State of the Art" sections were selected by the SoMe Team for podcast and infographic promotion based on their perceived interest to emergency physicians. A control group was composed retrospectively of articles from the 2015 and 2016 issues with the highest Altmetric score that received standard Facebook and Twitter promotions. Studies on SoMe topics were excluded. Dissemination was quantified by January 1, 2017 Altmetric scores. Readership was measured by abstract and full-text views over a 3-month period. The number needed to view (NNV) was calculated by dividing abstract views by full-text views. RESULTS: Twenty-nine of 88 articles that met inclusion were included in the podcast (6), infographic (11), and control (12) groups. Descriptive statistics (mean, 95% confidence interval) were calculated for podcast (Altmetric: 61, 42-80; Abstract: 1795, 1135-2455; Full-text: 431, 0-1031), infographic (Altmetric: 31.5, 19-43; Abstract: 590, 361-819; Full-text: 65, 33-98), and control (Altmetric: 12, 8-15; Abstract: 257, 159-354; Full-Text: 73, 38-109) articles. The NNV was 4.2 for podcast, 9.0 for infographic, and 3.5 for control articles. Discussion Limitations included selection bias, the influence of SoMe promotion on the Altmetric scores, and a lack of generalizability to other journals. CONCLUSION: Collaboration with established SoMe websites using podcasts and infographics was associated with increased Altmetric scores and abstract views but not full-text article views.