Machine learning to extract communication and history-taking skills in OSCE transcripts.

OBJECTIVES: Observed Structured Clinical Exams (OSCEs) allow assessment of, and provide feedback to, medical students. Clinical examiners and standardised patients (SP) typically complete itemised checklists and global scoring scales, which have known shortcomings. In this study, we applied machine learning (ML) to label some communication skills and interview content information in OSCE transcripts and to compare several ML methodologies by performance and transferability. METHODS: One-hundred and twenty-one transcripts of two OSCE scenarios were manually annotated per utterance across 19 communication skills and content areas. Utterances were converted to two types of numeric sentence vector representations and were paired with three types of ML algorithms. First, ML models (MLMs) were evaluated using a five K-fold cross-validation technique on all transcripts in one scenario to generate precision and recall, and their harmonic mean, F1 scores. Second, ML models were trained on all 101 transcripts from scenario 1 and tested for transferability on 20 scenario 2 transcripts. RESULTS: Performance testing in the K-fold cross-validation demonstrated relatively high mean F1 scores: median 0.87 and range 0.53-0.98 across all 19 labels. Transferability testing demonstrated success: F1 median 0.76 and range 0.46-0.97. The combination of a bi-directional long short-term memory neural network (biLSTM) algorithm with GenSen numeric sentence vector representations was associated with greater F1 scores across both performance and transferability (P < .005). CONCLUSIONS: We report the first application of ML in the context of student-SP OSCEs. We demonstrated that several MLMs automatically labelled OSCE transcripts for a range of interview content and some clinical communications skills. Some MLMs achieved greater performance and transferability. Optimised MLMs could provide automated and accurate assessment of OSCEs with potential to track student progress and identify areas for further practice.

Preparing Medical Students for Anti-racism at the Bedside: Teaching Skills to Mitigate Racism and Bias in Clinical Encounters.

Introduction: Systemic racism perpetuates health disparities and negatively impacts health care delivery and patient outcomes. Racism and bias can affect every aspect of clinical care, including history-taking, physical examination, laboratory interpretation, note-writing, oral presentation, and decision-making. Medical students must learn racism- and bias-mitigation skills early in their professional development to provide high-quality, equitable care. Methods: In November 2021, senior medical students and faculty with expertise in promoting health equity and justice in medicine designed and cotaught a Zoom-based, 75-minute, interactive session for second-year medical students. Participants prepared by reading assigned articles. Breakout rooms were used to facilitate small-group discussions. Session topics included use of a structural vulnerability assessment tool, examples of how bias can impact the physical exam, demonstration of how language can transmit bias, and skill practice using neutral instead of stigmatizing language. Results: Forty second-year medical students participated in the session. Thirty-one students (78%) completed Likert-type surveys evaluating reaction and learning. Results showed improvements in students' perceptions of their abilities to assess for structural factors that influence health, recognize ways bias can impact clinical encounters, and apply skills to minimize bias in clinical care and decision-making. Discussion: Providing opportunities for health care learners to think critically about how bias impacts patients and communities and equipping them with tools to begin dismantling exclusionary, racist practices in medicine are achievable and crucial to actualizing a just and equitable health system. This educational session can be adapted for training across health care professions and the educational continuum.

Workplace-Based Entrustment Scales for the Core EPAs: A Multisite Comparison of Validity Evidence for Two Proposed Instruments Using Structured Vignettes and Trained Raters.

PURPOSE: In undergraduate medical education (UME), competency-based medical education has been operationalized through the 13 Core Entrustable Professional Activities for Entering Residency (Core EPAs). Direct observation in the workplace using rigorous, valid, reliable measures is required to inform summative decisions about graduates' readiness for residency. The purpose of this study is to investigate the validity evidence of 2 proposed workplace-based entrustment scales. METHOD: The authors of this multisite, randomized, experimental study used structured vignettes and experienced raters to examine validity evidence of the Ottawa scale and the UME supervisory tool (Chen scale) in 2019. The authors used a series of 8 cases (6 developed de novo) depicting learners at preentrustable (less-developed) and entrustable (more-developed) skill levels across 5 Core EPAs. Participants from Core EPA pilot institutions rated learner performance using either the Ottawa or Chen scale. The authors used descriptive statistics and analysis of variance to examine data trends and compare ratings, conducted interrater reliability and generalizability studies to evaluate consistency among participants, and performed a content analysis of narrative comments. RESULTS: Fifty clinician-educators from 10 institutions participated, yielding 579 discrete EPA assessments. Both Ottawa and Chen scales differentiated between less- and more-developed skill levels (P < .001). The interclass correlation was good to excellent for all EPAs using Ottawa (range, 0.68-0.91) and fair to excellent using Chen (range, 0.54-0.83). Generalizability analysis revealed substantial variance in ratings attributable to the learner-EPA interaction (59.6% for Ottawa; 48.9% for Chen) suggesting variability for ratings was appropriately associated with performance on individual EPAs. CONCLUSIONS: In a structured setting, both the Ottawa and Chen scales distinguished between preentrustable and entrustable learners; however, the Ottawa scale demonstrated more desirable characteristics. These findings represent a critical step forward in developing valid, reliable instruments to measure learner progression toward entrustment for the Core EPAs.

Combining training in knowledge translation with quality improvement reduced 30-day heart failure readmissions in a community hospital: a case study.

RATIONALE, AIMS AND OBJECTIVES: Training programmes in evidence-based practice (EBP) frequently fail to translate their content into practice change and care improvement. We linked multidisciplinary training in EBP to an initiative to decrease 30-day readmissions among patients admitted to a community teaching hospital for heart failure (HF). METHODS: Hospital staff reflecting all services and disciplines relevant to care of patients with HF attended a 3-day innovative capacity building conference in evidence-based health care over a 3-year period beginning in 2009. The team, facilitated by a conference faculty member, applied a knowledge-to-action model taught at the conference. We reviewed published research, profiled our population and practice experience, developed a three-phase protocol and implemented it in late 2010. We tracked readmission rates, adverse clinical outcomes and programme cost. RESULTS: The protocol emphasized patient education, medication reconciliation and transition to community-based care. Senior administration approved a full-time nurse HF coordinator. Thirty-day HF readmissions decreased from 23.1% to 16.4% (adjusted OR = 0.64, 95% CI = 0.42-0.97) during the year following implementation. Corresponding rates in another hospital serving the same population but not part of the programme were 22.3% and 20.2% (adjusted OR = 0.87, 95% CI = 0.71-1.08). Adherence to mandated HF quality measures improved. Following a start-up cost of $15 000 US, programme expenses balanced potential savings from decreased HF readmissions. CONCLUSION: Training of a multidisciplinary hospital team in use of a knowledge translation model, combined with ongoing facilitation, led to implementation of a budget neutral programme that decreased HF readmissions.