Using Natural Language Processing to Automatically Assess Feedback Quality: Findings From 3 Surgical Residencies.

PURPOSE: Learning is markedly improved with high-quality feedback, yet assuring the quality of feedback is difficult to achieve at scale. Natural language processing (NLP) algorithms may be useful in this context as they can automatically classify large volumes of narrative data. However, it is unknown if NLP models can accurately evaluate surgical trainee feedback. This study evaluated which NLP techniques best classify the quality of surgical trainee formative feedback recorded as part of a workplace assessment. METHOD: During the 2016-2017 academic year, the SIMPL (Society for Improving Medical Professional Learning) app was used to record operative performance narrative feedback for residents at 3 university-based general surgery residency training programs. Feedback comments were collected for a sample of residents representing all 5 postgraduate year levels and coded for quality. In May 2019, the coded comments were then used to train NLP models to automatically classify the quality of feedback across 4 categories (effective, mediocre, ineffective, or other). Models included support vector machines (SVM), logistic regression, gradient boosted trees, naive Bayes, and random forests. The primary outcome was mean classification accuracy. RESULTS: The authors manually coded the quality of 600 recorded feedback comments. Those data were used to train NLP models to automatically classify the quality of feedback across 4 categories. The NLP model using an SVM algorithm yielded a maximum mean accuracy of 0.64 (standard deviation, 0.01). When the classification task was modified to distinguish only high-quality vs low-quality feedback, maximum mean accuracy was 0.83, again with SVM. CONCLUSIONS: To the authors' knowledge, this is the first study to examine the use of NLP for classifying feedback quality. SVM NLP models demonstrated the ability to automatically classify the quality of surgical trainee evaluations. Larger training datasets would likely further increase accuracy.

Association of cigarette smoking and adverse events in left ventricular assist device patients.

INTRODUCTION: Adverse events (AEs) associated with left ventricular assist devices (LVADs) cause significant morbidity and mortality. Little is known about patient-specific factors that contribute to rates of AEs. The purpose of this study was to assess the association of cigarette smoking history and AEs following LVAD implantation. METHODS: This study was a single-center, observational examination of 355 consecutive patients who underwent continuous-flow LVAD implantation from May 1, 2008 to July 1, 2018. Based on self-report, 348 patients with available data were categorized as never, former, or current smokers. Pre-LVAD implantation baseline characteristics were obtained, and summary characteristics were calculated. Hospitalizations for gastrointestinal bleeds, driveline infections, strokes, pump thromboses, and acute heart failure were evaluated. The Cox proportional hazard model was used to estimate the association of smoking and AE-related hospital admissions. The cumulative incidence competing risk method was used for survival analysis. RESULTS: Current (8.22%, p 0.006) and former (4.75%, p 0.026) smokers had a greater proportion of admissions for pump thrombosis compared to never smokers (2.22%). Former smoking was associated with admission for driveline infection (HR 2.43, CI 1.08-5.46, p 0.03) on multivariate analysis. There were no significant associations between smoking and the other AEs of interest. There was no difference in survival among the three groups. CONCLUSIONS: Smokers had a higher proportion of admissions for pump thrombosis compared to never smokers, and former smoking was associated with admission for driveline infections in patients with LVADs.