Making Use of Natural Language Processing to Better Understand Medical Students' Self-Assessment of Clinical Skills.

PROBLEM: Reflective practice is necessary for self-regulated learning. Helping medical students develop these skills can be challenging since they are difficult to observe. One common solution is to assign students' reflective self-assessments, which produce large quantities of narrative assessment data. Reflective self-assessments also provide feedback to faculty regarding students' understanding of content, reflective abilities, and areas for course improvement. To maximize student learning and feedback to faculty, reflective self-assessments must be reviewed and analyzed, activities that are often difficult for faculty due to the time-intensive and cumbersome nature of processing large quantities of narrative assessment data. APPROACH: The authors collected narrative assessment data (2,224 students' reflective self-assessments) from 344 medical students' reflective self-assessments. In academic years 2019-2020 and 2021-2022, students at the University of Cincinnati College of Medicine responded to 2 prompts (aspects that surprised students, areas for student improvement) after reviewing their standardized patient encounters. These free-text entries were analyzed using TopEx, an open-source natural language processing (NLP) tool, to identify common topics and themes, which faculty then reviewed. OUTCOMES: TopEx expedited theme identification in students' reflective self-assessments, unveiling 10 themes for prompt 1 such as question organization and history analysis, and 8 for prompt 2, including sensitive histories and exam efficiency. Using TopEx offered a user-friendly, time-saving analysis method without requiring complex NLP implementations. The authors discerned 4 education enhancement implications: aggregating themes for future student reflection, revising self-assessments for common improvement areas, adjusting curriculum to guide students better, and aiding faculty in providing targeted upcoming feedback. NEXT STEPS: The University of Cincinnati College of Medicine aims to refine and expand the utilization of TopEx for deeper narrative assessment analysis, while other institutions may model or extend this approach to uncover broader educational insights and drive curricular advancements.

Improving Care Delivery: Location Timestamps to Enhance Process Measurement of a Clinical Workflow.

Traditional quality improvement (QI) strategies to describe workflow processes rely primarily upon qualitative methods or human-driven observations. These methods may be limited in scope and accuracy when applied to time-based workflow processes. This study sought to evaluate the utility of integrating objective time measurements to augment traditional QI strategies using procedural sedation workflow in a pediatric emergency department as an archetype. METHODS: We applied the FOCUS-Plan-Do-Check-Act framework to reduce the time from arrival to sedation for long-bone fractures. First, we added supplementary framework-defining steps to repeat the Clarifying and Understanding steps. We then extracted objective time-based data from an electronic health record (EHR) system and a real-time locating system (RTLS). We then compared and contrasted the findings of traditional surveys with analyses of timed steps within the sedation workflow. RESULTS: When identifying the source of delays, traditional survey techniques yielded ambiguous and even conflicting results based on clinical roles. The timestamps supported 5 measurable clinical role of subworkflows. By measuring the time to completion for 54 sedation cases, workflow patterns and significant bottlenecks were identified. CONCLUSIONS: Analyzing the time to complete individual tasks provided a more nuanced description of workflow delays and clarity when traditional survey results conflicted. Augmenting traditional QI process maps with EHR and RTLS timestamps better explained workflow bottlenecks, informing the QI team when selecting targets for subsequent Plan-Do-Check-Act work.

Machine Learning for Detection of Correct Peripherally Inserted Central Catheter Tip Position from Radiology Reports in Infants.

BACKGROUND: In critically ill infants, the position of a peripherally inserted central catheter (PICC) must be confirmed frequently, as the tip may move from its original position and run the risk of hyperosmolar vascular damage or extravasation into surrounding spaces. Automated detection of PICC tip position holds great promise for alerting bedside clinicians to noncentral PICCs. OBJECTIVES: This research seeks to use natural language processing (NLP) and supervised machine learning (ML) techniques to predict PICC tip position based primarily on text analysis of radiograph reports from infants with an upper extremity PICC. METHODS: Radiographs, containing a PICC line in infants under 6 months of age, were manually classified into 12 anatomical locations based on the radiologist's textual report of the PICC line's tip. After categorization, we performed a 70/30 train/test split and benchmarked the performance of seven different (neural network, support vector machine, the naïve Bayes, decision tree, random forest, AdaBoost, and K-nearest neighbors) supervised ML algorithms. After optimization, we calculated accuracy, precision, and recall of each algorithm's ability to correctly categorize the stated location of the PICC tip. RESULTS: A total of 17,337 radiographs met criteria for inclusion and were labeled manually. Interrater agreement was 99.1%. Support vector machines and neural networks yielded accuracies as high as 98% in identifying PICC tips in central versus noncentral position (binary outcome) and accuracies as high as 95% when attempting to categorize the individual anatomical location (12-category outcome). CONCLUSION: Our study shows that ML classifiers can automatically extract the anatomical location of PICC tips from radiology reports. Two ML classifiers, support vector machine (SVM) and a neural network, obtained top accuracies in both binary and multiple category predictions. Implementing these algorithms in a neonatal intensive care unit as a clinical decision support system may help clinicians address PICC line position.