Post-Processing Automatic Transcriptions with Machine Learning for Verbal Fluency Scoring.

Objective: To compare verbal fluency scores derived from manual transcriptions to those obtained using automatic speech recognition enhanced with machine learning classifiers. Methods: Using Amazon Web Services, we automatically transcribed verbal fluency recordings from 1400 individuals who performed both animal and letter F verbal fluency tasks. We manually adjusted timings and contents of the automatic transcriptions to obtain "gold standard" transcriptions. To make automatic scoring possible, we trained machine learning classifiers to discern between valid and invalid utterances. We then calculated and compared verbal fluency scores from the manual and automatic transcriptions. Results: For both animal and letter fluency tasks, we achieved good separation of valid versus invalid utterances. Verbal fluency scores calculated based on automatic transcriptions showed high correlation with those calculated after manual correction. Conclusion: Many techniques for scoring verbal fluency word lists require accurate transcriptions with word timings. We show that machine learning methods can be applied to improve off-the-shelf ASR for this purpose. These automatically derived scores may be satisfactory for some applications. Low correlations among some of the scores indicate the need for improvement in automatic speech recognition before a fully automatic approach can be reliably implemented.

Prediction of coronary artery bypass graft outcomes using a single surgical note: An artificial intelligence-based prediction model study.

BACKGROUND: Healthcare providers currently calculate risk of the composite outcome of morbidity or mortality associated with a coronary artery bypass grafting (CABG) surgery through manual input of variables into a logistic regression-based risk calculator. This study indicates that automated artificial intelligence (AI)-based techniques can instead calculate risk. Specifically, we present novel numerical embedding techniques that enable NLP (natural language processing) models to achieve higher performance than the risk calculator using a single preoperative surgical note. METHODS: The most recent preoperative surgical consult notes of 1,738 patients who received an isolated CABG from July 1, 2014 to November 1, 2022 at a single institution were analyzed. The primary outcome was the Society of Thoracic Surgeons defined composite outcome of morbidity or mortality (MM). We tested three numerical-embedding techniques on the widely used TextCNN classification model: 1a) Basic embedding, treat numbers as word tokens; 1b) Basic embedding with a dataloader that Replaces out-of-context (ROOC) numbers with a tag, where context is defined as within a number of tokens of specified keywords; 2) ScaleNum, an embedding technique that scales in-context numbers via a learned sigmoid-linear-log function; and 3) AttnToNum, a ScaleNum-derivative that updates the ScaleNum embeddings via multi-headed attention applied to local context. Predictive performance was measured via area under the receiver operating characteristic curve (AUC) on holdout sets from 10 random-split experiments. For eXplainable-AI (X-AI), we calculate SHapley Additive exPlanation (SHAP) values at an ngram resolution (SHAP-N). While the analyses focus on TextCNN, we execute an analogous performance pipeline with a long short-term memory (LSTM) model to test if the numerical embedding advantage is robust to model architecture. RESULTS: A total of 567 (32.6%) patients had MM following CABG. The embedding performances are as follows with the TextCNN architecture: 1a) Basic, mean AUC 0.788 [95% CI (confidence interval): 0.768-0.809]; 1b) ROOC, 0.801 [CI: 0.788-0.815]; 2) ScaleNum, 0.808 [CI: 0.785-0.821]; and 3) AttnToNum, 0.821 [CI: 0.806-0.834]. The LSTM architecture produced a similar trend. Permutation tests indicate that AttnToNum outperforms the other embedding techniques, though not statistically significant verse ScaleNum (p-value of .07). SHAP-N analyses indicate that the model learns to associate low blood urine nitrate (BUN) and creatinine values with survival. A correlation analysis of the attention-updated numerical embeddings indicates that AttnToNum learns to incorporate both number magnitude and local context to derive semantic similarities. CONCLUSION: This research presents both quantitative and clinical novel contributions. Quantitatively, we contribute two new embedding techniques: AttnToNum and ScaleNum. Both can embed strictly positive and bounded numerical values, and both surpass basic embeddings in predictive performance. The results suggest AttnToNum outperforms ScaleNum. With regards to clinical research, we show that AI methods can predict outcomes after CABG using a single preoperative note at a performance that matches or surpasses the current risk calculator. These findings reveal the potential role of NLP in automated registry reporting and quality improvement.

Red Flag/Blue Flag visualization of a common CNN for text classification.

A shallow convolutional neural network (CNN), TextCNN, has become nearly ubiquitous for classification among clinical and medical text. This research presents a novel eXplainable-AI (X-AI) software, Red Flag/Blue Flag (RFBF), designed for binary classification with TextCNN. RFBF visualizes each convolutional filter's discriminative capability. This is a more informative approach than direct assessment of logit contribution, features that overfit to train set nuances on smaller datasets may indiscriminately activate large logits on validation samples from both classes. RFBF enables model diagnosis, term feature verification, and overfit prevention. We present 3 use cases of (1) filter consistency assessment; (2) predictive performance improvement; and (3) estimation of information leakage between train and holdout sets. The use cases derive from experiments on TextCNN for binary prediction of surgical misadventure outcomes from physician-authored operative notes. Due to TextCNN's prevalence, this X-AI can benefit clinical text research, and hence improve patient outcomes.

A better way: training for direct observations in healthcare.

Direct observation is valuable for identifying latent threats and elucidating system complexity in clinical environments. This approach facilitates prospective risk assessment and reveals workarounds, near-misses and recurrent safety problems difficult to diagnose retrospectively or via outcome data alone. As observers are an instrument of data collection, developing effective and comprehensive observer training is critical to ensuring the reliability of the data collection and reproducibility of the research. However, methodological rigour for ensuring these data collection properties remains a key challenge in direct observation research in healthcare. Although prior literature has offered key considerations for observational research in healthcare, operationalising these recommendations may pose a challenge and unless guidance is also provided on observer training. In this article, we offer guidelines for training non-clinical observers to conduct direct observations including conducting a training needs analysis, incorporating practice observations and evaluating observers and inter-rater reliability. The operationalisation of these guidelines is described in the context of a 5-year multisite observational study investigating technology integration in the operating room. We also discuss novel tools developed during the course our project to support data collection and examine inter-rater reliability among observers in direct observation studies.

Research and Exploratory Analysis Driven-Time-data Visualization (read-tv) software.

MOTIVATION: Research & Exploratory Analysis Driven Time-data Visualization (read-tv) is an open source R Shiny application for visualizing irregularly and regularly spaced longitudinal data. read-tv provides unique filtering and changepoint analysis (CPA) features. The need for these analyses was motivated by research of surgical work-flow disruptions in operating room settings. Specifically, for the analysis of the causes and characteristics of periods of high disruption-rates, which are associated with adverse surgical outcomes. MATERIALS AND METHODS: read-tv is a graphical application, and the main component of a package of the same name. read-tv generates and evaluates code to filter and visualize data. Users can view the visualization code from within the application, which facilitates reproducibility. The data input requirements are simple, a table with a time column with no missing values. The input can either be in the form of a file, or an in-memory dataframe- which is effective for rapid visualization during curation. RESULTS: We used read-tv to automatically detect surgical disruption cascades. We found that the most common disruption type during a cascade was training, followed by equipment. DISCUSSION: read-tv fills a need for visualization software of surgical disruptions and other longitudinal data. Every visualization is reproducible, the exact source code that read-tv executes to create a visualization is available from within the application. read-tv is generalizable, it can plot any tabular dataset given the simple requirements that there is a numeric, datetime, or datetime string column with no missing values. Finally, the tab-based architecture of read-tv is easily extensible, it is relatively simple to add new functionality by implementing a tab in the source code. CONCLUSION: read-tv enables quick identification of patterns through customizable longitudinal plots; faceting; CPA; and user-specified filters. The package is available on GitHub under an MIT license.

Using Flow Disruptions to Examine System Safety in Robotic-Assisted Surgery: Protocol for a Stepped Wedge Crossover Design.

BACKGROUND: The integration of high technology into health care systems is intended to provide new treatment options and improve the quality, safety, and efficiency of care. Robotic-assisted surgery is an example of high technology integration in health care, which has become ubiquitous in many surgical disciplines. OBJECTIVE: This study aims to understand and measure current robotic-assisted surgery processes in a systematic, quantitative, and replicable manner to identify latent systemic threats and opportunities for improvement based on our observations and to implement and evaluate interventions. This 5-year study will follow a human factors engineering approach to improve the safety and efficiency of robotic-assisted surgery across 4 US hospitals. METHODS: The study uses a stepped wedge crossover design with 3 interventions, introduced in different sequences at each of the hospitals over four 8-month phases. Robotic-assisted surgery procedures will be observed in the following specialties: urogynecology, gynecology, urology, bariatrics, general, and colorectal. We will use the data collected from observations, surveys, and interviews to inform interventions focused on teamwork, task design, and workplace design. We intend to evaluate attitudes toward each intervention, safety culture, subjective workload for each case, effectiveness of each intervention (including through direct observation of a sample of surgeries in each observational phase), operating room duration, length of stay, and patient safety incident reports. Analytic methods will include statistical data analysis, point process analysis, and thematic content analysis. RESULTS: The study was funded in September 2018 and approved by the institutional review board of each institution in May and June of 2019 (CSMC and MDRH: Pro00056245; VCMC: STUDY 270; MUSC: Pro00088741). After refining the 3 interventions in phase 1, data collection for phase 2 (baseline data) began in November 2019 and was scheduled to continue through June 2020. However, data collection was suspended in March 2020 due to the COVID-19 pandemic. We collected a total of 65 observations across the 4 sites before the pandemic. Data collection for phase 2 was resumed in October 2020 at 2 of the 4 sites. CONCLUSIONS: This will be the largest direct observational study of surgery ever conducted with data collected on 680 robotic surgery procedures at 4 different institutions. The proposed interventions will be evaluated using individual-level (workload and attitude), process-level (perioperative duration and flow disruption), and organizational-level (safety culture and complications) measures. An implementation science framework is also used to investigate the causes of success or failure of each intervention at each site and understand the potential spread of the interventions. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): DERR1-10.2196/25284.

Predicting naming responses based on pre-articulatory electrical activity in individuals with aphasia.

OBJECTIVE: To investigate whether pre-articulatory neural activity could be used to predict correct vs. incorrect naming responses in individuals with post-stroke aphasia. METHODS: We collected 64-channel high density electroencephalography (hdEEG) data from 5 individuals with chronic post-stroke aphasia (2 female/3 male, median age: 54 years) during naming of 80 concrete images. We applied machine learning on continuous wavelet transformed hdEEG data separately for alpha and beta energy bands (200 ms pre-stimulus to 1500 ms post-stimulus, but before articulation), and determined whether electrode/time-range/energy (ETE) combinations were predictive of correct vs incorrect responses for each participant. RESULTS: The five participants correctly named between 30% and 70% of the 80 stimuli correctly. We observed that pre-articulatory scalp EEG ETE combinations could predict correct vs incorrect responses with accuracies ranging from 63% to 80%. For all but one participant, the prediction accuracies were statistically better than chance. CONCLUSIONS: Our findings indicate that pre-articulatory neural activity may be used to predict correct vs incorrect naming responses for some individuals with aphasia. SIGNIFICANCE: The individualized pre-articulatory neural pattern associated with correct naming responses could be used to both predict naming problems in aphasia and lead to the development of brain stimulation strategies for treatment.

Using machine learning to classify temporal lobe epilepsy based on diffusion MRI.

BACKGROUND: It is common for patients diagnosed with medial temporal lobe epilepsy (TLE) to have extrahippocampal damage. However, it is unclear whether microstructural extrahippocampal abnormalities are consistent enough to enable classification using diffusion MRI imaging. Therefore, we implemented a support vector machine (SVM)-based method to predict TLE from three different imaging modalities: mean kurtosis (MK), mean diffusivity (MD), and fractional anisotropy (FA). While MD and FA can be calculated from traditional diffusion tensor imaging (DTI), MK requires diffusion kurtosis imaging (DKI). METHODS: Thirty-two TLE patients and 36 healthy controls underwent DKI imaging. To measure predictive capability, a fivefold cross-validation (CV) was repeated for 1000 iterations. An ensemble of SVM models, each with a different regularization value, was trained with the subject images in the training set, and had performance assessed on the test set. The different regularization values were determined using a Bayesian-based method. RESULTS: Mean kurtosis achieved higher accuracy than both FA and MD on every iteration, and had far superior average accuracy: 0.82 (MK), 0.68 (FA), and 0.51 (MD). Finally, the MK voxels with the highest coefficients in the predictive models were distributed within the inferior medial aspect of the temporal lobes. CONCLUSION: These results corroborate our earlier publications which indicated that DKI shows more promise in identifying TLE-associated pathological features than DTI. Also, the locations of the contributory MK voxels were in areas with high fiber crossing and complex fiber anatomy. These traits result in non-Gaussian water diffusion, and hence render DTI less likely to detect abnormalities. If the location of consistent microstructural abnormalities can be better understood, then it may be possible in the future to identify the various phenotypes of TLE. This is important since treatment outcome varies dependent on type of TLE.

Mapping Language Networks Using the Structural and Dynamic Brain Connectomes.

Lesion-symptom mapping is often employed to define brain structures that are crucial for human behavior. Even though poststroke deficits result from gray matter damage as well as secondary white matter loss, the impact of structural disconnection is overlooked by conventional lesion-symptom mapping because it does not measure loss of connectivity beyond the stroke lesion. This study describes how traditional lesion mapping can be combined with structural connectome lesion symptom mapping (CLSM) and connectome dynamics lesion symptom mapping (CDLSM) to relate residual white matter networks to behavior. Using data from a large cohort of stroke survivors with aphasia, we observed improved prediction of aphasia severity when traditional lesion symptom mapping was combined with CLSM and CDLSM. Moreover, only CLSM and CDLSM disclosed the importance of temporal-parietal junction connections in aphasia severity. In summary, connectome measures can uniquely reveal brain networks that are necessary for function, improving the traditional lesion symptom mapping approach.

Activity associated with speech articulation measured through direct cortical recordings.

The insula has been credited with a role in a number of functions, including speech production. Here, we recorded electrocorticography (ECoG) signals from the left insula during pseudoword articulation in two patients undergoing pre-surgical monitoring for the management of medically-intractable epilepsy. Event-related band power (ERBP) activity from electrodes implanted in the superior precentral gyrus of the insula (SPGI) was compared to that of other left hemisphere regions implicated in speech production. Results showed that SPGI contacts demonstrated significantly greater ERBP within the high-gamma frequency range (75-150Hz) during articulation compared to a listening condition. However, frontal and post-central regions demonstrated significantly greater responses to the articulation task compared to the SPGI. Results suggest the SPGI is active during articulation, but frontal and post-central regions demonstrate significantly more robust responses. Given the small sample size, and number of electrodes implanted in the SPGI, further study is warranted to confirm these findings.