Predicting Readmission or Death After Discharge From the ICU: External Validation and Retraining of a Machine Learning Model.

OBJECTIVES: Many machine learning (ML) models have been developed for application in the ICU, but few models have been subjected to external validation. The performance of these models in new settings therefore remains unknown. The objective of this study was to assess the performance of an existing decision support tool based on a ML model predicting readmission or death within 7 days after ICU discharge before, during, and after retraining and recalibration. DESIGN: A gradient boosted ML model was developed and validated on electronic health record data from 2004 to 2021. We performed an independent validation of this model on electronic health record data from 2011 to 2019 from a different tertiary care center. SETTING: Two ICUs in tertiary care centers in The Netherlands. PATIENTS: Adult patients who were admitted to the ICU and stayed for longer than 12 hours. INTERVENTIONS: None. MEASUREMENTS AND MAIN RESULTS: We assessed discrimination by area under the receiver operating characteristic curve (AUC) and calibration (slope and intercept). We retrained and recalibrated the original model and assessed performance via a temporal validation design. The final retrained model was cross-validated on all data from the new site. Readmission or death within 7 days after ICU discharge occurred in 577 of 10,052 ICU admissions (5.7%) at the new site. External validation revealed moderate discrimination with an AUC of 0.72 (95% CI 0.67-0.76). Retrained models showed improved discrimination with AUC 0.79 (95% CI 0.75-0.82) for the final validation model. Calibration was poor initially and good after recalibration via isotonic regression. CONCLUSIONS: In this era of expanding availability of ML models, external validation and retraining are key steps to consider before applying ML models to new settings. Clinicians and decision-makers should take this into account when considering applying new ML models to their local settings.

Functional brain network and trail making test changes following major surgery and postoperative delirium: a prospective, multicentre, observational cohort study.

BACKGROUND: Delirium is a frequent complication after surgery in older adults and is associated with an increased risk of long-term cognitive impairment and dementia. Disturbances in functional brain networks were previously reported during delirium. We hypothesised that alterations in functional brain networks persist after remission of postoperative delirium and that functional brain network alterations are associated with long-term cognitive impairment. METHODS: In this prospective, multicentre, observational cohort study, we included older patients who underwent clinical assessments (including the Trail Making Test B [TMT-B]) and resting-state functional MRI (rs-fMRI) before and 3 months after elective surgery. Delirium was assessed on the first seven postoperative days. RESULTS: Of the 554 enrolled patients, 246 remained after strict motion correction, of whom 38 (16%) developed postoperative delirium. The rs-fMRI functional connectivity strength increased 3 months after surgery in the total study population (ß=0.006; 95% confidence interval [CI]: 0.001-0.011; P=0.013), but it decreased after postoperative delirium (ß=-0.015; 95% CI: -0.028 to 0.002; P=0.023). No difference in TMT-B scores was found at follow-up between patients with and without postoperative delirium. Patients with decreased functional connectivity strength declined in TMT-B scores compared with those who did not (ß=11.04; 95% CI: 0.85-21.2; P=0.034). CONCLUSIONS: Postoperative delirium was associated with decreased brain functional connectivity strength after 3 months, suggesting that delirium has a long-lasting impact on brain networks. The decreased connectivity strength was associated with significant cognitive deterioration after major surgery. CLINICAL TRIAL REGISTRATION: NCT02265263.

Analyzing patient experiences using natural language processing: development and validation of the artificial intelligence patient reported experience measure (AI-PREM).

BACKGROUND: Evaluating patients' experiences is essential when incorporating the patients' perspective in improving healthcare. Experiences are mainly collected using closed-ended questions, although the value of open-ended questions is widely recognized. Natural language processing (NLP) can automate the analysis of open-ended questions for an efficient approach to patient-centeredness. METHODS: We developed the Artificial Intelligence Patient-Reported Experience Measures (AI-PREM) tool, consisting of a new, open-ended questionnaire, an NLP pipeline to analyze the answers using sentiment analysis and topic modeling, and a visualization to guide physicians through the results. The questionnaire and NLP pipeline were iteratively developed and validated in a clinical context. RESULTS: The final AI-PREM consisted of five open-ended questions about the provided information, personal approach, collaboration between healthcare professionals, organization of care, and other experiences. The AI-PREM was sent to 867 vestibular schwannoma patients, 534 of which responded. The sentiment analysis model attained an F1 score of 0.97 for positive texts and 0.63 for negative texts. There was a 90% overlap between automatically and manually extracted topics. The visualization was hierarchically structured into three stages: the sentiment per question, the topics per sentiment and question, and the original patient responses per topic. CONCLUSIONS: The AI-PREM tool is a comprehensive method that combines a validated, open-ended questionnaire with a well-performing NLP pipeline and visualization. Thematically organizing and quantifying patient feedback reduces the time invested by healthcare professionals to evaluate and prioritize patient experiences without being confined to the limited answer options of closed-ended questions.

Different phenotypes of neuropsychiatric systemic lupus erythematosus are related to a distinct pattern of structural changes on brain MRI.

OBJECTIVES: The underlying structural brain correlates of neuropsychiatric involvement in systemic lupus erythematosus (NPSLE) remain unclear, thus hindering correct diagnosis. We compared brain tissue volumes between a clinically well-defined cohort of patients with NPSLE and SLE patients with neuropsychiatric syndromes not attributed to SLE (non-NPSLE). Within the NPSLE patients, we also examined differences between patients with two distinct disease phenotypes: ischemic and inflammatory. METHODS: In this prospective (May 2007 to April 2015) cohort study, we included 38 NPSLE patients (26 inflammatory and 12 ischemic) and 117 non-NPSLE patients. All patients underwent a 3-T brain MRI scan that was used to automatically determine white matter, grey matter, white matter hyperintensities (WMH) and total brain volumes. Group differences in brain tissue volumes were studied with linear regression analyses corrected for age, gender, and total intracranial volume and expressed as B values and 95% confidence intervals. RESULTS: NPSLE patients showed higher WMH volume compared to non-NPSLE patients (p = 0.004). NPSLE inflammatory patients showed lower total brain (p = 0.014) and white matter volumes (p = 0.020), and higher WMH volume (p = 0.002) compared to non-NPSLE patients. Additionally, NPSLE inflammatory patients showed lower white matter (p = 0.020) and total brain volumes (p = 0.038) compared to NPSLE ischemic patients. CONCLUSION: We showed that different phenotypes of NPSLE were related to distinct patterns of underlying structural brain MRI changes. Especially the inflammatory phenotype of NPSLE was associated with the most pronounced brain volume changes, which might facilitate the diagnostic process in SLE patients with neuropsychiatric symptoms. KEY POINTS: • Neuropsychiatric systemic lupus erythematosus (NPSLE) patients showed a higher WMH volume compared to SLE patients with neuropsychiatric syndromes not attributed to SLE (non-NPSLE). • NPSLE patients with inflammatory phenotype showed a lower total brain and white matter volume, and a higher volume of white matter hyperintensities, compared to non-NPSLE patients. • NPSLE patients with inflammatory phenotype showed lower white matter and total brain volumes compared to NPSLE patients with ischemic phenotype.

MRI Markers of Neurodegenerative and Neurovascular Changes in Relation to Postoperative Delirium and Postoperative Cognitive Decline.

Postoperative delirium (POD) and postoperative cognitive decline (POCD) are common in elderly patients. The aim of the present review was to explore the association of neurodegenerative and neurovascular changes with the occurrence of POD and POCD. Fifteen MRI studies were identified by combining multiple search terms for POD, POCD, and brain imaging. These studies described a total of 1,422 patients and were all observational in design. Neurodegenerative changes (global and regional brain volumes) did not show a consistent association with the occurrence of POD (four studies) or POCD (two studies). In contrast, neurovascular changes (white matter hyperintensities and cerebral infarcts) were more consistently associated with the occurrence of POD (seven studies) and POCD (five studies). In conclusion, neurovascular changes appear to be consistently associated with the occurrence of POD and POCD, and may identify patients at increased risk of these conditions. Larger prospective studies are needed to study the consistency of these findings and to unravel the underlying pathophysiological mechanisms.

Impact of a Digital Scribe System on Clinical Documentation Time and Quality: Usability Study.

BACKGROUND: Physicians spend approximately half of their time on administrative tasks, which is one of the leading causes of physician burnout and decreased work satisfaction. The implementation of natural language processing-assisted clinical documentation tools may provide a solution. OBJECTIVE: This study investigates the impact of a commercially available Dutch digital scribe system on clinical documentation efficiency and quality. METHODS: Medical students with experience in clinical practice and documentation (n=22) created a total of 430 summaries of mock consultations and recorded the time they spent on this task. The consultations were summarized using 3 methods: manual summaries, fully automated summaries, and automated summaries with manual editing. We then randomly reassigned the summaries and evaluated their quality using a modified version of the Physician Documentation Quality Instrument (PDQI-9). We compared the differences between the 3 methods in descriptive statistics, quantitative text metrics (word count and lexical diversity), the PDQI-9, Recall-Oriented Understudy for Gisting Evaluation scores, and BERTScore. RESULTS: The median time for manual summarization was 202 seconds against 186 seconds for editing an automatic summary. Without editing, the automatic summaries attained a poorer PDQI-9 score than manual summaries (median PDQI-9 score 25 vs 31, P<.001, ANOVA test). Automatic summaries were found to have higher word counts but lower lexical diversity than manual summaries (P<.001, independent t test). The study revealed variable impacts on PDQI-9 scores and summarization time across individuals. Generally, students viewed the digital scribe system as a potentially useful tool, noting its ease of use and time-saving potential, though some criticized the summaries for their greater length and rigid structure. CONCLUSIONS: This study highlights the potential of digital scribes in improving clinical documentation processes by offering a first summary draft for physicians to edit, thereby reducing documentation time without compromising the quality of patient records. Furthermore, digital scribes may be more beneficial to some physicians than to others and could play a role in improving the reusability of clinical documentation. Future studies should focus on the impact and quality of such a system when used by physicians in clinical practice.

Applying natural language processing to patient messages to identify depression concerns in cancer patients.

OBJECTIVE: This study aims to explore and develop tools for early identification of depression concerns among cancer patients by leveraging the novel data source of messages sent through a secure patient portal. MATERIALS AND METHODS: We developed classifiers based on logistic regression (LR), support vector machines (SVMs), and 2 Bidirectional Encoder Representations from Transformers (BERT) models (original and Reddit-pretrained) on 6600 patient messages from a cancer center (2009-2022), annotated by a panel of healthcare professionals. Performance was compared using AUROC scores, and model fairness and explainability were examined. We also examined correlations between model predictions and depression diagnosis and treatment. RESULTS: BERT and RedditBERT attained AUROC scores of 0.88 and 0.86, respectively, compared to 0.79 for LR and 0.83 for SVM. BERT showed bigger differences in performance across sex, race, and ethnicity than RedditBERT. Patients who sent messages classified as concerning had a higher chance of receiving a depression diagnosis, a prescription for antidepressants, or a referral to the psycho-oncologist. Explanations from BERT and RedditBERT differed, with no clear preference from annotators. DISCUSSION: We show the potential of BERT and RedditBERT in identifying depression concerns in messages from cancer patients. Performance disparities across demographic groups highlight the need for careful consideration of potential biases. Further research is needed to address biases, evaluate real-world impacts, and ensure responsible integration into clinical settings. CONCLUSION: This work represents a significant methodological advancement in the early identification of depression concerns among cancer patients. Our work contributes to a route to reduce clinical burden while enhancing overall patient care, leveraging BERT-based models.

Identification of Distinct Brain MRI Phenotypes and Their Association With Long-Term Dementia Risk in Community-Dwelling Older Adults.

BACKGROUND AND OBJECTIVES: Individual brain MRI markers only show at best a modest association with long-term occurrence of dementia. Therefore, it is challenging to accurately identify individuals at increased risk for dementia. We aimed to identify different brain MRI phenotypes by hierarchical clustering analysis based on combined neurovascular and neurodegenerative brain MRI markers and to determine the long-term dementia risk within the brain MRI phenotype subgroups. METHODS: Hierarchical clustering analysis based on 32 combined neurovascular and neurodegenerative brain MRI markers in community-dwelling individuals of the Age-Gene/Environment Susceptibility Reykjavik Study was applied to identify brain MRI phenotypes. A Cox proportional hazards regression model was used to determine the long-term risk for dementia per subgroup. RESULTS: We included 3,056 participants and identified 15 subgroups with distinct brain MRI phenotypes. The phenotypes ranged from limited burden, mostly irregular white matter hyperintensity (WMH) shape and cerebral atrophy, mostly irregularly WMHs and microbleeds, mostly cortical infarcts and atrophy, mostly irregularly shaped WMH and cerebral atrophy to multiburden subgroups. Each subgroup showed different long-term risks for dementia (min-max range hazard ratios [HRs] 1.01-6.18; mean time to follow-up 9.9 ± 2.6 years); especially the brain MRI phenotype with mainly WMHs and atrophy showed a large increased risk (HR 6.18, 95% CI 3.37-11.32). DISCUSSION: Distinct brain MRI phenotypes can be identified in community-dwelling older adults. Our results indicate that distinct brain MRI phenotypes are related to varying long-term risks of developing dementia. Brain MRI phenotypes may in the future assist in an improved understanding of the structural correlates of dementia predisposition.

Diagnosis Classification in the Emergency Room Using Natural Language Processing.

Diagnosis classification in the emergency room (ER) is a complex task. We developed several natural language processing classification models, looking both at the full classification task of 132 diagnostic categories and at several clinically applicable samples consisting of two diagnoses that are hard to distinguish.

Postoperative delirium is associated with grey matter brain volume loss.

Delirium is associated with long-term cognitive dysfunction and with increased brain atrophy. However, it is unclear whether these problems result from or predisposes to delirium. We aimed to investigate preoperative to postoperative brain changes, as well as the role of delirium in these changes over time. We investigated the effects of surgery and postoperative delirium with brain MRIs made before and 3 months after major elective surgery in 299 elderly patients, and an MRI with a 3 months follow-up MRI in 48 non-surgical control participants. To study the effects of surgery and delirium, we compared brain volumes, white matter hyperintensities and brain infarcts between baseline and follow-up MRIs, using multiple regression analyses adjusting for possible confounders. Within the patients group, 37 persons (12%) developed postoperative delirium. Surgical patients showed a greater decrease in grey matter volume than non-surgical control participants [linear regression: B (95% confidence interval) = -0.65% of intracranial volume (-1.01 to -0.29, P < 0.005)]. Within the surgery group, delirium was associated with a greater decrease in grey matter volume [B (95% confidence interval): -0.44% of intracranial volume (-0.82 to -0.06, P = 0.02)]. Furthermore, within the patients, delirium was associated with a non-significantly increased risk of a new postoperative brain infarct [logistic regression: odds ratio (95% confidence interval): 2.8 (0.7-11.1), P = 0.14]. Our study was the first to investigate the association between delirium and preoperative to postoperative brain volume changes, suggesting that delirium is associated with increased progression of grey matter volume loss.