Predictive Performance of Machine Learning-Based Models for Poststroke Clinical Outcomes in Comparison With Conventional Prognostic Scores: Multicenter, Hospital-Based Observational Study.

BACKGROUND: Although machine learning is a promising tool for making prognoses, the performance of machine learning in predicting outcomes after stroke remains to be examined. OBJECTIVE: This study aims to examine how much data-driven models with machine learning improve predictive performance for poststroke outcomes compared with conventional stroke prognostic scores and to elucidate how explanatory variables in machine learning-based models differ from the items of the stroke prognostic scores. METHODS: We used data from 10,513 patients who were registered in a multicenter prospective stroke registry in Japan between 2007 and 2017. The outcomes were poor functional outcome (modified Rankin Scale score >2) and death at 3 months after stroke. Machine learning-based models were developed using all variables with regularization methods, random forests, or boosted trees. We selected 3 stroke prognostic scores, namely, ASTRAL (Acute Stroke Registry and Analysis of Lausanne), PLAN (preadmission comorbidities, level of consciousness, age, neurologic deficit), and iScore (Ischemic Stroke Predictive Risk Score) for comparison. Item-based regression models were developed using the items of these 3 scores. The model performance was assessed in terms of discrimination and calibration. To compare the predictive performance of the data-driven model with that of the item-based model, we performed internal validation after random splits of identical populations into 80% of patients as a training set and 20% of patients as a test set; the models were developed in the training set and were validated in the test set. We evaluated the contribution of each variable to the models and compared the predictors used in the machine learning-based models with the items of the stroke prognostic scores. RESULTS: The mean age of the study patients was 73.0 (SD 12.5) years, and 59.1% (6209/10,513) of them were men. The area under the receiver operating characteristic curves and the area under the precision-recall curves for predicting poststroke outcomes were higher for machine learning-based models than for item-based models in identical populations after random splits. Machine learning-based models also performed better than item-based models in terms of the Brier score. Machine learning-based models used different explanatory variables, such as laboratory data, from the items of the conventional stroke prognostic scores. Including these data in the machine learning-based models as explanatory variables improved performance in predicting outcomes after stroke, especially poststroke death. CONCLUSIONS: Machine learning-based models performed better in predicting poststroke outcomes than regression models using the items of conventional stroke prognostic scores, although they required additional variables, such as laboratory data, to attain improved performance. Further studies are warranted to validate the usefulness of machine learning in clinical settings.

Development of Machine Learning Prediction Models for Self-Extubation After Delirium Using Emergency Department Data.

Delirium is common in the emergency department, and once it develops, there is a risk of self-extubation of drains and tubes, so it is critical to predict delirium before it occurs. Machine learning was used to create two prediction models in this study: one for predicting the occurrence of delirium and one for predicting self-extubation after delirium. Each model showed high discriminative performance, indicating the possibility of selecting high-risk cases. Visualization of predictors using Shapley additive explanation (SHAP), a machine learning interpretability method, showed that the predictors of delirium were different from those of self-extubation after delirium. Data-driven decisions, rather than empirical decisions, on whether or not to use physical restraints or other actions that cause patient suffering will result in improved value in medical care.

Comparing the Robustness of ResNet, Swin-Transformer, and MLP-Mixer under Unique Distribution Shifts in Fundus Images.

BACKGROUND: Diabetic retinopathy (DR) is the leading cause of visual impairment and blindness. Consequently, numerous deep learning models have been developed for the early detection of DR. Safety-critical applications employed in medical diagnosis must be robust to distribution shifts. Previous studies have focused on model performance under distribution shifts using natural image datasets such as ImageNet, CIFAR-10, and SVHN. However, there is a lack of research specifically investigating the performance using medical image datasets. To address this gap, we investigated trends under distribution shifts using fundus image datasets. METHODS: We used the EyePACS dataset for DR diagnosis, introduced noise specific to fundus images, and evaluated the performance of ResNet, Swin-Transformer, and MLP-Mixer models under a distribution shift. The discriminative ability was evaluated using the Area Under the Receiver Operating Characteristic curve (ROC-AUC), while the calibration ability was evaluated using the monotonic sweep calibration error (ECE sweep). RESULTS: Swin-Transformer exhibited a higher ROC-AUC than ResNet under all types of noise and displayed a smaller reduction in the ROC-AUC due to noise. ECE sweep did not show a consistent trend across different model architectures. CONCLUSIONS: Swin-Transformer consistently demonstrated superior discrimination compared to ResNet. This trend persisted even under unique distribution shifts in the fundus images.

Temporal Generalizability of Machine Learning Models for Predicting Postoperative Delirium Using Electronic Health Record Data: Model Development and Validation Study.

BACKGROUND: Although machine learning models demonstrate significant potential in predicting postoperative delirium, the advantages of their implementation in real-world settings remain unclear and require a comparison with conventional models in practical applications. OBJECTIVE: The objective of this study was to validate the temporal generalizability of decision tree ensemble and sparse linear regression models for predicting delirium after surgery compared with that of the traditional logistic regression model. METHODS: The health record data of patients hospitalized at an advanced emergency and critical care medical center in Kumamoto, Japan, were collected electronically. We developed a decision tree ensemble model using extreme gradient boosting (XGBoost) and a sparse linear regression model using least absolute shrinkage and selection operator (LASSO) regression. To evaluate the predictive performance of the model, we used the area under the receiver operating characteristic curve (AUROC) and the Matthews correlation coefficient (MCC) to measure discrimination and the slope and intercept of the regression between predicted and observed probabilities to measure calibration. The Brier score was evaluated as an overall performance metric. We included 11,863 consecutive patients who underwent surgery with general anesthesia between December 2017 and February 2022. The patients were divided into a derivation cohort before the COVID-19 pandemic and a validation cohort during the COVID-19 pandemic. Postoperative delirium was diagnosed according to the confusion assessment method. RESULTS: A total of 6497 patients (68.5, SD 14.4 years, women n=2627, 40.4%) were included in the derivation cohort, and 5366 patients (67.8, SD 14.6 years, women n=2105, 39.2%) were included in the validation cohort. Regarding discrimination, the XGBoost model (AUROC 0.87-0.90 and MCC 0.34-0.44) did not significantly outperform the LASSO model (AUROC 0.86-0.89 and MCC 0.34-0.41). The logistic regression model (AUROC 0.84-0.88, MCC 0.33-0.40, slope 1.01-1.19, intercept -0.16 to 0.06, and Brier score 0.06-0.07), with 8 predictors (age, intensive care unit, neurosurgery, emergency admission, anesthesia time, BMI, blood loss during surgery, and use of an ambulance) achieved good predictive performance. CONCLUSIONS: The XGBoost model did not significantly outperform the LASSO model in predicting postoperative delirium. Furthermore, a parsimonious logistic model with a few important predictors achieved comparable performance to machine learning models in predicting postoperative delirium.

Development of artificial intelligence prognostic model for surgically resected non-small cell lung cancer.

There are great expectations for artificial intelligence (AI) in medicine. We aimed to develop an AI prognostic model for surgically resected non-small cell lung cancer (NSCLC). This study enrolled 1049 patients with pathological stage I-IIIA surgically resected NSCLC at Kyushu University. We set 17 clinicopathological factors and 30 preoperative and 22 postoperative blood test results as explanatory variables. Disease-free survival (DFS), overall survival (OS), and cancer-specific survival (CSS) were set as objective variables. The eXtreme Gradient Boosting (XGBoost) was used as the machine learning algorithm. The median age was 69 (23-89) years, and 605 patients (57.7%) were male. The numbers of patients with pathological stage IA, IB, IIA, IIB, and IIIA were 553 (52.7%), 223 (21.4%), 100 (9.5%), 55 (5.3%), and 118 (11.2%), respectively. The 5-year DFS, OS, and CSS rates were 71.0%, 82.8%, and 88.7%, respectively. Our AI prognostic model showed that the areas under the curve of the receiver operating characteristic curves of DFS, OS, and CSS at 5 years were 0.890, 0.926, and 0.960, respectively. The AI prognostic model using XGBoost showed good prediction accuracy and provided accurate predictive probability of postoperative prognosis of NSCLC.

Non-linear association between body weight and functional outcome after acute ischemic stroke.

This study aimed to determine whether body weight is associated with functional outcome after acute ischemic stroke. We measured the body mass index (BMI) and assessed clinical outcomes in patients with acute ischemic stroke. The BMI was categorized into underweight (< 18.5 kg/m2), normal weight (18.5-22.9 kg/m2), overweight (23.0-24.9 kg/m2), and obesity (≥ 25.0 kg/m2). The association between BMI and a poor functional outcome (modified Rankin Scale [mRS] score: 3-6) was evaluated. We included 11,749 patients with acute ischemic stroke (70.3 ± 12.2 years, 36.1% women). The risk of a 3-month poor functional outcome was higher for underweight, lower for overweight, and did not change for obesity in reference to a normal weight even after adjusting for covariates by logistic regression analysis. Restricted cubic splines and SHapley Additive exPlanation values in eXtreme Gradient Boosting model also showed non-linear relationships. Associations between BMI and a poor functional outcome were maintained even after excluding death (mRS score: 3-5) or including mild disability (mRS score: 2-6) as the outcome. The associations were strong in older patients, non-diabetic patients, and patients with mild stroke. Body weight has a non-linear relationship with the risk of a poor functional outcome after acute ischemic stroke.

Diagnostic model for preschool workers' unwillingness to continue working: Developed using machine-learning techniques.

The turnover of kindergarten teachers has drastically increased in the past 10 years. Reducing the turnover rates among preschool workers has become an important issue worldwide. Parents have avoided enrolling children in preschools due to insufficient care, which affects their ability to work. Therefore, this study developed a diagnostic model to understand preschool workers' unwillingness to continue working. A total of 1002 full-time preschool workers were divided into 2 groups. Predictors were drawn from general questionnaires, including those for mental health. We compared 3 algorithms: the least absolute shrinkage and selection operator, eXtreme Gradient Boosting, and logistic regression. Additionally, the SHapley Additive exPlanation was used to visualize the relationship between years of work experience and intention to continue working. The logistic regression model was adopted as the diagnostic model, and the predictors were "not living with children," "human relation problems with boss," "high risk of mental distress," and "work experience." The developed risk score and the optimal cutoff value were 14 points. By using the diagnostic model to determine workers' unwillingness to continue working, supervisors can intervene with workers who are experiencing difficulties at work and can help resolve their problems.

Explanation of machine learning models using shapley additive explanation and application for real data in hospital.

BACKGROUND AND OBJECTIVE: When using machine learning techniques in decision-making processes, the interpretability of the models is important. In the present paper, we adopted the Shapley additive explanation (SHAP), which is based on fair profit allocation among many stakeholders depending on their contribution, for interpreting a gradient-boosting decision tree model using hospital data. METHODS: For better interpretability, we propose two novel techniques as follows: (1) a new metric of feature importance using SHAP and (2) a technique termed feature packing, which packs multiple similar features into one grouped feature to allow an easier understanding of the model without reconstruction of the model. We then compared the explanation results between the SHAP framework and existing methods using cerebral infarction data from our hospital. RESULTS: The interpretation by SHAP was mostly consistent with that by the existing methods. We showed how the A/G ratio works as an important prognostic factor for cerebral infarction using proposed techniques. CONCLUSION: Our techniques are useful for interpreting machine learning models and can uncover the underlying relationships between features and outcome.

Impact of a learning health system on acute care and medical complications after intracerebral hemorrhage.

INTRODUCTION: Patients with stroke often experience pneumonia during the acute stage after stroke onset. Oral care may be effective in reducing the risk of stroke-associated pneumonia (SAP). We aimed to determine the changes in oral care, as well as the incidence of SAP, in patients with intracerebral hemorrhage, following implementation of a learning health system in our hospital. METHODS: We retrospectively analyzed the data of 1716 patients with intracerebral hemorrhage who were hospitalized at a single stroke center in Japan between January 2012 and December 2018. Data were stratified on the basis of three periods of evolving oral care: period A, during which conventional, empirically driven oral care was provided (n = 725); period B, during which standardized oral care was introduced, with SAP prophylaxis based on known risk factors (n = 469); and period C, during which oral care was risk-appropriate based on learning health system data (n = 522). Logistic regression analysis was performed to evaluate associations between each of the three treatment approaches and the risk of SAP. RESULTS: Among the included patients, the mean age was 71.3 ± 13.6 years; 52.6% of patients were men. During the course of each period, the frequency of oral care within 24 hours of admission increased (P < .001), as did the adherence rate to oral care ≥3 times per day (P < .001). After adjustment for confounding factors, a change in the risk of SAP was not observed in period B; however, the risk significantly decreased in period C (odds ratio 0.61; 95% confidence interval 0.43-0.87) compared with period A. These associations were maintained for SAP diagnosed using strict clinical criteria or after exclusion of 174 patients who underwent neurosurgical treatment. CONCLUSIONS: Risk-appropriate care informed by the use of learning health system data could improve care and potentially reduce the risk of SAP in patients with intracerebral hemorrhage in the acute stage.

Stroke Prognostic Scores and Data-Driven Prediction of Clinical Outcomes After Acute Ischemic Stroke.

Background and Purpose- Several stroke prognostic scores have been developed to predict clinical outcomes after stroke. This study aimed to develop and validate novel data-driven predictive models for clinical outcomes by referring to previous prognostic scores in patients with acute ischemic stroke in a real-world setting. Methods- We used retrospective data of 4237 patients with acute ischemic stroke who were hospitalized in a single stroke center in Japan between January 2012 and August 2017. We first validated point-based stroke prognostic scores (preadmission comorbidities, level of consciousness, age, and neurological deficit [PLAN] score, ischemic stroke predictive risk score [IScore], and acute stroke registry and analysis of Lausanne [ASTRAL] score in all patients; Houston intraarterial recanalization therapy [HIAT] score, totaled health risks in vascular events [THRIVE] score, and stroke prognostication using age and National Institutes of Health Stroke Scale-100 [SPAN-100] in patients who received reperfusion therapy) in our cohort. We then developed predictive models using all available data by linear regression or decision tree ensembles (random forest and gradient boosting decision tree) and evaluated their area under the receiver operating characteristic curve for clinical outcomes after repeated random splits. Results- The mean (SD) age of the patients was 74.7 (12.9) years and 58.3% were men. Area under the receiver operating characteristic curves (95% CIs) of prognostic scores in our cohort were 0.92 PLAN score (0.90-0.93), 0.86 for IScore (0.85-0.87), 0.85 for ASTRAL score (0.83-0.86), 0.69 for HIAT score (0.62-0.75), 0.70 for THRIVE score (0.64-0.76), and 0.70 for SPAN-100 (0.63-0.76) for poor functional outcomes, and 0.87 for PLAN score (0.85-0.90), 0.88 for IScore (0.86-0.91), and 0.88 ASTRAL score (0.85-0.91) for in-hospital mortality. Internal validation of data-driven prediction models showed that their area under the receiver operating characteristic curves ranged between 0.88 and 0.94 for poor functional outcomes and between 0.84 and 0.88 for in-hospital mortality. Ensemble models of a decision tree tended to outperform linear regression models in predicting poor functional outcomes but not in predicting in-hospital mortality. Conclusions- Stroke prognostic scores perform well in predicting clinical outcomes after stroke. Data-driven models may be an alternative tool for predicting poststroke clinical outcomes in a real-world setting.

Extracting Predictive Indicator for Prognosis of Cerebral Infarction Using Machine Learning Techniques.

Identifying important predicative indicators for prognosis is useful since these factors help for understanding diseases and determining treatments for patients. We extracted important factors for prognosis of cerebral infarction from EHR. We analyzed EHR data of 1,697 patients with 1,602 variables using gradient boosting decision tree. Extracted factors include not only well-known factors such as NIHSS but also new factors such as albumin-globulin ratio.