Attention-based neural networks for clinical prediction modelling on electronic health records.

BACKGROUND: Deep learning models have had a lot of success in various fields. However, on structured data they have struggled. Here we apply four state-of-the-art supervised deep learning models using the attention mechanism and compare against logistic regression and XGBoost using discrimination, calibration and clinical utility. METHODS: We develop the models using a general practitioners database. We implement a recurrent neural network, a transformer with and without reverse distillation and a graph neural network. We measure discrimination using the area under the receiver operating characteristic curve (AUC) and the area under the precision recall curve (AUPRC). We assess smooth calibration using restricted cubic splines and clinical utility with decision curve analysis. RESULTS: Our results show that deep learning approaches can improve discrimination up to 2.5% points AUC and 7.4% points AUPRC. However, on average the baselines are competitive. Most models are similarly calibrated as the baselines except for the graph neural network. The transformer using reverse distillation shows the best performance in clinical utility on two out of three prediction problems over most of the prediction thresholds. CONCLUSION: In this study, we evaluated various approaches in supervised learning using neural networks and attention. Here we do a rigorous comparison, not only looking at discrimination but also calibration and clinical utility. There is value in using deep learning models on electronic health record data since it can improve discrimination and clinical utility while providing good calibration. However, good baseline methods are still competitive.

External validation of existing dementia prediction models on observational health data.

BACKGROUND: Many dementia prediction models have been developed, but only few have been externally validated, which hinders clinical uptake and may pose a risk if models are applied to actual patients regardless. Externally validating an existing prediction model is a difficult task, where we mostly rely on the completeness of model reporting in a published article. In this study, we aim to externally validate existing dementia prediction models. To that end, we define model reporting criteria, review published studies, and externally validate three well reported models using routinely collected health data from administrative claims and electronic health records. METHODS: We identified dementia prediction models that were developed between 2011 and 2020 and assessed if they could be externally validated given a set of model criteria. In addition, we externally validated three of these models (Walters' Dementia Risk Score, Mehta's RxDx-Dementia Risk Index, and Nori's ADRD dementia prediction model) on a network of six observational health databases from the United States, United Kingdom, Germany and the Netherlands, including the original development databases of the models. RESULTS: We reviewed 59 dementia prediction models. All models reported the prediction method, development database, and target and outcome definitions. Less frequently reported by these 59 prediction models were predictor definitions (52 models) including the time window in which a predictor is assessed (21 models), predictor coefficients (20 models), and the time-at-risk (42 models). The validation of the model by Walters (development c-statistic: 0.84) showed moderate transportability (0.67-0.76 c-statistic). The Mehta model (development c-statistic: 0.81) transported well to some of the external databases (0.69-0.79 c-statistic). The Nori model (development AUROC: 0.69) transported well (0.62-0.68 AUROC) but performed modestly overall. Recalibration showed improvements for the Walters and Nori models, while recalibration could not be assessed for the Mehta model due to unreported baseline hazard. CONCLUSION: We observed that reporting is mostly insufficient to fully externally validate published dementia prediction models, and therefore, it is uncertain how well these models would work in other clinical settings. We emphasize the importance of following established guidelines for reporting clinical prediction models. We recommend that reporting should be more explicit and have external validation in mind if the model is meant to be applied in different settings.

Comparing penalization methods for linear models on large observational health data.

OBJECTIVE: This study evaluates regularization variants in logistic regression (L1, L2, ElasticNet, Adaptive L1, Adaptive ElasticNet, Broken adaptive ridge [BAR], and Iterative hard thresholding [IHT]) for discrimination and calibration performance, focusing on both internal and external validation. MATERIALS AND METHODS: We use data from 5 US claims and electronic health record databases and develop models for various outcomes in a major depressive disorder patient population. We externally validate all models in the other databases. We use a train-test split of 75%/25% and evaluate performance with discrimination and calibration. Statistical analysis for difference in performance uses Friedman's test and critical difference diagrams. RESULTS: Of the 840 models we develop, L1 and ElasticNet emerge as superior in both internal and external discrimination, with a notable AUC difference. BAR and IHT show the best internal calibration, without a clear external calibration leader. ElasticNet typically has larger model sizes than L1. Methods like IHT and BAR, while slightly less discriminative, significantly reduce model complexity. CONCLUSION: L1 and ElasticNet offer the best discriminative performance in logistic regression for healthcare predictions, maintaining robustness across validations. For simpler, more interpretable models, L0-based methods (IHT and BAR) are advantageous, providing greater parsimony and calibration with fewer features. This study aids in selecting suitable regularization techniques for healthcare prediction models, balancing performance, complexity, and interpretability.

Using clinical text to refine unspecific condition codes in Dutch general practitioner EHR data.

OBJECTIVE: Observational studies using electronic health record (EHR) databases often face challenges due to unspecific clinical codes that can obscure detailed medical information, hindering precise data analysis. In this study, we aimed to assess the feasibility of refining these unspecific condition codes into more specific codes in a Dutch general practitioner (GP) EHR database by leveraging the available clinical free text. METHODS: We utilized three approaches for text classification-search queries, semi-supervised learning, and supervised learning-to improve the specificity of ten unspecific International Classification of Primary Care (ICPC-1) codes. Two text representations and three machine learning algorithms were evaluated for the (semi-)supervised models. Additionally, we measured the improvement achieved by the refinement process on all code occurrences in the database. RESULTS: The classification models performed well for most codes. In general, no single classification approach consistently outperformed the others. However, there were variations in the relative performance of the classification approaches within each code and in the use of different text representations and machine learning algorithms. Class imbalance and limited training data affected the performance of the (semi-)supervised models, yet the simple search queries remained particularly effective. Ultimately, the developed models improved the specificity of over half of all the unspecific code occurrences in the database. CONCLUSIONS: Our findings show the feasibility of using information from clinical text to improve the specificity of unspecific condition codes in observational healthcare databases, even with a limited range of machine-learning techniques and modest annotated training sets. Future work could investigate transfer learning, integration of structured data, alternative semi-supervised methods, and validation of models across healthcare settings. The improved level of detail enriches the interpretation of medical information and can benefit observational research and patient care.

Challenges of Estimating Global Feature Importance in Real-World Health Care Data.

Feature importance is often used to explain clinical prediction models. In this work, we examine three challenges using experiments with electronic health record data: computational feasibility, choosing between methods, and interpretation of the resulting explanation. This work aims to create awareness of the disagreement between feature importance methods and underscores the need for guidance to practitioners how to deal with these discrepancies.

Does Using a Stacking Ensemble Method to Combine Multiple Base Learners Within a Database Improve Model Transportability?

We investigated a stacking ensemble method that combines multiple base learners within a database. The results on external validation across four large databases suggest a stacking ensemble could improve model transportability.

Patient specific intracranial neural signatures of obsessions and compulsions in the ventral striatum.

Objective. Deep brain stimulation is a treatment option for patients with refractory obsessive-compulsive disorder. A new generation of stimulators hold promise for closed loop stimulation, with adaptive stimulation in response to biologic signals. Here we aimed to discover a suitable biomarker in the ventral striatum in patients with obsessive compulsive disorder using local field potentials.Approach.We induced obsessions and compulsions in 11 patients undergoing deep brain stimulation treatment using a symptom provocation task. Then we trained machine learning models to predict symptoms using the recorded intracranial signal from the deep brain stimulation electrodes.Main results.Average areas under the receiver operating characteristics curve were 62.1% for obsessions and 78.2% for compulsions for patient specific models. For obsessions it reached over 85% in one patient, whereas performance was near chance level when the model was trained across patients. Optimal performances for obsessions and compulsions was obtained at different recording sites.Significance. The results from this study suggest that closed loop stimulation may be a viable option for obsessive-compulsive disorder, but that intracranial biomarkers are patient and not disorder specific.Clinical Trial:Netherlands trial registry NL7486.

Use of unstructured text in prognostic clinical prediction models: a systematic review.

OBJECTIVE: This systematic review aims to assess how information from unstructured text is used to develop and validate clinical prognostic prediction models. We summarize the prediction problems and methodological landscape and determine whether using text data in addition to more commonly used structured data improves the prediction performance. MATERIALS AND METHODS: We searched Embase, MEDLINE, Web of Science, and Google Scholar to identify studies that developed prognostic prediction models using information extracted from unstructured text in a data-driven manner, published in the period from January 2005 to March 2021. Data items were extracted, analyzed, and a meta-analysis of the model performance was carried out to assess the added value of text to structured-data models. RESULTS: We identified 126 studies that described 145 clinical prediction problems. Combining text and structured data improved model performance, compared with using only text or only structured data. In these studies, a wide variety of dense and sparse numeric text representations were combined with both deep learning and more traditional machine learning methods. External validation, public availability, and attention for the explainability of the developed models were limited. CONCLUSION: The use of unstructured text in the development of prognostic prediction models has been found beneficial in addition to structured data in most studies. The text data are source of valuable information for prediction model development and should not be neglected. We suggest a future focus on explainability and external validation of the developed models, promoting robust and trustworthy prediction models in clinical practice.

Predicting mortality of individual patients with COVID-19: a multicentre Dutch cohort.

OBJECTIVE: Develop and validate models that predict mortality of patients diagnosed with COVID-19 admitted to the hospital. DESIGN: Retrospective cohort study. SETTING: A multicentre cohort across 10 Dutch hospitals including patients from 27 February to 8 June 2020. PARTICIPANTS: SARS-CoV-2 positive patients (age ≥18) admitted to the hospital. MAIN OUTCOME MEASURES: 21-day all-cause mortality evaluated by the area under the receiver operator curve (AUC), sensitivity, specificity, positive predictive value and negative predictive value. The predictive value of age was explored by comparison with age-based rules used in practice and by excluding age from the analysis. RESULTS: 2273 patients were included, of whom 516 had died or discharged to palliative care within 21 days after admission. Five feature sets, including premorbid, clinical presentation and laboratory and radiology values, were derived from 80 features. Additionally, an Analysis of Variance (ANOVA)-based data-driven feature selection selected the 10 features with the highest F values: age, number of home medications, urea nitrogen, lactate dehydrogenase, albumin, oxygen saturation (%), oxygen saturation is measured on room air, oxygen saturation is measured on oxygen therapy, blood gas pH and history of chronic cardiac disease. A linear logistic regression and non-linear tree-based gradient boosting algorithm fitted the data with an AUC of 0.81 (95% CI 0.77 to 0.85) and 0.82 (0.79 to 0.85), respectively, using the 10 selected features. Both models outperformed age-based decision rules used in practice (AUC of 0.69, 0.65 to 0.74 for age >70). Furthermore, performance remained stable when excluding age as predictor (AUC of 0.78, 0.75 to 0.81). CONCLUSION: Both models showed good performance and had better test characteristics than age-based decision rules, using 10 admission features readily available in Dutch hospitals. The models hold promise to aid decision-making during a hospital bed shortage.

Structural neuroimaging biomarkers for obsessive-compulsive disorder in the ENIGMA-OCD consortium: medication matters.

No diagnostic biomarkers are available for obsessive-compulsive disorder (OCD). Here, we aimed to identify magnetic resonance imaging (MRI) biomarkers for OCD, using 46 data sets with 2304 OCD patients and 2068 healthy controls from the ENIGMA consortium. We performed machine learning analysis of regional measures of cortical thickness, surface area and subcortical volume and tested classification performance using cross-validation. Classification performance for OCD vs. controls using the complete sample with different classifiers and cross-validation strategies was poor. When models were validated on data from other sites, model performance did not exceed chance-level. In contrast, fair classification performance was achieved when patients were grouped according to their medication status. These results indicate that medication use is associated with substantial differences in brain anatomy that are widely distributed, and indicate that clinical heterogeneity contributes to the poor performance of structural MRI as a disease marker.

Mapping Cortical and Subcortical Asymmetry in Obsessive-Compulsive Disorder: Findings From the ENIGMA Consortium.

BACKGROUND: Lateralized dysfunction has been suggested in obsessive-compulsive disorder (OCD). However, it is currently unclear whether OCD is characterized by abnormal patterns of brain structural asymmetry. Here we carried out what is by far the largest study of brain structural asymmetry in OCD. METHODS: We studied a collection of 16 pediatric datasets (501 patients with OCD and 439 healthy control subjects), as well as 30 adult datasets (1777 patients and 1654 control subjects) from the OCD Working Group within the ENIGMA (Enhancing Neuro Imaging Genetics through Meta Analysis) Consortium. Asymmetries of the volumes of subcortical structures, and of measures of regional cortical thickness and surface areas, were assessed based on T1-weighted magnetic resonance imaging scans, using harmonized image analysis and quality control protocols. We investigated possible alterations of brain asymmetry in patients with OCD. We also explored potential associations of asymmetry with specific aspects of the disorder and medication status. RESULTS: In the pediatric datasets, the largest case-control differences were observed for volume asymmetry of the thalamus (more leftward; Cohen's d = 0.19) and the pallidum (less leftward; d = -0.21). Additional analyses suggested putative links between these asymmetry patterns and medication status, OCD severity, or anxiety and depression comorbidities. No significant case-control differences were found in the adult datasets. CONCLUSIONS: The results suggest subtle changes of the average asymmetry of subcortical structures in pediatric OCD, which are not detectable in adults with the disorder. These findings may reflect altered neurodevelopmental processes in OCD.

Cortical Abnormalities Associated With Pediatric and Adult Obsessive-Compulsive Disorder: Findings From the ENIGMA Obsessive-Compulsive Disorder Working Group.

OBJECTIVE: Brain imaging studies of structural abnormalities in OCD have yielded inconsistent results, partly because of limited statistical power, clinical heterogeneity, and methodological differences. The authors conducted meta- and mega-analyses comprising the largest study of cortical morphometry in OCD ever undertaken. METHOD: T1-weighted MRI scans of 1,905 OCD patients and 1,760 healthy controls from 27 sites worldwide were processed locally using FreeSurfer to assess cortical thickness and surface area. Effect sizes for differences between patients and controls, and associations with clinical characteristics, were calculated using linear regression models controlling for age, sex, site, and intracranial volume. RESULTS: In adult OCD patients versus controls, we found a significantly lower surface area for the transverse temporal cortex and a thinner inferior parietal cortex. Medicated adult OCD patients also showed thinner cortices throughout the brain. In pediatric OCD patients compared with controls, we found significantly thinner inferior and superior parietal cortices, but none of the regions analyzed showed significant differences in surface area. However, medicated pediatric OCD patients had lower surface area in frontal regions. Cohen's d effect sizes varied from -0.10 to -0.33. CONCLUSIONS: The parietal cortex was consistently implicated in both adults and children with OCD. More widespread cortical thickness abnormalities were found in medicated adult OCD patients, and more pronounced surface area deficits (mainly in frontal regions) were found in medicated pediatric OCD patients. These cortical measures represent distinct morphological features and may be differentially affected during different stages of development and illness, and possibly moderated by disease profile and medication.

An Empirical Comparison of Meta- and Mega-Analysis With Data From the ENIGMA Obsessive-Compulsive Disorder Working Group.

Objective: Brain imaging communities focusing on different diseases have increasingly started to collaborate and to pool data to perform well-powered meta- and mega-analyses. Some methodologists claim that a one-stage individual-participant data (IPD) mega-analysis can be superior to a two-stage aggregated data meta-analysis, since more detailed computations can be performed in a mega-analysis. Before definitive conclusions regarding the performance of either method can be drawn, it is necessary to critically evaluate the methodology of, and results obtained by, meta- and mega-analyses. Methods: Here, we compare the inverse variance weighted random-effect meta-analysis model with a multiple linear regression mega-analysis model, as well as with a linear mixed-effects random-intercept mega-analysis model, using data from 38 cohorts including 3,665 participants of the ENIGMA-OCD consortium. We assessed the effect sizes and standard errors, and the fit of the models, to evaluate the performance of the different methods. Results: The mega-analytical models showed lower standard errors and narrower confidence intervals than the meta-analysis. Similar standard errors and confidence intervals were found for the linear regression and linear mixed-effects random-intercept models. Moreover, the linear mixed-effects random-intercept models showed better fit indices compared to linear regression mega-analytical models. Conclusions: Our findings indicate that results obtained by meta- and mega-analysis differ, in favor of the latter. In multi-center studies with a moderate amount of variation between cohorts, a linear mixed-effects random-intercept mega-analytical framework appears to be the better approach to investigate structural neuroimaging data.

Changes in scalp potentials and spatial smoothing effects of inclusion of dura layer in human head models for EEG simulations.

The dura layer which covers the brain is less conductive than the CSF (cerebrospinal fluid) and also more conductive than the skull bone. This could significantly influence the flow of volume currents from cortex to the scalp surface which will also change the magnitude and spatial profiles of scalp potentials. This was examined with a 3-D finite element method (FEM) model of an adult subject constructed from 192 segmented axial magnetic resonance (MR) slices with 256×256 pixel resolution. The voxel resolution was 1×1×1 mm. The model included the dura layer. In addition, other major tissues were also identified. The electrical conductivities of various tissues were obtained from the literature. The conductivities of dura and CSF were 0.001 S/m and 0.06 S/m, respectively. The electrical activity of the cortex was represented by 144,000 distributed dipolar sources with orientations normal to the local cortical surface. The dipolar intensity was in the range of 0.0-0.4 mA meter with a uniform random distribution. Scalp potentials were simulated for two head models with an adaptive finite element solver. One model had the dura layer and in the other model, dura layer was replaced with the CSF. Spatial contour plots of potentials on the cortical surface, dural surface and the scalp surface were made. With the inclusion of the dura layer, scalp potentials decrease by about 20%. The contours of gyri and sulci structures were visible in the spatial profiles of the cortical potentials which were smoothed out on the dural surface and were not visible on the scalp surface. These results suggest that dura layer should be included for an accurate modeling of scalp and cortical potentials.