Relationship between prediction accuracy and feature importance reliability: An empirical and theoretical study.

There is significant interest in using neuroimaging data to predict behavior. The predictive models are often interpreted by the computation of feature importance, which quantifies the predictive relevance of an imaging feature. Tian and Zalesky (2021) suggest that feature importance estimates exhibit low split-half reliability, as well as a trade-off between prediction accuracy and feature importance reliability across parcellation resolutions. However, it is unclear whether the trade-off between prediction accuracy and feature importance reliability is universal. Here, we demonstrate that, with a sufficient sample size, feature importance (operationalized as Haufe-transformed weights) can achieve fair to excellent split-half reliability. With a sample size of 2600 participants, Haufe-transformed weights achieve average intra-class correlation coefficients of 0.75, 0.57 and 0.53 for cognitive, personality and mental health measures respectively. Haufe-transformed weights are much more reliable than original regression weights and univariate FC-behavior correlations. Original regression weights are not reliable even with 2600 participants. Intriguingly, feature importance reliability is strongly positively correlated with prediction accuracy across phenotypes. Within a particular behavioral domain, there is no clear relationship between prediction performance and feature importance reliability across regression models. Furthermore, we show mathematically that feature importance reliability is necessary, but not sufficient, for low feature importance error. In the case of linear models, lower feature importance error is mathematically related to lower prediction error. Therefore, higher feature importance reliability might yield lower feature importance error and higher prediction accuracy. Finally, we discuss how our theoretical results relate with the reliability of imaging features and behavioral measures. Overall, the current study provides empirical and theoretical insights into the relationship between prediction accuracy and feature importance reliability.

Comparison between gradients and parcellations for functional connectivity prediction of behavior.

Resting-state functional connectivity (RSFC) is widely used to predict behavioral measures. To predict behavioral measures, representing RSFC with parcellations and gradients are the two most popular approaches. Here, we compare parcellation and gradient approaches for RSFC-based prediction of a broad range of behavioral measures in the Human Connectome Project (HCP) and Adolescent Brain Cognitive Development (ABCD) datasets. Among the parcellation approaches, we consider group-average "hard" parcellations (Schaefer et al., 2018), individual-specific "hard" parcellations (Kong et al., 2021a), and an individual-specific "soft" parcellation (spatial independent component analysis with dual regression; Beckmann et al., 2009). For gradient approaches, we consider the well-known principal gradients (Margulies et al., 2016) and the local gradient approach that detects local RSFC changes (Laumann et al., 2015). Across two regression algorithms, individual-specific hard-parcellation performs the best in the HCP dataset, while the principal gradients, spatial independent component analysis and group-average "hard" parcellations exhibit similar performance. On the other hand, principal gradients and all parcellation approaches perform similarly in the ABCD dataset. Across both datasets, local gradients perform the worst. Finally, we find that the principal gradient approach requires at least 40 to 60 gradients to perform as well as parcellation approaches. While most principal gradient studies utilize a single gradient, our results suggest that incorporating higher order gradients can provide significant behaviorally relevant information. Future work will consider the inclusion of additional parcellation and gradient approaches for comparison.

Proportional intracranial volume correction differentially biases behavioral predictions across neuroanatomical features, sexes, and development.

Individual differences in brain anatomy can be used to predict variations in cognitive ability. Most studies to date have focused on broad population-level trends, but the extent to which the observed predictive features are shared across sexes and age groups remains to be established. While it is standard practice to account for intracranial volume (ICV) using proportion correction in both regional and whole-brain morphometric analyses, in the context of brain-behavior predictions the possible differential impact of ICV correction on anatomical features and subgroups within the population has yet to be systematically investigated. In this work, we evaluate the effect of proportional ICV correction on sex-independent and sex-specific predictive models of individual cognitive abilities across multiple anatomical properties (surface area, gray matter volume, and cortical thickness) in healthy young adults (Human Connectome Project; n = 1013, 548 females) and typically developing children (Adolescent Brain Cognitive Development study; n = 1823, 979 females). We demonstrate that ICV correction generally reduces predictive accuracies derived from surface area and gray matter volume, while increasing predictive accuracies based on cortical thickness in both adults and children. Furthermore, the extent to which predictive models generalize across sexes and age groups depends on ICV correction: models based on surface area and gray matter volume are more generalizable without ICV correction, while models based on cortical thickness are more generalizable with ICV correction. Finally, the observed neuroanatomical features predictive of cognitive abilities are unique across age groups regardless of ICV correction, but whether they are shared or unique across sexes (within age groups) depends on ICV correction. These findings highlight the importance of considering individual differences in ICV, and show that proportional ICV correction does not remove the effects of cranial volume from anatomical measurements and can introduce ICV bias where previously there was none. ICV correction choices affect not just the strength of the relationships captured, but also the conclusions drawn regarding the neuroanatomical features that underlie those relationships.

Comparison of individualized behavioral predictions across anatomical, diffusion and functional connectivity MRI.

A fundamental goal across the neurosciences is the characterization of relationships linking brain anatomy, functioning, and behavior. Although various MRI modalities have been developed to probe these relationships, direct comparisons of their ability to predict behavior have been lacking. Here, we compared the ability of anatomical T1, diffusion and functional MRI (fMRI) to predict behavior at an individual level. Cortical thickness, area and volume were extracted from anatomical T1 images. Diffusion Tensor Imaging (DTI) and approximate Neurite Orientation Dispersion and Density Imaging (NODDI) models were fitted to the diffusion images. The resulting metrics were projected to the Tract-Based Spatial Statistics (TBSS) skeleton. We also ran probabilistic tractography for the diffusion images, from which we extracted the stream count, average stream length, and the average of each DTI and NODDI metric across tracts connecting each pair of brain regions. Functional connectivity (FC) was extracted from both task and resting-state fMRI. Individualized prediction of a wide range of behavioral measures were performed using kernel ridge regression, linear ridge regression and elastic net regression. Consistency of the results were investigated with the Human Connectome Project (HCP) and Adolescent Brain Cognitive Development (ABCD) datasets. In both datasets, FC-based models gave the best prediction performance, regardless of regression model or behavioral measure. This was especially true for the cognitive component. Furthermore, all modalities were able to predict cognition better than other behavioral components. Combining all modalities improved prediction of cognition, but not other behavioral components. Finally, across all behaviors, combining resting and task FC yielded prediction performance similar to combining all modalities. Overall, our study suggests that in the case of healthy children and young adults, behaviorally-relevant information in T1 and diffusion features might reflect a subset of the variance captured by FC.

Evaluation of trigeminal nerve tractography using two-fold-accelerated simultaneous multi-slice readout-segmented echo planar diffusion tensor imaging.

OBJECTIVES: Simultaneous multi-slice (SMS) imaging with short repetition time (TR) accelerates diffusion tensor imaging (DTI) acquisitions. However, its impact when combined with readout-segmented echo planar imaging (RESOLVE) on the cranial nerves given the challenging skull base/posterior fossa terrain is unexplored. We evaluated the reliability of trigeminal nerve DTI metrics using SMS with RESOLVE-DTI. METHODS: Eight healthy controls and six patients with unilateral trigeminal neuralgia (TN) underwent brain MRI scan. Three different RESOLVE-DTI protocols were performed on a 3-T MRI system: non-SMS (TR = 4330 ms), SMS with identical TR (4330 ms), and SMS with short TR (2400 ms). Pontine signal-to-noise ratio (SNR) and DTI metrics of the trigeminal nerve streamlines tracked by two independent raters using deterministic tractography and standardized tracking protocol were obtained. These were statistically analyzed and compared across the three protocols using intra-rater and inter-rater intraclass correlation coefficients (ICCs), one-way analysis of variance (ANOVA), post hoc analysis, and linear regression. RESULTS: On visual screening, there were no artifacts across the trigeminal nerves. All data also cleared objective image quality assurance analysis. Pontine SNR was similar for the two SMS protocols and higher for the non-SMS RESOLVE-DTI (F(2,36) = 4.40, p = 0.02). Intra-rater and inter-rater ICCs were very good (> 0.85). Trigeminal nerve DTI metrics were consistently measured by the three protocols, revealing significant linear relationships between non-SMS- and SMS-derived DTI metrics. CONCLUSION: SMS RESOLVE-DTI enables fast and reliable evaluation of microstructural integrity of the trigeminal nerve, with potential application in the clinical management of TN. KEY POINTS: • Readout-segmented diffusion-weighted echo planar imaging (RESOLVE-DTI) reduces image distortion artifacts in the posterior fossa but its long acquisition time limits clinical utility. • Simultaneous multi-slice (SMS) imaging combined with RESOLVE-DTI provides reliable trigeminal nerve tractography with potential applications in trigeminal neuralgia. • Two-fold-accelerated RESOLVE-DTI yields comparable trigeminal nerve streamlines and DTI metrics while near-halving acquisition time.

MULTIMODAL NEURAL CORRELATES OF CHILDHOOD PSYCHOPATHOLOGY.

Complex structural and functional changes occurring in typical and atypical development necessitate multidimensional approaches to better understand the risk of developing psychopathology. Here, we simultaneously examined structural and functional brain network patterns in relation to dimensions of psychopathology in the Adolescent Brain Cognitive Development dataset. Several components were identified, recapitulating the psychopathology hierarchy, with the general psychopathology (p) factor explaining most covariance with multimodal imaging features, while the internalizing, externalizing, and neurodevelopmental dimensions were each associated with distinct morphological and functional connectivity signatures. Connectivity signatures associated with the p factor and neurodevelopmental dimensions followed the sensory-to-transmodal axis of cortical organization, which is related to the emergence of complex cognition and risk for psychopathology. Results were consistent in two separate data subsamples, supporting generalizability, and robust to variations in analytical parameters. Our findings help in better understanding biological mechanisms underpinning dimensions of psychopathology, and could provide brain-based vulnerability markers.

Ventral attention network connectivity is linked to cortical maturation and cognitive ability in childhood.

The human brain experiences functional changes through childhood and adolescence, shifting from an organizational framework anchored within sensorimotor and visual regions into one that is balanced through interactions with later-maturing aspects of association cortex. Here, we link this profile of functional reorganization to the development of ventral attention network connectivity across independent datasets. We demonstrate that maturational changes in cortical organization link preferentially to within-network connectivity and heightened degree centrality in the ventral attention network, whereas connectivity within network-linked vertices predicts cognitive ability. This connectivity is associated closely with maturational refinement of cortical organization. Children with low ventral attention network connectivity exhibit adolescent-like topographical profiles, suggesting that attentional systems may be relevant in understanding how brain functions are refined across development. These data suggest a role for attention networks in supporting age-dependent shifts in cortical organization and cognition across childhood and adolescence.

MRI economics: Balancing sample size and scan duration in brain wide association studies.

A pervasive dilemma in neuroimaging is whether to prioritize sample size or scan time given fixed resources. Here, we systematically investigate this trade-off in the context of brain-wide association studies (BWAS) using functional magnetic resonance imaging (fMRI). We find that total scan duration (sample size x scan time per participant) robustly explains individual-level phenotypic prediction accuracy via a logarithmic model, suggesting that sample size and scan time are broadly interchangeable up to 20-30 min of data. However, the returns of scan time diminish relative to sample size, which we explain with principled theoretical derivations. When accounting for fixed overhead costs associated with each participant (e.g., recruitment, non-imaging measures), prediction accuracy in many small-scale and some large-scale BWAS might benefit from longer scan time than typically assumed. These results generalize across phenotypic domains, scanners, acquisition protocols, racial groups, mental disorders, age groups, as well as resting-state and task-state functional connectivity. Overall, our study emphasizes the importance of scan time, which is ignored in standard power calculations. Standard power calculations maximize sample size, at the expense of scan time, which can result in sub-optimal prediction accuracies and inefficient use of resources. Our empirically informed reference is available for future study design: WEB_APPLICATION_LINK.

Distinct brain network features predict internalizing and externalizing traits in children and adults.

Internalizing and externalizing traits are two distinct classes of behaviors in psychiatry. However, whether shared or unique brain network features predict internalizing and externalizing behaviors in children and adults remain poorly understood. Using a sample of 2262 children from the Adolescent Brain Cognitive Development (ABCD) study and 752 adults from the Human Connectome Project (HCP), we show that network features predicting internalizing and externalizing behavior are, at least in part, dissociable in children, but not in adults. In ABCD children, traits within internalizing and externalizing behavioral categories are predicted by more similar network features concatenated across task and resting states than those between different categories. We did not observe this pattern in HCP adults. Distinct network features predict internalizing and externalizing behaviors in ABCD children and HCP adults. These data reveal shared and unique brain network features accounting for individual variation within broad internalizing and externalizing categories across developmental stages.

Serial deep gray nuclear DTI changes in Parkinson's disease over twelve years.

Background: Deep gray nuclear pathology relates to motor deterioration in idiopathic Parkinson's disease (PD). Inconsistent deep nuclear diffusion tensor imaging (DTI) findings in cross-sectional or short-term longitudinal studies have been reported. Long-term studies in PD are clinically challenging; decade-long deep nuclear DTI data are nonexistent. We investigated serial DTI changes and clinical utility in a case-control PD cohort of 149 subjects (72 patients/77 controls) over 12 years. Methods: Participating subjects underwent brain MRI at 1.5T; DTI metrics from segmented masks of caudate, putamen, globus pallidus and thalamus were extracted from three timepoints with 6-year gaps. Patients underwent clinical assessment, including Unified Parkinson Disease Rating Scale Part 3 (UPDRS-III) and Hoehn and Yahr (H&Y) staging. A multivariate linear mixed-effects regression model with adjustments for age and gender was used to assess between-group differences in DTI metrics at each timepoint. Partial Pearson correlation analysis was used to correlate clinical motor scores with DTI metrics over time. Results: MD progressively increased over time and was higher in the putamen (p < 0.001) and globus pallidus (p = 0.002). FA increased (p < 0.05) in the thalamus at year six, and decreased in the putamen and globus pallidus at year 12. Putaminal (p = 0.0210), pallidal (p = 0.0066) and caudate MD (p < 0.0001) correlated with disease duration. Caudate MD (p < 0.05) also correlated with UPDRS-III and H&Y scores. Conclusion: Pallido-putaminal MD showed differential neurodegeneration in PD over 12 years on longitudinal DTI; putaminal and thalamic FA changes were complex. Caudate MD could serve as a surrogate marker to track late PD progression.

Multilayer meta-matching: translating phenotypic prediction models from multiple datasets to small data.

Resting-state functional connectivity (RSFC) is widely used to predict phenotypic traits in individuals. Large sample sizes can significantly improve prediction accuracies. However, for studies of certain clinical populations or focused neuroscience inquiries, small-scale datasets often remain a necessity. We have previously proposed a "meta-matching" approach to translate prediction models from large datasets to predict new phenotypes in small datasets. We demonstrated large improvement of meta-matching over classical kernel ridge regression (KRR) when translating models from a single source dataset (UK Biobank) to the Human Connectome Project Young Adults (HCP-YA) dataset. In the current study, we propose two meta-matching variants ("meta-matching with dataset stacking" and "multilayer meta-matching") to translate models from multiple source datasets across disparate sample sizes to predict new phenotypes in small target datasets. We evaluate both approaches by translating models trained from five source datasets (with sample sizes ranging from 862 participants to 36,834 participants) to predict phenotypes in the HCP-YA and HCP-Aging datasets. We find that multilayer meta-matching modestly outperforms meta-matching with dataset stacking. Both meta-matching variants perform better than the original "meta-matching with stacking" approach trained only on the UK Biobank. All meta-matching variants outperform classical KRR and transfer learning by a large margin. In fact, KRR is better than classical transfer learning when less than 50 participants are available for finetuning, suggesting the difficulty of classical transfer learning in the very small sample regime. The multilayer meta-matching model is publicly available at GITHUB_LINK.

Shared and unique brain network features predict cognitive, personality, and mental health scores in the ABCD study.

How individual differences in brain network organization track behavioral variability is a fundamental question in systems neuroscience. Recent work suggests that resting-state and task-state functional connectivity can predict specific traits at the individual level. However, most studies focus on single behavioral traits, thus not capturing broader relationships across behaviors. In a large sample of 1858 typically developing children from the Adolescent Brain Cognitive Development (ABCD) study, we show that predictive network features are distinct across the domains of cognitive performance, personality scores and mental health assessments. On the other hand, traits within each behavioral domain are predicted by similar network features. Predictive network features and models generalize to other behavioral measures within the same behavioral domain. Although tasks are known to modulate the functional connectome, predictive network features are similar between resting and task states. Overall, our findings reveal shared brain network features that account for individual variation within broad domains of behavior in childhood.

Cross-ethnicity/race generalization failure of behavioral prediction from resting-state functional connectivity.

Algorithmic biases that favor majority populations pose a key challenge to the application of machine learning for precision medicine. Here, we assessed such bias in prediction models of behavioral phenotypes from brain functional magnetic resonance imaging. We examined the prediction bias using two independent datasets (preadolescent versus adult) of mixed ethnic/racial composition. When predictive models were trained on data dominated by white Americans (WA), out-of-sample prediction errors were generally higher for African Americans (AA) than for WA. This bias toward WA corresponds to more WA-like brain-behavior association patterns learned by the models. When models were trained on AA only, compared to training only on WA or an equal number of AA and WA participants, AA prediction accuracy improved but stayed below that for WA. Overall, the results point to the need for caution and further research regarding the application of current brain-behavior prediction models in minority populations.

Clinical correlates of white matter lesions in Parkinson's disease using automated multi-modal segmentation measures.

BACKGROUND: Age-related white matter lesions (WML) are common, impact neuronal connectivity, and affect motor function and cognition. In addition to pathological nigrostriatal losses, WML are also common co-morbidities in Parkinson's disease (PD) that affect postural stability and gait. Automated brain volume measures are increasingly incorporated into the clinical reporting workflow to facilitate precision in medicine. Recently, multi-modal segmentation algorithms have been developed to overcome challenges with WML quantification based on single-modality input. OBJECTIVE: We evaluated WML volumes and their distribution in a case-control cohort of PD patients to predict the domain-specific clinical severity using a fully automated multi-modal segmentation algorithm. METHODS: Fifty-five subjects comprising of twenty PD patients and thirty-five age- and gender-matched control subjects underwent standardized motor/gait and cognitive assessments and brain MRI. Spatially differentiated WML obtained using automated segmentation algorithms on multi-modal MPRAGE and FLAIR images were used to predict domain-specific clinical severity. Preliminary statistical analysis focused on describing the relationship between WML and clinical scores, and the distribution of WML by brain regions. Subsequent stepwise regressions were performed to predict each clinical score using WML volumes in different brain regions, while controlling for age. RESULTS: WML volume strongly correlates with both motor and cognitive dysfunctions in PD patients (p < 0.05), with differential impact in the frontal lobe and periventricular regions on cognitive domains (p < 0.01) and severity of motor deficits (p < 0.01), respectively. CONCLUSION: Automated multi-modal segmentation algorithms may facilitate precision medicine through regional WML load quantification, which show potential as imaging biomarkers for predicting domain-specific disease severity in PD.

Corrigendum: Increased Activation of Default Mode Network in Early Parkinson's With Excessive Daytime Sleepiness.

[This corrects the article DOI: 10.3389/fnins.2019.01334.].

Increased Activation of Default Mode Network in Early Parkinson's With Excessive Daytime Sleepiness.

BACKGROUND AND OBJECTIVES: The underlying neuropathology of excessive daytime sleepiness (EDS) remains elusive in Parkinson's disease (PD). We aim to investigate neural network changes that underlie EDS in PD. METHODS: Early PD patients comprising eighty-one patients without EDS (EDS-) and seventeen patients with EDS (EDS+) received a resting state functional MRI scan and the Epworth Sleepiness Scale (ESS). Connectivities within the default mode network (DMN), motor and basal ganglia networks were compared between the EDS+ and EDS- groups. Correlations between network connectivity and the severity of EDS were investigated through linear regression. RESULTS: EDS+ patients displayed a trend of increased network connectivity of the posterior DMN (pDMN). A significant positive correlation was found between connectivity of the ventromedial prefrontal cortex in the pDMN and ESS. CONCLUSION: EDS+ patients are likely to display increased activation in the DMN, suggesting neural compensation in early PD or impaired attentiveness due to mechanisms such as mind-wandering.