Predicting the future of neuroimaging predictive models in mental health.

Predictive modeling using neuroimaging data has the potential to improve our understanding of the neurobiology underlying psychiatric disorders and putatively information interventions. Accordingly, there is a plethora of literature reviewing published studies, the mathematics underlying machine learning, and the best practices for using these approaches. As our knowledge of mental health and machine learning continue to evolve, we instead aim to look forward and "predict" topics that we believe will be important in current and future studies. Some of the most discussed topics in machine learning, such as bias and fairness, the handling of dirty data, and interpretable models, may be less familiar to the broader community using neuroimaging-based predictive modeling in psychiatry. In a similar vein, transdiagnostic research and targeting brain-based features for psychiatric intervention are modern topics in psychiatry that predictive models are well-suited to tackle. In this work, we target an audience who is a researcher familiar with the fundamental procedures of machine learning and who wishes to increase their knowledge of ongoing topics in the field. We aim to accelerate the utility and applications of neuroimaging-based predictive models for psychiatric research by highlighting and considering these topics. Furthermore, though not a focus, these ideas generalize to neuroimaging-based predictive modeling in other clinical neurosciences and predictive modeling with different data types (e.g., digital health data).

Leveraging edge-centric networks complements existing network-level inference for functional connectomes.

The human connectome is modular with distinct brain regions clustering together to form large-scale communities, or networks. This concept has recently been leveraged in novel inferencing procedures by averaging the edge-level statistics within networks to induce more powerful inferencing at the network level. However, these networks are constructed based on the similarity between pairs of nodes. Emerging work has described novel edge-centric networks, which instead use the similarity between pairs of edges to construct networks. In this work, we use these edge-centric networks in a network-level inferencing procedure and compare this novel method to traditional inferential procedures and the network-level procedure using node-centric networks. We use data from the Human Connectome Project, the Healthy Brain Network, and the Philadelphia Neurodevelopmental Cohort and use a resampling technique with various sample sizes (n=40, 80, 120) to probe the power and specificity of each method. Across datasets and sample sizes, using the edge-centric networks outperforms using node-centric networks for inference as well as edge-level FDR correction and NBS. Additionally, the edge-centric networks were found to be more consistent in clustering effect sizes of similar values as compared to node-centric networks, although node-centric networks often had a lower average within-network effect size variability. Together, these findings suggest that using edge-centric networks for network-level inference can procure relatively powerful results while remaining similarly accurate to the underlying edge-level effects across networks, complementing previous inferential methods.

Large-scale differences in functional organization of left- and right-handed individuals using whole-brain, data-driven analysis of connectivity.

Handedness influences differences in lateralization of language areas as well as dominance of motor and somatosensory cortices. However, differences in whole-brain functional connectivity (i.e., functional connectomes) due to handedness have been relatively understudied beyond pre-specified networks of interest. Here, we compared functional connectomes of left- and right-handed individuals at the whole brain level. We explored differences in functional connectivity of previously established regions of interest, and showed differences between primarily left- and primarily right-handed individuals in the motor, somatosensory, and language areas using functional connectivity. We then proceeded to investigate these differences in the whole brain and found that the functional connectivity of left- and right-handed individuals are not specific to networks of interest, but extend across every region of the brain. In particular, we found that connections between and within the cerebellum show distinct patterns of connectivity. To put these effects into context, we show that the effect sizes associated with handedness differences account for a similar amount of individual differences in the connectome as sex differences. Together these results shed light on regions of the brain beyond those traditionally explored that contribute to differences in the functional organization of left- and right-handed individuals and underscore that handedness effects are neurobiologically meaningful in addition to being statistically significant.

Data leakage inflates prediction performance in connectome-based machine learning models.

Predictive modeling is a central technique in neuroimaging to identify brain-behavior relationships and test their generalizability to unseen data. However, data leakage undermines the validity of predictive models by breaching the separation between training and test data. Leakage is always an incorrect practice but still pervasive in machine learning. Understanding its effects on neuroimaging predictive models can inform how leakage affects existing literature. Here, we investigate the effects of five forms of leakage-involving feature selection, covariate correction, and dependence between subjects-on functional and structural connectome-based machine learning models across four datasets and three phenotypes. Leakage via feature selection and repeated subjects drastically inflates prediction performance, whereas other forms of leakage have minor effects. Furthermore, small datasets exacerbate the effects of leakage. Overall, our results illustrate the variable effects of leakage and underscore the importance of avoiding data leakage to improve the validity and reproducibility of predictive modeling.

Brain-phenotype predictions can survive across diverse real-world data.

Recent work suggests that machine learning models predicting psychiatric treatment outcomes based on clinical data may fail when applied to unharmonized samples. Neuroimaging predictive models offer the opportunity to incorporate neurobiological information, which may be more robust to dataset shifts. Yet, among the minority of neuroimaging studies that undertake any form of external validation, there is a notable lack of attention to generalization across dataset-specific idiosyncrasies. Research settings, by design, remove the between-site variations that real-world and, eventually, clinical applications demand. Here, we rigorously test the ability of a range of predictive models to generalize across three diverse, unharmonized samples: the Philadelphia Neurodevelopmental Cohort (n=1291), the Healthy Brain Network (n=1110), and the Human Connectome Project in Development (n=428). These datasets have high inter-dataset heterogeneity, encompassing substantial variations in age distribution, sex, racial and ethnic minority representation, recruitment geography, clinical symptom burdens, fMRI tasks, sequences, and behavioral measures. We demonstrate that reproducible and generalizable brain-behavior associations can be realized across diverse dataset features with sample sizes in the hundreds. Results indicate the potential of functional connectivity-based predictive models to be robust despite substantial inter-dataset variability. Notably, for the HCPD and HBN datasets, the best predictions were not from training and testing in the same dataset (i.e., cross-validation) but across datasets. This result suggests that training on diverse data may improve prediction in specific cases. Overall, this work provides a critical foundation for future work evaluating the generalizability of neuroimaging predictive models in real-world scenarios and clinical settings.

The effects of data leakage on connectome-based machine learning models.

Predictive modeling has now become a central technique in neuroimaging to identify complex brain-behavior relationships and test their generalizability to unseen data. However, data leakage, which unintentionally breaches the separation between data used to train and test the model, undermines the validity of predictive models. Previous literature suggests that leakage is generally pervasive in machine learning, but few studies have empirically evaluated the effects of leakage in neuroimaging data. Although leakage is always an incorrect practice, understanding the effects of leakage on neuroimaging predictive models provides insight into the extent to which leakage may affect the literature. Here, we investigated the effects of leakage on machine learning models in two common neuroimaging modalities, functional and structural connectomes. Using over 400 different pipelines spanning four large datasets and three phenotypes, we evaluated five forms of leakage fitting into three broad categories: feature selection, covariate correction, and lack of independence between subjects. As expected, leakage via feature selection and repeated subjects drastically inflated prediction performance. Notably, other forms of leakage had only minor effects (e.g., leaky site correction) or even decreased prediction performance (e.g., leaky covariate regression). In some cases, leakage affected not only prediction performance, but also model coefficients, and thus neurobiological interpretations. Finally, we found that predictive models using small datasets were more sensitive to leakage. Overall, our results illustrate the variable effects of leakage on prediction pipelines and underscore the importance of avoiding data leakage to improve the validity and reproducibility of predictive modeling.

Power and reproducibility in the external validation of brain-phenotype predictions.

Identifying reproducible and generalizable brain-phenotype associations is a central goal of neuroimaging. Consistent with this goal, prediction frameworks evaluate brain-phenotype models in unseen data. Most prediction studies train and evaluate a model in the same dataset. However, external validation, or the evaluation of a model in an external dataset, provides a better assessment of robustness and generalizability. Despite the promise of external validation and calls for its usage, the statistical power of such studies has yet to be investigated. In this work, we ran over 60 million simulations across several datasets, phenotypes, and sample sizes to better understand how the sizes of the training and external datasets affect statistical power. We found that prior external validation studies used sample sizes prone to low power, which may lead to false negatives and effect size inflation. Furthermore, increases in the external sample size led to increased simulated power directly following theoretical power curves, whereas changes in the training dataset size offset the simulated power curves. Finally, we compared the performance of a model within a dataset to the external performance. The within-dataset performance was typically within r=0.2 of the cross-dataset performance, which could help decide how to power future external validation studies. Overall, our results illustrate the importance of considering the sample sizes of both the training and external datasets when performing external validation.

Cross Atlas Remapping via Optimal Transport (CAROT): Creating connectomes for different atlases when raw data is not available.

Open-source, publicly available neuroimaging datasets - whether from large-scale data collection efforts or pooled from multiple smaller studies - offer unprecedented sample sizes and promote generalization efforts. Releasing data can democratize science, increase the replicability of findings, and lead to discoveries. Partly due to patient privacy, computational, and data storage concerns, researchers typically release preprocessed data with the voxelwise time series parcellated into a map of predefined regions, known as an atlas. However, releasing preprocessed data also limits the choices available to the end-user. This is especially true for connectomics, as connectomes created from different atlases are not directly comparable. Since there exist several atlases with no gold standards, it is unrealistic to have processed, open-source data available from all atlases. Together, these limitations directly inhibit the potential benefits of open-source neuroimaging data. To address these limitations, we introduce Cross Atlas Remapping via Optimal Transport (CAROT) to find a mapping between two atlases. This approach allows data processed from one atlas to be directly transformed into a connectome based on another atlas without the need for raw data access. To validate CAROT, we compare reconstructed connectomes against their original counterparts (i.e., connectomes generated directly from an atlas), demonstrate the utility of transformed connectomes in downstream analyses, and show how a connectome-based predictive model can generalize to publicly available data that was processed with different atlases. Overall, CAROT can reconstruct connectomes from an extensive set of atlases - without needing the raw data - allowing already processed connectomes to be easily reused in a wide range of analyses while eliminating redundant processing efforts. We share this tool as both source code and as a stand-alone web application (http://carotproject.com/).

Heightened sensitivity to high-calorie foods in children at risk for obesity: insights from behavior, neuroimaging, and genetics.

Pediatric obesity is a major public health concern. Genetic susceptibility and increased availability of energy-dense food are known risk factors for obesity. However, the extent to which these factors jointly bias behavior and neural circuitry towards increased adiposity in children remains unclear. While undergoing fMRI, 108 children (ages 5-11y) performed a food-specific go/no-go task. Participants were instructed to either respond ("go") or inhibit responding ("no-go") to images of food or toys. Half of the runs depicted high-calorie foods (e.g., pizza) whereas the other half depicted low-calorie foods (e.g., salad). Children were also genotyped for a DNA polymorphism associated with energy intake and obesity (FTO rs9939609) to examine the influence of obesity risk on behavioral and brain responses to food. Participants demonstrated differences in behavioral sensitivity to high- and low-calorie food images depending on task demands. Participants were slower but more accurate at detecting high- (relative to low-) calorie foods when responding to a neutral stimulus (i.e., toys) and worse at detecting toys when responding to high-calorie foods. Inhibition failures were accompanied by salience network activity (anterior insula, dorsal anterior cingulate cortex), which was driven by false alarms to food images. Children at a greater genetic risk for obesity (dose-dependent model of the FTO genotype) demonstrated pronounced brain and behavioral relationships such that genetic risk was associated with heightened sensitivity to high-calorie food images and increased anterior insula activity. These findings suggest that high-calorie foods may be particularly salient to children at risk for developing eating habits that promote obesity.

Machine Learning and Prediction in Fetal, Infant, and Toddler Neuroimaging: A Review and Primer.

Predictive models in neuroimaging are increasingly designed with the intent to improve risk stratification and support interventional efforts in psychiatry. Many of these models have been developed in samples of children school-aged or older. Nevertheless, despite growing evidence that altered brain maturation during the fetal, infant, and toddler (FIT) period modulates risk for poor mental health outcomes in childhood, these models are rarely implemented in FIT samples. Applications of predictive modeling in children of these ages provide an opportunity to develop powerful tools for improved characterization of the neural mechanisms underlying development. To facilitate the broader use of predictive models in FIT neuroimaging, we present a brief primer and systematic review on the methods used in current predictive modeling FIT studies. Reflecting on current practices in more than 100 studies conducted over the past decade, we provide an overview of topics, modalities, and methods commonly used in the field and under-researched areas. We then outline ethical and future considerations for neuroimaging researchers interested in predicting health outcomes in early life, including researchers who may be relatively new to either advanced machine learning methods or using FIT data. Altogether, the last decade of FIT research in machine learning has provided a foundation for accelerating the prediction of early-life trajectories across the full spectrum of illness and health.

Foxa2 and Pet1 Direct and Indirect Synergy Drive Serotonergic Neuronal Differentiation.

Neuronal programming by forced expression of transcription factors (TFs) holds promise for clinical applications of regenerative medicine. However, the mechanisms by which TFs coordinate their activities on the genome and control distinct neuronal fates remain obscure. Using direct neuronal programming of embryonic stem cells, we dissected the contribution of a series of TFs to specific neuronal regulatory programs. We deconstructed the Ascl1-Lmx1b-Foxa2-Pet1 TF combination that has been shown to generate serotonergic neurons and found that stepwise addition of TFs to Ascl1 canalizes the neuronal fate into a diffuse monoaminergic fate. The addition of pioneer factor Foxa2 represses Phox2b to induce serotonergic fate, similar to in vivo regulatory networks. Foxa2 and Pet1 appear to act synergistically to upregulate serotonergic fate. Foxa2 and Pet1 co-bind to a small fraction of genomic regions but mostly bind to different regulatory sites. In contrast to the combinatorial binding activities of other programming TFs, Pet1 does not strictly follow the Foxa2 pioneer. These findings highlight the challenges in formulating generalizable rules for describing the behavior of TF combinations that program distinct neuronal subtypes.

A protocol for working with open-source neuroimaging datasets.

Large, publicly available neuroimaging datasets are becoming increasingly common, but their use presents challenges because of insufficient knowledge of the tool options for data processing and proper data organization. Here, we describe a protocol to lessen these barriers. We describe the steps for the search and download of the open-source dataset. We detail the steps for proper data management and practical guidelines for data analysis. Finally, we give instructions for data and result sharing on public repositories and preprint services. For complete details on the use and execution of this profile, please refer to Horien et al. (2021).

Functional Connectome-Based Predictive Modeling in Autism.

Autism is a heterogeneous neurodevelopmental condition, and functional magnetic resonance imaging-based studies have helped advance our understanding of its effects on brain network activity. We review how predictive modeling, using measures of functional connectivity and symptoms, has helped reveal key insights into this condition. We discuss how different prediction frameworks can further our understanding of the brain-based features that underlie complex autism symptomatology and consider how predictive models may be used in clinical settings. Throughout, we highlight aspects of study interpretation, such as data decay and sampling biases, that require consideration within the context of this condition. We close by suggesting exciting future directions for predictive modeling in autism.

Big data approaches to identifying sex differences in long-term memory.

Whether in neurotransmitters or large-scale circuits, sex differences have long been of interest in neuroscience. Spets and Slotnick conducted a meta-analysis of fMRI studies of long-term memory to identify sex differences in brain-behavior associations, demonstrating that sex differences are pervasive across many sub-types of long-term memory. Meta-analyses are a workhorse toward aggregating larger sample sizes to arrive at a more comprehensive understanding of such topics. However, more research is crucial to elucidate complex relationships in how fMRI signals translate to behavioral outcomes. We propose big data and open-science as a solution toward finding robust sex differences in brain-behavior associations.