Self-supervised multimodal learning for group inferences from MRI data: Discovering disorder-relevant brain regions and multimodal links.

In recent years, deep learning approaches have gained significant attention in predicting brain disorders using neuroimaging data. However, conventional methods often rely on single-modality data and supervised models, which provide only a limited perspective of the intricacies of the highly complex brain. Moreover, the scarcity of accurate diagnostic labels in clinical settings hinders the applicability of the supervised models. To address these limitations, we propose a novel self-supervised framework for extracting multiple representations from multimodal neuroimaging data to enhance group inferences and enable analysis without resorting to labeled data during pre-training. Our approach leverages Deep InfoMax (DIM), a self-supervised methodology renowned for its efficacy in learning representations by estimating mutual information without the need for explicit labels. While DIM has shown promise in predicting brain disorders from single-modality MRI data, its potential for multimodal data remains untapped. This work extends DIM to multimodal neuroimaging data, allowing us to identify disorder-relevant brain regions and explore multimodal links. We present compelling evidence of the efficacy of our multimodal DIM analysis in uncovering disorder-relevant brain regions, including the hippocampus, caudate, insula, - and multimodal links with the thalamus, precuneus, and subthalamus hypothalamus. Our self-supervised representations demonstrate promising capabilities in predicting the presence of brain disorders across a spectrum of Alzheimer's phenotypes. Comparative evaluations against state-of-the-art unsupervised methods based on autoencoders, canonical correlation analysis, and supervised models highlight the superiority of our proposed method in achieving improved classification performance, capturing joint information, and interpretability capabilities. The computational efficiency of the decoder-free strategy enhances its practical utility, as it saves compute resources without compromising performance. This work offers a significant step forward in addressing the challenge of understanding multimodal links in complex brain disorders, with potential applications in neuroimaging research and clinical diagnosis.

Chromatic fusion: Generative multimodal neuroimaging data fusion provides multi-informed insights into schizophrenia.

This work proposes a novel generative multimodal approach to jointly analyze multimodal data while linking the multimodal information to colors. We apply our proposed framework, which disentangles multimodal data into private and shared sets of features from pairs of structural (sMRI), functional (sFNC and ICA), and diffusion MRI data (FA maps). With our approach, we find that heterogeneity in schizophrenia is potentially a function of modality pairs. Results show (1) schizophrenia is highly multimodal and includes changes in specific networks, (2) non-linear relationships with schizophrenia are observed when interpolating among shared latent dimensions, and (3) we observe a decrease in the modularity of functional connectivity and decreased visual-sensorimotor connectivity for schizophrenia patients for the FA-sFNC and sMRI-sFNC modality pairs, respectively. Additionally, our results generally indicate decreased fractional corpus callosum anisotropy, and decreased spatial ICA map and voxel-based morphometry strength in the superior frontal lobe as found in the FA-sFNC, sMRI-FA, and sMRI-ICA modality pair clusters. In sum, we introduce a powerful new multimodal neuroimaging framework designed to provide a rich and intuitive understanding of the data which we hope challenges the reader to think differently about how modalities interact.

Chromatic fusion: generative multimodal neuroimaging data fusion provides multi-informed insights into schizophrenia.

This work proposes a novel generative multimodal approach to jointly analyze multimodal data while linking the multimodal information to colors. By linking colors to private and shared information from modalities, we introduce chromatic fusion, a framework that allows for intuitively interpreting multimodal data. We test our framework on structural, functional, and diffusion modality pairs. In this framework, we use a multimodal variational autoencoder to learn separate latent subspaces; a private space for each modality, and a shared space between both modalities. These subspaces are then used to cluster subjects, and colored based on their distance from the variational prior, to obtain meta-chromatic patterns (MCPs). Each subspace corresponds to a different color, red is the private space of the first modality, green is the shared space, and blue is the private space of the second modality. We further analyze the most schizophrenia-enriched MCPs for each modality pair and find that distinct schizophrenia subgroups are captured by schizophrenia-enriched MCPs for different modality pairs, emphasizing schizophrenia's heterogeneity. For the FA-sFNC, sMRI-ICA, and sMRI-ICA MCPs, we generally find decreased fractional corpus callosum anisotropy and decreased spatial ICA map and voxel-based morphometry strength in the superior frontal lobe for schizophrenia patients. To additionally highlight the importance of the shared space between modalities, we perform a robustness analysis of the latent dimensions in the shared space across folds. These robust latent dimensions are subsequently correlated with schizophrenia to reveal that for each modality pair, multiple shared latent dimensions strongly correlate with schizophrenia. In particular, for FA-sFNC and sMRI-sFNC shared latent dimensions, we respectively observe a reduction in the modularity of the functional connectivity and a decrease in visual-sensorimotor connectivity for schizophrenia patients. The reduction in modularity couples with increased fractional anisotropy in the left part of the cerebellum dorsally. The reduction in the visual-sensorimotor connectivity couples with a reduction in the voxel-based morphometry generally but increased dorsal cerebellum voxel-based morphometry. Since the modalities are trained jointly, we can also use the shared space to try and reconstruct one modality from the other. We show that cross-reconstruction is possible with our network and is generally much better than depending on the variational prior. In sum, we introduce a powerful new multimodal neuroimaging framework designed to provide a rich and intuitive understanding of the data that we hope challenges the reader to think differently about how modalities interact.

Mind the gap: functional network connectivity interpolation between schizophrenia patients and controls using a variational autoencoder.

Mental disorders such as schizophrenia have been challenging to characterize due in part to their heterogeneous presentation in individuals. Most studies have focused on identifying groups differences and have typically ignored the heterogeneous patterns within groups. Here we propose a novel approach based on a variational autoencoder (VAE) to interpolate static functional network connectivity (sFNC) across individuals, with group-specific patterns between schizophrenia patients and controls captured simultaneously. We then visualize the original sFNC in a 2D grid according to the samples in the VAE latent space. We observe a high correspondence between the generated and the original sFNC. The proposed framework facilitates data visualization and can potentially be applied to predict the stage that a subject falls within a disorder continuum as well as characterize individual heterogeneity within and between groups.

Neurodevelopmental Trajectories in Children With Internalizing, Externalizing and Emotion Dysregulation Symptoms.

Introduction: Childhood and adolescence are crucial periods for brain and behavioral development. However, it is not yet clear how and when deviations from typical brain development are related to broad domains of psychopathology. Methods: Using three waves of neuroimaging data within the population-based Generation R Study sample, spanning a total age range of 6-16 years, we applied normative modeling to establish typical development curves for (sub-)cortical volume in 37 brain regions, and cortical thickness in 32 brain regions. Z-scores representing deviations from typical development were extracted and related to internalizing, externalizing and dysregulation profile (DP) symptoms. Results: Normative modeling showed regional differences in developmental trajectories. Psychopathology symptoms were related to negative deviations from typical development for cortical volume in widespread regions of the cortex and subcortex, and to positive deviations from typical development for cortical thickness in the orbitofrontal, frontal pole, pericalcarine and posterior cingulate regions of the cortex. Discussion: Taken together, this study charts developmental curves across the cerebrum for (sub-)cortical volume and cortical thickness. Our findings show that psychopathology symptoms, are associated with widespread differences in brain development, in which those with DP symptoms are most heavily affected.

Fusing multimodal neuroimaging data with a variational autoencoder.

Neuroimaging studies often collect multimodal data. These modalities contain both shared and mutually exclusive information about the brain. This work aims to find a scalable and interpretable method to fuse the information of multiple neuroimaging modalities into a lower-dimensional latent space using a variational autoencoder (VAE). To assess whether the encoder-decoder pair retains meaningful information, this work evaluates the representations using a schizophrenia classification task. The linear classifier, trained on the representations obtained through dimensionality reduction, achieves an area under the curve of the receiver operating characteristic (ROC-AUC) of 0.8609. Thus, training on a multimodal dataset with functional brain networks and a structural magnetic resonance imaging (sMRI) scan, leads to dimensionality reduction that retains meaningful information. The proposed dimensionality reduction outperforms both early and late fusion principal component analysis on the classification task.Clinical relevance - This work examines the interplay between neuroimaging modalities and their relation to mental disorders. This allows for more complex and rigorous analysis of multimodal neuroimaging data throughout clinical settings.

Environment-Wide Association Study (En WAS) of Prenatal and Perinatal Factors Associated With Autistic Traits: A Population-Based Study.

A combination of genetic and environmental factors contributes to the origins of autism spectrum disorder (ASD). While a number of studies have described specific environmental factors associating with emerging ASD, studies that compare and contrast multiple environmental factors in the same study are lacking. Thus, the goal of this study was to perform a prospective, data-driven environmental-wide association study of pre- and perinatal factors associated with the later development of autistic symptoms in childhood. The participants included 3891 6-year-old children from a birth cohort with pre- and perinatal data. Autistic symptoms were measured using the Social Responsiveness Scale in all children. Prior to any analyses, the sample was randomly split into a discovery set (2920) and a test set (921). Multiple linear regression analyses were performed for each of 920 variables, correcting for six of the most common covariates in epidemiological studies. We found 111 different pre- and perinatal factors associated with autistic traits during childhood. In secondary analyses where we controlled for parental psychopathology, 23 variables in the domains of family and interpersonal relationships were associated with the development of autistic symptoms during childhood. In conclusion, a data-driven approach was used to identify a number of pre- and perinatal risk factors associating with higher childhood autistic symptoms. These factors include measures of parental psychopathology and family and interpersonal relationships. These measures could potentially be used for the early identification of those at increased risk to develop ASD. LAY SUMMARY: A combination of genetic and environmental factors contributes to the development of autism spectrum disorder (ASD). Each environmental factor may affect the risk of ASD. In a study on 6-year-old children, a number of pre- and perinatal risk factors were identified that are associated with autistic symptoms in childhood. These factors include measures of parental psychopathology and family and interpersonal relationships. These variables could potentially serve as markers to identify those at increased risk to develop ASD or autistic symptoms. Autism Res 2020, 13: 1582-1600. © 2020 International Society for Autism Research, Wiley Periodicals, Inc.