Self-supervised multimodal learning for group inferences from MRI data: Discovering disorder-relevant brain regions and multimodal links.

In recent years, deep learning approaches have gained significant attention in predicting brain disorders using neuroimaging data. However, conventional methods often rely on single-modality data and supervised models, which provide only a limited perspective of the intricacies of the highly complex brain. Moreover, the scarcity of accurate diagnostic labels in clinical settings hinders the applicability of the supervised models. To address these limitations, we propose a novel self-supervised framework for extracting multiple representations from multimodal neuroimaging data to enhance group inferences and enable analysis without resorting to labeled data during pre-training. Our approach leverages Deep InfoMax (DIM), a self-supervised methodology renowned for its efficacy in learning representations by estimating mutual information without the need for explicit labels. While DIM has shown promise in predicting brain disorders from single-modality MRI data, its potential for multimodal data remains untapped. This work extends DIM to multimodal neuroimaging data, allowing us to identify disorder-relevant brain regions and explore multimodal links. We present compelling evidence of the efficacy of our multimodal DIM analysis in uncovering disorder-relevant brain regions, including the hippocampus, caudate, insula, - and multimodal links with the thalamus, precuneus, and subthalamus hypothalamus. Our self-supervised representations demonstrate promising capabilities in predicting the presence of brain disorders across a spectrum of Alzheimer's phenotypes. Comparative evaluations against state-of-the-art unsupervised methods based on autoencoders, canonical correlation analysis, and supervised models highlight the superiority of our proposed method in achieving improved classification performance, capturing joint information, and interpretability capabilities. The computational efficiency of the decoder-free strategy enhances its practical utility, as it saves compute resources without compromising performance. This work offers a significant step forward in addressing the challenge of understanding multimodal links in complex brain disorders, with potential applications in neuroimaging research and clinical diagnosis.

Chromatic fusion: Generative multimodal neuroimaging data fusion provides multi-informed insights into schizophrenia.

This work proposes a novel generative multimodal approach to jointly analyze multimodal data while linking the multimodal information to colors. We apply our proposed framework, which disentangles multimodal data into private and shared sets of features from pairs of structural (sMRI), functional (sFNC and ICA), and diffusion MRI data (FA maps). With our approach, we find that heterogeneity in schizophrenia is potentially a function of modality pairs. Results show (1) schizophrenia is highly multimodal and includes changes in specific networks, (2) non-linear relationships with schizophrenia are observed when interpolating among shared latent dimensions, and (3) we observe a decrease in the modularity of functional connectivity and decreased visual-sensorimotor connectivity for schizophrenia patients for the FA-sFNC and sMRI-sFNC modality pairs, respectively. Additionally, our results generally indicate decreased fractional corpus callosum anisotropy, and decreased spatial ICA map and voxel-based morphometry strength in the superior frontal lobe as found in the FA-sFNC, sMRI-FA, and sMRI-ICA modality pair clusters. In sum, we introduce a powerful new multimodal neuroimaging framework designed to provide a rich and intuitive understanding of the data which we hope challenges the reader to think differently about how modalities interact.

Revisiting Functional Dysconnectivity: a Review of Three Model Frameworks in Schizophrenia.

PURPOSE OF REVIEW: Over the last decade, evidence suggests that a combination of behavioral and neuroimaging findings can help illuminate changes in functional dysconnectivity in schizophrenia. We review the recent connectivity literature considering several vital models, considering connectivity findings, and relationships with clinical symptoms. We reviewed resting state fMRI studies from 2017 to 2023. We summarized the role of two sets of brain networks (cerebello-thalamo-cortical (CTCC) and the triple network set) across three hypothesized models of schizophrenia etiology (neurodevelopmental, vulnerability-stress, and neurotransmitter hypotheses). RECENT FINDINGS: The neurotransmitter and neurodevelopmental models best explained CTCC-subcortical dysfunction, which was consistently connected to symptom severity and motor symptoms. Triple network dysconnectivity was linked to deficits in executive functioning, and the salience network (SN)-default mode network dysconnectivity was tied to disordered thought and attentional deficits. This paper links behavioral symptoms of schizophrenia (symptom severity, motor, executive functioning, and attentional deficits) to various hypothesized mechanisms.

Statelets: Capturing recurrent transient variations in dynamic functional network connectivity.

Dynamic functional network connectivity (dFNC) analysis is a widely used approach for capturing brain activation patterns, connectivity states, and network organization. However, a typical sliding window plus clustering (SWC) approach for analyzing dFNC models the system through a fixed sequence of connectivity states. SWC assumes connectivity patterns span throughout the brain, but they are relatively spatially constrained and temporally short-lived in practice. Thus, SWC is neither designed to capture transient dynamic changes nor heterogeneity across subjects/time. We propose a state-space time series summarization framework called "statelets" to address these shortcomings. It models functional connectivity dynamics at fine-grained timescales, adapting time series motifs to changes in connectivity strength, and constructs a concise yet informative representation of the original data that conveys easily comprehensible information about the phenotypes. We leverage the earth mover distance in a nonstandard way to handle scale differences and utilize kernel density estimation to build a probability density profile for local motifs. We apply the framework to study dFNC of patients with schizophrenia (SZ) and healthy control (HC). Results demonstrate SZ subjects exhibit reduced modularity in their brain network organization relative to HC. Statelets in the HC group show an increased recurrence across the dFNC time-course compared to the SZ. Analyzing the consistency of the connections across time reveals significant differences within visual, sensorimotor, and default mode regions where HC subjects show higher consistency than SZ. The introduced approach also enables handling dynamic information in cross-modal and multimodal applications to study healthy and disordered brains.

Privacy-preserving quality control of neuroimaging datasets in federated environments.

Privacy concerns for rare disease data, institutional or IRB policies, access to local computational or storage resources or download capabilities are among the reasons that may preclude analyses that pool data to a single site. A growing number of multisite projects and consortia were formed to function in the federated environment to conduct productive research under constraints of this kind. In this scenario, a quality control tool that visualizes decentralized data in its entirety via global aggregation of local computations is especially important, as it would allow the screening of samples that cannot be jointly evaluated otherwise. To solve this issue, we present two algorithms: decentralized data stochastic neighbor embedding, dSNE, and its differentially private counterpart, DP-dSNE. We leverage publicly available datasets to simultaneously map data samples located at different sites according to their similarities. Even though the data never leaves the individual sites, dSNE does not provide any formal privacy guarantees. To overcome that, we rely on differential privacy: a formal mathematical guarantee that protects individuals from being identified as contributors to a dataset. We implement DP-dSNE with AdaCliP, a method recently proposed to add less noise to the gradients per iteration. We introduce metrics for measuring the embedding quality and validate our algorithms on these metrics against their centralized counterpart on two toy datasets. Our validation on six multisite neuroimaging datasets shows promising results for the quality control tasks of visualization and outlier detection, highlighting the potential of our private, decentralized visualization approach.

Three-way parallel group independent component analysis: Fusion of spatial and spatiotemporal magnetic resonance imaging data.

Advances in imaging acquisition techniques allow multiple imaging modalities to be collected from the same subject. Each individual modality offers limited yet unique views of the functional, structural, or dynamic temporal features of the brain. Multimodal fusion provides effective ways to leverage these complementary perspectives from multiple modalities. However, the majority of current multimodal fusion approaches involving functional magnetic resonance imaging (fMRI) are limited to 3D feature summaries that do not incorporate its rich temporal information. Thus, we propose a novel three-way parallel group independent component analysis (pGICA) fusion method that incorporates the first-level 4D fMRI data (temporal information included) by parallelizing group ICA into parallel ICA via a unified optimization framework. A new variability matrix was defined to capture subject-wise functional variability and then link it to the mixing matrices of the other two modalities. Simulation results show that the three-way pGICA provides highly accurate cross-modality linkage estimation under both weakly and strongly correlated conditions, as well as comparable source estimation under different noise levels. Results using real brain imaging data identified one linked functional-structural-diffusion component associated to differences between schizophrenia and controls. This was replicated in an independent cohort, and the identified components were also correlated with major cognitive domains. Functional network connectivity revealed visual-subcortical and default mode-cerebellum pairs that discriminate between schizophrenia and controls. Overall, both simulation and real data results support the use of three-way pGICA to identify multimodal spatiotemporal links and to pursue the study of brain disorders under a single unifying multimodal framework.

A Correlated Noise-assisted Decentralized Differentially Private Estimation Protocol, and its application to fMRI Source Separation.

Blind source separation algorithms such as independent component analysis (ICA) are widely used in the analysis of neuroimaging data. To leverage larger sample sizes, different data holders/sites may wish to collaboratively learn feature representations. However, such datasets are often privacy-sensitive, precluding centralized analyses that pool the data at one site. In this work, we propose a differentially private algorithm for performing ICA in a decentralized data setting. Due to the high dimension and small sample size, conventional approaches to decentralized differentially private algorithms suffer in terms of utility. When centralizing the data is not possible, we investigate the benefit of enabling limited collaboration in the form of generating jointly distributed random noise. We show that such (anti) correlated noise improves the privacy-utility trade-off, and can reach the same level of utility as the corresponding non-private algorithm for certain parameter choices. We validate this benefit using synthetic and real neuroimaging datasets. We conclude that it is possible to achieve meaningful utility while preserving privacy, even in complex signal processing systems.

Decentralized dynamic functional network connectivity: State analysis in collaborative settings.

As neuroimaging data increase in complexity and related analytical problems follow suite, more researchers are drawn to collaborative frameworks that leverage data sets from multiple data-collection sites to balance out the complexity with an increased sample size. Although centralized data-collection approaches have dominated the collaborative scene, a number of decentralized approaches-those that avoid gathering data at a shared central store-have grown in popularity. We expect the prevalence of decentralized approaches to continue as privacy risks and communication overhead become increasingly important for researchers. In this article, we develop, implement and evaluate a decentralized version of one such widely used tool: dynamic functional network connectivity. Our resulting algorithm, decentralized dynamic functional network connectivity (ddFNC), synthesizes a new, decentralized group independent component analysis algorithm (dgICA) with algorithms for decentralized k-means clustering. We compare both individual decentralized components and the full resulting decentralized analysis pipeline against centralized counterparts on the same data, and show that both provide comparable performance. Additionally, we perform several experiments which evaluate the communication overhead and convergence behavior of various decentralization strategies and decentralized clustering algorithms. Our analysis indicates that ddFNC is a fine candidate for facilitating decentralized collaboration between neuroimaging researchers, and stands ready for the inclusion of privacy-enabling modifications, such as differential privacy.

Decentralized temporal independent component analysis: Leveraging fMRI data in collaborative settings.

The field of neuroimaging has recently witnessed a strong shift towards data sharing; however, current collaborative research projects may be unable to leverage institutional architectures that collect and store data in local, centralized data centers. Additionally, though research groups are willing to grant access for collaborations, they often wish to maintain control of their data locally. These concerns may stem from research culture as well as privacy and accountability concerns. In order to leverage the potential of these aggregated larger data sets, we require tools that perform joint analyses without transmitting the data. Ideally, these tools would have similar performance and ease of use as their current centralized counterparts. In this paper, we propose and evaluate a new Algorithm, decentralized joint independent component analysis (djICA), which meets these technical requirements. djICA shares only intermediate statistics about the data, plausibly retaining privacy of the raw information to local sites, thus making it amenable to further privacy protections, for example via differential privacy. We validate our method on real functional magnetic resonance imaging (fMRI) data and show that it enables collaborative large-scale temporal ICA of fMRI, a rich vein of analysis as of yet largely unexplored, and which can benefit from the larger-N studies enabled by a decentralized approach. We show that djICA is robust to different distributions of data over sites, and that the temporal components estimated with djICA show activations similar to the temporal functional modes analyzed in previous work, thus solidifying djICA as a new, decentralized method oriented toward the frontiers of temporal independent component analysis.

Reading the (functional) writing on the (structural) wall: Multimodal fusion of brain structure and function via a deep neural network based translation approach reveals novel impairments in schizophrenia.

This work presents a novel approach to finding linkage/association between multimodal brain imaging data, such as structural MRI (sMRI) and functional MRI (fMRI). Motivated by the machine translation domain, we employ a deep learning model, and consider two different imaging views of the same brain like two different languages conveying some common facts. That analogy enables finding linkages between two modalities. The proposed translation-based fusion model contains a computing layer that learns "alignments" (or links) between dynamic connectivity features from fMRI data and static gray matter patterns from sMRI data. The approach is evaluated on a multi-site dataset consisting of eyes-closed resting state imaging data collected from 298 subjects (age- and gender matched 154 healthy controls and 144 patients with schizophrenia). Results are further confirmed on an independent dataset consisting of eyes-open resting state imaging data from 189 subjects (age- and gender matched 91 healthy controls and 98 patients with schizophrenia). We used dynamic functional connectivity (dFNC) states as the functional features and ICA-based sources from gray matter densities as the structural features. The dFNC states characterized by weakly correlated intrinsic connectivity networks (ICNs) were found to have stronger association with putamen and insular gray matter pattern, while the dFNC states of profuse strongly correlated ICNs exhibited stronger links with the gray matter pattern in precuneus, posterior cingulate cortex (PCC), and temporal cortex. Further investigation with the estimated link strength (or alignment score) showed significant group differences between healthy controls and patients with schizophrenia in several key regions including temporal lobe, and linked these to connectivity states showing less occupancy in healthy controls. Moreover, this novel approach revealed significant correlation between a cognitive score (attention/vigilance) and the function/structure alignment score that was not detected when data modalities were considered separately.

Task-specific feature extraction and classification of fMRI volumes using a deep neural network initialized with a deep belief network: Evaluation using sensorimotor tasks.

Feedforward deep neural networks (DNNs), artificial neural networks with multiple hidden layers, have recently demonstrated a record-breaking performance in multiple areas of applications in computer vision and speech processing. Following the success, DNNs have been applied to neuroimaging modalities including functional/structural magnetic resonance imaging (MRI) and positron-emission tomography data. However, no study has explicitly applied DNNs to 3D whole-brain fMRI volumes and thereby extracted hidden volumetric representations of fMRI that are discriminative for a task performed as the fMRI volume was acquired. Our study applied fully connected feedforward DNN to fMRI volumes collected in four sensorimotor tasks (i.e., left-hand clenching, right-hand clenching, auditory attention, and visual stimulus) undertaken by 12 healthy participants. Using a leave-one-subject-out cross-validation scheme, a restricted Boltzmann machine-based deep belief network was pretrained and used to initialize weights of the DNN. The pretrained DNN was fine-tuned while systematically controlling weight-sparsity levels across hidden layers. Optimal weight-sparsity levels were determined from a minimum validation error rate of fMRI volume classification. Minimum error rates (mean±standard deviation; %) of 6.9 (±3.8) were obtained from the three-layer DNN with the sparsest condition of weights across the three hidden layers. These error rates were even lower than the error rates from the single-layer network (9.4±4.6) and the two-layer network (7.4±4.1). The estimated DNN weights showed spatial patterns that are remarkably task-specific, particularly in the higher layers. The output values of the third hidden layer represented distinct patterns/codes of the 3D whole-brain fMRI volume and encoded the information of the tasks as evaluated from representational similarity analysis. Our reported findings show the ability of the DNN to classify a single fMRI volume based on the extraction of hidden representations of fMRI volumes associated with tasks across multiple hidden layers. Our study may be beneficial to the automatic classification/diagnosis of neuropsychiatric and neurological diseases and prediction of disease severity and recovery in (pre-) clinical settings using fMRI volumes without requiring an estimation of activation patterns or ad hoc statistical evaluation.

Tracking whole-brain connectivity dynamics in the resting state.

Spontaneous fluctuations are a hallmark of recordings of neural signals, emergent over time scales spanning milliseconds and tens of minutes. However, investigations of intrinsic brain organization based on resting-state functional magnetic resonance imaging have largely not taken into account the presence and potential of temporal variability, as most current approaches to examine functional connectivity (FC) implicitly assume that relationships are constant throughout the length of the recording. In this work, we describe an approach to assess whole-brain FC dynamics based on spatial independent component analysis, sliding time window correlation, and k-means clustering of windowed correlation matrices. The method is applied to resting-state data from a large sample (n = 405) of young adults. Our analysis of FC variability highlights particularly flexible connections between regions in lateral parietal and cingulate cortex, and argues against a labeling scheme where such regions are treated as separate and antagonistic entities. Additionally, clustering analysis reveals unanticipated FC states that in part diverge strongly from stationary connectivity patterns and challenge current descriptions of interactions between large-scale networks. Temporal trends in the occurrence of different FC states motivate theories regarding their functional roles and relationships with vigilance/arousal. Overall, we suggest that the study of time-varying aspects of FC can unveil flexibility in the functional coordination between different neural systems, and that the exploitation of these dynamics in further investigations may improve our understanding of behavioral shifts and adaptive processes.

A statistically motivated framework for simulation of stochastic data fusion models applied to multimodal neuroimaging.

Multimodal fusion is becoming more common as it proves to be a powerful approach to identify complementary information from multimodal datasets. However, simulation of joint information is not straightforward. Published approaches mostly employ limited, provisional designs that often break the link between the model assumptions and the data for the sake of demonstrating properties of fusion techniques. This work introduces a new approach to synthetic data generation which allows full-compliance between data and model while still representing realistic spatiotemporal features in accordance with the current neuroimaging literature. The focus is on the simulation of joint information for the verification of stochastic linear models, particularly those used in multimodal data fusion of brain imaging data. Our first goal is to obtain a benchmark ground-truth in which estimation errors due to model mismatch are minimal or none. Then we move on to assess how estimation is affected by gradually increasing model discrepancies toward a more realistic dataset. The key aspect of our approach is that it permits complete control over the type and level of model mismatch, allowing for more educated inferences about the limitations and caveats of select stochastic linear models. As a result, impartial comparison of models is possible based on their performance in multiple different scenarios. Our proposed method uses the commonly overlooked theory of copulas to enable full control of the type and level of dependence/association between modalities, with no occurrence of spurious multimodal associations. Moreover, our approach allows for arbitrary single-modality marginal distributions for any fixed choice of dependence/association between multimodal features. Using our simulation framework, we can rigorously challenge the assumptions of several existing multimodal fusion approaches. Our study brings a new perspective to the problem of simulating multimodal data that can be used for ground-truth verification of various stochastic multimodal models available in the literature, and reveals some important aspects that are not captured or are overlooked by ad hoc simulations that lack a firm statistical motivation.

Functional and effective connectivity of stopping.

Behavioral inhibition often is studied by comparing the electroencephalographic responses to stop and to go signals. Most studies simply assess amplitude differences of the N200 and P300 event-related potentials, which seem to best correspond to increased activity in the theta and delta frequency bands, respectively. However, neither have reliable indicators for successful behavioral inhibition been identified nor have the causal dependencies of stop-related neurocognitive processes been addressed yet. By studying functional and effective connectivity underlying stopping behavior, this study opens new directions for the investigation of behavioral inhibition. Group independent component analysis was used to infer functionally coherent networks from electroencephalographic data, which were recorded from healthy human participants during processing of a stop signal task. Then, the temporal dynamics of causal dependencies between independent components were identified by means of Bayesian network estimations. The mean clustering coefficient and the characteristic path length measure indicated time windows between 130 and 180 ms and between 420 and 500 ms to express significantly different connectivity profiles between conditions. Three components showed significant correlations between 120 and 260 ms with stop signal reaction times and the number of failed stops. Two of these components acted as sources of causal flow, one capturing P300/delta characteristics while the other was characterized by alpha power depletion putatively representing the evaluation or processing of stimulus features. Although results suggest that the P300 and associated delta activity seem to be statistically dependent on earlier processes associated with behavioral inhibition, the time window critical for inhibition coincides with early changes in causal patterns and largely precedes peak amplitude differences between go and stop trials. Altogether, utilizing the analysis of stopping-related connectivity, previously undetected patterns emerged that warrant further investigation.

Restricted Boltzmann machines for neuroimaging: an application in identifying intrinsic networks.

Matrix factorization models are the current dominant approach for resolving meaningful data-driven features in neuroimaging data. Among them, independent component analysis (ICA) is arguably the most widely used for identifying functional networks, and its success has led to a number of versatile extensions to group and multimodal data. However there are indications that ICA may have reached a limit in flexibility and representational capacity, as the majority of such extensions are case-driven, custom-made solutions that are still contained within the class of mixture models. In this work, we seek out a principled and naturally extensible approach and consider a probabilistic model known as a restricted Boltzmann machine (RBM). An RBM separates linear factors from functional brain imaging data by fitting a probability distribution model to the data. Importantly, the solution can be used as a building block for more complex (deep) models, making it naturally suitable for hierarchical and multimodal extensions that are not easily captured when using linear factorizations alone. We investigate the capability of RBMs to identify intrinsic networks and compare its performance to that of well-known linear mixture models, in particular ICA. Using synthetic and real task fMRI data, we show that RBMs can be used to identify networks and their temporal activations with accuracy that is equal or greater than that of factorization models. The demonstrated effectiveness of RBMs supports its use as a building block for deeper models, a significant prospect for future neuroimaging research.

High-order interactions observed in multi-task intrinsic networks are dominant indicators of aberrant brain function in schizophrenia.

Identifying the complex activity relationships present in rich, modern neuroimaging data sets remains a key challenge for neuroscience. The problem is hard because (a) the underlying spatial and temporal networks may be nonlinear and multivariate and (b) the observed data may be driven by numerous latent factors. Further, modern experiments often produce data sets containing multiple stimulus contexts or tasks processed by the same subjects. Fusing such multi-session data sets may reveal additional structure, but raises further statistical challenges. We present a novel analysis method for extracting complex activity networks from such multifaceted imaging data sets. Compared to previous methods, we choose a new point in the trade-off space, sacrificing detailed generative probability models and explicit latent variable inference in order to achieve robust estimation of multivariate, nonlinear group factors ("network clusters"). We apply our method to identify relationships of task-specific intrinsic networks in schizophrenia patients and control subjects from a large fMRI study. After identifying network-clusters characterized by within- and between-task interactions, we find significant differences between patient and control groups in interaction strength among networks. Our results are consistent with known findings of brain regions exhibiting deviations in schizophrenic patients. However, we also find high-order, nonlinear interactions that discriminate groups but that are not detected by linear, pairwise methods. We additionally identify high-order relationships that provide new insights into schizophrenia but that have not been found by traditional univariate or second-order methods. Overall, our approach can identify key relationships that are missed by existing analysis methods, without losing the ability to find relationships that are known to be important.

A deep learning approach for mental health quality prediction using functional network connectivity and assessment data.

While one can characterize mental health using questionnaires, such tools do not provide direct insight into the underlying biology. By linking approaches that visualize brain activity to questionnaires in the context of individualized prediction, we can gain new insights into the biology and behavioral aspects of brain health. Resting-state fMRI (rs-fMRI) can be used to identify biomarkers of these conditions and study patterns of abnormal connectivity. In this work, we estimate mental health quality for individual participants using static functional network connectivity (sFNC) data from rs-fMRI. The deep learning model uses the sFNC data as input to predict four categories of mental health quality and visualize the neural patterns indicative of each group. We used guided gradient class activation maps (guided Grad-CAM) to identify the most discriminative sFNC patterns. The effectiveness of this model was validated using the UK Biobank dataset, in which we showed that our approach outperformed four alternative models by 4-18% accuracy. The proposed model's performance evaluation yielded a classification accuracy of 76%, 78%, 88%, and 98% for the excellent, good, fair, and poor mental health categories, with poor mental health accuracy being the highest. The findings show distinct sFNC patterns across each group. The patterns associated with excellent mental health consist of the cerebellar-subcortical regions, whereas the most prominent areas in the poor mental health category are in the sensorimotor and visual domains. Thus the combination of rs-fMRI and deep learning opens a promising path for developing a comprehensive framework to evaluate and measure mental health. Moreover, this approach had the potential to guide the development of personalized interventions and enable the monitoring of treatment response. Overall this highlights the crucial role of advanced imaging modalities and deep learning algorithms in advancing our understanding and management of mental health.

Chromatic fusion: generative multimodal neuroimaging data fusion provides multi-informed insights into schizophrenia.

This work proposes a novel generative multimodal approach to jointly analyze multimodal data while linking the multimodal information to colors. By linking colors to private and shared information from modalities, we introduce chromatic fusion, a framework that allows for intuitively interpreting multimodal data. We test our framework on structural, functional, and diffusion modality pairs. In this framework, we use a multimodal variational autoencoder to learn separate latent subspaces; a private space for each modality, and a shared space between both modalities. These subspaces are then used to cluster subjects, and colored based on their distance from the variational prior, to obtain meta-chromatic patterns (MCPs). Each subspace corresponds to a different color, red is the private space of the first modality, green is the shared space, and blue is the private space of the second modality. We further analyze the most schizophrenia-enriched MCPs for each modality pair and find that distinct schizophrenia subgroups are captured by schizophrenia-enriched MCPs for different modality pairs, emphasizing schizophrenia's heterogeneity. For the FA-sFNC, sMRI-ICA, and sMRI-ICA MCPs, we generally find decreased fractional corpus callosum anisotropy and decreased spatial ICA map and voxel-based morphometry strength in the superior frontal lobe for schizophrenia patients. To additionally highlight the importance of the shared space between modalities, we perform a robustness analysis of the latent dimensions in the shared space across folds. These robust latent dimensions are subsequently correlated with schizophrenia to reveal that for each modality pair, multiple shared latent dimensions strongly correlate with schizophrenia. In particular, for FA-sFNC and sMRI-sFNC shared latent dimensions, we respectively observe a reduction in the modularity of the functional connectivity and a decrease in visual-sensorimotor connectivity for schizophrenia patients. The reduction in modularity couples with increased fractional anisotropy in the left part of the cerebellum dorsally. The reduction in the visual-sensorimotor connectivity couples with a reduction in the voxel-based morphometry generally but increased dorsal cerebellum voxel-based morphometry. Since the modalities are trained jointly, we can also use the shared space to try and reconstruct one modality from the other. We show that cross-reconstruction is possible with our network and is generally much better than depending on the variational prior. In sum, we introduce a powerful new multimodal neuroimaging framework designed to provide a rich and intuitive understanding of the data that we hope challenges the reader to think differently about how modalities interact.

NeuroCrypt: Machine Learning Over Encrypted Distributed Neuroimaging Data.

The field of neuroimaging can greatly benefit from building machine learning models to detect and predict diseases, and discover novel biomarkers, but much of the data collected at various organizations and research centers is unable to be shared due to privacy or regulatory concerns (especially for clinical data or rare disorders). In addition, aggregating data across multiple large studies results in a huge amount of duplicated technical debt and the resources required can be challenging or impossible for an individual site to build. Training on the data distributed across organizations can result in models that generalize much better than models trained on data from any of organizations alone. While there are approaches for decentralized sharing, these often do not provide the highest possible guarantees of sample privacy that only cryptography can provide. In addition, such approaches are often focused on probabilistic solutions. In this paper, we propose an approach that leverages the potential of datasets spread among a number of data collecting organizations by performing joint analyses in a secure and deterministic manner when only encrypted data is shared and manipulated. The approach is based on secure multiparty computation which refers to cryptographic protocols that enable distributed computation of a function over distributed inputs without revealing additional information about the inputs. It enables multiple organizations to train machine learning models on their joint data and apply the trained models to encrypted data without revealing their sensitive data to the other parties. In our proposed approach, organizations (or sites) securely collaborate to build a machine learning model as it would have been trained on the aggregated data of all the organizations combined. Importantly, the approach does not require a trusted party (i.e. aggregator), each contributing site plays an equal role in the process, and no site can learn individual data of any other site. We demonstrate effectiveness of the proposed approach, in a range of empirical evaluations using different machine learning algorithms including logistic regression and convolutional neural network models on human structural and functional magnetic resonance imaging datasets.

Interpreting models interpreting brain dynamics.

Brain dynamics are highly complex and yet hold the key to understanding brain function and dysfunction. The dynamics captured by resting-state functional magnetic resonance imaging data are noisy, high-dimensional, and not readily interpretable. The typical approach of reducing this data to low-dimensional features and focusing on the most predictive features comes with strong assumptions and can miss essential aspects of the underlying dynamics. In contrast, introspection of discriminatively trained deep learning models may uncover disorder-relevant elements of the signal at the level of individual time points and spatial locations. Yet, the difficulty of reliable training on high-dimensional low sample size datasets and the unclear relevance of the resulting predictive markers prevent the widespread use of deep learning in functional neuroimaging. In this work, we introduce a deep learning framework to learn from high-dimensional dynamical data while maintaining stable, ecologically valid interpretations. Results successfully demonstrate that the proposed framework enables learning the dynamics of resting-state fMRI directly from small data and capturing compact, stable interpretations of features predictive of function and dysfunction.