Cross-validation and permutations in MVPA: Validity of permutation strategies and power of cross-validation schemes.

Multi-Voxel Pattern Analysis (MVPA) is a well established tool to disclose weak, distributed effects in brain activity patterns. The generalization ability is assessed by testing the learning model on new, unseen data. However, when limited data is available, the decoding success is estimated using cross-validation. There is general consensus on assessing statistical significance of cross-validated accuracy with non-parametric permutation tests. In this work we focus on the false positive control of different permutation strategies and on the statistical power of different cross-validation schemes. With simulations, we show that estimating the entire cross-validation error on each permuted dataset is the only statistically valid permutation strategy. Furthermore, using both simulations and real data from the HCP WU-Minn 3T fMRI dataset, we show that, among the different cross-validation schemes, a repeated split-half cross-validation is the most powerful, despite achieving slightly lower classification accuracy, when compared to other schemes. Our findings provide additional insights into the optimization of the experimental design for MVPA, highlighting the benefits of having many short runs.

A Comparison of Independent Component Analysis (ICA) of fMRI and Electrical Source Imaging (ESI) in Focal Epilepsy Reveals Misclassification Using a Classifier.

Interictal epileptiform discharges (IEDs) can produce haemodynamic responses that can be detected by electroencephalography-functional magnetic resonance imaging (EEG-fMRI) using different analysis methods such as the general linear model (GLM) of IEDs or independent component analysis (ICA). The IEDs can also be mapped by electrical source imaging (ESI) which has been demonstrated to be useful in presurgical evaluation in a high proportion of cases with focal IEDs. ICA advantageously does not require IEDs or a model of haemodynamic responses but its use in EEG-fMRI of epilepsy has been limited by its ability to separate and select epileptic components. Here, we evaluated the performance of a classifier that aims to filter all non-BOLD responses and we compared the spatial and temporal features of the selected independent components (ICs). The components selected by the classifier were compared to those components selected by a strong spatial correlation with ESI maps of IED sources. Both sets of ICs were subsequently compared to a temporal model derived from the convolution of the IEDs (derived from the simultaneously acquired EEG) with a standard haemodynamic response. Selected ICs were compared to the patients' clinical information in 13 patients with focal epilepsy. We found that the misclassified ICs clearly related to IED in 16/25 cases. We also found that the classifier failed predominantly due to the increased spectral range of fMRIs temporal responses to IEDs. In conclusion, we show that ICA can be an efficient approach to separate responses related to epilepsy but that contemporary classifiers need to be retrained for epilepsy data. Our findings indicate that, for ICA to contribute to the analysis of data without IEDs to improve its sensitivity, classification strategies based on data features other than IC time course frequency is required.

Multivariate linear regression of high-dimensional fMRI data with multiple target variables.

Multivariate regression is increasingly used to study the relation between fMRI spatial activation patterns and experimental stimuli or behavioral ratings. With linear models, informative brain locations are identified by mapping the model coefficients. This is a central aspect in neuroimaging, as it provides the sought-after link between the activity of neuronal populations and subject's perception, cognition or behavior. Here, we show that mapping of informative brain locations using multivariate linear regression (MLR) may lead to incorrect conclusions and interpretations. MLR algorithms for high dimensional data are designed to deal with targets (stimuli or behavioral ratings, in fMRI) separately, and the predictive map of a model integrates information deriving from both neural activity patterns and experimental design. Not accounting explicitly for the presence of other targets whose associated activity spatially overlaps with the one of interest may lead to predictive maps of troublesome interpretation. We propose a new model that can correctly identify the spatial patterns associated with a target while achieving good generalization. For each target, the training is based on an augmented dataset, which includes all remaining targets. The estimation on such datasets produces both maps and interaction coefficients, which are then used to generalize. The proposed formulation is independent of the regression algorithm employed. We validate this model on simulated fMRI data and on a publicly available dataset. Results indicate that our method achieves high spatial sensitivity and good generalization and that it helps disentangle specific neural effects from interaction with predictive maps associated with other targets.