Categorical working memory codes in human visual cortex.

Working memory contents are represented in neural activity patterns across multiple regions of the cortical hierarchy. A division of labor has been proposed where more anterior regions harbor increasingly abstract and categorical representations while the most detailed representations are held in primary sensory cortices. Here, using fMRI and multivariate encoding modeling, we demonstrate that for color stimuli categorical codes are already present at the level of extrastriate visual cortex (V4 and VO1), even when subjects are neither implicitly nor explicitly encouraged to categorize the stimuli. Importantly, this categorical coding was observed during working memory, but not during perception. Thus, visual working memory is likely to rely at least in part on categorical representations. SIGNIFICANCE STATEMENT: Working memory is the representational basis for human cognition. Recent work has demonstrated that numerous regions across the human brain can represent the contents of working memory. We use fMRI brain scanning and machine learning methods to demonstrate that different regions can represent the same content differently during working memory. Reading out the neural codes used to store working memory contents, we show that already in sensory cortex, areas V4 and VO1 represent color in a categorical format rather than a purely sensory fashion. Thereby, we provide a better understanding of how different regions of the brain might serve working memory and cognition.

Instruction effects on randomness in sequence generation.

Randomness is a fundamental property of human behavior. It occurs both in the form of intrinsic random variability, say when repetitions of a task yield slightly different behavioral outcomes, or in the form of explicit randomness, say when a person tries to avoid being predicted in a game of rock, paper and scissors. Randomness has frequently been studied using random sequence generation tasks (RSG). A key finding has been that humans are poor at deliberately producing random behavior. At the same time, it has been shown that people might be better randomizers if randomness is only an implicit (rather than an explicit) requirement of the task. We therefore hypothesized that randomization performance might vary with the exact instructions with which randomness is elicited. To test this, we acquired data from a large online sample (n = 388), where every participant made 1,000 binary choices based on one of the following instructions: choose either randomly, freely, irregularly, according to an imaginary coin toss or perform a perceptual guessing task. Our results show significant differences in randomness between the conditions as quantified by conditional entropy and estimated Markov order. The randomization scores were highest in the conditions where people were asked to be irregular or mentally simulate a random event (coin toss) thus yielding recommendations for future studies on randomization behavior.

Decoding verbal working memory representations of Chinese characters from Broca's area.

Representations of sensory working memory can be found across the entire neocortex. But how are verbal working memory (VWM) contents retained in the human brain? Here we used fMRI and multi-voxel pattern analyses to study Chinese native speakers (15 males, 13 females) memorizing Chinese characters. Chinese characters are uniquely suitable to study VWM because verbal encoding is encouraged by their complex visual appearance and monosyllabic pronunciation. We found that activity patterns in Broca's area and left premotor cortex carried information about the memorized characters. These language-related areas carried (1) significantly more information about cued characters than those not cued for memorization, (2) significantly more information on the left than the right hemisphere and (3) significantly more information about Chinese symbols than complex visual patterns which are hard to verbalize. In contrast, early visual cortex carries a comparable amount of information about cued and uncued stimuli and is thus unlikely to be involved in memory retention. This study provides evidence for verbal working memory maintenance in a distributed network of language-related brain regions, consistent with distributed accounts of WM. The results also suggest that Broca's area and left premotor cortex form the articulatory network which serves articulatory rehearsal in the retention of verbal working memory contents.

Inverse transformed encoding models - a solution to the problem of correlated trial-by-trial parameter estimates in fMRI decoding.

Techniques of multivariate pattern analysis (MVPA) can be used to decode the discrete experimental condition or a continuous modulator variable from measured brain activity during a particular trial. In functional magnetic resonance imaging (fMRI), trial-wise response amplitudes are sometimes estimated from the measured signal using a general linear model (GLM) with one onset regressor for each trial. When using rapid event-related designs with trials closely spaced in time, those estimates are highly variable and serially correlated due to the temporally extended shape of the hemodynamic response function (HRF). Here, we describe inverse transformed encoding modelling (ITEM), a principled approach of accounting for those serial correlations and decoding from the resulting estimates, at low computational cost and with no loss in statistical power. We use simulated data to show that ITEM outperforms the current standard approach in terms of decoding accuracy and analyze empirical data to demonstrate that ITEM is capable of visual reconstruction from fMRI signals.

MACS - a new SPM toolbox for model assessment, comparison and selection.

BACKGROUND: In cognitive neuroscience, functional magnetic resonance imaging (fMRI) data are widely analyzed using general linear models (GLMs). However, model quality of GLMs for fMRI is rarely assessed, in part due to the lack of formal measures for statistical model inference. NEW METHOD: We introduce a new SPM toolbox for model assessment, comparison and selection (MACS) of GLMs applied to fMRI data. MACS includes classical, information-theoretic and Bayesian methods of model assessment previously applied to GLMs for fMRI as well as recent methodological developments of model selection and model averaging in fMRI data analysis. RESULTS: The toolbox - which is freely available from GitHub - directly builds on the Statistical Parametric Mapping (SPM) software package and is easy-to-use, general-purpose, modular, readable and extendable. We validate the toolbox by reproducing model selection and model averaging results from earlier publications. COMPARISON WITH EXISTING METHODS: A previous toolbox for model diagnosis in fMRI has been discontinued and other approaches to model comparison between GLMs have not been translated into reusable computational resources in the past. CONCLUSIONS: Increased attention on model quality will lead to lower false-positive rates in cognitive neuroscience and increased application of the MACS toolbox will increase the reproducibility of GLM analyses and is likely to increase the replicability of fMRI studies.

Cortical specialization for attended versus unattended working memory.

Items held in working memory can be either attended or not, depending on their current behavioral relevance. It has been suggested that unattended contents might be solely retained in an activity-silent form. Instead, we demonstrate here that encoding unattended contents involves a division of labor. While visual cortex only maintains attended items, intraparietal areas and the frontal eye fields represent both attended and unattended items.

The same analysis approach: Practical protection against the pitfalls of novel neuroimaging analysis methods.

Standard neuroimaging data analysis based on traditional principles of experimental design, modelling, and statistical inference is increasingly complemented by novel analysis methods, driven e.g. by machine learning methods. While these novel approaches provide new insights into neuroimaging data, they often have unexpected properties, generating a growing literature on possible pitfalls. We propose to meet this challenge by adopting a habit of systematic testing of experimental design, analysis procedures, and statistical inference. Specifically, we suggest to apply the analysis method used for experimental data also to aspects of the experimental design, simulated confounds, simulated null data, and control data. We stress the importance of keeping the analysis method the same in main and test analyses, because only this way possible confounds and unexpected properties can be reliably detected and avoided. We describe and discuss this Same Analysis Approach in detail, and demonstrate it in two worked examples using multivariate decoding. With these examples, we reveal two sources of error: A mismatch between counterbalancing (crossover designs) and cross-validation which leads to systematic below-chance accuracies, and linear decoding of a nonlinear effect, a difference in variance.

View-Independent Working Memory Representations of Artificial Shapes in Prefrontal and Posterior Regions of the Human Brain.

Traditional views of visual working memory postulate that memorized contents are stored in dorsolateral prefrontal cortex using an adaptive and flexible code. In contrast, recent studies proposed that contents are maintained by posterior brain areas using codes akin to perceptual representations. An important question is whether this reflects a difference in the level of abstraction between posterior and prefrontal representations. Here, we investigated whether neural representations of visual working memory contents are view-independent, as indicated by rotation-invariance. Using functional magnetic resonance imaging and multivariate pattern analyses, we show that when subjects memorize complex shapes, both posterior and frontal brain regions maintain the memorized contents using a rotation-invariant code. Importantly, we found the representations in frontal cortex to be localized to the frontal eye fields rather than dorsolateral prefrontal cortices. Thus, our results give evidence for the view-independent storage of complex shapes in distributed representations across posterior and frontal brain regions.

How to improve parameter estimates in GLM-based fMRI data analysis: cross-validated Bayesian model averaging.

In functional magnetic resonance imaging (fMRI), model quality of general linear models (GLMs) for first-level analysis is rarely assessed. In recent work (Soch et al., 2016: "How to avoid mismodelling in GLM-based fMRI data analysis: cross-validated Bayesian model selection", NeuroImage, vol. 141, pp. 469-489; http://dx.doi.org/10.1016/j.neuroimage.2016.07.047), we have introduced cross-validated Bayesian model selection (cvBMS) to infer the best model for a group of subjects and use it to guide second-level analysis. While this is the optimal approach given that the same GLM has to be used for all subjects, there is a much more efficient procedure when model selection only addresses nuisance variables and regressors of interest are included in all candidate models. In this work, we propose cross-validated Bayesian model averaging (cvBMA) to improve parameter estimates for these regressors of interest by combining information from all models using their posterior probabilities. This is particularly useful as different models can lead to different conclusions regarding experimental effects and the most complex model is not necessarily the best choice. We find that cvBMS can prevent not detecting established effects and that cvBMA can be more sensitive to experimental effects than just using even the best model in each subject or the model which is best in a group of subjects.

How to avoid mismodelling in GLM-based fMRI data analysis: cross-validated Bayesian model selection.

Voxel-wise general linear models (GLMs) are a standard approach for analyzing functional magnetic resonance imaging (fMRI) data. An advantage of GLMs is that they are flexible and can be adapted to the requirements of many different data sets. However, the specification of first-level GLMs leaves the researcher with many degrees of freedom which is problematic given recent efforts to ensure robust and reproducible fMRI data analysis. Formal model comparisons that allow a systematic assessment of GLMs are only rarely performed. On the one hand, too simple models may underfit data and leave real effects undiscovered. On the other hand, too complex models might overfit data and also reduce statistical power. Here we present a systematic approach termed cross-validated Bayesian model selection (cvBMS) that allows to decide which GLM best describes a given fMRI data set. Importantly, our approach allows for non-nested model comparison, i.e. comparing more than two models that do not just differ by adding one or more regressors. It also allows for spatially heterogeneous modelling, i.e. using different models for different parts of the brain. We validate our method using simulated data and demonstrate potential applications to empirical data. The increased use of model comparison and model selection should increase the reliability of GLM results and reproducibility of fMRI studies.

Valid population inference for information-based imaging: From the second-level t-test to prevalence inference.

In multivariate pattern analysis of neuroimaging data, 'second-level' inference is often performed by entering classification accuracies into a t-test vs chance level across subjects. We argue that while the random-effects analysis implemented by the t-test does provide population inference if applied to activation differences, it fails to do so in the case of classification accuracy or other 'information-like' measures, because the true value of such measures can never be below chance level. This constraint changes the meaning of the population-level null hypothesis being tested, which becomes equivalent to the global null hypothesis that there is no effect in any subject in the population. Consequently, rejecting it only allows to infer that there are some subjects in which there is an information effect, but not that it generalizes, rendering it effectively equivalent to fixed-effects analysis. This statement is supported by theoretical arguments as well as simulations. We review possible alternative approaches to population inference for information-based imaging, converging on the idea that it should not target the mean, but the prevalence of the effect in the population. One method to do so, 'permutation-based information prevalence inference using the minimum statistic', is described in detail and applied to empirical data.

The point of no return in vetoing self-initiated movements.

In humans, spontaneous movements are often preceded by early brain signals. One such signal is the readiness potential (RP) that gradually arises within the last second preceding a movement. An important question is whether people are able to cancel movements after the elicitation of such RPs, and if so until which point in time. Here, subjects played a game where they tried to press a button to earn points in a challenge with a brain-computer interface (BCI) that had been trained to detect their RPs in real time and to emit stop signals. Our data suggest that subjects can still veto a movement even after the onset of the RP. Cancellation of movements was possible if stop signals occurred earlier than 200 ms before movement onset, thus constituting a point of no return.

MRI-based diagnostic biomarkers for early onset pediatric multiple sclerosis.

Currently, it is unclear whether pediatric multiple sclerosis (PMS) is a pathoetiologically homogeneous disease phenotype due to clinical and epidemiological differences between early and late onset PMS (EOPMS and LOPMS). Consequently, the question was raised whether diagnostic guidelines need to be complemented by specific EOPMS markers. To search for such markers, we analyzed cerebral MRI images acquired with standard protocols using computer-based classification techniques. Specifically, we applied classification algorithms to gray (GM) and white matter (WM) tissue probability parameters of small brain regions derived from T2-weighted MRI images of EOPMS patients (onset <12 years), LOPMS patients (onset ≥12 years), and healthy controls (HC). This was done for PMS subgroups matched for disease duration and participant age independently. As expected, maximal diagnostic information for distinguishing PMS patients and HC was found in a periventricular WM area containing lesions (87.1% accuracy, p < 2.2 × 10(-5)). MRI-based biomarkers specific for EOPMS were identified in prefrontal cortex. Specifically, a coordinate in middle frontal gyrus contained maximal diagnostic information (77.3%, p = 1.8 × 10(-4)). Taken together, we were able to identify biomarkers reflecting pathognomonic processes specific for MS patients with very early onset. Especially GM involvement in the separation between PMS subgroups suggests that conventional MRI contains a richer set of diagnostically informative features than previously assumed.

Multimodal prediction of conversion to Alzheimer's disease based on incomplete biomarkers.

BACKGROUND: This study investigates the prediction of mild cognitive impairment-to-Alzheimer's disease (MCI-to-AD) conversion based on extensive multimodal data with varying degrees of missing values. METHODS: Based on Alzheimer's Disease Neuroimaging Initiative data from MCI-patients including all available modalities, we predicted the conversion to AD within 3 years. Different ways of replacing missing data in combination with different classification algorithms are compared. The performance was evaluated on features prioritized by experts and automatically selected features. RESULTS: The conversion to AD could be predicted with a maximal accuracy of 73% using support vector machines and features chosen by experts. Among data modalities, neuropsychological, magnetic resonance imaging, and positron emission tomography data were most informative. The best single feature was the functional activities questionnaire. CONCLUSION: Extensive multimodal and incomplete data can be adequately handled by a combination of missing data substitution, feature selection, and classification.

The neural code for face orientation in the human fusiform face area.

Humans recognize faces and objects with high speed and accuracy regardless of their orientation. Recent studies have proposed that orientation invariance in face recognition involves an intermediate representation where neural responses are similar for mirror-symmetric views. Here, we used fMRI, multivariate pattern analysis, and computational modeling to investigate the neural encoding of faces and vehicles at different rotational angles. Corroborating previous studies, we demonstrate a representation of face orientation in the fusiform face-selective area (FFA). We go beyond these studies by showing that this representation is category-selective and tolerant to retinal translation. Critically, by controlling for low-level confounds, we found the representation of orientation in FFA to be compatible with a linear angle code. Aspects of mirror-symmetric coding cannot be ruled out when FFA mean activity levels are considered as a dimension of coding. Finally, we used a parametric family of computational models, involving a biased sampling of view-tuned neuronal clusters, to compare different face angle encoding models. The best fitting model exhibited a predominance of neuronal clusters tuned to frontal views of faces. In sum, our findings suggest a category-selective and monotonic code of face orientation in the human FFA, in line with primate electrophysiology studies that observed mirror-symmetric tuning of neural responses at higher stages of the visual system, beyond the putative homolog of human FFA.

Robust artifactual independent component classification for BCI practitioners.

OBJECTIVE: EEG artifacts of non-neural origin can be separated from neural signals by independent component analysis (ICA). It is unclear (1) how robustly recently proposed artifact classifiers transfer to novel users, novel paradigms or changed electrode setups, and (2) how artifact cleaning by a machine learning classifier impacts the performance of brain-computer interfaces (BCIs). APPROACH: Addressing (1), the robustness of different strategies with respect to the transfer between paradigms and electrode setups of a recently proposed classifier is investigated on offline data from 35 users and 3 EEG paradigms, which contain 6303 expert-labeled components from two ICA and preprocessing variants. Addressing (2), the effect of artifact removal on single-trial BCI classification is estimated on BCI trials from 101 users and 3 paradigms. MAIN RESULTS: We show that (1) the proposed artifact classifier generalizes to completely different EEG paradigms. To obtain similar results under massively reduced electrode setups, a proposed novel strategy improves artifact classification. Addressing (2), ICA artifact cleaning has little influence on average BCI performance when analyzed by state-of-the-art BCI methods. When slow motor-related features are exploited, performance varies strongly between individuals, as artifacts may obstruct relevant neural activity or are inadvertently used for BCI control. SIGNIFICANCE: Robustness of the proposed strategies can be reproduced by EEG practitioners as the method is made available as an EEGLAB plug-in.

Searchlight-based multi-voxel pattern analysis of fMRI by cross-validated MANOVA.

Multi-voxel pattern analysis (MVPA) is a fruitful and increasingly popular complement to traditional univariate methods of analyzing neuroimaging data. We propose to replace the standard 'decoding' approach to searchlight-based MVPA, measuring the performance of a classifier by its accuracy, with a method based on the multivariate form of the general linear model. Following the well-established methodology of multivariate analysis of variance (MANOVA), we define a measure that directly characterizes the structure of multi-voxel data, the pattern distinctness D. Our measure is related to standard multivariate statistics, but we apply cross-validation to obtain an unbiased estimate of its population value, independent of the amount of data or its partitioning into 'training' and 'test' sets. The estimate D^ can therefore serve not only as a test statistic, but also as an interpretable measure of multivariate effect size. The pattern distinctness generalizes the Mahalanobis distance to an arbitrary number of classes, but also the case where there are no classes of trials because the design is described by parametric regressors. It is defined for arbitrary estimable contrasts, including main effects (pattern differences) and interactions (pattern changes). In this way, our approach makes the full analytical power of complex factorial designs known from univariate fMRI analyses available to MVPA studies. Moreover, we show how the results of a factorial analysis can be used to obtain a measure of pattern stability, the equivalent of 'cross-decoding'.

Content-specific coordination of listeners' to speakers' EEG during communication.

Cognitive neuroscience has recently begun to extend its focus from the isolated individual mind to two or more individuals coordinating with each other. In this study we uncover a coordination of neural activity between the ongoing electroencephalogram (EEG) of two people-a person speaking and a person listening. The EEG of one set of twelve participants ("speakers") was recorded while they were narrating short stories. The EEG of another set of twelve participants ("listeners") was recorded while watching audiovisual recordings of these stories. Specifically, listeners watched the superimposed videos of two speakers simultaneously and were instructed to attend either to one or the other speaker. This allowed us to isolate neural coordination due to processing the communicated content from the effects of sensory input. We find several neural signatures of communication: First, the EEG is more similar among listeners attending to the same speaker than among listeners attending to different speakers, indicating that listeners' EEG reflects content-specific information. Secondly, listeners' EEG activity correlates with the attended speakers' EEG, peaking at a time delay of about 12.5 s. This correlation takes place not only between homologous, but also between non-homologous brain areas in speakers and listeners. A semantic analysis of the stories suggests that listeners coordinate with speakers at the level of complex semantic representations, so-called "situation models". With this study we link a coordination of neural activity between individuals directly to verbally communicated information.

Multi-scale classification of disease using structural MRI and wavelet transform.

Recently, multivariate analysis algorithms have become a popular tool to diagnose neurological diseases based on neuroimaging data. Most studies, however, are biased for one specific scale, namely the scale given by the spatial resolution (i.e. dimension) of the data. In the present study, we propose to use the dual-tree complex wavelet transform to extract information on different spatial scales from structural MRI data and show its relevance for disease classification. Based on the magnitude representation of the complex wavelet coefficients calculated from the MR images, we identified a new class of features taking scale, directionality and potentially local information into account simultaneously. By using a linear support vector machine, these features were shown to discriminate significantly between spatially normalized MR images of 41 patients suffering from multiple sclerosis and 26 healthy controls. Interestingly, the decoding accuracies varied strongly among the different scales and it turned out that scales containing low frequency information were partly superior to scales containing high frequency information. Usually, this type of information is neglected since most decoding studies use only the original scale of the data. In conclusion, our proposed method has not only a high potential to assist in the diagnostic process of multiple sclerosis, but can be applied to other diseases or general decoding problems in structural or functional MRI.