Categorical working memory codes in human visual cortex.

Working memory contents are represented in neural activity patterns across multiple regions of the cortical hierarchy. A division of labor has been proposed where more anterior regions harbor increasingly abstract and categorical representations while the most detailed representations are held in primary sensory cortices. Here, using fMRI and multivariate encoding modeling, we demonstrate that for color stimuli categorical codes are already present at the level of extrastriate visual cortex (V4 and VO1), even when subjects are neither implicitly nor explicitly encouraged to categorize the stimuli. Importantly, this categorical coding was observed during working memory, but not during perception. Thus, visual working memory is likely to rely at least in part on categorical representations. SIGNIFICANCE STATEMENT: Working memory is the representational basis for human cognition. Recent work has demonstrated that numerous regions across the human brain can represent the contents of working memory. We use fMRI brain scanning and machine learning methods to demonstrate that different regions can represent the same content differently during working memory. Reading out the neural codes used to store working memory contents, we show that already in sensory cortex, areas V4 and VO1 represent color in a categorical format rather than a purely sensory fashion. Thereby, we provide a better understanding of how different regions of the brain might serve working memory and cognition.

Decoding verbal working memory representations of Chinese characters from Broca's area.

Representations of sensory working memory can be found across the entire neocortex. But how are verbal working memory (VWM) contents retained in the human brain? Here we used fMRI and multi-voxel pattern analyses to study Chinese native speakers (15 males, 13 females) memorizing Chinese characters. Chinese characters are uniquely suitable to study VWM because verbal encoding is encouraged by their complex visual appearance and monosyllabic pronunciation. We found that activity patterns in Broca's area and left premotor cortex carried information about the memorized characters. These language-related areas carried (1) significantly more information about cued characters than those not cued for memorization, (2) significantly more information on the left than the right hemisphere and (3) significantly more information about Chinese symbols than complex visual patterns which are hard to verbalize. In contrast, early visual cortex carries a comparable amount of information about cued and uncued stimuli and is thus unlikely to be involved in memory retention. This study provides evidence for verbal working memory maintenance in a distributed network of language-related brain regions, consistent with distributed accounts of WM. The results also suggest that Broca's area and left premotor cortex form the articulatory network which serves articulatory rehearsal in the retention of verbal working memory contents.

Inverse transformed encoding models - a solution to the problem of correlated trial-by-trial parameter estimates in fMRI decoding.

Techniques of multivariate pattern analysis (MVPA) can be used to decode the discrete experimental condition or a continuous modulator variable from measured brain activity during a particular trial. In functional magnetic resonance imaging (fMRI), trial-wise response amplitudes are sometimes estimated from the measured signal using a general linear model (GLM) with one onset regressor for each trial. When using rapid event-related designs with trials closely spaced in time, those estimates are highly variable and serially correlated due to the temporally extended shape of the hemodynamic response function (HRF). Here, we describe inverse transformed encoding modelling (ITEM), a principled approach of accounting for those serial correlations and decoding from the resulting estimates, at low computational cost and with no loss in statistical power. We use simulated data to show that ITEM outperforms the current standard approach in terms of decoding accuracy and analyze empirical data to demonstrate that ITEM is capable of visual reconstruction from fMRI signals.

View-Independent Working Memory Representations of Artificial Shapes in Prefrontal and Posterior Regions of the Human Brain.

Traditional views of visual working memory postulate that memorized contents are stored in dorsolateral prefrontal cortex using an adaptive and flexible code. In contrast, recent studies proposed that contents are maintained by posterior brain areas using codes akin to perceptual representations. An important question is whether this reflects a difference in the level of abstraction between posterior and prefrontal representations. Here, we investigated whether neural representations of visual working memory contents are view-independent, as indicated by rotation-invariance. Using functional magnetic resonance imaging and multivariate pattern analyses, we show that when subjects memorize complex shapes, both posterior and frontal brain regions maintain the memorized contents using a rotation-invariant code. Importantly, we found the representations in frontal cortex to be localized to the frontal eye fields rather than dorsolateral prefrontal cortices. Thus, our results give evidence for the view-independent storage of complex shapes in distributed representations across posterior and frontal brain regions.

The point of no return in vetoing self-initiated movements.

In humans, spontaneous movements are often preceded by early brain signals. One such signal is the readiness potential (RP) that gradually arises within the last second preceding a movement. An important question is whether people are able to cancel movements after the elicitation of such RPs, and if so until which point in time. Here, subjects played a game where they tried to press a button to earn points in a challenge with a brain-computer interface (BCI) that had been trained to detect their RPs in real time and to emit stop signals. Our data suggest that subjects can still veto a movement even after the onset of the RP. Cancellation of movements was possible if stop signals occurred earlier than 200 ms before movement onset, thus constituting a point of no return.

The same analysis approach: Practical protection against the pitfalls of novel neuroimaging analysis methods.

Standard neuroimaging data analysis based on traditional principles of experimental design, modelling, and statistical inference is increasingly complemented by novel analysis methods, driven e.g. by machine learning methods. While these novel approaches provide new insights into neuroimaging data, they often have unexpected properties, generating a growing literature on possible pitfalls. We propose to meet this challenge by adopting a habit of systematic testing of experimental design, analysis procedures, and statistical inference. Specifically, we suggest to apply the analysis method used for experimental data also to aspects of the experimental design, simulated confounds, simulated null data, and control data. We stress the importance of keeping the analysis method the same in main and test analyses, because only this way possible confounds and unexpected properties can be reliably detected and avoided. We describe and discuss this Same Analysis Approach in detail, and demonstrate it in two worked examples using multivariate decoding. With these examples, we reveal two sources of error: A mismatch between counterbalancing (crossover designs) and cross-validation which leads to systematic below-chance accuracies, and linear decoding of a nonlinear effect, a difference in variance.

How to improve parameter estimates in GLM-based fMRI data analysis: cross-validated Bayesian model averaging.

In functional magnetic resonance imaging (fMRI), model quality of general linear models (GLMs) for first-level analysis is rarely assessed. In recent work (Soch et al., 2016: "How to avoid mismodelling in GLM-based fMRI data analysis: cross-validated Bayesian model selection", NeuroImage, vol. 141, pp. 469-489; http://dx.doi.org/10.1016/j.neuroimage.2016.07.047), we have introduced cross-validated Bayesian model selection (cvBMS) to infer the best model for a group of subjects and use it to guide second-level analysis. While this is the optimal approach given that the same GLM has to be used for all subjects, there is a much more efficient procedure when model selection only addresses nuisance variables and regressors of interest are included in all candidate models. In this work, we propose cross-validated Bayesian model averaging (cvBMA) to improve parameter estimates for these regressors of interest by combining information from all models using their posterior probabilities. This is particularly useful as different models can lead to different conclusions regarding experimental effects and the most complex model is not necessarily the best choice. We find that cvBMS can prevent not detecting established effects and that cvBMA can be more sensitive to experimental effects than just using even the best model in each subject or the model which is best in a group of subjects.

Valid population inference for information-based imaging: From the second-level t-test to prevalence inference.

In multivariate pattern analysis of neuroimaging data, 'second-level' inference is often performed by entering classification accuracies into a t-test vs chance level across subjects. We argue that while the random-effects analysis implemented by the t-test does provide population inference if applied to activation differences, it fails to do so in the case of classification accuracy or other 'information-like' measures, because the true value of such measures can never be below chance level. This constraint changes the meaning of the population-level null hypothesis being tested, which becomes equivalent to the global null hypothesis that there is no effect in any subject in the population. Consequently, rejecting it only allows to infer that there are some subjects in which there is an information effect, but not that it generalizes, rendering it effectively equivalent to fixed-effects analysis. This statement is supported by theoretical arguments as well as simulations. We review possible alternative approaches to population inference for information-based imaging, converging on the idea that it should not target the mean, but the prevalence of the effect in the population. One method to do so, 'permutation-based information prevalence inference using the minimum statistic', is described in detail and applied to empirical data.

How to avoid mismodelling in GLM-based fMRI data analysis: cross-validated Bayesian model selection.

Voxel-wise general linear models (GLMs) are a standard approach for analyzing functional magnetic resonance imaging (fMRI) data. An advantage of GLMs is that they are flexible and can be adapted to the requirements of many different data sets. However, the specification of first-level GLMs leaves the researcher with many degrees of freedom which is problematic given recent efforts to ensure robust and reproducible fMRI data analysis. Formal model comparisons that allow a systematic assessment of GLMs are only rarely performed. On the one hand, too simple models may underfit data and leave real effects undiscovered. On the other hand, too complex models might overfit data and also reduce statistical power. Here we present a systematic approach termed cross-validated Bayesian model selection (cvBMS) that allows to decide which GLM best describes a given fMRI data set. Importantly, our approach allows for non-nested model comparison, i.e. comparing more than two models that do not just differ by adding one or more regressors. It also allows for spatially heterogeneous modelling, i.e. using different models for different parts of the brain. We validate our method using simulated data and demonstrate potential applications to empirical data. The increased use of model comparison and model selection should increase the reliability of GLM results and reproducibility of fMRI studies.

The neural code for face orientation in the human fusiform face area.

Humans recognize faces and objects with high speed and accuracy regardless of their orientation. Recent studies have proposed that orientation invariance in face recognition involves an intermediate representation where neural responses are similar for mirror-symmetric views. Here, we used fMRI, multivariate pattern analysis, and computational modeling to investigate the neural encoding of faces and vehicles at different rotational angles. Corroborating previous studies, we demonstrate a representation of face orientation in the fusiform face-selective area (FFA). We go beyond these studies by showing that this representation is category-selective and tolerant to retinal translation. Critically, by controlling for low-level confounds, we found the representation of orientation in FFA to be compatible with a linear angle code. Aspects of mirror-symmetric coding cannot be ruled out when FFA mean activity levels are considered as a dimension of coding. Finally, we used a parametric family of computational models, involving a biased sampling of view-tuned neuronal clusters, to compare different face angle encoding models. The best fitting model exhibited a predominance of neuronal clusters tuned to frontal views of faces. In sum, our findings suggest a category-selective and monotonic code of face orientation in the human FFA, in line with primate electrophysiology studies that observed mirror-symmetric tuning of neural responses at higher stages of the visual system, beyond the putative homolog of human FFA.

Searchlight-based multi-voxel pattern analysis of fMRI by cross-validated MANOVA.

Multi-voxel pattern analysis (MVPA) is a fruitful and increasingly popular complement to traditional univariate methods of analyzing neuroimaging data. We propose to replace the standard 'decoding' approach to searchlight-based MVPA, measuring the performance of a classifier by its accuracy, with a method based on the multivariate form of the general linear model. Following the well-established methodology of multivariate analysis of variance (MANOVA), we define a measure that directly characterizes the structure of multi-voxel data, the pattern distinctness D. Our measure is related to standard multivariate statistics, but we apply cross-validation to obtain an unbiased estimate of its population value, independent of the amount of data or its partitioning into 'training' and 'test' sets. The estimate D^ can therefore serve not only as a test statistic, but also as an interpretable measure of multivariate effect size. The pattern distinctness generalizes the Mahalanobis distance to an arbitrary number of classes, but also the case where there are no classes of trials because the design is described by parametric regressors. It is defined for arbitrary estimable contrasts, including main effects (pattern differences) and interactions (pattern changes). In this way, our approach makes the full analytical power of complex factorial designs known from univariate fMRI analyses available to MVPA studies. Moreover, we show how the results of a factorial analysis can be used to obtain a measure of pattern stability, the equivalent of 'cross-decoding'.

Instruction effects on randomness in sequence generation.

Randomness is a fundamental property of human behavior. It occurs both in the form of intrinsic random variability, say when repetitions of a task yield slightly different behavioral outcomes, or in the form of explicit randomness, say when a person tries to avoid being predicted in a game of rock, paper and scissors. Randomness has frequently been studied using random sequence generation tasks (RSG). A key finding has been that humans are poor at deliberately producing random behavior. At the same time, it has been shown that people might be better randomizers if randomness is only an implicit (rather than an explicit) requirement of the task. We therefore hypothesized that randomization performance might vary with the exact instructions with which randomness is elicited. To test this, we acquired data from a large online sample (n = 388), where every participant made 1,000 binary choices based on one of the following instructions: choose either randomly, freely, irregularly, according to an imaginary coin toss or perform a perceptual guessing task. Our results show significant differences in randomness between the conditions as quantified by conditional entropy and estimated Markov order. The randomization scores were highest in the conditions where people were asked to be irregular or mentally simulate a random event (coin toss) thus yielding recommendations for future studies on randomization behavior.

Multi-scale classification of disease using structural MRI and wavelet transform.

Recently, multivariate analysis algorithms have become a popular tool to diagnose neurological diseases based on neuroimaging data. Most studies, however, are biased for one specific scale, namely the scale given by the spatial resolution (i.e. dimension) of the data. In the present study, we propose to use the dual-tree complex wavelet transform to extract information on different spatial scales from structural MRI data and show its relevance for disease classification. Based on the magnitude representation of the complex wavelet coefficients calculated from the MR images, we identified a new class of features taking scale, directionality and potentially local information into account simultaneously. By using a linear support vector machine, these features were shown to discriminate significantly between spatially normalized MR images of 41 patients suffering from multiple sclerosis and 26 healthy controls. Interestingly, the decoding accuracies varied strongly among the different scales and it turned out that scales containing low frequency information were partly superior to scales containing high frequency information. Usually, this type of information is neglected since most decoding studies use only the original scale of the data. In conclusion, our proposed method has not only a high potential to assist in the diagnostic process of multiple sclerosis, but can be applied to other diseases or general decoding problems in structural or functional MRI.

Flicker-light induced visual phenomena: frequency dependence and specificity of whole percepts and percept features.

Flickering light induces visual hallucinations in human observers. Despite a long history of the phenomenon, little is known about the dependence of flicker-induced subjective impressions on the flicker frequency. We investigate this question using Ganzfeld stimulation and an experimental paradigm combining a continuous frequency scan (1-50 Hz) with a focus on re-occurring, whole percepts. On the single-subject level, we find a high degree of frequency stability of percepts. To generalize across subjects, we apply two rating systems, (1) a set of complex percept classes derived from subjects' reports and (2) an enumeration of elementary percept features, and determine distributions of occurrences over flicker frequency. We observe a stronger frequency specificity for complex percept classes than elementary percept features. Comparing the similarity relations among percept categories to those among frequency profiles, we observe that though percepts are preferentially induced by particular frequencies, the frequency does not unambiguously determine the experienced percept.

Mental states as macrostates emerging from brain electrical dynamics.

Psychophysiological correlations form the basis for different medical and scientific disciplines, but the nature of this relation has not yet been fully understood. One conceptual option is to understand the mental as "emerging" from neural processes in the specific sense that psychology and physiology provide two different descriptions of the same system. Stating these descriptions in terms of coarser- and finer-grained system states (macro- and microstates), the two descriptions may be equally adequate if the coarse-graining preserves the possibility to obtain a dynamical rule for the system. To test the empirical viability of our approach, we describe an algorithm to obtain a specific form of such a coarse-graining from data, and illustrate its operation using a simulated dynamical system. We then apply the method to an electroencephalographic recording, where we are able to identify macrostates from the physiological data that correspond to mental states of the subject.

Ganzfeld-induced hallucinatory experience, its phenomenology and cerebral electrophysiology.

Ganzfeld, i.e., exposure to an unstructured, uniform stimulation field, elicits in most observers pseudo-hallucinatory percepts, and may even induce global functional state changes ('altered states of consciousness'). The present paper gives a comprehensive overview of the phenomenology of subjective experience in the ganzfeld and its electrophysiological correlates. Laboratory techniques for visual or multi-modal ganzfeld induction are explained. The spectrum of ganzfeld-induced phenomena, ranging from elementary percepts to complex, vivid, dream-like imagery is described, and the latter illustrated by transcripts of subjects' reports. Similarities and differences to related sensory/perceptual phenomena are also discussed. Earlier findings on electrophysiological correlates of the ganzfeld are reviewed. Our own studies of electroencephalographic (EEG) activity in the ganzfeld are presented in some detail, and a re-analysis of data on EEG correlates of hallucinatory percepts in statu nascendi is reported. The results do not support the hypothesis of the hypnagogic origin of the percepts; the ganzfeld-induced steady-state is an activated state, and the spectral EEG dynamics in the alpha frequency range reveals processes of attention shifts and percept formation. The final section is devoted to the controversial topic of allegedly anomalous communication between human subjects ('ganzfeld telepathy'). It is shown that the use of ganzfeld in this research field relies partly on unsupported hypotheses concerning ganzfeld-induced states, partly on a weak conceptual background of the experimental procedure. The rôle of a particular belief system shared by the participants and experimenters is critically discussed.

MACS - a new SPM toolbox for model assessment, comparison and selection.

BACKGROUND: In cognitive neuroscience, functional magnetic resonance imaging (fMRI) data are widely analyzed using general linear models (GLMs). However, model quality of GLMs for fMRI is rarely assessed, in part due to the lack of formal measures for statistical model inference. NEW METHOD: We introduce a new SPM toolbox for model assessment, comparison and selection (MACS) of GLMs applied to fMRI data. MACS includes classical, information-theoretic and Bayesian methods of model assessment previously applied to GLMs for fMRI as well as recent methodological developments of model selection and model averaging in fMRI data analysis. RESULTS: The toolbox - which is freely available from GitHub - directly builds on the Statistical Parametric Mapping (SPM) software package and is easy-to-use, general-purpose, modular, readable and extendable. We validate the toolbox by reproducing model selection and model averaging results from earlier publications. COMPARISON WITH EXISTING METHODS: A previous toolbox for model diagnosis in fMRI has been discontinued and other approaches to model comparison between GLMs have not been translated into reusable computational resources in the past. CONCLUSIONS: Increased attention on model quality will lead to lower false-positive rates in cognitive neuroscience and increased application of the MACS toolbox will increase the reproducibility of GLM analyses and is likely to increase the replicability of fMRI studies.

Cortical specialization for attended versus unattended working memory.

Items held in working memory can be either attended or not, depending on their current behavioral relevance. It has been suggested that unattended contents might be solely retained in an activity-silent form. Instead, we demonstrate here that encoding unattended contents involves a division of labor. While visual cortex only maintains attended items, intraparietal areas and the frontal eye fields represent both attended and unattended items.

On the meaning and interpretation of global descriptors of brain electrical activity. Including a reply to X. Pei et al.

Global descriptors of the brain's electrical activity, Sigma, Phi, and Omega, provide a comprehensive characterisation of brain functional states. Recently, Pei et al. [Pei, X., Zheng, C., Zhang, A., Duan, F., Bin, G., 2005. Discussion on "Towards a quantitative characterisation of functional states of the brain: from the nonlinear methodology to the global linear description" by J. Wackermann. Int. J. Psychophysiol. 56, 201-207] discussed the effects of signal power on the global measure of spatial complexity, Omega, and suggested a modification consisting in epoch-wise and channel-wise normalisation of input data to unit power. In the present paper, the basic principles of the global approach are reviewed, and the issues of Pei et al.'s approach are assessed. The original and the modified measures of spatial complexity are compared in two case studies. Numerical simulation shows that both methods veridically estimate small numbers of signal sources, but systematically underestimate as the number increases; the modified method yields a minor relative improvement. A study on real EEG data shows that the two measures sensibly differ only where artefactual inhomogeneities in channel variances affect the data; a combined procedure, consisting in record-wise equalisation of channel variances before Omega calculations, is suggested as the optimal strategy. Differences between the original objectives of the global methodology and the proposed modifications are pointed out and critically discussed.

Detecting synchronization clusters in multivariate time series via coarse-graining of Markov chains.

Synchronization cluster analysis is an approach to the detection of underlying structures in data sets of multivariate time series, starting from a matrix R of bivariate synchronization indices. A previous method utilized the eigenvectors of R for cluster identification, analogous to several recent attempts at group identification using eigenvectors of the correlation matrix. All of these approaches assumed a one-to-one correspondence of dominant eigenvectors and clusters, which has however been shown to be wrong in important cases. We clarify the usefulness of eigenvalue decomposition for synchronization cluster analysis by translating the problem into the language of stochastic processes, and derive an enhanced clustering method harnessing recent insights from the coarse-graining of finite-state Markov processes. We illustrate the operation of our method using a simulated system of coupled Lorenz oscillators, and we demonstrate its superior performance over the previous approach. Finally we investigate the question of robustness of the algorithm against small sample size, which is important with regard to field applications.