Flexible sensory-motor mapping rules manifest in correlated variability of stimulus and action codes across the brain.

Humans and non-human primates can flexibly switch between different arbitrary mappings from sensation to action to solve a cognitive task. It has remained unknown how the brain implements such flexible sensory-motor mapping rules. Here, we uncovered a dynamic reconfiguration of task-specific correlated variability between sensory and motor brain regions. Human participants switched between two rules for reporting visual orientation judgments during fMRI recordings. Rule switches were either signaled explicitly or inferred by the participants from ambiguous cues. We used behavioral modeling to reconstruct the time course of their belief about the active rule. In both contexts, the patterns of correlations between ongoing fluctuations in stimulus- and action-selective activity across visual- and action-related brain regions tracked participants' belief about the active rule. The rule-specific correlation patterns broke down around the time of behavioral errors. We conclude that internal beliefs about task state are instantiated in brain-wide, selective patterns of correlated variability.

Adaptive circuit dynamics across human cortex during evidence accumulation in changing environments.

Many decisions under uncertainty entail the temporal accumulation of evidence that informs about the state of the environment. When environments are subject to hidden changes in their state, maximizing accuracy and reward requires non-linear accumulation of evidence. How this adaptive, non-linear computation is realized in the brain is unknown. We analyzed human behavior and cortical population activity (measured with magnetoencephalography) recorded during visual evidence accumulation in a changing environment. Behavior and decision-related activity in cortical regions involved in action planning exhibited hallmarks of adaptive evidence accumulation, which could also be implemented by a recurrent cortical microcircuit. Decision dynamics in action-encoding parietal and frontal regions were mirrored in a frequency-specific modulation of the state of the visual cortex that depended on pupil-linked arousal and the expected probability of change. These findings link normative decision computations to recurrent cortical circuit dynamics and highlight the adaptive nature of decision-related feedback to the sensory cortex.

Large-scale dynamics of perceptual decision information across human cortex.

Perceptual decisions entail the accumulation of sensory evidence for a particular choice towards an action plan. An influential framework holds that sensory cortical areas encode the instantaneous sensory evidence and downstream, action-related regions accumulate this evidence. The large-scale distribution of this computation across the cerebral cortex has remained largely elusive. Here, we develop a regionally-specific magnetoencephalography decoding approach to exhaustively map the dynamics of stimulus- and choice-specific signals across the human cortical surface during a visual decision. Comparison with the evidence accumulation dynamics inferred from behavior disentangles stimulus-dependent and endogenous components of choice-predictive activity across the visual cortical hierarchy. We find such an endogenous component in early visual cortex (including V1), which is expressed in a low (<20 Hz) frequency band and tracks, with delay, the build-up of choice-predictive activity in (pre-) motor regions. Our results are consistent with choice- and frequency-specific cortical feedback signaling during decision formation.

Fixation-pattern similarity analysis reveals adaptive changes in face-viewing strategies following aversive learning.

Animals can effortlessly adapt their behavior by generalizing from past aversive experiences, allowing to avoid harm in novel situations. We studied how visual information was sampled by eye-movements during this process called fear generalization, using faces organized along a circular two-dimensional perceptual continuum. During learning, one face was conditioned to predict a harmful event, whereas the most dissimilar face stayed neutral. This introduced an adversity gradient along one specific dimension, while the other, unspecific dimension was defined solely by perceptual similarity. Aversive learning changed scanning patterns selectively along the adversity-related dimension, but not the orthogonal dimension. This effect was mainly located within the eye region of faces. Our results provide evidence for adaptive changes in viewing strategies of faces following aversive learning. This is compatible with the view that these changes serve to sample information in a way that allows discriminating between safe and adverse for a better threat prediction.

Entorhinal cortex receptive fields are modulated by spatial attention, even without movement.

Grid cells in the entorhinal cortex allow for the precise decoding of position in space. Along with potentially playing an important role in navigation, grid cells have recently been hypothesized to make a general contribution to mental operations. A prerequisite for this hypothesis is that grid cell activity does not critically depend on physical movement. Here, we show that movement of covert attention, without any physical movement, also elicits spatial receptive fields with a triangular tiling of space. In monkeys trained to maintain central fixation while covertly attending to a stimulus moving in the periphery we identified a significant population (20/141, 14% neurons at a FDR <5%) of entorhinal cells with spatially structured receptive fields. This contrasts with recordings obtained in the hippocampus, where grid-like representations were not observed. Our results provide evidence that neurons in macaque entorhinal cortex do not rely on physical movement.

An extensive dataset of eye movements during viewing of complex images.

We present a dataset of free-viewing eye-movement recordings that contains more than 2.7 million fixation locations from 949 observers on more than 1000 images from different categories. This dataset aggregates and harmonizes data from 23 different studies conducted at the Institute of Cognitive Science at Osnabrück University and the University Medical Center in Hamburg-Eppendorf. Trained personnel recorded all studies under standard conditions with homogeneous equipment and parameter settings. All studies allowed for free eye-movements, and differed in the age range of participants (~7-80 years), stimulus sizes, stimulus modifications (phase scrambled, spatial filtering, mirrored), and stimuli categories (natural and urban scenes, web sites, fractal, pink-noise, and ambiguous artistic figures). The size and variability of viewing behavior within this dataset presents a strong opportunity for evaluating and comparing computational models of overt attention, and furthermore, for thoroughly quantifying strategies of viewing behavior. This also makes the dataset a good starting point for investigating whether viewing strategies change in patient groups.

Differential Contribution of Low- and High-level Image Content to Eye Movements in Monkeys and Humans.

Oculomotor selection exerts a fundamental impact on our experience of the environment. To better understand the underlying principles, researchers typically rely on behavioral data from humans, and electrophysiological recordings in macaque monkeys. This approach rests on the assumption that the same selection processes are at play in both species. To test this assumption, we compared the viewing behavior of 106 humans and 11 macaques in an unconstrained free-viewing task. Our data-driven clustering analyses revealed distinct human and macaque clusters, indicating species-specific selection strategies. Yet, cross-species predictions were found to be above chance, indicating some level of shared behavior. Analyses relying on computational models of visual saliency indicate that such cross-species commonalities in free viewing are largely due to similar low-level selection mechanisms, with only a small contribution by shared higher level selection mechanisms and with consistent viewing behavior of monkeys being a subset of the consistent viewing behavior of humans.

STN-DBS Reduces Saccadic Hypometria but Not Visuospatial Bias in Parkinson's Disease Patients.

In contrast to its well-established role in alleviating skeleto-motor symptoms in Parkinson's disease, little is known about the impact of deep brain stimulation (DBS) of the subthalamic nucleus (STN) on oculomotor control and attention. Eye-tracking data of 17 patients with left-hemibody symptom onset was compared with 17 age-matched control subjects. Free-viewing of natural images was assessed without stimulation as baseline and during bilateral DBS. To examine the involvement of ventral STN territories in oculomotion and spatial attention, we employed unilateral stimulation via the left and right ventralmost contacts respectively. When DBS was off, patients showed shorter saccades and a rightward viewing bias compared with controls. Bilateral stimulation in therapeutic settings improved saccadic hypometria but not the visuospatial bias. At a group level, unilateral ventral stimulation yielded no consistent effects. However, the evaluation of electrode position within normalized MNI coordinate space revealed that the extent of early exploration bias correlated with the precise stimulation site within the left subthalamic area. These results suggest that oculomotor impairments "but not higher-level exploration patterns" are effectively ameliorable by DBS in therapeutic settings. Our findings highlight the relevance of the STN topography in selecting contacts for chronic stimulation especially upon appearance of visuospatial attention deficits.

Dissociation between saliency signals and activity in early visual cortex.

Saliency is a measure that describes how attention is guided by local stimulus properties. Some hypotheses assign its computation to specific topographically organized areas of early human visual cortex. However, in most stimuli, saliency is correlated with luminance contrast, which in turn is known to correlate with activity in these early areas. Thus, any observed correlation of local activity with saliency might be due to the area encoding luminance contrast. Here we disentangle encoding of local luminance contrast and saliency by using stimuli where the two properties are uncorrelated. First, we conducted an eye-tracking study to verify that both negative and positive contrast modifications located in individual quadrants of the visual field increase saliency. Second, subjects viewed identical stimuli while fMRI signals were recorded. We find that positive contrast modifications induce a robust increase of activity in V1-V3 and hV4. However, negative contrast modifications lead to a reduced (V1, V2) or comparable (V3, hV4) activity level compared to unmodified quadrants. Furthermore, even with linear multivariate pattern-classification techniques, it is not possible to decode the location of the salient quadrant independent of the type of the contrast modification. Instead, decoding of the contrast-modified location is only possible separately for the two modification types in V1-V3. These findings suggest that the BOLD activity in V1-V3 is dominated by contrast-dependent processes and does not include the contrast invariance necessary for the computation of feature-invariant saliency.

Predictions in the light of your own action repertoire as a general computational principle.

We argue that brains generate predictions only within the constraints of the action repertoire. This makes the computational complexity tractable and fosters a step-by-step parallel development of sensory and motor systems. Hence, it is more of a benefit than a literal constraint and may serve as a universal normative principle to understand sensorimotor coupling and interactions with the world.

Saccadic momentum and facilitation of return saccades contribute to an optimal foraging strategy.

The interest in saccadic IOR is funneled by the hypothesis that it serves a clear functional purpose in the selection of fixation points: the facilitation of foraging. In this study, we arrive at a different interpretation of saccadic IOR. First, we find that return saccades are performed much more often than expected from the statistical properties of saccades and saccade pairs. Second, we find that fixation durations before a saccade are modulated by the relative angle of the saccade, but return saccades show no sign of an additional temporal inhibition. Thus, we do not find temporal saccadic inhibition of return. Interestingly, we find that return locations are more salient, according to empirically measured saliency (locations that are fixated by many observers) as well as stimulus dependent saliency (defined by image features), than regular fixation locations. These results and the finding that return saccades increase the match of individual trajectories with a grand total priority map evidences the return saccades being part of a fixation selection strategy that trades off exploration and exploitation.

Measures and limits of models of fixation selection.

Models of fixation selection are a central tool in the quest to understand how the human mind selects relevant information. Using this tool in the evaluation of competing claims often requires comparing different models' relative performance in predicting eye movements. However, studies use a wide variety of performance measures with markedly different properties, which makes a comparison difficult. We make three main contributions to this line of research: First we argue for a set of desirable properties, review commonly used measures, and conclude that no single measure unites all desirable properties. However the area under the ROC curve (a classification measure) and the KL-divergence (a distance measure of probability distributions) combine many desirable properties and allow a meaningful comparison of critical model performance. We give an analytical proof of the linearity of the ROC measure with respect to averaging over subjects and demonstrate an appropriate correction of entropy-based measures like KL-divergence for small sample sizes in the context of eye-tracking data. Second, we provide a lower bound and an upper bound of these measures, based on image-independent properties of fixation data and between subject consistency respectively. Based on these bounds it is possible to give a reference frame to judge the predictive power of a model of fixation selection. We provide open-source python code to compute the reference frame. Third, we show that the upper, between subject consistency bound holds only for models that predict averages of subject populations. Departing from this we show that incorporating subject-specific viewing behavior can generate predictions which surpass that upper bound. Taken together, these findings lay out the required information that allow a well-founded judgment of the quality of any model of fixation selection and should therefore be reported when a new model is introduced.

Investigating task-dependent top-down effects on overt visual attention.

Different tasks can induce different viewing behavior, yet it is still an open question how or whether at all high-level task information interacts with the bottom-up processing of stimulus-related information. Two possible causal routes are considered in this paper. Firstly, the weak top-down hypothesis, according to which top-down effects are mediated by changes of feature weights in the bottom-up system. Secondly, the strong top-down hypothesis, which proposes that top-down information acts independently of the bottom-up process. To clarify the influences of these different routes, viewing behavior was recorded on web pages for three different tasks: free viewing, content awareness, and information search. The data reveal significant task-dependent differences in viewing behavior that are accompanied by minor changes in feature-fixation correlations. Extensive computational modeling shows that these small but significant changes are insufficient to explain the observed differences in viewing behavior. Collectively, the results show that task-dependent differences in the current setting are not mediated by a reweighting of features in the bottom-up hierarchy, ruling out the weak top-down hypothesis. Consequently, the strong top-down hypothesis is the most viable explanation for the observed data.