Planning to revisit: Neural activity in refixation precursors.

Eye tracking studies suggest that refixations-fixations to locations previously visited-serve to recover information lost or missed during earlier exploration of a visual scene. These studies have largely ignored the role of precursor fixations-previous fixations on locations the eyes return to later. We consider the possibility that preparations to return later are already made during precursor fixations. This process would mark precursor fixations as a special category of fixations, that is, distinct in neural activity from other fixation categories such as refixations and fixations to locations visited only once. To capture the neural signals associated with fixation categories, we analyzed electroencephalograms (EEGs) and eye movements recorded simultaneously in a free-viewing contour search task. We developed a methodological pipeline involving regression-based deconvolution modeling, allowing our analyses to account for overlapping EEG responses owing to the saccade sequence and other oculomotor covariates. We found that precursor fixations were preceded by the largest saccades among the fixation categories. Independent of the effect of saccade length, EEG amplitude was enhanced in precursor fixations compared with the other fixation categories 200 to 400 ms after fixation onsets, most noticeably over the occipital areas. We concluded that precursor fixations play a pivotal role in visual perception, marking the continuous occurrence of transitions between exploratory and exploitative modes of eye movement in natural viewing behavior.

Humans predict the forest, not the trees: statistical learning of spatiotemporal structure in visual scenes.

The human brain is capable of using statistical regularities to predict future inputs. In the real world, such inputs typically comprise a collection of objects (e.g. a forest constitutes numerous trees). The present study aimed to investigate whether perceptual anticipation relies on lower-level or higher-level information. Specifically, we examined whether the human brain anticipates each object in a scene individually or anticipates the scene as a whole. To explore this issue, we first trained participants to associate co-occurring objects within fixed spatial arrangements. Meanwhile, participants implicitly learned temporal regularities between these displays. We then tested how spatial and temporal violations of the structure modulated behavior and neural activity in the visual system using fMRI. We found that participants only showed a behavioral advantage of temporal regularities when the displays conformed to their previously learned spatial structure, demonstrating that humans form configuration-specific temporal expectations instead of predicting individual objects. Similarly, we found suppression of neural responses for temporally expected compared with temporally unexpected objects in lateral occipital cortex only when the objects were embedded within expected configurations. Overall, our findings indicate that humans form expectations about object configurations, demonstrating the prioritization of higher-level over lower-level information in temporal expectation.

WildLab: A naturalistic free viewing experiment reveals previously unknown electroencephalography signatures of face processing.

Neural mechanisms of face perception are predominantly studied in well-controlled experimental settings that involve random stimulus sequences and fixed eye positions. Although powerful, the employed paradigms are far from what constitutes natural vision. Here, we demonstrate the feasibility of ecologically more valid experimental paradigms using natural viewing behaviour, by combining a free viewing paradigm on natural scenes, free of photographer bias, with advanced data processing techniques that correct for overlap effects and co-varying non-linear dependencies of multiple eye movement parameters. We validate this approach by replicating classic N170 effects in neural responses, triggered by fixation onsets (fixation event-related potentials [fERPs]). Importantly, besides finding a strong correlation between both experiments, our more natural stimulus paradigm yielded smaller variability between subjects than the classic setup. Moving beyond classic temporal and spatial effect locations, our experiment furthermore revealed previously unknown signatures of face processing: This includes category-specific modulation of the event-related potential (ERP)'s amplitude even before fixation onset, as well as adaptation effects across subsequent fixations depending on their history.

Coordinating With a Robot Partner Affects Neural Processing Related to Action Monitoring.

Robots start to play a role in our social landscape, and they are progressively becoming responsive, both physically and socially. It begs the question of how humans react to and interact with robots in a coordinated manner and what the neural underpinnings of such behavior are. This exploratory study aims to understand the differences in human-human and human-robot interactions at a behavioral level and from a neurophysiological perspective. For this purpose, we adapted a collaborative dynamical paradigm from the literature. We asked 12 participants to hold two corners of a tablet while collaboratively guiding a ball around a circular track either with another participant or a robot. In irregular intervals, the ball was perturbed outward creating an artificial error in the behavior, which required corrective measures to return to the circular track again. Concurrently, we recorded electroencephalography (EEG). In the behavioral data, we found an increased velocity and positional error of the ball from the track in the human-human condition vs. human-robot condition. For the EEG data, we computed event-related potentials. We found a significant difference between human and robot partners driven by significant clusters at fronto-central electrodes. The amplitudes were stronger with a robot partner, suggesting a different neural processing. All in all, our exploratory study suggests that coordinating with robots affects action monitoring related processing. In the investigated paradigm, human participants treat errors during human-robot interaction differently from those made during interactions with other humans. These results can improve communication between humans and robot with the use of neural activity in real-time.

#EEGManyLabs: Investigating the replicability of influential EEG experiments.

There is growing awareness across the neuroscience community that the replicability of findings about the relationship between brain activity and cognitive phenomena can be improved by conducting studies with high statistical power that adhere to well-defined and standardised analysis pipelines. Inspired by recent efforts from the psychological sciences, and with the desire to examine some of the foundational findings using electroencephalography (EEG), we have launched #EEGManyLabs, a large-scale international collaborative replication effort. Since its discovery in the early 20th century, EEG has had a profound influence on our understanding of human cognition, but there is limited evidence on the replicability of some of the most highly cited discoveries. After a systematic search and selection process, we have identified 27 of the most influential and continually cited studies in the field. We plan to directly test the replicability of key findings from 20 of these studies in teams of at least three independent laboratories. The design and protocol of each replication effort will be submitted as a Registered Report and peer-reviewed prior to data collection. Prediction markets, open to all EEG researchers, will be used as a forecasting tool to examine which findings the community expects to replicate. This project will update our confidence in some of the most influential EEG findings and generate a large open access database that can be used to inform future research practices. Finally, through this international effort, we hope to create a cultural shift towards inclusive, high-powered multi-laboratory collaborations.

Regression-based analysis of combined EEG and eye-tracking data: Theory and applications.

Fixation-related potentials (FRPs), neural responses aligned to the end of saccades, are a promising tool for studying the dynamics of attention and cognition under natural viewing conditions. In the past, four methodological problems have complicated the analysis of such combined eye-tracking/electroencephalogram experiments: (1) the synchronization of data streams, (2) the removal of ocular artifacts, (3) the condition-specific temporal overlap between the brain responses evoked by consecutive fixations, and (4) the fact that numerous low-level stimulus and saccade properties also influence the postsaccadic neural responses. Although effective solutions exist for the first two problems, the latter two are only beginning to be addressed. In the current paper, we present and review a unified regression-based framework for FRP analysis that allows us to deconvolve overlapping potentials while also controlling for both linear and nonlinear confounds on the FRP waveform. An open software implementation is provided for all procedures. We then demonstrate the advantages of this proposed (non)linear deconvolution modeling approach for data from three commonly studied paradigms: face perception, scene viewing, and reading. First, for a traditional event-related potential (ERP) face recognition experiment, we show how this technique can separate stimulus ERPs from overlapping muscle and brain potentials produced by small (micro)saccades on the face. Second, in natural scene viewing, we model and isolate multiple nonlinear effects of saccade parameters on the FRP. Finally, for a natural sentence reading experiment using the boundary paradigm, we show how it is possible to study the neural correlates of parafoveal preview after removing spurious overlap effects caused by the associated difference in average fixation time. Our results suggest a principal way of measuring reliable eye movement-related brain activity during natural vision.

Opposite effects of choice history and evidence history resolve a paradox of sequential choice bias.

Perceptual decisions are biased toward previous decisions. Earlier research suggests that this choice repetition bias is increased after previous decisions of high confidence, as inferred from response time measures (Urai, Braun, & Donner, 2017), but also when previous decisions were based on weak sensory evidence (Akaishi, Umeda, Nagase, & Sakai, 2014). As weak sensory evidence is typically associated with low confidence, these previous findings appear conflicting. To resolve this conflict, we set out to investigate the effect of decision confidence on choice repetition more directly by measuring explicit confidence ratings in a motion coherence discrimination task. Moreover, we explored how choice and evidence history jointly affect subsequent perceptual choices. We found that participants were more likely to repeat previous choices of high subjective confidence, as well as previous fast choices, confirming the boost of choice repetition with decision confidence. Furthermore, we discovered that current choices were biased away from the previous evidence direction and that this effect grew with previous evidence strength. These findings point toward simultaneous biases of choice repetition, modulated by decision confidence, and evidence adaptation, modulated by the strength of evidence, which bias current perceptual decisions in opposite directions.

How does the method change what we measure? Comparing virtual reality and text-based surveys for the assessment of moral decisions in traffic dilemmas.

The question of how self-driving cars should behave in dilemma situations has recently attracted a lot of attention in science, media and society. A growing number of publications amass insight into the factors underlying the choices we make in such situations, often using forced-choice paradigms closely linked to the trolley dilemma. The methodology used to address these questions, however, varies widely between studies, ranging from fully immersive virtual reality settings to completely text-based surveys. In this paper we compare virtual reality and text-based assessments, analyzing the effect that different factors in the methodology have on decisions and emotional response of participants. We present two studies, comparing a total of six different conditions varying across three dimensions: The level of abstraction, the use of virtual reality, and time-constraints. Our results show that the moral decisions made in this context are not strongly influenced by the assessment, and the compared methods ultimately appear to measure very similar constructs. Furthermore, we add to the pool of evidence on the underlying factors of moral judgment in traffic dilemmas, both in terms of general preferences, i.e., features of the particular situation and potential victims, as well as in terms of individual differences between participants, such as their age and gender.

Unfold: an integrated toolbox for overlap correction, non-linear modeling, and regression-based EEG analysis.

Electrophysiological research with event-related brain potentials (ERPs) is increasingly moving from simple, strictly orthogonal stimulation paradigms towards more complex, quasi-experimental designs and naturalistic situations that involve fast, multisensory stimulation and complex motor behavior. As a result, electrophysiological responses from subsequent events often overlap with each other. In addition, the recorded neural activity is typically modulated by numerous covariates, which influence the measured responses in a linear or non-linear fashion. Examples of paradigms where systematic temporal overlap variations and low-level confounds between conditions cannot be avoided include combined electroencephalogram (EEG)/eye-tracking experiments during natural vision, fast multisensory stimulation experiments, and mobile brain/body imaging studies. However, even "traditional," highly controlled ERP datasets often contain a hidden mix of overlapping activity (e.g., from stimulus onsets, involuntary microsaccades, or button presses) and it is helpful or even necessary to disentangle these components for a correct interpretation of the results. In this paper, we introduce unfold, a powerful, yet easy-to-use MATLAB toolbox for regression-based EEG analyses that combines existing concepts of massive univariate modeling ("regression-ERPs"), linear deconvolution modeling, and non-linear modeling with the generalized additive model into one coherent and flexible analysis framework. The toolbox is modular, compatible with EEGLAB and can handle even large datasets efficiently. It also includes advanced options for regularization and the use of temporal basis functions (e.g., Fourier sets). We illustrate the advantages of this approach for simulated data as well as data from a standard face recognition experiment. In addition to traditional and non-conventional EEG/ERP designs, unfold can also be applied to other overlapping physiological signals, such as pupillary or electrodermal responses. It is available as open-source software at http://www.unfoldtoolbox.org.

A new comprehensive eye-tracking test battery concurrently evaluating the Pupil Labs glasses and the EyeLink 1000.

Eye-tracking experiments rely heavily on good data quality of eye-trackers. Unfortunately, it is often the case that only the spatial accuracy and precision values are available from the manufacturers. These two values alone are not sufficient to serve as a benchmark for an eye-tracker: Eye-tracking quality deteriorates during an experimental session due to head movements, changing illumination or calibration decay. Additionally, different experimental paradigms require the analysis of different types of eye movements; for instance, smooth pursuit movements, blinks or microsaccades, which themselves cannot readily be evaluated by using spatial accuracy or precision alone. To obtain a more comprehensive description of properties, we developed an extensive eye-tracking test battery. In 10 different tasks, we evaluated eye-tracking related measures such as: the decay of accuracy, fixation durations, pupil dilation, smooth pursuit movement, microsaccade classification, blink classification, or the influence of head motion. For some measures, true theoretical values exist. For others, a relative comparison to a reference eye-tracker is needed. Therefore, we collected our gaze data simultaneously from a remote EyeLink 1000 eye-tracker as the reference and compared it with the mobile Pupil Labs glasses. As expected, the average spatial accuracy of 0.57° for the EyeLink 1000 eye-tracker was better than the 0.82° for the Pupil Labs glasses (N = 15). Furthermore, we classified less fixations and shorter saccade durations for the Pupil Labs glasses. Similarly, we found fewer microsaccades using the Pupil Labs glasses. The accuracy over time decayed only slightly for the EyeLink 1000, but strongly for the Pupil Labs glasses. Finally, we observed that the measured pupil diameters differed between eye-trackers on the individual subject level but not on the group level. To conclude, our eye-tracking test battery offers 10 tasks that allow us to benchmark the many parameters of interest in stereotypical eye-tracking situations and addresses a common source of confounds in measurement errors (e.g., yaw and roll head movements). All recorded eye-tracking data (including Pupil Labs' eye videos), the stimulus code for the test battery, and the modular analysis pipeline are freely available (https://github.com/behinger/etcomp).

The Social Situation Affects How We Process Feedback About Our Actions.

Humans achieve their goals in joint action tasks either by cooperation or competition. In the present study, we investigated the neural processes underpinning error and monetary rewards processing in such cooperative and competitive situations. We used electroencephalography (EEG) and analyzed event-related potentials (ERPs) triggered by feedback in both social situations. 26 dyads performed a joint four-alternative forced choice (4AFC) visual task either cooperatively or competitively. At the end of each trial, participants received performance feedback about their individual and joint errors and accompanying monetary rewards. Furthermore, the outcome, i.e., resulting positive, negative, or neutral rewards, was dependent on the pay-off matrix, defining the social situation either as cooperative or competitive. We used linear mixed effects models to analyze the feedback-related-negativity (FRN) and used the Threshold-free cluster enhancement (TFCE) method to explore activations of all electrodes and times. We found main effects of the outcome and social situation, but no interaction at mid-line frontal electrodes. The FRN was more negative for losses than wins in both social situations. However, the FRN amplitudes differed between social situations. Moreover, we compared monetary with neutral outcomes in both social situations. Our exploratory TFCE analysis revealed that processing of feedback differs between cooperative and competitive situations at right temporo-parietal electrodes where the cooperative situation elicited more positive amplitudes. Further, the differences induced by the social situations were stronger in participants with higher scores on a perspective taking test. In sum, our results replicate previous studies about the FRN and extend them by comparing neurophysiological responses to positive and negative outcomes in a task that simultaneously engages two participants in competitive and cooperative situations.

Probing the temporal dynamics of the exploration-exploitation dilemma of eye movements.

When scanning a visual scene, we are in a constant decision process regarding whether to further exploit the information content at the current fixation or to go on and explore the scene. The balance of these two processes determines the distribution of fixation durations. Using a gaze-contingent paradigm, we experimentally interrupt this process to probe its state. Here, we developed a guided-viewing task where only a single 3° aperture of an image ("bubble") is displayed. Subjects had to fixate the bubble for an experimentally controlled time (forced fixation time). Then, the previously fixated bubble disappeared, and one to five bubbles emerged at different locations. The subjects freely selected one of these by performing a saccade toward it. By repeating this procedure, the subjects explored the image. We modeled the resulting saccadic reaction times (choice times) from bubble offset to saccade onset using a Bayesian linear mixed model. We observed an exponential decay between the forced fixation time and the choice time: Short fixation durations elicited longer choice times. In trials with multiple bubbles, the choice time increased monotonically with the number of possible future targets. Additionally, we found only weak influences of the saccade amplitude, low-level stimulus properties, and saccade angle on the choice times. The exponential decay of the choice times suggests that the sampling and processing of the current stimulus were exhausted for long fixation durations, biasing toward faster exploration. This observation also shows that the decision process took into account processing demands at the current fixation location.

Humans treat unreliable filled-in percepts as more real than veridical ones.

Humans often evaluate sensory signals according to their reliability for optimal decision-making. However, how do we evaluate percepts generated in the absence of direct input that are, therefore, completely unreliable? Here, we utilize the phenomenon of filling-in occurring at the physiological blind-spots to compare partially inferred and veridical percepts. Subjects chose between stimuli that elicit filling-in, and perceptually equivalent ones presented outside the blind-spots, looking for a Gabor stimulus without a small orthogonal inset. In ambiguous conditions, when the stimuli were physically identical and the inset was absent in both, subjects behaved opposite to optimal, preferring the blind-spot stimulus as the better example of a collinear stimulus, even though no relevant veridical information was available. Thus, a percept that is partially inferred is paradoxically considered more reliable than a percept based on external input. In other words: Humans treat filled-in inferred percepts as more real than veridical ones.

Extensive training leads to temporal and spatial shifts of cortical activity underlying visual category selectivity.

The human visual system is able to distinguish naturally occurring categories with exceptional speed and accuracy. At the same time, it exhibits substantial plasticity, permitting the seamless and fast learning of entirely novel categories. Here we investigate the interplay of these two processes by asking how category selectivity emerges and develops from initial to extended category learning. For this purpose, we combine a rapid event-related MEG adaptation paradigm, an extension of fMRI adaptation to high temporal resolution, a novel spatiotemporal analysis approach to separate adaptation effects from other effect origins, and source localization. The results demonstrate a spatiotemporal shift of cortical activity underlying category selectivity: after initial category acquisition, the onset of category selectivity was observed starting at 275ms together with stronger activity in prefrontal cortex. Following extensive training over 22 sessions, adding up to more than 16.600 trials, the earliest category effects occurred at a markedly shorter latency of 113ms and were accompanied by stronger occipitotemporal activity. Our results suggest that the brain balances plasticity and efficiency by relying on different mechanisms to recognize new and re-occurring categories.

Melanopsin Variants as Intrinsic Optogenetic On and Off Switches for Transient versus Sustained Activation of G Protein Pathways.

G-protein-coupled receptors (GPCRs) represent the major protein family for cellular modulation in mammals. Therefore, various strategies have been developed to analyze the function of GPCRs involving pharmaco- and optogenetic approaches [1, 2]. However, a tool that combines precise control of the activation and deactivation of GPCR pathways and/or neuronal firing with limited phototoxicity is still missing. We compared the biophysical properties and optogenetic application of a human and a mouse melanopsin variant (hOpn4L and mOpn4L) on the control of Gi/o and Gq pathways in heterologous expression systems and mouse brain. We found that GPCR pathways can be switched on/off by blue/yellow light. The proteins differ in their kinetics and wavelength dependence to activate and deactivate G protein pathways. Whereas mOpn4L is maximally activated by very short light pulses, leading to sustained G protein activation, G protein responses of hOpn4L need longer light pulses to be activated and decline in amplitude. Based on the different biophysical properties, brief light activation of mOpn4L is sufficient to induce sustained neuronal firing in cerebellar Purkinje cells (PC), whereas brief light activation of hOpn4L induces AP firing, which declines in frequency over time. Most importantly, mOpn4L-induced sustained firing can be switched off by yellow light. Based on the biophysical properties, hOpn4L and mOpn4L represent the first GPCR optogenetic tools, which can be used to switch GPCR pathways/neuronal firing on an off with temporal precision and limited phototoxicity. We suggest to name these tools moMo and huMo for future optogenetic applications.

Predictions of Visual Content across Eye Movements and Their Modulation by Inferred Information.

The brain is proposed to operate through probabilistic inference, testing and refining predictions about the world. Here, we search for neural activity compatible with the violation of active predictions, learned from the contingencies between actions and the consequent changes in sensory input. We focused on vision, where eye movements produce stimuli shifts that could, in principle, be predicted. We compared, in humans, error signals to saccade-contingent changes of veridical and inferred inputs by contrasting the electroencephalographic activity after saccades to a stimulus presented inside or outside the blind spot. We observed early (<250 ms) and late (>250 ms) error signals after stimulus change, indicating the violation of sensory and associative predictions, respectively. Remarkably, the late response was diminished for blind-spot trials. These results indicate that predictive signals occur across multiple levels of the visual hierarchy, based on generative models that differentiate between signals that originate from the outside world and those that are inferred.

Kinesthetic and vestibular information modulate alpha activity during spatial navigation: a mobile EEG study.

In everyday life, spatial navigation involving locomotion provides congruent visual, vestibular, and kinesthetic information that need to be integrated. Yet, previous studies on human brain activity during navigation focus on stationary setups, neglecting vestibular and kinesthetic feedback. The aim of our work is to uncover the influence of those sensory modalities on cortical processing. We developed a fully immersive virtual reality setup combined with high-density mobile electroencephalography (EEG). Participants traversed one leg of a triangle, turned on the spot, continued along the second leg, and finally indicated the location of their starting position. Vestibular and kinesthetic information was provided either in combination, as isolated sources of information, or not at all within a 2 × 2 full factorial intra-subjects design. EEG data were processed by clustering independent components, and time-frequency spectrograms were calculated. In parietal, occipital, and temporal clusters, we detected alpha suppression during the turning movement, which is associated with a heightened demand of visuo-attentional processing and closely resembles results reported in previous stationary studies. This decrease is present in all conditions and therefore seems to generalize to more natural settings. Yet, in incongruent conditions, when different sensory modalities did not match, the decrease is significantly stronger. Additionally, in more anterior areas we found that providing only vestibular but no kinesthetic information results in alpha increase. These observations demonstrate that stationary experiments omit important aspects of sensory feedback. Therefore, it is important to develop more natural experimental settings in order to capture a more complete picture of neural correlates of spatial navigation.