Bimodal moment-by-moment coupling in perceptual multistability.

Multistable perception occurs in all sensory modalities, and there is ongoing theoretical debate about whether there are overarching mechanisms driving multistability across modalities. Here we study whether multistable percepts are coupled across vision and audition on a moment-by-moment basis. To assess perception simultaneously for both modalities without provoking a dual-task situation, we query auditory perception by direct report, while measuring visual perception indirectly via eye movements. A support-vector-machine (SVM)-based classifier allows us to decode visual perception from the eye-tracking data on a moment-by-moment basis. For each timepoint, we compare visual percept (SVM output) and auditory percept (report) and quantify the co-occurrence of integrated (one-object) or segregated (two-objects) interpretations in the two modalities. Our results show an above-chance coupling of auditory and visual perceptual interpretations. By titrating stimulus parameters toward an approximately symmetric distribution of integrated and segregated percepts for each modality and individual, we minimize the amount of coupling expected by chance. Because of the nature of our task, we can rule out that the coupling stems from postperceptual levels (i.e., decision or response interference). Our results thus indicate moment-by-moment perceptual coupling in the resolution of visual and auditory multistability, lending support to theories that postulate joint mechanisms for multistable perception across the senses.

Attention deployment in natural scenes: Higher-order scene statistics rather than semantics modulate the N2pc component.

Which properties of a natural scene affect visual search? We consider the alternative hypotheses that low-level statistics, higher-level statistics, semantics, or layout affect search difficulty in natural scenes. Across three experiments (n = 20 each), we used four different backgrounds that preserve distinct scene properties: (a) natural scenes (all experiments); (b) 1/f noise (pink noise, which preserves only low-level statistics and was used in Experiments 1 and 2); (c) textures that preserve low-level and higher-level statistics but not semantics or layout (Experiments 2 and 3); and (d) inverted (upside-down) scenes that preserve statistics and semantics but not layout (Experiment 2). We included "split scenes" that contained different backgrounds left and right of the midline (Experiment 1, natural/noise; Experiment 3, natural/texture). Participants searched for a Gabor patch that occurred at one of six locations (all experiments). Reaction times were faster for targets on noise and slower on inverted images, compared to natural scenes and textures. The N2pc component of the event-related potential, a marker of attentional selection, had a shorter latency and a higher amplitude for targets in noise than for all other backgrounds. The background contralateral to the target had an effect similar to that on the target side: noise led to faster reactions and shorter N2pc latencies than natural scenes, although we observed no difference in N2pc amplitude. There were no interactions between the target side and the non-target side. Together, this shows that-at least when searching simple targets without own semantic content-natural scenes are more effective distractors than noise and that this results from higher-order statistics rather than from semantics or layout.

Multisensory cues for walking in virtual reality: humans combine conflicting visual and self-motion information to reproduce distances.

When humans walk, it is important for them to have some measure of the distance they have traveled. Typically, many cues from different modalities are available, as humans perceive both the environment around them (for example, through vision and haptics) and their own walking. Here, we investigate the contribution of visual cues and nonvisual self-motion cues to distance reproduction when walking on a treadmill through a virtual environment by separately manipulating the speed of a treadmill belt and of the virtual environment. Using mobile eye tracking, we also investigate how our participants sampled the visual information through gaze. We show that, as predicted, both modalities affected how participants (N = 28) reproduced a distance. Participants weighed nonvisual self-motion cues more strongly than visual cues, corresponding also to their respective reliabilities, but with some interindividual variability. Those who looked more toward those parts of the visual scene that contained cues to speed and distance tended also to weigh visual information more strongly, although this correlation was nonsignificant, and participants generally directed their gaze toward visually informative areas of the scene less than expected. As measured by motion capture, participants adjusted their gait patterns to the treadmill speed but not to walked distance. In sum, we show in a naturalistic virtual environment how humans use different sensory modalities when reproducing distances and how the use of these cues differs between participants and depends on information sampling.NEW & NOTEWORTHY Combining virtual reality with treadmill walking, we measured the relative importance of visual cues and nonvisual self-motion cues for distance reproduction. Participants used both cues but put more weight on self-motion; weight on visual cues had a trend to correlate with looking at visually informative areas. Participants overshot distances, especially when self-motion was slow; they adjusted steps to self-motion cues but not to visual cues. Our work thus quantifies the multimodal contributions to distance reproduction.

Slipping while counting: gaze-gait interactions during perturbed walking under dual-task conditions.

Walking is a complex task. To prevent falls and injuries, gait needs to constantly adjust to the environment. This requires information from various sensory systems; in turn, moving through the environment continuously changes available sensory information. Visual information is available from a distance, and therefore most critical when negotiating difficult terrain. To effectively sample visual information, humans adjust their gaze to the terrain or-in laboratory settings-when facing motor perturbations. During activities of daily living, however, only a fraction of sensory and cognitive resources can be devoted to ensuring safe gait. How do humans deal with challenging walking conditions when they face high cognitive load? Young, healthy participants (N = 24) walked on a treadmill through a virtual, but naturalistic environment. Occasionally, their gait was experimentally perturbed, inducing slipping. We varied cognitive load by asking participants in some blocks to count backward in steps of seven; orthogonally, we varied whether visual cues indicated upcoming perturbations. We replicated earlier findings on how humans adjust their gaze and their gait rapidly and flexibly on various time scales: eye and head movements responded in a partially compensatory pattern and visual cues mostly affected eye movements. Interestingly, the cognitive task affected mainly head orientation. During the cognitive task, we found no clear signs of a less stable gait or of a cautious gait mode, but evidence that participants adapted their gait less to the perturbations than without secondary task. In sum, cognitive load affects head orientation and impairs the ability to adjust to gait perturbations.

Effects of interpreting a dynamic geometric cue as gaze on attention allocation.

Gaze is a powerful cue for directing attention. We investigate the interpretation of an abstract figure as gaze modulates its efficacy as an attentional cue. In each trial, two vertical lines on a central disk moved to one side (left or right). Independent of this "feature-cued" side, a target (black disk) subsequently appeared on one side. After 300 trials (phase 1), participants watched a video of a human avatar walking away. For one group, the avatar wore a helmet that visually matched the central disk and looked at black disks to either side. The other group's video was unrelated to the cueing task. After another 300 trials (phase 2), videos were swapped between groups; 300 further trials (phase 3) followed. In all phases, participants responded more quickly for targets appearing on the feature-cued side. There was a significant interaction between group and phase for reaction times: In phase 3, the group who had just watched the avatar with the helmet had a reduced advantage to the feature-cued side. Hence, interpreting the disk as a turning head seen from behind counteracts the cueing by the motion of the disk. This suggests that the mere perceptual interpretation of an abstract stimulus as gaze yields social cueing effects.

Action-based predictions affect visual perception, neural processing, and pupil size, regardless of temporal predictability.

Sensory consequences of one's own action are often perceived as less intense, and lead to reduced neural responses, compared to externally generated stimuli. Presumably, such sensory attenuation is due to predictive mechanisms based on the motor command (efference copy). However, sensory attenuation has also been observed outside the context of voluntary action, namely when stimuli are temporally predictable. Here, we aimed at disentangling the effects of motor and temporal predictability-based mechanisms on the attenuation of sensory action consequences. During fMRI data acquisition, participants (N = 25) judged which of two visual stimuli was brighter. In predictable blocks, the stimuli appeared temporally aligned with their button press (active) or aligned with an automatically generated cue (passive). In unpredictable blocks, stimuli were presented with a variable delay after button press/cue, respectively. Eye tracking was performed to investigate pupil-size changes and to ensure proper fixation. Self-generated stimuli were perceived as darker and led to less neural activation in visual areas than their passive counterparts, indicating sensory attenuation for self-generated stimuli independent of temporal predictability. Pupil size was larger during self-generated stimuli, which correlated negatively with the blood oxygenation level dependent (BOLD) response: the larger the pupil, the smaller the BOLD amplitude in visual areas. Our results suggest that sensory attenuation in visual cortex is driven by action-based predictive mechanisms rather than by temporal predictability. This effect may be related to changes in pupil diameter. Altogether, these results emphasize the role of the efference copy in the processing of sensory action consequences.

Low-high-low or high-low-high? Pattern effects on sequential auditory scene analysis.

Sequential auditory scene analysis (ASA) is often studied using sequences of two alternating tones, such as ABAB or ABA_, with "_" denoting a silent gap, and "A" and "B" sine tones differing in frequency (nominally low and high). Many studies implicitly assume that the specific arrangement (ABAB vs ABA_, as well as low-high-low vs high-low-high within ABA_) plays a negligible role, such that decisions about the tone pattern can be governed by other considerations. To explicitly test this assumption, a systematic comparison of different tone patterns for two-tone sequences was performed in three different experiments. Participants were asked to report whether they perceived the sequences as originating from a single sound source (integrated) or from two interleaved sources (segregated). Results indicate that core findings of sequential ASA, such as an effect of frequency separation on the proportion of integrated and segregated percepts, are similar across the different patterns during prolonged listening. However, at sequence onset, the integrated percept was more likely to be reported by the participants in ABA_low-high-low than in ABA_high-low-high sequences. This asymmetry is important for models of sequential ASA, since the formation of percepts at onset is an integral part of understanding how auditory interpretations build up.

Perceptual Difficulty Persistently Increases Dominance in Binocular Rivalry-Even Without a Task.

A major objective of perception is the reduction of uncertainty about the outside world. Eye-movement research has demonstrated that attention and oculomotor control can subserve the function of decreasing uncertainty in vision. Here, we ask whether a similar effect exists for awareness in binocular rivalry, when two distinct stimuli presented to the two eyes compete for awareness. We tested whether this competition can be biased by uncertainty about the stimuli and their relevance for a perceptual task. Specifically, we have stimuli that are perceptually difficult (i.e., carry high perceptual uncertainty) compete with stimuli that are perceptually easy (low perceptual uncertainty). Using a no-report paradigm and reading the dominant stimulus continuously from the observers' eye movements, we find that the perceptually difficult stimulus becomes more dominant than the easy stimulus. This difference is enhanced by the stimuli's relevance for the task. In trials with task, the difference in dominance emerges quickly, peaks before the response, and then persists throughout the trial (further 10 s). However, the difference is already present in blocks before task instruction and still observable when the stimuli have ceased to be task relevant. This shows that perceptual uncertainty persistently increases perceptual dominance, and this is magnified by task relevance.

Icy road ahead-rapid adjustments of gaze-gait interactions during perturbed naturalistic walking.

Most humans can walk effortlessly across uniform terrain even when they do not pay much attention to it. However, most natural terrain is far from uniform, and we need visual information to maintain stable gait. Recent advances in mobile eye-tracking technology have made it possible to study, in natural environments, how terrain affects gaze and thus the sampling of visual information. However, natural environments provide only limited experimental control, and some conditions cannot safely be tested. Typical laboratory setups, in contrast, are far from natural settings for walking. We used a setup consisting of a dual-belt treadmill, 240\(^\circ\) projection screen, floor projection, three-dimensional optical motion tracking, and mobile eye tracking to investigate eye, head, and body movements during perturbed and unperturbed walking in a controlled yet naturalistic environment. In two experiments (N = 22 each), we simulated terrain difficulty by repeatedly inducing slipping through accelerating either of the two belts rapidly and unpredictably (Experiment 1) or sometimes following visual cues (Experiment 2). We quantified the distinct roles of eye and head movements for adjusting gaze on different time scales. While motor perturbations mainly influenced head movements, eye movements were primarily affected by the presence of visual cues. This was true both immediately following slips and-to a lesser extent-over the course of entire 5-min blocks. We find adapted gaze parameters already after the first perturbation in each block, with little transfer between blocks. In conclusion, gaze-gait interactions in experimentally perturbed yet naturalistic walking are adaptive, flexible, and effector specific.

Anterior insula reflects surprise in value-based decision-making and perception.

The brain has been theorized to employ inferential processes to overcome the problem of uncertainty. Inference is thought to underlie neural processes, including in disparate domains such as value-based decision-making and perception. Value-based decision-making commonly involves deliberation, a time-consuming process that requires conscious consideration of decision variables. Perception, by contrast, is thought to be automatic and effortless. Both processes may call on a general neural system to resolve for uncertainty however. We addressed this question by directly comparing uncertainty signals in visual perception and an economic task using fMRI. We presented the same individuals with different versions of a bi-stable figure (Necker's cube) and with a gambling task during fMRI acquisition. We experimentally varied uncertainty, either on perceptual state or financial outcome. We found that inferential errors indexed by a formal account of surprise in the gambling task yielded BOLD responses in the anterior insula, in line with earlier findings. Moreover, we found perceptual uncertainty and surprise in the Necker Cube task yielded similar responses in the anterior insula. These results suggest that uncertainty, irrespective of domain, correlates to a common brain region, the anterior insula. These findings provide empirical evidence that the brain interacts with its environment through inferential processes.

Picture-evoked changes in pupil size predict learning success in children.

Episodic memory, the ability to remember past events in time and place, develops during childhood. Much knowledge about the underlying neuronal mechanisms has been gained from methods not suitable for children. We applied pupillometry to study memory encoding and recognition mechanisms. Children aged 8 and 9 years (n = 24) and adults (n = 24) studied a set of visual scenes to later distinguish them from new pictures. Children performed worse than adults, demonstrating immature episodic memory. During memorization, picture-related changes in pupil diameter predicted later successful recognition. This prediction effect was also observed on a single-trial level. During retrieval, novel pictures showed stronger pupil constriction than familiar pictures in both age groups. The statistically independent effects of objective familiarity (previously presented pictures) versus subjective familiarity (pictures evaluated as familiar independent of the prior presentation) suggest dissociable underlying brain mechanisms. In addition, we isolated principal components of the picture-related pupil response that were differently affected by the memorization and retrieval effects. Results are discussed in the context of the maturation of the medial temporal lobe and prefrontal networks. Our results demonstrate the dissociation of distinct contributions to episodic memory with a psychophysiological method that is suitable for a wide age spectrum.

Fixation durations in natural scene viewing are guided by peripheral scene content.

Fixation durations provide insights into processing demands. We investigated factors controlling fixation durations during scene viewing in two experiments. In Experiment 1, we tested the degree to which fixation durations adapt to global scene processing difficulty by manipulating the contrast (from original contrast to isoluminant) and saturation (original vs. grayscale) of the entire scene. We observed longer fixation durations for lower levels of contrast, and longer fixation durations for grayscale than for color scenes. Thus fixation durations were globally slowed as visual information became more and more degraded, making scene processing increasingly more difficult. In Experiment 2, we investigated two possible sources for this slow-down. We used "checkerboard" stimuli in which unmodified patches alternated with patches from which luminance information had been removed (isoluminant patches). Fixation durations showed an inverted immediacy effect (longer, rather than shorter, fixation durations on unmodified patches) along with a parafoveal-on-foveal effect (shorter fixation durations, when an unmodified patch was fixated next). This effect was stronger when the currently fixated patch was isoluminant as opposed to unmodified. Our results suggest that peripheral scene information substantially affects fixation durations and are consistent with the notion of competition among the current and potential future fixation locations.

Continuous flash suppression: Manual action affects eye movements but not the reported percept.

Diverse paradigms, including ambiguous stimuli and mental imagery, have suggested a shared representation between motor and perceptual domains. We examined the effects of manual action on ambiguous perception in a continuous flash suppression (CFS) experiment. Specifically, we asked participants to try to perceive a suppressed grating while rotating a manipulandum. In one condition, the grating's motion was fully controlled by the manipulandum movement; in another condition the coupling was weak; and in a third condition, no movement was executed. We found no effect of the movement condition on the subjectively reported visibility of the grating, which is in contrast to previous studies that allowed for more top-down influence. However, we did observe an effect on eye movements: the gain of the optokinetic nystagmus induced by the grating was modulated by its coupling to the manual movement. Our results (a) indicate that action-to-perception transfer can occur on different levels of perceptual organization, (b) demonstrate that CFS involves the shared representations between action and perception differently than paradigms used in earlier studies, and

Cognition modulates action-to-perception transfer in ambiguous perception.

Can cognition penetrate action-to-perception transfer? Participants observed a structure-from-motion cylinder of ambiguous rotation direction. Beforehand, they experienced one of two mechanical models: An unambiguous cylinder was connected to a rod by either a belt (cylinder and rod rotating in the same direction) or by gears (both rotating in opposite directions). During ambiguous cylinder presentation, mechanics and rod were invisible, making both conditions visually identical. Observers inferred the rod's direction from their moment-by-moment subjective perceptual interpretation of the ambiguous cylinder. They reported the (hidden) rod's direction by rotating a manipulandum in either the same or the opposite direction. With respect to their effect on perceptual stability, the resulting match/nonmatch between perceived cylinder rotation and manipulandum rotation showed a significant interaction with the cognitive model they had previously been biased with. For the "belt" model, congruency between cylinder perception and manual action is induced by same-direction report. Here, we found that same-direction movement stabilized the perceived motion direction, replicating a known congruency effect. For the "gear" model, congruency between perception and action is-in contrast-induced by opposite-direction report. Here, no effect of perception-action congruency was found: Perceptual congruency and cognitive model nullified each other. Hence, an observer's internal model of a machine's operation guides action-to-perception transfer.

Using binocular rivalry to tag foreground sounds: Towards an objective visual measure for auditory multistability.

In binocular rivalry, paradigms have been proposed for unobtrusive moment-by-moment readout of observers' perceptual experience ("no-report paradigms"). Here, we take a first step to extend this concept to auditory multistability. Observers continuously reported which of two concurrent tone sequences they perceived in the foreground: high-pitch (1008 Hz) or low-pitch (400 Hz) tones. Interstimulus intervals were either fixed per sequence (Experiments 1 and 2) or random with tones alternating (Experiment 3). A horizontally drifting grating was presented to each eye; to induce binocular rivalry, gratings had distinct colors and motion directions. To associate each grating with one tone sequence, a pattern on the grating jumped vertically whenever the respective tone occurred. We found that the direction of the optokinetic nystagmus (OKN)-induced by the visually dominant grating-could be used to decode the tone (high/low) that was perceived in the foreground well above chance. This OKN-based readout improved after observers had gained experience with the auditory task (Experiments 1 and 2) and for simpler auditory tasks (Experiment 3). We found no evidence that the visual stimulus affected auditory multistability. Although decoding performance is still far from perfect, our paradigm may eventually provide a continuous estimate of the currently dominant percept in auditory multistability.

Eye movements of patients with schizophrenia in a natural environment.

Alterations of eye movements in schizophrenia patients have been widely described for laboratory settings. For example, gain during smooth tracking is reduced, and fixation patterns differ between patients and healthy controls. The question remains, whether such results are related to the specifics of the experimental environment, or whether they transfer to natural settings. Twenty ICD-10 diagnosed schizophrenia patients and 20 healthy age-matched controls participated in the study, each performing four different oculomotor tasks corresponding to natural everyday behavior in an indoor environment: (I) fixating stationary targets, (II) sitting in a hallway with free gaze, (III) walking down the hallway, and (IV) visually tracking a target on the floor while walking straight-ahead. In all conditions, eye movements were continuously recorded binocularly by a mobile lightweight eye tracker (EyeSeeCam). When patients looked at predefined targets, they showed more fixations with reduced durations than controls. The opposite was true when participants were sitting in a hallway with free gaze. During visual tracking, patients showed a significantly greater root-mean-square error (representing the mean deviation from optimal) of retinal target velocity. Different from previous results on smooth-pursuit eye movements obtained in laboratory settings, no such difference was found for velocity gain. Taken together, we have identified significant differences in fundamental oculomotor parameters between schizophrenia patients and healthy controls during natural behavior in a real environment. Moreover, our data provide evidence that in natural settings, patients overcome some impairments, which might be present only in laboratory studies, by as of now unknown compensatory mechanisms or strategies.

Salient in space, salient in time: Fixation probability predicts fixation duration during natural scene viewing.

During natural scene viewing, humans typically attend and fixate selected locations for about 200-400 ms. Two variables characterize such "overt" attention: the probability of a location being fixated, and the fixation's duration. Both variables have been widely researched, but little is known about their relation. We use a two-step approach to investigate the relation between fixation probability and duration. In the first step, we use a large corpus of fixation data. We demonstrate that fixation probability (empirical salience) predicts fixation duration across different observers and tasks. Linear mixed-effects modeling shows that this relation is explained neither by joint dependencies on simple image features (luminance, contrast, edge density) nor by spatial biases (central bias). In the second step, we experimentally manipulate some of these features. We find that fixation probability from the corpus data still predicts fixation duration for this new set of experimental data. This holds even if stimuli are deprived of low-level images features, as long as higher level scene structure remains intact. Together, this shows a robust relation between fixation duration and probability, which does not depend on simple image features. Moreover, the study exemplifies the combination of empirical research on a large corpus of data with targeted experimental manipulations.

Binocular rivalry: frontal activity relates to introspection and action but not to perception.

When two dissimilar stimuli are presented to the eyes, perception alternates between multiple interpretations, a phenomenon dubbed binocular rivalry. Numerous recent imaging studies have attempted to unveil neural substrates underlying multistable perception. However, these studies had a conceptual constraint: access to observers' perceptual state relied on their introspection and active report. Here, we investigated to what extent neural correlates of binocular rivalry in healthy humans are confounded by this subjective measure and by action. We used the optokinetic nystagmus and pupil size to objectively and continuously map perceptual alternations for binocular-rivalry stimuli. Combining these two measures with fMRI allowed us to assess the neural correlates of binocular rivalry time locked to the perceptual alternations in the absence of active report. When observers were asked to actively report their percept, our objective measures matched the report. In this active condition, objective measures and subjective reporting revealed that occipital, parietal, and frontal areas underlie the processing of binocular rivalry, replicating earlier findings. Furthermore, objective measures provided additional statistical power due to their continuous nature. Importantly, when observers passively experienced rivalry without reporting perceptual alternations, a different picture emerged: differential neural activity in frontal areas was absent, whereas activation in occipital and parietal regions persisted. Our results question the popular view of a driving role of frontal areas in the initiation of perceptual alternations during binocular rivalry. Instead, we conclude that frontal areas are associated with active report and introspection rather than with rivalry per se.

Evidence from pupillometry and fMRI indicates reduced neural response during vicarious social pain but not physical pain in autism.

Autism spectrum disorder (ASD) is characterized by substantial social deficits. The notion that dysfunctions in neural circuits involved in sharing another's affect explain these deficits is appealing, but has received only modest experimental support. Here we evaluated a complex paradigm on the vicarious social pain of embarrassment to probe social deficits in ASD as to whether it is more potent than paradigms currently in use. To do so we acquired pupillometry and fMRI in young adults with ASD and matched healthy controls. During a simple vicarious physical pain task no differences emerged between groups in behavior, pupillometry, and neural activation of the anterior insula (AIC) and anterior cingulate cortex (ACC). In contrast, processing complex vicarious social pain yielded reduced responses in ASD on all physiological measures of sharing another's affect. The reduced activity within the AIC was thereby explained by the severity of autistic symptoms in the social and affective domain. Additionally, behavioral responses lacked correspondence with the anterior cingulate and anterior insula cortex activity found in controls. Instead, behavioral responses in ASD were associated with hippocampal activity. The observed dissociation echoes the clinical observations that deficits in ASD are most pronounced in complex social situations and simple tasks may not probe the dysfunctions in neural pathways involved in sharing affect. Our results are highly relevant because individuals with ASD may have preserved abilities to share another's physical pain but still have problems with the vicarious representation of more complex emotions that matter in life.