Visuo-vestibular heading perception: a model system to study multi-sensory decision making.

Integrating noisy signals across time as well as sensory modalities, a process named multi-sensory decision making (MSDM), is an essential strategy for making more accurate and sensitive decisions in complex environments. Although this field is just emerging, recent extraordinary works from different perspectives, including computational theory, psychophysical behaviour and neurophysiology, begin to shed new light onto MSDM. In the current review, we focus on MSDM by using a model system of visuo-vestibular heading. Combining well-controlled behavioural paradigms on virtual-reality systems, single-unit recordings, causal manipulations and computational theory based on spiking activity, recent progress reveals that vestibular signals contain complex temporal dynamics in many brain regions, including unisensory, multi-sensory and sensory-motor association areas. This challenges the brain for cue integration across time and sensory modality such as optic flow which mainly contains a motion velocity signal. In addition, new evidence from the higher-level decision-related areas, mostly in the posterior and frontal/prefrontal regions, helps revise our conventional thought on how signals from different sensory modalities may be processed, converged, and moment-by-moment accumulated through neural circuits for forming a unified, optimal perceptual decision. This article is part of the theme issue 'Decision and control processes in multisensory perception'.

Heading perception from optic flow occurs at both perceptual representation and working memory stages with EEG evidence.

Psychophysical studies have demonstrated that heading perception from optic flow occurs in perceptual and post-perceptual stages. The post-perception stage is a complex concept, containing working memory. The current study examined whether working memory was involved in heading perception from optic flow by asking participants to conduct a heading perception task and recording their scalp EEG. On each trial, an optic flow display was presented, followed by a blank display. Participants were then asked to report their perceived heading. We know that participants would tend to automatically forget previous headings when they learned that previously presented headings were unrelated to the current heading perception to save cognitive resources. As a result, we could not decode previous headings from the EEG data of current trials. More importantly, if we successfully decoded previous headings when the blank display (optic flow) was presented, then working memory (perceptual representation stage) was involved in heading perception. Our results showed that the decoding accuracy was significantly higher than the chance level when the optic flow and blank displays were presented. Therefore, the current study provided electrophysiological evidence that heading perception from optic flow occurred in the perceptual representation and working memory stages, against the previous perceptual claim.

Heading perception from optic flow is affected by heading distribution.

Recent studies have revealed a central tendency in the perception of physical features. That is, the perceived feature was biased toward the mean of recently experienced features (i.e., previous feature distribution). However, no study explored whether the central tendency was in heading perception or not. In this study, we conducted three experiments to answer this question. The results showed that the perceived heading was not biased toward the mean of the previous heading distribution, suggesting that the central tendency was not in heading perception. However, the perceived headings were overall biased toward the left side, where headings rarely appeared in the right-heavied distribution (Experiment 3), suggesting that heading perception from optic flow was affected by previously seen headings. It indicated that the participants learned the heading distributions and used them to adjust their heading perception. Our study revealed that heading perception from optic flow was not purely perceptual and that postperceptual stages (e.g., attention and working memory) might be involved in the heading perception from optic flow.

The Speed of Optic Flow Stimuli Influences Body Sway.

Optic flow is a perceptual cue processed for self-motion control. The aim of this study was to investigate whether postural control is modulated by the speed of radial optic flow stimuli. The experiments were performed on 20 healthy volunteers using stabilometry and surface electromyography (EMG). The subjects were instructed to fixate a central fixation point while radial optic flow stimuli were presented full field, in the foveal and in the peripheral visual field at different dots speed (8, 11, 14, 17 and 20°/s). Fixation in the dark was used as control stimulus. The EMG analysis showed that male and female subjects reacted to the stimuli with different muscle activity (main effects for gender, muscle and laterality: p < 0.001). The analysis of the center of pressure (COP) parameters showed that optic flow stimuli had a different effect on the left and right limbs of males and females (main effects of laterality: p < 0.015; interaction effects of gender and laterality: p < 0.016). The low speed of optic flow stimuli (8 and 11°/s) evoked non-uniform directions of oscillations especially in peripheral stimulation in all subjects, meaning that optic flow simulating slow self-motion stabilizes body sway.

Temporal synchrony effects of optic flow and vestibular inputs on multisensory heading perception.

Precise heading perception requires integration of optic flow and vestibular cues, yet the two cues often carry distinct temporal dynamics that may confound cue integration benefit. Here, we varied temporal offset between the two sensory inputs while macaques discriminated headings around straight ahead. We find the best heading performance does not occur under natural condition of synchronous inputs with zero offset but rather when visual stimuli are artificially adjusted to lead vestibular by a few hundreds of milliseconds. This amount exactly matches the lag between the vestibular acceleration and visual speed signals as measured from single-unit-activity in frontal and posterior parietal cortices. Manually aligning cues in these areas best facilitates integration with some nonlinear gain modulation effects. These findings are consistent with predictions from a model by which the brain integrates optic flow speed with a faster vestibular acceleration signal for sensing instantaneous heading direction during self-motion in the environment.

Humans navigate with stereo olfaction.

Human navigation relies on inputs to our paired eyes and ears. Although we also have two nasal passages, there has been little empirical indication that internostril differences yield directionality in human olfaction without involving the trigeminal system. By using optic flow that captures the pattern of apparent motion of surface elements in a visual scene, we demonstrate through formal psychophysical testing that a moderate binaral concentration disparity of a nontrigeminal odorant consistently biases recipients' perceived direction of self-motion toward the higher-concentration side, despite that they cannot verbalize which nostril smells a stronger odor. We further show that the effect depends on the internostril ratio of odor concentrations and not the numeric difference in concentration between the two nostrils. Taken together, our findings provide behavioral evidence that humans smell in stereo and subconsciously utilize stereo olfactory cues in spatial navigation.

Causal Evidence of Motion Signals in Macaque Middle Temporal Area Weighted-Pooled for Global Heading Perception.

Accurate heading perception relies on visual information integrated across a wide field, that is, optic flow. Numerous computational studies have speculated how local visual information might be pooled by the brain to compute heading, but these hypotheses lack direct neurophysiological support. In the current study, we instructed human and monkey subjects to judge heading directions based on global optic flow. We showed that a local perturbation cue applied within only a small part of the visual field could bias the subjects' heading judgments, and shift the neuronal tuning in the macaque middle temporal (MT) area at the same time. Electrical microstimulation in MT significantly biased the animals' heading judgments predictable from the tuning of the stimulated neurons. Masking the visual stimuli within these neurons' receptive fields could not remove the stimulation effect, indicating a sufficient role of the MT signals pooled by downstream neurons for global heading estimation. Interestingly, this pooling is not homogeneous because stimulating neurons with excitatory surrounds produced relatively larger effects than stimulating neurons with inhibitory surrounds. Thus our data not only provide direct causal evidence, but also new insights into the neural mechanisms of pooling local motion information for global heading estimation.

Clustering of heading selectivity and perception-related activity in the ventral intraparietal area.

The ventral intraparietal area (VIP) of the macaque brain is a multimodal cortical region, with many cells tuned to both optic flow and vestibular stimuli. Responses of many VIP neurons also show robust correlations with perceptual judgments during a fine heading discrimination task. Previous studies have shown that heading tuning based on optic flow is represented in a clustered fashion in VIP. However, it is unknown whether vestibular self-motion selectivity is clustered in VIP. Moreover, it is not known whether stimulus- and choice-related signals in VIP show clustering in the context of a heading discrimination task. To address these issues, we compared the response characteristics of isolated single units (SUs) with those of the undifferentiated multiunit (MU) activity corresponding to several neighboring neurons recorded from the same microelectrode. We find that MU activity typically shows selectivity similar to that of simultaneously recorded SUs, for both the vestibular and visual stimulus conditions. In addition, the choice-related activity of MU signals, as quantified using choice probabilities, is correlated with the choice-related activity of SUs. Overall, these findings suggest that both sensory and choice-related signals regarding self-motion are clustered in VIP. NEW & NOTEWORTHY We demonstrate, for the first time, that the vestibular tuning of ventral intraparietal area (VIP) neurons in response to both translational and rotational motion is clustered. In addition, heading discriminability and choice-related activity are also weakly clustered in VIP.

Evidence for a Causal Contribution of Macaque Vestibular, But Not Intraparietal, Cortex to Heading Perception.

Multisensory convergence of visual and vestibular signals has been observed within a network of cortical areas involved in representing heading. Vestibular-dominant heading tuning has been found in the macaque parietoinsular vestibular cortex (PIVC) and the adjacent visual posterior sylvian (VPS) area, whereas relatively balanced visual/vestibular tuning was encountered in the ventral intraparietal (VIP) area and visual-dominant tuning was found in the dorsal medial superior temporal (MSTd) area. Although the respective functional roles of these areas remain unclear, perceptual deficits in heading discrimination following reversible chemical inactivation of area MSTd area suggested that areas with vestibular-dominant heading tuning also contribute to behavior. To explore the roles of other areas in heading perception, muscimol injections were used to reversibly inactivate either the PIVC or the VIP area bilaterally in macaques. Inactivation of the anterior PIVC increased psychophysical thresholds when heading judgments were based on either optic flow or vestibular cues, although effects were stronger for vestibular stimuli. All behavioral deficits recovered within 36 h. Visual deficits were larger following inactivation of the posterior portion of the PIVC, likely because these injections encroached upon the VPS area, which contains neurons with optic flow tuning (unlike the PIVC). In contrast, VIP inactivation led to no behavioral deficits, despite the fact that VIP neurons show much stronger choice-related activity than MSTd neurons. These results suggest that the VIP area either provides a parallel and partially redundant pathway for this task, or does not participate in heading discrimination. In contrast, the PIVC/VPS area, along with the MSTd area, make causal contributions to heading perception based on either vestibular or visual signals. SIGNIFICANCE STATEMENT: Multisensory vestibular and visual signals are found in multiple cortical areas, but their causal contribution to self-motion perception has been previously tested only in the dorsal medial superior temporal (MSTd) area. In these experiments, we show that inactivation of the parietoinsular vestibular cortex (PIVC) also results in causal deficits during heading discrimination for both visual and vestibular cues. In contrast, ventral intraparietal (VIP) area inactivation led to no behavioral deficits, despite the fact that VIP neurons show much stronger choice-related activity than MSTd or PIVC neurons. These results demonstrate that choice-related activity does not always imply a causal role in sensory perception.

The temporal dynamics of heading perception in the presence of moving objects.

Many forms of locomotion rely on the ability to accurately perceive one's direction of locomotion (i.e., heading) based on optic flow. Although accurate in rigid environments, heading judgments may be biased when independently moving objects are present. The aim of this study was to systematically investigate the conditions in which moving objects influence heading perception, with a focus on the temporal dynamics and the mechanisms underlying this bias. Subjects viewed stimuli simulating linear self-motion in the presence of a moving object and judged their direction of heading. Experiments 1 and 2 revealed that heading perception is biased when the object crosses or almost crosses the observer's future path toward the end of the trial, but not when the object crosses earlier in the trial. Nonetheless, heading perception is not based entirely on the instantaneous optic flow toward the end of the trial. This was demonstrated in Experiment 3 by varying the portion of the earlier part of the trial leading up to the last frame that was presented to subjects. When the stimulus duration was long enough to include the part of the trial before the moving object crossed the observer's path, heading judgments were less biased. The findings suggest that heading perception is affected by the temporal evolution of optic flow. The time course of dorsal medial superior temporal area (MSTd) neuron responses may play a crucial role in perceiving heading in the presence of moving objects, a property not captured by many existing models.