Cortical Integration of Vestibular and Visual Cues for Navigation, Visual Processing, and Perception.

Despite increasing evidence of its involvement in several key functions of the cerebral cortex, the vestibular sense rarely enters our consciousness. Indeed, the extent to which these internal signals are incorporated within cortical sensory representation and how they might be relied upon for sensory-driven decision-making, during, for example, spatial navigation, is yet to be understood. Recent novel experimental approaches in rodents have probed both the physiological and behavioral significance of vestibular signals and indicate that their widespread integration with vision improves both the cortical representation and perceptual accuracy of self-motion and orientation. Here, we summarize these recent findings with a focus on cortical circuits involved in visual perception and spatial navigation and highlight the major remaining knowledge gaps. We suggest that vestibulo-visual integration reflects a process of constant updating regarding the status of self-motion, and access to such information by the cortex is used for sensory perception and predictions that may be implemented for rapid, navigation-related decision-making.

Perceptual Inference, Learning, and Attention in a Multisensory World.

Adaptive behavior in a complex, dynamic, and multisensory world poses some of the most fundamental computational challenges for the brain, notably inference, decision-making, learning, binding, and attention. We first discuss how the brain integrates sensory signals from the same source to support perceptual inference and decision-making by weighting them according to their momentary sensory uncertainties. We then show how observers solve the binding or causal inference problem-deciding whether signals come from common causes and should hence be integrated or else be treated independently. Next, we describe the multifarious interplay between multisensory processing and attention. We argue that attentional mechanisms are crucial to compute approximate solutions to the binding problem in naturalistic environments when complex time-varying signals arise from myriad causes. Finally, we review how the brain dynamically adapts multisensory processing to a changing world across multiple timescales.

Neuronal Population Encoding of Identity in Primate Prefrontal Cortex.

The ventrolateral prefrontal cortex (VLPFC) shows robust activation during the perception of faces and voices. However, little is known about what categorical features of social stimuli drive neural activity in this region. Since perception of identity and expression are critical social functions, we examined whether neural responses to naturalistic stimuli were driven by these two categorical features in the prefrontal cortex. We recorded single neurons in the VLPFC, while two male rhesus macaques (Macaca mulatta) viewed short audiovisual videos of unfamiliar conspecifics making expressions of aggressive, affiliative, and neutral valence. Of the 285 neurons responsive to the audiovisual stimuli, 111 neurons had a main effect (two-way ANOVA) of identity, expression, or their interaction in their stimulus-related firing rates; however, decoding of expression and identity using single-unit firing rates rendered poor accuracy. Interestingly, when decoding from pseudo-populations of recorded neurons, the accuracy for both expression and identity increased with population size, suggesting that the population transmitted information relevant to both variables. Principal components analysis of mean population activity across time revealed that population responses to the same identity followed similar trajectories in the response space, facilitating segregation from other identities. Our results suggest that identity is a critical feature of social stimuli that dictates the structure of population activity in the VLPFC, during the perception of vocalizations and their corresponding facial expressions. These findings enhance our understanding of the role of the VLPFC in social behavior.

Sex Differences in Low-Level Multisensory Integration in Developmental Dyslexia.

Reading acquisition involves the integration of auditory and visual stimuli. Thus, low-level audiovisual multisensory integration might contribute to disrupted reading in developmental dyslexia. Although dyslexia is more frequently diagnosed in males and emerging evidence indicates that the neural basis of dyslexia might differ between sexes, previous studies examining multisensory integration did not evaluate potential sex differences nor tested its neural correlates. In the current study on 88 adolescents and young adults, we found that only males with dyslexia showed a deficit in multisensory integration of simple nonlinguistic stimuli. At the neural level, both females and males with dyslexia presented smaller differences in response to multisensory compared to those in response to unisensory conditions in the N1 and N2 components (early components of event-related potentials associated with sensory processing) than the control group. Additionally, in a subsample of 80 participants matched for nonverbal IQ, only males with dyslexia exhibited smaller differences in the left hemisphere in response to multisensory compared to those in response to unisensory conditions in the N1 component. Our study indicates that deficits of multisensory integration seem to be more severe in males than females with dyslexia. This provides important insights into sex-modulated cognitive processes that might confer vulnerability to reading difficulties.

Visual Stimuli Modulate Local Field Potentials But Drive No High-Frequency Activity in Human Auditory Cortex.

Neuroimaging studies suggest cross-sensory visual influences in human auditory cortices (ACs). Whether these influences reflect active visual processing in human ACs, which drives neuronal firing and concurrent broadband high-frequency activity (BHFA; >70 Hz), or whether they merely modulate sound processing is still debatable. Here, we presented auditory, visual, and audiovisual stimuli to 16 participants (7 women, 9 men) with stereo-EEG depth electrodes implanted near ACs for presurgical monitoring. Anatomically normalized group analyses were facilitated by inverse modeling of intracranial source currents. Analyses of intracranial event-related potentials (iERPs) suggested cross-sensory responses to visual stimuli in ACs, which lagged the earliest auditory responses by several tens of milliseconds. Visual stimuli also modulated the phase of intrinsic low-frequency oscillations and triggered 15-30 Hz event-related desynchronization in ACs. However, BHFA, a putative correlate of neuronal firing, was not significantly increased in ACs after visual stimuli, not even when they coincided with auditory stimuli. Intracranial recordings demonstrate cross-sensory modulations, but no indication of active visual processing in human ACs.

Modality-Independent Effect of Gravity in Shaping the Internal Representation of 3D Space for Visual and Haptic Object Perception.

Visual and haptic perceptions of 3D shape are plagued by distortions, which are influenced by nonvisual factors, such as gravitational vestibular signals. Whether gravity acts directly on the visual or haptic systems or at a higher, modality-independent level of information processing remains unknown. To test these hypotheses, we examined visual and haptic 3D shape perception by asking male and female human subjects to perform a "squaring" task in upright and supine postures and in microgravity. Subjects adjusted one edge of a 3D object to match the length of another in each of the three canonical reference planes, and we recorded the matching errors to obtain a characterization of the perceived 3D shape. The results show opposing, body-centered patterns of errors for visual and haptic modalities, whose amplitudes are negatively correlated, suggesting that they arise in distinct, modality-specific representations that are nevertheless linked at some level. On the other hand, weightlessness significantly modulated both visual and haptic perceptual distortions in the same way, indicating a common, modality-independent origin for gravity's effects. Overall, our findings show a link between modality-specific visual and haptic perceptual distortions and demonstrate a role of gravity-related signals on a modality-independent internal representation of the body and peripersonal 3D space used to interpret incoming sensory inputs.

Balancing the Senses: Electrophysiological Responses Reveal the Interplay between Somatosensory and Visual Processing During Body-Related Multisensory Conflict.

In the study of bodily awareness, the predictive coding theory has revealed that our brain continuously modulates sensory experiences to integrate them into a unitary body representation. Indeed, during multisensory illusions (e.g., the rubber hand illusion, RHI), the synchronous stroking of the participant's concealed hand and a fake visible one creates a visuotactile conflict, generating a prediction error. Within the predictive coding framework, through sensory processing modulation, prediction errors are solved, inducing participants to feel as if touches originated from the fake hand, thus ascribing the fake hand to their own body. Here, we aimed to address sensory processing modulation under multisensory conflict, by disentangling somatosensory and visual stimuli processing that are intrinsically associated during the illusion induction. To this aim, we designed two EEG experiments, in which somatosensory- (SEPs; Experiment 1; N = 18; F = 10) and visual-evoked potentials (VEPs; Experiment 2; N = 18; F = 9) were recorded in human males and females following the RHI. Our results show that, in both experiments, ERP amplitude is significantly modulated in the illusion as compared with both control and baseline conditions, with a modality-dependent diametrical pattern showing decreased SEP amplitude and increased VEP amplitude. Importantly, both somatosensory and visual modulations occur in long-latency time windows previously associated with tactile and visual awareness, thus explaining the illusion of perceiving touch at the sight location. In conclusion, we describe a diametrical modulation of somatosensory and visual processing as the neural mechanism that allows maintaining a stable body representation, by restoring visuotactile congruency under the occurrence of multisensory conflicts.

Subcortical contributions to the sense of body ownership.

The sense of body ownership (i.e. the feeling that our body or its parts belong to us) plays a key role in bodily self-consciousness and is believed to stem from multisensory integration. Experimental paradigms such as the rubber hand illusion have been developed to allow the controlled manipulation of body ownership in laboratory settings, providing effective tools for investigating malleability in the sense of body ownership and the boundaries that distinguish self from other. Neuroimaging studies of body ownership converge on the involvement of several cortical regions, including the premotor cortex and posterior parietal cortex. However, relatively less attention has been paid to subcortical structures that may also contribute to body ownership perception, such as the cerebellum and putamen. Here, on the basis of neuroimaging and neuropsychological observations, we provide an overview of relevant subcortical regions and consider their potential role in generating and maintaining a sense of ownership over the body. We also suggest novel avenues for future research targeting the role of subcortical regions in making sense of the body as our own.

Neural correlates of object identity and reward outcome in the sensory cortical-hippocampal hierarchy: coding of motivational information in perirhinal cortex.

Neural circuits support behavioral adaptations by integrating sensory and motor information with reward and error-driven learning signals, but it remains poorly understood how these signals are distributed across different levels of the corticohippocampal hierarchy. We trained rats on a multisensory object-recognition task and compared visual and tactile responses of simultaneously recorded neuronal ensembles in somatosensory cortex, secondary visual cortex, perirhinal cortex, and hippocampus. The sensory regions primarily represented unisensory information, whereas hippocampus was modulated by both vision and touch. Surprisingly, the sensory cortices and the hippocampus coded object-specific information, whereas the perirhinal cortex did not. Instead, perirhinal cortical neurons signaled trial outcome upon reward-based feedback. A majority of outcome-related perirhinal cells responded to a negative outcome (reward omission), whereas a minority of other cells coded positive outcome (reward delivery). Our results highlight a distributed neural coding of multisensory variables in the cortico-hippocampal hierarchy. Notably, the perirhinal cortex emerges as a crucial region for conveying motivational outcomes, whereas distinct functions related to object identity are observed in the sensory cortices and hippocampus.

Enhancement of speech-in-noise comprehension through vibrotactile stimulation at the syllabic rate.

SignificanceSyllables are important building blocks of speech. They occur at a rate between 4 and 8 Hz, corresponding to the theta frequency range of neural activity in the cerebral cortex. When listening to speech, the theta activity becomes aligned to the syllabic rhythm, presumably aiding in parsing a speech signal into distinct syllables. However, this neural activity cannot only be influenced by sound, but also by somatosensory information. Here, we show that the presentation of vibrotactile signals at the syllabic rate can enhance the comprehension of speech in background noise. We further provide evidence that this multisensory enhancement of speech comprehension reflects the multisensory integration of auditory and tactile information in the auditory cortex.

Predictive coding, multisensory integration, and attentional control: A multicomponent framework for lucid dreaming.

Lucid dreaming (LD) is a mental state in which we realize not being awake but are dreaming while asleep. It often involves vivid, perceptually intense dream images as well as peculiar kinesthetic sensations, such as flying, levitating, or out-of-body experiences. LD is in the cross-spotlight of cognitive neuroscience and sleep research as a particular case to study consciousness, cognition, and the neural background of dream experiences. Here, we present a multicomponent framework for the study and understanding of neurocognitive mechanisms and phenomenological aspects of LD. We propose that LD is associated with prediction error signals arising during sleep and occurring at higher or lower levels of the processing hierarchy. Prediction errors are resolved by generating a superordinate self-model able to integrate ambiguous stimuli arriving from sensory periphery and higher-order cortical regions. While multisensory integration enables lucidity maintenance and contributes to peculiar kinesthetic experiences, attentional control facilitates multisensory integration by dynamically regulating the balance between the influence of top-down mental models and the precision weighting of bottom-up sensory inputs. Our novel framework aims to link neural correlates of LD with current concepts of sleep and arousal regulation and provide testable predictions on interindividual differences in LD as well as neurocognitive mechanisms inducing lucid dreams.

Multiple representations of the body schema for the same body part.

Purposeful motor actions depend on the brain's representation of the body, called the body schema, and disorders of the body schema have been reported to show motor deficits. The body schema has been assumed for almost a century to be a common body representation supporting all types of motor actions, and previous studies have considered only a single motor action. Although we often execute multiple motor actions, how the body schema operates during such actions is unknown. To address this issue, I developed a technique to measure the body schema during multiple motor actions. Participants made simultaneous eye and reach movements to the same location of 10 landmarks on their hand. By analyzing the internal configuration of the locations of these points for each of the eye and reach movements, I produced maps of the mental representation of hand shape. Despite these two movements being simultaneously directed to the same bodily location, the resulting hand map (i.e., a part of the body schema) was much more distorted for reach movements than for eye movements. Furthermore, the weighting of visual and proprioceptive bodily cues to build up this part of the body schema differed for each effector. These results demonstrate that the body schema is organized as multiple effector-specific body representations. I propose that the choice of effector toward one's body can determine which body representation in the brain is observed and that this visualization approach may offer a new way to understand patients' body schema.

Hierarchical unimodal processing within the primary somatosensory cortex during a bimodal detection task.

Do sensory cortices process more than one sensory modality? To answer these questions, scientists have generated a wide variety of studies at distinct space-time scales in different animal models, and often shown contradictory conclusions. Some conclude that this process occurs in early sensory cortices, but others that this occurs in areas central to sensory cortices. Here, we sought to determine whether sensory neurons process and encode physical stimulus properties of different modalities (tactile and acoustic). For this, we designed a bimodal detection task where the senses of touch and hearing compete from trial to trial. Two Rhesus monkeys performed this novel task, while neural activity was recorded in areas 3b and 1 of the primary somatosensory cortex (S1). We analyzed neurons' coding properties and variability, organizing them by their receptive field's position relative to the stimulation zone. Our results indicate that neurons of areas 3b and 1 are unimodal, encoding only the tactile modality in both the firing rate and variability. Moreover, we found that neurons in area 3b carried more information about the periodic stimulus structure than those in area 1, possessed lower response and coding latencies, and had a lower intrinsic time scale. In sum, these differences reveal a hidden processing-based hierarchy. Finally, using a powerful nonlinear dimensionality reduction algorithm, we show that the activity from areas 3b and 1 can be separated, establishing a clear division in the functionality of these two subareas of S1.

Modality-specific effects of threat on self-motion perception.

BACKGROUND: Threat and individual differences in threat-processing bias perception of stimuli in the environment. Yet, their effect on perception of one's own (body-based) self-motion in space is unknown. Here, we tested the effects of threat on self-motion perception using a multisensory motion simulator with concurrent threatening or neutral auditory stimuli. RESULTS: Strikingly, threat had opposite effects on vestibular and visual self-motion perception, leading to overestimation of vestibular, but underestimation of visual self-motions. Trait anxiety tended to be associated with an enhanced effect of threat on estimates of self-motion for both modalities. CONCLUSIONS: Enhanced vestibular perception under threat might stem from shared neural substrates with emotional processing, whereas diminished visual self-motion perception may indicate that a threatening stimulus diverts attention away from optic flow integration. Thus, threat induces modality-specific biases in everyday experiences of self-motion.

A Crayfish-Inspired Sensor Fusion Platform for Super Additive Integration of Visual, Chemical, and Tactile Information.

The concept of cross-sensor modulation, wherein one sensor modality can influence another's response, is often overlooked in traditional sensor fusion architectures, leading to missed opportunities for enhancing data accuracy and robustness. In contrast, biological systems, such as aquatic animals like crayfish, demonstrate superior sensor fusion through multisensory integration. These organisms adeptly integrate visual, tactile, and chemical cues to perform tasks such as evading predators and locating prey. Drawing inspiration from this, we propose a neuromorphic platform that integrates graphene-based chemitransistors, monolayer molybdenum disulfide (MoS2) based photosensitive memtransistors, and triboelectric tactile sensors to achieve "Super-Additive" responses to weak chemical, visual, and tactile cues and demonstrate contextual response modulation, also referred to as the "Inverse Effectiveness Effect." We hold the view that integrating bio-inspired sensor fusion principles across various modalities holds promise for a wide range of applications.

Learning-Induced Odor Modulation of Neuronal Activity in Auditory Cortex.

Sensory cortices, even of primary regions, are not purely unisensory. Rather, cortical neurons in sensory cortex show various forms of multisensory interactions. While some multisensory interactions naturally co-occur, the combination of others will co-occur through experience. In real life, learning and experience will result in conjunction with seemingly disparate sensory information that ultimately becomes behaviorally relevant, impacting perception, cognition, and action. Here we describe a novel auditory discrimination task in mice, designed to manipulate the expectation of upcoming trials using olfactory cues. We show that, after learning, female mice display a transient period of several days during which they exploit odor-mediated expectations for making correct decisions. Using two-photon calcium imaging of single neurons in auditory cortex (ACx) during behavior, we found that the behavioral effects of odor-mediated expectations are accompanied by an odor-induced modulation of neuronal activity. Further, we find that these effects are manifested differentially, based on the response preference of individual cells. A significant portion of effects, but not all, are consistent with a predictive coding framework. Our data show that learning novel odor-sound associations evoke changes in ACx. We suggest that behaviorally relevant multisensory environments mediate contextual effects as early as ACx.SIGNIFICANCE STATEMENT Natural environments are composed of multisensory objects. It remains unclear whether and how animals learn the regularities of congruent multisensory associations and how these may impact behavior and neural activity. We tested how learned odor-sound associations affected single-neuron responses in auditory cortex. We introduce a novel auditory discrimination task for mice in which odors set different contexts of expectation to upcoming trials. We show that, although the task can be solved purely by sounds, odor-mediated expectation impacts performance. We further show that odors cause a modulation of neuronal activity in auditory cortex, which is correlated with behavior. These results suggest that learning prompts an interaction of odor and sound information as early as sensory cortex.

Neural Substrates of Body Ownership and Agency during Voluntary Movement.

Body ownership and the sense of agency are two central aspects of bodily self-consciousness. While multiple neuroimaging studies have investigated the neural correlates of body ownership and agency separately, few studies have investigated the relationship between these two aspects during voluntary movement when such experiences naturally combine. By eliciting the moving rubber hand illusion with active or passive finger movements during functional magnetic resonance imaging, we isolated activations reflecting the sense of body ownership and agency, respectively, as well as their interaction, and assessed their overlap and anatomic segregation. We found that perceived hand ownership was associated with activity in premotor, posterior parietal, and cerebellar regions, whereas the sense of agency over the movements of the hand was related to activity in the dorsal premotor cortex and superior temporal cortex. Moreover, one section of the dorsal premotor cortex showed overlapping activity for ownership and agency, and somatosensory cortical activity reflected the interaction of ownership and agency with higher activity when both agency and ownership were experienced. We further found that activations previously attributed to agency in the left insular cortex and right temporoparietal junction reflected the synchrony or asynchrony of visuoproprioceptive stimuli rather than agency. Collectively, these results reveal the neural bases of agency and ownership during voluntary movement. Although the neural representations of these two experiences are largely distinct, there are interactions and functional neuroanatomical overlap during their combination, which has bearing on theories on bodily self-consciousness.SIGNIFICANCE STATEMENT How does the brain generate the sense of being in control of bodily movement (agency) and the sense that body parts belong to one's body (body ownership)? Using fMRI and a bodily illusion triggered by movement, we found that agency is associated with activity in premotor cortex and temporal cortex, and body ownership with activity in premotor, posterior parietal, and cerebellar regions. The activations reflecting the two sensations were largely distinct, but there was overlap in premotor cortex and an interaction in somatosensory cortex. These findings advance our understanding of the neural bases of and interplay between agency and body ownership during voluntary movement, which has implications for the development of advanced controllable prosthetic limbs that feel like real limbs.

Enhancing learning outcomes through multisensory integration: A fMRI study of audio-visual training in virtual reality.

The integration of information from different sensory modalities is a fundamental process that enhances perception and performance in real and virtual environments (VR). Understanding these mechanisms, especially during learning tasks that exploit novel multisensory cue combinations provides opportunities for the development of new rehabilitative interventions. This study aimed to investigate how functional brain changes support behavioural performance improvements during an audio-visual (AV) learning task. Twenty healthy participants underwent a 30 min daily VR training for four weeks. The task was an AV adaptation of a 'scanning training' paradigm that is commonly used in hemianopia rehabilitation. Functional magnetic resonance imaging (fMRI) and performance data were collected at baseline, after two and four weeks of training, and four weeks post-training. We show that behavioural performance, operationalised as mean reaction time reduction in VR, significantly improves. In separate tests in a controlled laboratory environment, we showed that the behavioural performance gains in the VR training environment transferred to a significant mean RT reduction for the trained AV voluntary task on a computer screen. Enhancements were observed in both the visual-only and AV conditions, with the latter demonstrating a faster response time supported by the presence of audio cues. The behavioural learning effect also transfers to two additional tasks that were tested: a visual search task and an involuntary visual task. Our fMRI results reveal an increase in functional activation (BOLD signal) in multisensory brain regions involved in early-stage AV processing: the thalamus, the caudal inferior parietal lobe and cerebellum. These functional changes were only observed for the trained, multisensory, task and not for unimodal visual stimulation. Functional activation changes in the thalamus were significantly correlated to behavioural performance improvements. This study demonstrates that incorporating spatial auditory cues to voluntary visual training in VR leads to augmented brain activation changes in multisensory integration, resulting in measurable performance gains across tasks. The findings highlight the potential of VR-based multisensory training as an effective method for enhancing cognitive function and as a potentially valuable tool in rehabilitative programmes.

Unravelling the multisensory learning advantage: Different patterns of within and across frequency-specific interactions drive uni- and multisensory neuroplasticity.

In the field of learning theory and practice, the superior efficacy of multisensory learning over uni-sensory is well-accepted. However, the underlying neural mechanisms at the macro-level of the human brain remain largely unexplored. This study addresses this gap by providing novel empirical evidence and a theoretical framework for understanding the superiority of multisensory learning. Through a cognitive, behavioral, and electroencephalographic assessment of carefully controlled uni-sensory and multisensory training interventions, our study uncovers a fundamental distinction in their neuroplastic patterns. A multilayered network analysis of pre- and post- training EEG data allowed us to model connectivity within and across different frequency bands at the cortical level. Pre-training EEG analysis unveils a complex network of distributed sources communicating through cross-frequency coupling, while comparison of pre- and post-training EEG data demonstrates significant differences in the reorganizational patterns of uni-sensory and multisensory learning. Uni-sensory training primarily modifies cross-frequency coupling between lower and higher frequencies, whereas multisensory training induces changes within the beta band in a more focused network, implying the development of a unified representation of audiovisual stimuli. In combination with behavioural and cognitive findings this suggests that, multisensory learning benefits from an automatic top-down transfer of training, while uni-sensory training relies mainly on limited bottom-up generalization. Our findings offer a compelling theoretical framework for understanding the advantage of multisensory learning.

The multisensory and multidimensional nature of object representation.

Recent functional magnetic resonance imaging (fMRI) experiments revealed similar neural representations across different types of two-dimensional (2-D) visual stimuli; however, real three-dimensional (3-D) objects affording action differentially affect neural activation and behavioral results relative to 2-D objects. Recruitment of multiple sensory regions during unisensory (visual, haptic, and auditory) object shape tasks suggests that shape representation may be modality invariant. This mini-review explores the overlapping neural regions involved in object shape representation, across 2-D, 3-D, visual, and haptic experiments.