Neural correlates of audiovisual narrative speech perception in children and adults on the autism spectrum: A functional magnetic resonance imaging study.

Autistic individuals show substantially reduced benefit from observing visual articulations during audiovisual speech perception, a multisensory integration deficit that is particularly relevant to social communication. This has mostly been studied using simple syllabic or word-level stimuli and it remains unclear how altered lower-level multisensory integration translates to the processing of more complex natural multisensory stimulus environments in autism. Here, functional neuroimaging was used to examine neural correlates of audiovisual gain (AV-gain) in 41 autistic individuals to those of 41 age-matched non-autistic controls when presented with a complex audiovisual narrative. Participants were presented with continuous narration of a story in auditory-alone, visual-alone, and both synchronous and asynchronous audiovisual speech conditions. We hypothesized that previously identified differences in audiovisual speech processing in autism would be characterized by activation differences in brain regions well known to be associated with audiovisual enhancement in neurotypicals. However, our results did not provide evidence for altered processing of auditory alone, visual alone, audiovisual conditions or AV- gain in regions associated with the respective task when comparing activation patterns between groups. Instead, we found that autistic individuals responded with higher activations in mostly frontal regions where the activation to the experimental conditions was below baseline (de-activations) in the control group. These frontal effects were observed in both unisensory and audiovisual conditions, suggesting that these altered activations were not specific to multisensory processing but reflective of more general mechanisms such as an altered disengagement of Default Mode Network processes during the observation of the language stimulus across conditions.

Older adults preserve audiovisual integration through enhanced cortical activations, not by recruiting new regions.

Effective interactions with the environment rely on the integration of multisensory signals: Our brains must efficiently combine signals that share a common source, and segregate those that do not. Healthy ageing can change or impair this process. This functional magnetic resonance imaging study assessed the neural mechanisms underlying age differences in the integration of auditory and visual spatial cues. Participants were presented with synchronous audiovisual signals at various degrees of spatial disparity and indicated their perceived sound location. Behaviourally, older adults were able to maintain localisation accuracy. At the neural level, they integrated auditory and visual cues into spatial representations along dorsal auditory and visual processing pathways similarly to their younger counterparts but showed greater activations in a widespread system of frontal, temporal, and parietal areas. According to multivariate Bayesian decoding, these areas encoded critical stimulus information beyond that which was encoded in the brain areas commonly activated by both groups. Surprisingly, however, the boost in information provided by these areas with age-related activation increases was comparable across the 2 age groups. This dissociation-between comparable information encoded in brain activation patterns across the 2 age groups, but age-related increases in regional blood-oxygen-level-dependent responses-contradicts the widespread notion that older adults recruit new regions as a compensatory mechanism to encode task-relevant information. Instead, our findings suggest that activation increases in older adults reflect nonspecific or modulatory mechanisms related to less efficient or slower processing, or greater demands on attentional resources.

Cortical depth profiles in primary visual cortex for illusory and imaginary experiences.

Visual illusions and mental imagery are non-physical sensory experiences that involve cortical feedback processing in the primary visual cortex. Using laminar functional magnetic resonance imaging (fMRI) in two studies, we investigate if information about these internal experiences is visible in the activation patterns of different layers of primary visual cortex (V1). We find that imagery content is decodable mainly from deep layers of V1, whereas seemingly 'real' illusory content is decodable mainly from superficial layers. Furthermore, illusory content shares information with perceptual content, whilst imagery content does not generalise to illusory or perceptual information. Together, our results suggest that illusions and imagery, which differ immensely in their subjective experiences, also involve partially distinct early visual microcircuits. However, overlapping microcircuit recruitment might emerge based on the nuanced nature of subjective conscious experience.

Attention Drives Visual Processing and Audiovisual Integration During Multimodal Communication.

During communication in real-life settings, our brain often needs to integrate auditory and visual information and at the same time actively focus on the relevant sources of information, while ignoring interference from irrelevant events. The interaction between integration and attention processes remains poorly understood. Here, we use rapid invisible frequency tagging and magnetoencephalography to investigate how attention affects auditory and visual information processing and integration, during multimodal communication. We presented human participants (male and female) with videos of an actress uttering action verbs (auditory; tagged at 58 Hz) accompanied by two movie clips of hand gestures on both sides of fixation (attended stimulus tagged at 65 Hz; unattended stimulus tagged at 63 Hz). Integration difficulty was manipulated by a lower-order auditory factor (clear/degraded speech) and a higher-order visual semantic factor (matching/mismatching gesture). We observed an enhanced neural response to the attended visual information during degraded speech compared to clear speech. For the unattended information, the neural response to mismatching gestures was enhanced compared to matching gestures. Furthermore, signal power at the intermodulation frequencies of the frequency tags, indexing nonlinear signal interactions, was enhanced in the left frontotemporal and frontal regions. Focusing on the left inferior frontal gyrus, this enhancement was specific for the attended information, for those trials that benefitted from integration with a matching gesture. Together, our results suggest that attention modulates audiovisual processing and interaction, depending on the congruence and quality of the sensory input.

Effects of sensory modality and task relevance on omitted stimulus potentials.

Omitted stimulus potentials (OSPs) occur when a sensory stimulus is unexpectedly omitted. They are thought to reflect predictions about upcoming sensory events. The present study examined how OSPs differ across the sensory modalities of predicted stimuli. Twenty-nine university students were asked to press a mouse button at a regular interval of 1-2 s, which was immediately followed by either a visual or auditory stimulus in different blocks. The stimuli were sometimes omitted (p = 0.2), to which event-related potentials (ERPs) were recorded. The results showed that stimulus omissions in both modalities elicited ERP waveforms consisting of three components, oN1, oN2, and oP3. The peak latencies of these components were shorter in the auditory modality than in the visual modality. The amplitudes of OSPs were larger when participants were told that the omission indicated their poor performance (i.e., they pressed a button at an irregular interval) than when it was irrelevant to their performance. These findings suggest that OSPs occur from around 100 ms in a modality-specific manner and increase in amplitude depending on the task relevance of stimulus omissions.

Inducing visual attention through audiovisual stimuli: Can synchronous sound be a salient event?

We present an experimental research aiming to explore how spatial attention may be biased through auditory stimuli. In particular, we investigate how synchronous sound and image may affect attention and increase the saliency of the audiovisual event. We have designed and implemented an experimental study where subjects, wearing an eye-tracking system, were examined regarding their gaze toward the audiovisual stimuli being displayed. The audiovisual stimuli were specifically tailored for this experiment, consisting of videos contrasting in terms of Synch Points (i.e., moments where a visual event is associated with a visible trigger movement, synchronous with its correspondent sound). While consistency across audiovisual sensory modalities revealed to be an attention-drawing feature, when combined with synchrony, it clearly emphasized the biasing, triggering orienting, that is, focal attention towards the particular scene that contains the Synch Point. Consequently, results revealed synchrony to be a saliency factor, contributing to the strengthening of the focal attention.

Comparison of uni- and multimodal motion stimulation on visual neglect: A proof-of-concept study.

Spatial neglect is characterized by the failure to attend stimuli presented in the contralesional space. Typically, the visual modality is more severely impaired than the auditory one. This dissociation offers the possibility of cross-modal interactions, whereby auditory stimuli may have beneficial effects on the visual modality. A new auditory motion stimulation method with music dynamically moving from the right to the left hemispace has recently been shown to improve visual neglect. The aim of the present study was twofold: a) to compare the effects of unimodal auditory against visual motion stimulation, i.e., smooth pursuit training, which is an established therapeutical approach in neglect therapy and b) to explore whether a combination of auditory + visual motion stimulation, i.e., multimodal motion stimulation, would be more effective than unimodal auditory or visual motion stimulation. 28 patients with left-sided neglect due to a first-ever, right-hemispheric subacute stroke were included. Patients either received auditory, visual, or multimodal motion stimulation. The between-group effect of each motion stimulation condition as well as a control group without motion stimulation was investigated by means of a one-way ANOVA with the patient's visual exploration behaviour as an outcome variable. Our results showed that unimodal auditory motion stimulation is equally effective as unimodal visual motion stimulation: both interventions significantly improved neglect compared to the control group. Multimodal motion stimulation also significantly improved neglect, however, did not show greater improvement than unimodal auditory or visual motion stimulation alone. Besides the established visual motion stimulation, this proof-of-concept study suggests that auditory motion stimulation seems to be an alternative promising therapeutic approach to improve visual attention in neglect patients. Multimodal motion stimulation does not lead to any additional therapeutic gain. In neurorehabilitation, the implementation of either auditory or visual motion stimulation seems therefore reasonable.

Exploring the effect of photobiomodulation and gamma visual stimulation induced by 808 nm and visible LED in Alzheimer's disease mouse model.

Although photobiomodulation (PBM) and gamma visual stimulatqion (GVS) have been overwhelmingly explored in the recent time as a possible light stimulation (LS) means of Alzheimer's disease (AD) therapy, their effects have not been assessed at once. In our research, the AD mouse model was stimulated using light with various parameters [continuous wave (PBM) or 40 Hz pulsed visible LED (GVS) or 40 Hz pulsed 808 nm LED (PBM and GVS treatment)]]. The brain slices collected from the LS treated AD model mice were evaluated using (i) fluorescence microscopy to image thioflavine-S labeled amy-loid-ß (Aß) plaques (the main hallmark of AD), or (ii) two-photon excited fluorescence (TPEF) imaging of unlabeled Aß plaques, showing that the amount of Aß plaques was reduced after LS treatment. The imaging results correlated well with the results of Morris water maze (MWM) test, which demonstrated that the spatial learning and memory abilities of LS treated mice were noticeably higher than those of untreated mice. The LS effect was also assessed by in vivo nonlinear optical imaging, revealing that the cerebral amyloid angiopathy decreased spe-cifically as a result of 40 Hz pulsed 808 nm irradiation, on the contrary, the angiopathy reversed after visible 40 Hz pulsed light treatment. The obtained results provide useful reference for further optimization of the LS (PBM or GVS) parameters to achieve efficient phototherapy of AD.

The neural oscillations in delta- and theta-bands contribute to divided attention in audiovisual integration.

One of key mechanisms implicated in multisensory processing is neural oscillations in distinct frequency band. Many studies explored the modulation of attention by recording the electroencephalography signals when subjects attended one modality, and ignored the other modality input. However, when attention is directed toward one modality, it may be not always possible to shut out completely inputs from a different modality. Since many situations require division of attention between audition and vision, it is imperative to investigate the neural mechanisms underlying processing of concurrent auditory and visual sensory streams. In the present study, we designed a task of audiovisual semantic discrimination, in which the subjects were asked to share attention to both auditory and visual stimuli. We explored the contribution of neural oscillations in lower-frequency to the modulation of divided attention on audiovisual integration. Our results implied that theta-band activity contributes to the early modulation of divided attention, and delta-band activity contributes to the late modulation of divided attention to audiovisual integration. Moreover, the fronto-central delta- and theta-bands activity is likely a marker of divided attention in audiovisual integration, and the neural oscillation on delta- and theta-bands is conducive to allocating attention resources to dual-tasking involving task-coordinating abilities.

Visual Stimuli Modulate Local Field Potentials But Drive No High-Frequency Activity in Human Auditory Cortex.

Neuroimaging studies suggest cross-sensory visual influences in human auditory cortices (ACs). Whether these influences reflect active visual processing in human ACs, which drives neuronal firing and concurrent broadband high-frequency activity (BHFA; >70 Hz), or whether they merely modulate sound processing is still debatable. Here, we presented auditory, visual, and audiovisual stimuli to 16 participants (7 women, 9 men) with stereo-EEG depth electrodes implanted near ACs for presurgical monitoring. Anatomically normalized group analyses were facilitated by inverse modeling of intracranial source currents. Analyses of intracranial event-related potentials (iERPs) suggested cross-sensory responses to visual stimuli in ACs, which lagged the earliest auditory responses by several tens of milliseconds. Visual stimuli also modulated the phase of intrinsic low-frequency oscillations and triggered 15-30 Hz event-related desynchronization in ACs. However, BHFA, a putative correlate of neuronal firing, was not significantly increased in ACs after visual stimuli, not even when they coincided with auditory stimuli. Intracranial recordings demonstrate cross-sensory modulations, but no indication of active visual processing in human ACs.

Entrainment and resonance effects with a new mobile audio-visual stimulation device.

Entrainment and photic driving effects triggered by repetitive visual stimulation are long-established in clinical and therapeutic scenarios. Nonetheless, such stimulation patterns are typically bound to stationary clinical and laboratory applications. We investigated the effects of repetitive stimulation with a new dynamic auditory-visual stimulation pattern using a novel consumer-grade stimulation device for home application.Fourteen volunteers (study group) received 16 sessions of combined auditory-visual stimulation during four weeks. An additional control group (seven volunteers) received auditory-only stimulation for two sessions. From 64-channel electroencephalography recordings, we compared individual alpha peak frequencies (iAPF) between week one and week four as well as power values from the time-frequency analysis.The novel stimulation device yielded stable entrainment and resonance effects for all investigated stimulation frequencies. Both groups showed no differences in their iAPFs between weeks one and four. The power comparison suggests that there are similar entrainment and resonance effects in week one and week four within the study group.We conclude that the novel portable consumer-grade stimulation device is suitable for home-based auditory-visual stimulation leading to consistent entrainment and resonance effects over the course of 16 stimulation sessions over four weeks.

Competing influence of visual speech on auditory neural adaptation.

Visual information from a speaker's face enhances auditory neural processing and speech recognition. To determine whether auditory memory can be influenced by visual speech, the degree of auditory neural adaptation of an auditory syllable preceded by an auditory, visual, or audiovisual syllable was examined using EEG. Consistent with previous findings and additional adaptation of auditory neurons tuned to acoustic features, stronger adaptation of N1, P2 and N2 auditory evoked responses was observed when the auditory syllable was preceded by an auditory compared to a visual syllable. However, although stronger than when preceded by a visual syllable, lower adaptation was observed when the auditory syllable was preceded by an audiovisual compared to an auditory syllable. In addition, longer N1 and P2 latencies were then observed. These results further demonstrate that visual speech acts on auditory memory but suggest competing visual influences in the case of audiovisual stimulation.

Optogenetic activation of visual thalamus generates artificial visual percepts.

The lateral geniculate nucleus (LGN), a retinotopic relay center where visual inputs from the retina are processed and relayed to the visual cortex, has been proposed as a potential target for artificial vision. At present, it is unknown whether optogenetic LGN stimulation is sufficient to elicit behaviorally relevant percepts, and the properties of LGN neural responses relevant for artificial vision have not been thoroughly characterized. Here, we demonstrate that tree shrews pretrained on a visual detection task can detect optogenetic LGN activation using an AAV2-CamKIIα-ChR2 construct and readily generalize from visual to optogenetic detection. Simultaneous recordings of LGN spiking activity and primary visual cortex (V1) local field potentials (LFPs) during optogenetic LGN stimulation show that LGN neurons reliably follow optogenetic stimulation at frequencies up to 60 Hz and uncovered a striking phase locking between the V1 LFP and the evoked spiking activity in LGN. These phase relationships were maintained over a broad range of LGN stimulation frequencies, up to 80 Hz, with spike field coherence values favoring higher frequencies, indicating the ability to relay temporally precise information to V1 using light activation of the LGN. Finally, V1 LFP responses showed sensitivity values to LGN optogenetic activation that were similar to the animal's behavioral performance. Taken together, our findings confirm the LGN as a potential target for visual prosthetics in a highly visual mammal closely related to primates.

Effects of visual-electrotactile stimulation feedback on brain functional connectivity during motor imagery practice.

The use of neurofeedback is an important aspect of effective motor rehabilitation as it offers real-time sensory information to promote neuroplasticity. However, there is still limited knowledge about how the brain's functional networks reorganize in response to such feedback. To address this gap, this study investigates the reorganization of the brain network during motor imagery tasks when subject to visual stimulation or visual-electrotactile stimulation feedback. This study can provide healthcare professionals with a deeper understanding of the changes in the brain network and help develop successful treatment approaches for brain-computer interface-based motor rehabilitation applications. We examine individual edges, nodes, and the entire network, and use the minimum spanning tree algorithm to construct a brain network representation using a functional connectivity matrix. Furthermore, graph analysis is used to detect significant features in the brain network that might arise in response to the feedback. Additionally, we investigate the power distribution of brain activation patterns using power spectral analysis and evaluate the motor imagery performance based on the classification accuracy. The results showed that the visual and visual-electrotactile stimulation feedback induced subject-specific changes in brain activation patterns and network reorganization in the [Formula: see text] band. Thus, the visual-electrotactile stimulation feedback significantly improved the integration of information flow between brain regions associated with motor-related commands and higher-level cognitive functions, while reducing cognitive workload in the sensory areas of the brain and promoting positive emotions. Despite these promising results, neither neurofeedback modality resulted in a significant improvement in classification accuracy, compared with the absence of feedback. These findings indicate that multimodal neurofeedback can modulate imagery-mediated rehabilitation by enhancing motor-cognitive communication and reducing cognitive effort. In future interventions, incorporating this technique to ease cognitive demands for participants could be crucial for maintaining their motivation to engage in rehabilitation.

Motion-Binding Property Contributes to Accurate Temporal-Order Perception in Audiovisual Synchrony.

Temporal perception in multisensory processing is important for an accurate and efficient understanding of the physical world. In general, it is executed in a dynamic environment in our daily lives. In particular, the motion-binding property is important for correctly identifying moving objects in the external environment. However, how this property affects multisensory temporal perception remains unclear. We investigate whether the motion-binding property influences audiovisual temporal integration. The study subjects performed four types of temporal-order judgment (TOJ) task experiments using three types of perception. In Experiment 1, the subjects conducted audiovisual TOJ tasks in the motion-binding condition, between two flashes, and in the simultaneous condition, in which the two flashes are perceived as simultaneous stimuli without motion. In Experiment 2, subjects conducted audiovisual TOJ tasks in the motion-binding condition and the short and long successive interval condition, in which the two stimuli are perceived as successive with no motion. The results revealed that the point of subjective simultaneity (PSS) and the just-noticeable difference (JND) in the motion-binding condition differed significantly from those in the simultaneous and short and long successive interval conditions. Specifically, the PSS in the motion-binding condition was shifted toward a sound-lead stimulus in which the PSS became closer to zero (i.e., physical simultaneity) and the JND became narrower compared to other conditions. This suggests that the motion-binding property contributes to accurate temporal integration in multisensory processing by precisely encoding the temporal order of the physical stimuli.

What Makes the Detection of Movement Different Within the Autistic Traits Spectrum? Evidence From the Audiovisual Depth Paradigm.

Atypical sensory processing is now considered a diagnostic feature of autism. Although multisensory integration (MSI) may have cascading effects on the development of higher-level skills such as socio-communicative functioning, there is a clear lack of understanding of how autistic individuals integrate multiple sensory inputs. Multisensory dynamic information is a more ecological construct than static stimuli, reflecting naturalistic sensory experiences given that our environment involves moving stimulation of more than one sensory modality at a time. In particular, depth movement informs about crucial social (approaching to interact) and non-social (avoiding threats/collisions) information. As autistic characteristics are distributed on a spectrum over clinical and general populations, our work aimed to explore the multisensory integration of depth cues in the autistic personality spectrum, using a go/no-go detection task. The autistic profile of 38 participants from the general population was assessed using questionnaires extensively used in the literature. Participants performed a detection task of auditory and/or visual depth moving stimuli compared to static stimuli. We found that subjects with high-autistic traits overreacted to depth movement and exhibited faster reaction times to audiovisual cues, particularly when the audiovisual stimuli were looming and/or were presented at a fast speed. These results provide evidence of sensory particularities in people with high-autistic traits and suggest that low-level stages of multisensory integration could operate differently all along the autistic personality spectrum.

The role of conflict processing in multisensory perception: behavioural and electroencephalography evidence.

To form coherent multisensory perceptual representations, the brain must solve a causal inference problem: to decide if two sensory cues originated from the same event and should be combined, or if they came from different events and should be processed independently. According to current models of multisensory integration, during this process, the integrated (common cause) and segregated (different causes) internal perceptual models are entertained. In the present study, we propose that the causal inference process involves competition between these alternative perceptual models that engages the brain mechanisms of conflict processing. To test this hypothesis, we conducted two experiments, measuring reaction times (RTs) and electroencephalography, using an audiovisual ventriloquist illusion paradigm with varying degrees of intersensory disparities. Consistent with our hypotheses, incongruent trials led to slower RTs and higher fronto-medial theta power, both indicative of conflict. We also predicted that intermediate disparities would yield slower RTs and higher theta power when compared to congruent stimuli and to large disparities, owing to the steeper competition between causal models. Although this prediction was only validated in the RT study, both experiments displayed the anticipated trend. In conclusion, our findings suggest a potential involvement of the conflict mechanisms in multisensory integration of spatial information. This article is part of the theme issue 'Decision and control processes in multisensory perception'.

Physiological Effects of Visual Stimulation by a Japanese Low Wooden Table: A Crossover Field Experiment.

The purpose of this study was to evaluate the physiological effects of visual stimulation by a unique Japanese low wooden table on the prefrontal cortex and autonomic nervous activities. A within-participants experiment with 26 male university students was conducted in a Japanese-style room. The visual stimuli were a low wooden table (WT) made of Japanese cypress and a low cloth-covered table (control) for an exposure time of 90 s. Near-infrared spectroscopy was used to measure the prefrontal cortex activity in the left and right prefrontal cortices as an indicator of oxyhemoglobin (oxy-Hb) concentration. Autonomic nervous activity was measured as an indicator of sympathetic (low-frequency/high-frequency component ratio, LF/HF), and parasympathetic (high-frequency components, HF) nervous activities were assessed by heart rate variability. Furthermore, the modified semantic differential method and the Profile of Mood States 2nd edition were used to measure psychological responses. Physiologically, the oxy-Hb concentration in the left prefrontal cortex and ln (LF/HF) were significantly lower during visual exposure to the WT than to the control. Psychologically, more comfortable, relaxed, and natural impressions, as well as improved mood states, were reported during visual stimulation to the WT than to the control. This study demonstrated that viewing a WT led to physiological relaxation and had a positive psychological effect on the participants.

Comparing a modified simple-conditional with the conditional-only methods in teaching Chinese children with autism.

Instruction in auditory-visual conditional discriminations for young children with autism spectrum disorder is typically based on either a conditional-only or a simple-conditional method. In this study, we evaluated a modified simple-conditional method in which we removed the steps for which visual comparisons were presented in isolation. We compared this modified simple-conditional method with the conditional-only method when teaching auditory-visual conditional discriminations to six young Chinese children with autism spectrum disorder. We included two efficiency measures: total sessions and time to mastery. Our results replicated the findings of previous research in that both methods were efficacious for all but one participant. Although efficacy outcomes were similar across methods, the conditional-only method was more efficient across participants according to sessions and time to mastery. Thus, the results add to support for the use of the conditional-only method to teach auditory-visual conditional discriminations.

2MDR, a Microcomputer-Controlled Visual Stimulation Device for Psychotherapy-Like Treatments of Mice.

Post-traumatic stress disorder and other mental disorders can be treated by an established psychotherapy called Eye Movement Desensitization and Reprocessing (EMDR). In EMDR, patients are confronted with traumatic memories while they are stimulated with alternating bilateral stimuli (ABS). How ABS affects the brain and whether ABS could be adapted to different patients or mental disorders is unknown. Interestingly, ABS reduced conditioned fear in mice. Yet, an approach to systematically test complex visual stimuli and compare respective differences in emotional processing based on semiautomated/automated behavioral analysis is lacking. We developed 2MDR (MultiModal Visual Stimulation to Desensitize Rodents), a novel, open-source, low-cost, customizable device that can be integrated in and transistor-transistor logic (TTL) controlled by commercial rodent behavioral setups. 2MDR allows the design and precise steering of multimodal visual stimuli in the head direction of freely moving mice. Optimized videography allows semiautomatic analysis of rodent behavior during visual stimulation. Detailed building, integration, and treatment instructions along with open-source software provide easy access for inexperienced users. Using 2MDR, we confirmed that EMDR-like ABS persistently improves fear extinction in mice and showed for the first time that ABS-mediated anxiolytic effects strongly depend on physical stimulus properties such as ABS brightness. 2MDR not only enables researchers to interfere with mouse behavior in an EMDR-like setting, but also demonstrates that visual stimuli can be used as a noninvasive brain stimulation to differentially alter emotional processing in mice.