EEG alpha power predicts the temporal sensitivity of multisensory perception.

Pre-stimulus electroencephalogram (EEG) oscillations, especially in the alpha range (8-13 Hz), can affect the sensitivity to temporal lags between modalities in multisensory perception. The effects of alpha power are often explained in terms of alpha's inhibitory functions, whereas effects of alpha frequency have bolstered theories of discrete perceptual cycles, where the length of a cycle, or window of integration, is determined by alpha frequency. Such studies typically employ visual detection paradigms with near-threshold or even illusory stimuli. It is unclear whether such results generalize to above-threshold stimuli. Here, we recorded EEG, while measuring temporal discrimination sensitivity in a temporal-order judgement task using above-threshold auditory and visual stimuli. We tested whether the power and instantaneous frequency of pre-stimulus oscillations predict audiovisual temporal discrimination sensitivity on a trial-by-trial basis. By applying a jackknife procedure to link single-trial pre-stimulus oscillatory power and instantaneous frequency to psychometric measures, we identified a posterior cluster where lower alpha power was associated with higher temporal sensitivity of audiovisual discrimination. No statistically significant relationship between instantaneous alpha frequency and temporal sensitivity was found. These results suggest that temporal sensitivity for above-threshold multisensory stimuli fluctuates from moment to moment and is indexed by modulations in alpha power.

Neurophysiological investigation of auditory intensity dependence in patients with Parkinson's disease.

There is accumulating evidence for auditory dysfunctions in patients with Parkinson's disease (PD). Moreover, a possible relationship has been suggested between altered auditory intensity processing and the hypophonic speech characteristics in PD. Nonetheless, further insight into the neurophysiological correlates of auditory intensity processing in patients with PD is needed primarily. In the present study, high-density EEG recordings were used to investigate intensity dependence of auditory evoked potentials (IDAEPs) in 14 patients with PD and 14 age- and gender-matched healthy control participants (HCs). Patients with PD were evaluated in both the on- and off-medication states. HCs were also evaluated twice. Significantly increased IDAEP of the N1/P2 was demonstrated in patients with PD evaluated in the on-medication state compared to HCs. Distinctive results were found for the N1 and P2 component. Regarding the N1 component, no differences in latency or amplitude were shown between patients with PD and HCs regardless of the medication state. In contrast, increased P2 amplitude was demonstrated in patients with PD evaluated in the on-medication state compared to the off-medication state and HCs. In addition to a dopaminergic deficiency, deficits in serotonergic neurotransmission in PD were shown based on increased IDAEP. Due to specific alterations of the N1-P2 complex, the current results suggest deficiencies in early-attentive inhibitory processing of auditory input in PD. This interpretation is consistent with the involvement of the basal ganglia and the role of dopaminergic and serotonergic neurotransmission in auditory gating.

Predictive coding and multisensory integration: an attentional account of the multisensory mind.

Multisensory integration involves a host of different cognitive processes, occurring at different stages of sensory processing. Here I argue that, despite recent insights suggesting that multisensory interactions can occur at very early latencies, the actual integration of individual sensory traces into an internally consistent mental representation is dependent on both top-down and bottom-up processes. Moreover, I argue that this integration is not limited to just sensory inputs, but that internal cognitive processes also shape the resulting mental representation. Studies showing that memory recall is affected by the initial multisensory context in which the stimuli were presented will be discussed, as well as several studies showing that mental imagery can affect multisensory illusions. This empirical evidence will be discussed from a predictive coding perspective, in which a central top-down attentional process is proposed to play a central role in coordinating the integration of all these inputs into a coherent mental representation.

Early multisensory interactions affect the competition among multiple visual objects.

In dynamic cluttered environments, audition and vision may benefit from each other in determining what deserves further attention and what does not. We investigated the underlying neural mechanisms responsible for attentional guidance by audiovisual stimuli in such an environment. Event-related potentials (ERPs) were measured during visual search through dynamic displays consisting of line elements that randomly changed orientation. Search accuracy improved when a target orientation change was synchronized with an auditory signal as compared to when the auditory signal was absent or synchronized with a distractor orientation change. The ERP data show that behavioral benefits were related to an early multisensory interaction over left parieto-occipital cortex (50-60 ms post-stimulus onset), which was followed by an early positive modulation (80-100 ms) over occipital and temporal areas contralateral to the audiovisual event, an enhanced N2pc (210-250 ms), and a contralateral negative slow wave (CNSW). The early multisensory interaction was correlated with behavioral search benefits, indicating that participants with a strong multisensory interaction benefited the most from the synchronized auditory signal. We suggest that an auditory signal enhances the neural response to a synchronized visual event, which increases the chances of selection in a multiple object environment.

Intermodal attention affects the processing of the temporal alignment of audiovisual stimuli.

The temporal asynchrony between inputs to different sensory modalities has been shown to be a critical factor influencing the interaction between such inputs. We used scalp-recorded event-related potentials (ERPs) to investigate the effects of attention on the processing of audiovisual multisensory stimuli as the temporal asynchrony between the auditory and visual inputs varied across the audiovisual integration window (i.e., up to 125 ms). Randomized streams of unisensory auditory stimuli, unisensory visual stimuli, and audiovisual stimuli (consisting of the temporally proximal presentation of the visual and auditory stimulus components) were presented centrally while participants attended to either the auditory or the visual modality to detect occasional target stimuli in that modality. ERPs elicited by each of the contributing sensory modalities were extracted by signal processing techniques from the combined ERP waveforms elicited by the multisensory stimuli. This was done for each of the five different 50-ms subranges of stimulus onset asynchrony (SOA: e.g., V precedes A by 125-75 ms, by 75-25 ms, etc.). The extracted ERPs for the visual inputs of the multisensory stimuli were compared among each other and with the ERPs to the unisensory visual control stimuli, separately when attention was directed to the visual or to the auditory modality. The results showed that the attention effects on the right-hemisphere visual P1 was largest when auditory and visual stimuli were temporally aligned. In contrast, the N1 attention effect was smallest at this latency, suggesting that attention may play a role in the processing of the relative temporal alignment of the constituent parts of multisensory stimuli. At longer latencies an occipital selection negativity for the attended versus unattended visual stimuli was also observed, but this effect did not vary as a function of SOA, suggesting that by that latency a stable representation of the auditory and visual stimulus components has been established.

Selective attention and audiovisual integration: is attending to both modalities a prerequisite for early integration?

Interactions between multisensory integration and attention were studied using a combined audiovisual streaming design and a rapid serial visual presentation paradigm. Event-related potentials (ERPs) following audiovisual objects (AV) were compared with the sum of the ERPs following auditory (A) and visual objects (V). Integration processes were expressed as the difference between these AV and (A + V) responses and were studied while attention was directed to one or both modalities or directed elsewhere. Results show that multisensory integration effects depend on the multisensory objects being fully attended--that is, when both the visual and auditory senses were attended. In this condition, a superadditive audiovisual integration effect was observed on the P50 component. When unattended, this effect was reversed; the P50 components of multisensory ERPs were smaller than the unisensory sum. Additionally, we found an enhanced late frontal negativity when subjects attended the visual component of a multisensory object. This effect, bearing a strong resemblance to the auditory processing negativity, appeared to reflect late attention-related processing that had spread to encompass the auditory component of the multisensory object. In conclusion, our results shed new light on how the brain processes multisensory auditory and visual information, including how attention modulates multisensory integration processes.

Good times for multisensory integration: Effects of the precision of temporal synchrony as revealed by gamma-band oscillations.

The synchronous occurrence of the unisensory components of a multisensory stimulus contributes to their successful merging into a coherent perceptual representation. Oscillatory gamma-band responses (GBRs, 30-80 Hz) have been linked to feature integration mechanisms and to multisensory processing, suggesting they may also be sensitive to the temporal alignment of multisensory stimulus components. Here we examined the effects on early oscillatory GBR brain activity of varying the precision of the temporal synchrony of the unisensory components of an audio-visual stimulus. Audio-visual stimuli were presented with stimulus onset asynchronies ranging from -125 to +125 ms. Randomized streams of auditory (A), visual (V), and audio-visual (AV) stimuli were presented centrally while subjects attended to either the auditory or visual modality to detect occasional targets. GBRs to auditory and visual components of multisensory AV stimuli were extracted for five subranges of asynchrony (e.g., A preceded by V by 100+/-25 ms, by 50+/-25 ms, etc.) and compared with GBRs to unisensory control stimuli. Robust multisensory interactions were observed in the early GBRs when the auditory and visual stimuli were presented with the closest synchrony. These effects were found over medial-frontal brain areas after 30-80 ms and over occipital brain areas after 60-120 ms. A second integration effect, possibly reflecting the perceptual separation of the two sensory inputs, was found over occipital areas when auditory inputs preceded visual by 100+/-25 ms. No significant interactions were observed for the other subranges of asynchrony. These results show that the precision of temporal synchrony can have an impact on early cross-modal interactions in human cortex.

Attentional capacity for processing concurrent stimuli is larger across sensory modalities than within a modality.

One finding in attention research is that visual and auditory attention mechanisms are linked together. Such a link would predict a central, amodal capacity limit in processing visual and auditory stimuli. Here we show that this is not the case. Letter streams were accompanied by asynchronously presented streams of auditory, visual, and audiovisual objects. Either the letter streams or the visual, auditory, or audiovisual parts of the object streams were attended. Attending to various aspects of the objects resulted in modulations of the letter-stream-elicited steady-state evoked potentials (SSVEPs). SSVEPs were larger when auditory objects were attended than when either visual objects alone or when auditory and visual object stimuli were attended together. SSVEP amplitudes were the same in the latter conditions, indicating that attentional capacity between modalities is larger than attentional capacity within one and the same modality.

Selective attention to spatial and non-spatial visual stimuli is affected differentially by age: effects on event-related brain potentials and performance data.

To assess selective attention processes in young and old adults, behavioral and event-related potential (ERP) measures were recorded. Streams of visual stimuli were presented from left or right locations (Experiment 1) or from a central location and comprising two different spatial frequencies (Experiment 2). In both experiments, results were compared in visual-only and visual+auditory stimulus context conditions. Participants were forced to respond fast in both experiments, while maintaining high accuracy. In Experiment 1, no behavioral effects of aging were found; however, an enlargement of the N1 component in the older age group suggested that older adults initial selection process was larger than that of young adults. A late frontal effect following the P300 elicited by attended non-targets was larger in the visual+auditory condition than in the visual-only condition in the old age group. This effect was interpreted as reflecting a memory update of the relevant target location. In Experiment 2, older adults made relatively more errors in the visual+auditory condition than in visual-only condition, more so than the young adults. Older adults' ERP data were also characterized by an enlargement of the occipital selection negativity, compared to the young age group. In contrast to experiment 1, no late frontal post-P3 effect could be found, suggesting that the memory trace of the relevant stimulus feature was updated less frequently, explaining the reduction in response accuracy in the visual+auditory stimulus context conditions.

Multisensory processing and oscillatory gamma responses: effects of spatial selective attention.

Here we describe an EEG study investigating the interactions between multisensory (audio-visual) integration and spatial attention, using oscillatory gamma-band responses (GBRs). The results include a comparison with previously reported event-related potential (ERP) findings from the same paradigm. Unisensory-auditory (A), unisensory-visual (V), and multisensory (AV) stimuli were presented to the left and right hemispaces while subjects attended to a designated side to detect deviant target stimuli in either sensory modality. For attended multisensory stimuli we observed larger evoked GBRs approximately 40-50 ms post-stimulus over medial-frontal brain areas compared with those same multisensory stimuli when unattended. Further analysis indicated that the integration effect and its attentional enhancement may be caused in part by a stimulus-triggered phase resetting of ongoing gamma-band responses. Interestingly, no such early interaction effects (<90 ms) could be found in the ERP waveforms, suggesting that oscillatory GBRs may be more sensitive than ERPs to these early latency attention effects. Moreover, no GBR attention effects could be found for the unisensory auditory or unisensory visual stimuli, suggesting that attention particularly affects the integrative processing of audiovisual stimuli at these early latencies.