Keeping in touch with the visual system: spatial alignment and multisensory integration of visual-somatosensory inputs.

Correlated sensory inputs coursing along the individual sensory processing hierarchies arrive at multisensory convergence zones in cortex where inputs are processed in an integrative manner. The exact hierarchical level of multisensory convergence zones and the timing of their inputs are still under debate, although increasingly, evidence points to multisensory integration (MSI) at very early sensory processing levels. While MSI is said to be governed by stimulus properties including space, time, and magnitude, violations of these rules have been documented. The objective of the current study was to determine, both psychophysically and electrophysiologically, whether differential visual-somatosensory (VS) integration patterns exist for stimuli presented to the same versus opposite hemifields. Using high-density electrical mapping and complementary psychophysical data, we examined multisensory integrative processing for combinations of visual and somatosensory inputs presented to both left and right spatial locations. We assessed how early during sensory processing VS interactions were seen in the event-related potential and whether spatial alignment of the visual and somatosensory elements resulted in differential integration effects. Reaction times to all VS pairings were significantly faster than those to the unisensory conditions, regardless of spatial alignment, pointing to engagement of integrative multisensory processing in all conditions. In support, electrophysiological results revealed significant differences between multisensory simultaneous VS and summed V + S responses, regardless of the spatial alignment of the constituent inputs. Nonetheless, multisensory effects were earlier in the aligned conditions, and were found to be particularly robust in the case of right-sided inputs (beginning at just 55 ms). In contrast to previous work on audio-visual and audio-somatosensory inputs, the current work suggests a degree of spatial specificity to the earliest detectable multisensory integrative effects in response to VS pairings.

Mapping phonemic processing zones along human perisylvian cortex: an electro-corticographic investigation.

The auditory system is organized such that progressively more complex features are represented across successive cortical hierarchical stages. Just when and where the processing of phonemes, fundamental elements of the speech signal, is achieved in this hierarchy remains a matter of vigorous debate. Non-invasive measures of phonemic representation have been somewhat equivocal. While some studies point to a primary role for middle/anterior regions of the superior temporal gyrus (STG), others implicate the posterior STG. Differences in stimulation, task and inter-individual anatomical/functional variability may account for these discrepant findings. Here, we sought to clarify this issue by mapping phonemic representation across left perisylvian cortex, taking advantage of the excellent sampling density afforded by intracranial recordings in humans. We asked whether one or both major divisions of the STG were sensitive to phonemic transitions. The high signal-to-noise characteristics of direct intracranial recordings allowed for analysis at the individual participant level, circumventing issues of inter-individual anatomic and functional variability that may have obscured previous findings at the group level of analysis. The mismatch negativity (MMN), an electrophysiological response elicited by changes in repetitive streams of stimulation, served as our primary dependent measure. Oddball configurations of pairs of phonemes, spectro-temporally matched non-phonemes, and simple tones were presented. The loci of the MMN clearly differed as a function of stimulus type. Phoneme representation was most robust over middle/anterior STG/STS, but was also observed over posterior STG/SMG. These data point to multiple phonemic processing zones along perisylvian cortex, both anterior and posterior to primary auditory cortex. This finding is considered within the context of a dual stream model of auditory processing in which functionally distinct ventral and dorsal auditory processing pathways may be engaged by speech stimuli.

Does the ventriloquist illusion assist selective listening?

Driver (1996) reported that the ventriloquist illusion can enhance selective listening of speech. Participants in his study listened to target and distractor words from a single loudspeaker while watching lip movements of the target words on a video monitor either above the loudspeaker or displaced to the left or right. He found that participants were more accurate in repeating the target words when the video was displaced from the loudspeaker than when the video was directly above the loudspeaker. Driver proposed that the ventriloquist illusion dragged the target sounds toward the location of the lip movements, freeing them from interference from the distractor words. However, successful attempts at replicating this finding are rare (we know of only three successful replications from 19 attempts). In five experiments, we found a weak advantage for selective listening from displaced lip movements only when there was a convincing ventriloquist illusion. We conclude that the ventriloquist illusion is necessary to confer the advantage for selective listening from displaced lip movements but that the phenomenon is a fleeting one at best.

Electrophysiological evidence for the action of a center-surround mechanism on semantic processing in the left hemisphere.

Physiological evidence was sought for a center-surround attentional mechanism (CSM), which has been proposed to assist in the retrieval of weakly activated items from semantic memory. The CSM operates by facilitating strongly related items in the "center" of the weakly activated area of semantic memory, and inhibiting less strongly related items in its "surround". In this study weak activation was created by having subjects acquire the meanings of new words to a recall criterion of only 50%. Subjects who attained this approximate criterion level of performance were subsequently included in a semantic priming task, during which ERPs were recorded. Primes were newly learned rare words, and targets were either synonyms, non-synonymously related words, or unrelated words. All stimuli were presented to the RVF/LH (right visual field/left hemisphere) or the LVF/RH (left visual field/right hemisphere). Under RVF/LH stimulation the newly learned word primes produced facilitation on N400 for synonym targets, and inhibition for related targets. No differences were observed under LVF/RH stimulation. The LH thus, supports a CSM, whereby a synonym in the "center" of attention, focused on the newly learned word, is facilitated, whereas a related word in the "surround" is inhibited. The data are consistent with the view of this laboratory that semantic memory is subserved by a spreading activation system in the LH. Also consistent with our view, there was no evidence of spreading activation in the RH. The findings are discussed in the context of additional recent theories of semantic memory. Finally, the adult right hemisphere may require more learning than the LH in order to demonstrate evidence of meaning acquisition.

Identifying temporal and causal contributions of neural processes underlying the Implicit Association Test (IAT).

The Implicit Association Test (IAT) is a popular behavioral measure that assesses the associative strength between outgroup members and stereotypical and counterstereotypical traits. Less is known, however, about the degree to which the IAT reflects automatic processing. Two studies examined automatic processing contributions to a gender-IAT using a data driven, social neuroscience approach. Performance on congruent (e.g., categorizing male names with synonyms of strength) and incongruent (e.g., categorizing female names with synonyms of strength) IAT blocks were separately analyzed using EEG (event-related potentials, or ERPs, and coherence; Study 1) and lesion (Study 2) methodologies. Compared to incongruent blocks, performance on congruent IAT blocks was associated with more positive ERPs that manifested in frontal and occipital regions at automatic processing speeds, occipital regions at more controlled processing speeds and was compromised by volume loss in the anterior temporal lobe (ATL), insula and medial PFC. Performance on incongruent blocks was associated with volume loss in supplementary motor areas, cingulate gyrus and a region in medial PFC similar to that found for congruent blocks. Greater coherence was found between frontal and occipital regions to the extent individuals exhibited more bias. This suggests there are separable neural contributions to congruent and incongruent blocks of the IAT but there is also a surprising amount of overlap. Given the temporal and regional neural distinctions, these results provide converging evidence that stereotypic associative strength assessed by the IAT indexes automatic processing to a degree.

Regularity extraction from non-adjacent sounds.

The regular behavior of sound sources helps us to make sense of the auditory environment. Regular patterns may, for instance, convey information on the identity of a sound source (such as the acoustic signature of a train moving on the rails). Yet typically, this signature overlaps in time with signals emitted from other sound sources. It is generally assumed that auditory regularity extraction cannot operate upon this mixture of signals because it only finds regularities between adjacent sounds. In this view, the auditory environment would be grouped into separate entities by means of readily available acoustic cues such as separation in frequency and location. Regularity extraction processes would then operate upon the resulting groups. Our new experimental evidence challenges this view. We presented two interleaved sound sequences which overlapped in frequency range and shared all acoustic parameters. The sequences only differed in their underlying regular patterns. We inserted deviants into one of the sequences to probe whether the regularity was extracted. In the first experiment, we found that these deviants elicited the mismatch negativity (MMN) component. Thus the auditory system was able to find the regularity between the non-adjacent sounds. Regularity extraction was not influenced by sequence cohesiveness as manipulated by the relative duration of tones and silent inter-tone-intervals. In the second experiment, we showed that a regularity connecting non-adjacent sounds was discovered only when the intervening sequence also contained a regular pattern, but not when the intervening sounds were randomly varying. This suggests that separate regular patterns are available to the auditory system as a cue for identifying signals coming from distinct sound sources. Thus auditory regularity extraction is not necessarily confined to a processing stage after initial sound grouping, but may precede grouping when other acoustic cues are unavailable.

The representation of unattended, segmented sounds: a mismatch negativity (MMN) study.

The detection of an irregular, potentially relevant change (deviance) in the regular, unattended acoustic environment is ensured by the automatic deviance detection mechanism. It underlies the formation of a regularity representation and a comparison of an incoming sound with this representation. A mismatch outcome of this comparison evokes the mismatch negativity (MMN) of the event-related potential. For unattended pure tones the automatic deviance detection mechanism operates most efficiently for initial sound parts, which is why these are suggested to contribute more to sound representation than later parts. A transient that physically segments the sound can overcome this temporal constraint in sound representation. Whether the resulting individual (initial and terminal) sound segments or the joined two-segments give rise to the regularity representation is addressed here. We took advantage that the MMN attenuation to the second of two successive deviances (deviance-repetition effect) is more pronounced when the deviances belong to the same unit of representation. We measured MMN for two deviances (frequency modulations) within segmented sounds that either occurred within the initial or the terminal segment, or that were split across both segments. Unexpectedly, we did not obtain a deviance-repetition effect. Instead, we obtained a temporal distance effect: With increasing temporal distance from deviance-onset relative to segment-onset the MMN amplitude decreased. Furthermore, this effect did not depend on whether the deviance occurred in the initial or in the terminal segment. Thus, (for the current approach) we suggest that the regularity representation is based on the individual rather than joined segments.

Right Hemispheric Contributions to Fine Auditory Temporal Discriminations: High-Density Electrical Mapping of the Duration Mismatch Negativity (MMN).

That language processing is primarily a function of the left hemisphere has led to the supposition that auditory temporal discrimination is particularly well-tuned in the left hemisphere, since speech discrimination is thought to rely heavily on the registration of temporal transitions. However, physiological data have not consistently supported this view. Rather, functional imaging studies often show equally strong, if not stronger, contributions from the right hemisphere during temporal processing tasks, suggesting a more complex underlying neural substrate. The mismatch negativity (MMN) component of the human auditory evoked-potential provides a sensitive metric of duration processing in human auditory cortex and lateralization of MMN can be readily assayed when sufficiently dense electrode arrays are employed. Here, the sensitivity of the left and right auditory cortex for temporal processing was measured by recording the MMN to small duration deviants presented to either the left or right ear. We found that duration deviants differing by just 15% (i.e. rare 115 ms tones presented in a stream of 100 ms tones) elicited a significant MMN for tones presented to the left ear (biasing the right hemisphere). However, deviants presented to the right ear elicited no detectable MMN for this separation. Further, participants detected significantly more duration deviants and committed fewer false alarms for tones presented to the left ear during a subsequent psychophysical testing session. In contrast to the prevalent model, these results point to equivalent if not greater right hemisphere contributions to temporal processing of small duration changes.

Perceptual load affects spatial and nonspatial visual selection processes: an event-related brain potential study.

One major question toward understanding selective attention regards the efficiency of selection. One theory contends that this efficiency in vision is determined primarily by the perceptual load (PL) imposed by the relevant stimuli; if this load is high enough to fill attentional capacity, irrelevant stimuli will be excluded before they interfere with task performance, but if this load is lower the spare capacity will be directed automatically to the irrelevant information, which will then interfere with task performance. The current study attempts to test and extend this theory in order to understand better the role of PL by examining its effects on event-related brain potentials (ERPs), voltage fluctuations recorded at the scalp that reflect underlying cognitive operations. Stimuli were presented one at a time, and subjects were instructed to respond to rare deviant stimuli that appeared within a relevant stimulus channel and to ignore stimuli in an irrelevant channel, where channel was defined by either spatial (left, right) or nonspatial (red, blue) attributes in separate tasks. PL was manipulated by varying the similarity between the target/deviant and standard stimulus, and increases in PL were found to increase the magnitude of the relevant-irrelevant difference waveforms in both tasks at predicted temporal windows. These findings suggest that PL affects attentional selection that is tonically maintained across many experimental trials, and does so not only when selection is spatially based but also when it is based upon nonspatial cues.

Auditory scene analysis: the interaction of stimulation rate and frequency separation on pre-attentive grouping.

Segregation of auditory inputs into meaningful acoustic groups is a key element of auditory scene analysis. Previously, we showed that two interwoven sets of tones differing widely along multiple feature dimensions (duration, pitch and location) were pre-attentively separated into different groups, and that tones separated in this manner did not elicit the mismatch negativity component with respect to each other. Grouping was studied with human subjects using a stimulus rate too slow to induce streaming. Here, we varied the separation of tone sequences along a single feature dimension, i.e. frequency. Frequency differences were either 24 Hz (small) or 1054 Hz (large). Two relatively slow stimulus rates were used (2.7 or 1 tone/s) to explicitly investigate grouping outside the so-called 'streaming effect', which requires rates of about 4 tones/s or faster. Two tones were presented in a quasi-random manner with embedded trains of one to four identical tones in a row. Deviants were defined as frequency switches after trains of four identical tones. Mismatch negativity was only elicited for small frequency switches at the slower stimulation rate. The data indicate that pre-attentive grouping of tones occurred when the frequency difference that separated them was large, regardless of stimulation rate. For small frequency differences, inputs were only grouped separately when the stimulation rate was relatively fast.

The effects of interstimulus interval on event-related indices of attention: an auditory selective attention test of perceptual load theory.

OBJECTIVE: We examined the impact of perceptual load by manipulating interstimulus interval (ISI) in two auditory selective attention studies that varied in the difficulty of the target discrimination. METHODS: In the paradigm, channels were separated by frequency and target/deviant tones were softer in intensity. Three ISI conditions were presented: fast (300ms), medium (600ms) and slow (900ms). Behavioral (accuracy and RT) and electrophysiological measures (Nd, P3b) were observed. RESULTS: In both studies, participants evidenced poorer accuracy during the fast ISI condition than the slow suggesting that ISI impacted task difficulty. However, none of the three measures of processing examined, Nd amplitude, P3b amplitude elicited by unattended deviant stimuli, or false alarms to unattended deviants, were impacted by ISI in the manner predicted by perceptual load theory. CONCLUSIONS: The prediction based on perceptual load theory, that there would be more processing of irrelevant stimuli under conditions of low as compared to high perceptual load, was not supported in these auditory studies. SIGNIFICANCE: Task difficulty/perceptual load impacts the processing of irrelevant stimuli in the auditory modality differently than predicted by perceptual load theory, and perhaps differently than in the visual modality.

Age-related changes in executive function: an event-related potential (ERP) investigation of task-switching.

Older adults have difficulty when executive control must be brought on line to coordinate ongoing behavior. To assess age-related alterations in executive processing, task-switching performance and event-related potential (ERP) activity were compared in young and older adults on switch, post-switch, pre-switch, and no-switch trials, ordered in demand for executive processes from greatest to least. In stimulus-locked averages for young adults, only switch trials elicited fronto-central P3 components, indicative of task-set attentional reallocation, whereas in older adults, three of the four trial types evinced frontal potentials. In response-locked averages, the amplitude of a medial frontal negativity (MFN), a component reflecting conflict monitoring and detection, increased as a function of executive demands in the ERPs of the young but not those of the older adults. These data suggest altered executive processing in older adults resulting in persistent recruitment of prefrontal processes for conditions that do not require them in the young.

Auditory processing in schizophrenia during the middle latency period (10-50 ms): high-density electrical mapping and source analysis reveal subcortical antecedents to early cortical deficits.

INTRODUCTION: Auditory sensory processing dysfunction is a core component of schizophrenia, with deficits occurring at 50 ms post-stimulus firmly established in the literature. Given that the initial afference of primary auditory cortex occurs at least 35 ms earlier, however, an essential question remains: how early in sensory processing do such deficits arise, and do they occur during initial cortical afference or earlier, which would implicate subcortical auditory dysfunction. OBJECTIVE: To establish the onset of the earliest deficits in auditory processing, we examined the time window demarcating the transition from subcortical to cortical processing: 10 ms to 50 ms during the so-called middle latency responses (MLRs). These remain to be adequately characterized in patients with schizophrenia. METHODS: We recorded auditory evoked potentials (AEPs) to simple tone-pips from 15 control subjects and 21 medicated patients with longer-term schizophrenia or schizoaffective disorder (illness duration 16 yr, standard deviation [SD] 9.4 yr), using high-density electrical scalp recordings. Between-group analyses assessed the integrity of the MLRs across groups. In addition, 2 source-localization models were conducted to address whether a distinction between subcortical and cortical generators of the MLRs can be made and whether evidence for differential dorsal and ventral pathway contributions to auditory processing deficits can be established. RESULTS: Robust auditory processing deficits were found for patients as early as 15 ms. Evidence for subcortical generators of the earliest MLR component (P20) was provided by source analysis. Topographical mapping and source localization also pointed to greater decrements in processing in the dorsal auditory pathway of patients, providing support for a theory of pervasive deficits that are organized along the lines of a dorsal-ventral distinction. CONCLUSIONS: Auditory sensory dysfunction in schizophrenia begins extremely early in processing, is evident during initial cortical afference and is also seen at earlier subcortical processing stages in the thalamus. The implication is that well-established sensory processing deficits in schizophrenia may be secondary to earlier subcortical dysfunction. Our findings do not preclude the possibility of even earlier deficits in auditory sensory processing during the auditory brainstem responses.

Development of auditory selective attention: event-related potential measures of channel selection and target detection.

In this study, we examined developmental changes in auditory selective attention using both electrophysiological (Nd, P3b) and behavioral measures while two groups of children (9- and 12-year-olds) and adults were engaged in a two-channel selective attention task. Channel was determined by frequency (1000 or 2000 Hz). Targets in one condition were shorter than the standards (duration target) and in the other were softer (intensity target). We found that the Nd onset and peak latencies for the children were significantly longer than for the adults. Nd amplitude, however, did not differ between the groups. Further, all groups evidenced P3b to attended targets but not to unattended deviants. Hits, reaction times, and false alarms to unattended deviants continued to evidence improvements through adolescence. Taken together, our data are most consistent with a model of developmental improvement in the speed and efficiency of attention allocation.

Seeing voices: High-density electrical mapping and source-analysis of the multisensory mismatch negativity evoked during the McGurk illusion.

Seeing a speaker's facial articulatory gestures powerfully affects speech perception, helping us overcome noisy acoustical environments. One particularly dramatic illustration of visual influences on speech perception is the "McGurk illusion", where dubbing an auditory phoneme onto video of an incongruent articulatory movement can often lead to illusory auditory percepts. This illusion is so strong that even in the absence of any real change in auditory stimulation, it activates the automatic auditory change-detection system, as indexed by the mismatch negativity (MMN) component of the auditory event-related potential (ERP). We investigated the putative left hemispheric dominance of McGurk-MMN using high-density ERPs in an oddball paradigm. Topographic mapping of the initial McGurk-MMN response showed a highly lateralized left hemisphere distribution, beginning at 175 ms. Subsequently, scalp activity was also observed over bilateral fronto-central scalp with a maximal amplitude at approximately 290 ms, suggesting later recruitment of right temporal cortices. Strong left hemisphere dominance was again observed during the last phase of the McGurk-MMN waveform (350-400 ms). Source analysis indicated bilateral sources in the temporal lobe just posterior to primary auditory cortex. While a single source in the right superior temporal gyrus (STG) accounted for the right hemisphere activity, two separate sources were required, one in the left transverse gyrus and the other in STG, to account for left hemisphere activity. These findings support the notion that visually driven multisensory illusory phonetic percepts produce an auditory-MMN cortical response and that left hemisphere temporal cortex plays a crucial role in this process.

Rapid targeting followed by sustained deployment of visual spatial attention.

We investigated preparatory attention processes when a spatial discrimination was required at a cued location, by measuring electroencephalography following a central symbolic cue to deploy spatial attention. Electroencephalography activity in response to the cue revealed three cue-related activations: an early-onset positivity following the P1 at posterior scalp sites contralateral to the cued location, followed by cue-related frontal scalp activity and later-onset sustained activity at posterior scalp sites contralateral to the cued location. The early contralateral positivity may reflect rapid targeting of the cued location. Our results also extend the findings of cue-related frontal activity followed by posterior activity contralateral to the cued location, found with nonspatial feature discriminations, to a task requiring a spatial discrimination.

Preattentively grouped tones do not elicit MMN with respect to each other.

To promote preattenive grouping of two sets of tones, one set of tones with a combination of frequency and ear of delivery was intermixed with another set of tones with a different combination of frequency and ear of delivery. The ERPs elicited by tones delivered to one ear that were preceded by three or four tones delivered in a row to the other ear were associated with an enhanced N1, due to the changes in frequency and ear of delivery with respect to the immediately preceding tones. However, no mismatch negativity (MMN) was obtained, even though these tones differed from the previious tones on the two dimensions of frequency and ear of delivery. The data were interpreted to signify that preattentively grouped sets of tones do not elicit MMN with respect to one another. This implies that once acoustic input has been preattentively grouped, the MMN system is dedicated to detecting changes that occur within but not between preattentively grouped stimuli.

Aging effects on the ERP correlates of involuntary attentional capture in speech sound analysis.

In an auditory oddball paradigm, 16 old and 16 young adults were asked to ignore binaurally presented disyllabic speech sounds and to watch a silent movie while event-related potentials (ERPs) were recorded. Four types of phonetic deviants occurred on, respectively: (1) the Stressed-First syllable, (2) the Unstressed-First syllable, (3) the Stressed-Second syllable and (4) the Unstressed-Second syllable. The nature of the phonetic change was the same across the four deviants. A P3a was seen only for deviant 3 in the elderly while observed for deviants 1 and 3 in the young. The mismatch negativities (MMNs) to deviants 1 and 2 were reduced compared with those to the other two deviants in both age groups. It is concluded that the ability to use salient prosody to involuntarily capture attention for speech sounds is preserved in normal aging. The lack of P3a response to deviant 1 in the elderly may result from the combined effects of backward masking and age-related temporal encoding inefficiency as indicated by reduced MMNs for the elderly.

Brain potentials to sexually suggestive whistles show meaning modulates the mismatch negativity.

Electroencephalographic data suggest that spoken words produce an enhanced output of the brain's automatic deviance detection system, as reflected by the mismatch negativity. Using meaningful and nonmeaningful whistles, we sought to distinguish the effect of semantic content on the brain's deviance detection system from language-specific stimulus features. In the meaningful condition, study participants heard a human 'wolf whistle', which is commonly interpreted as an unsolicited expression of sexual attention. In the nonmeaningful condition participants heard an acoustically identical, but digitally rearranged, version of the wolf whistle. The mismatch negativity amplitude was significantly larger when the infrequent stimulus was meaningful than when it was meaningless. These data suggest that enhanced mismatch negativity magnitude was due to the semantic valence of the eliciting deviant.

ERP correlates of involuntary attention capture by prosodic salience in speech.

This study addressed whether temporally salient (e.g., word onset) or prosodically salient (e.g., stressed syllables) information serves as a cue to capture attention in speech sound analysis. In an auditory oddball paradigm, 16 native English speakers were asked to ignore binaurally presented disyllabic speech sounds and watch a silent movie while ERPs were recorded. Four types of phonetic deviants were employed: a deviant syllable that was either stressed or unstressed and that occurred in either the first or second temporal position. The nature of the phonetic change (a change from a voiced consonant to its corresponding unvoiced consonant) was kept constant. MMNs were observed for all deviants. In contrast, the P3a was only seen when the deviance occurred on stressed syllables. The sensitivity of the P3a to the stress manipulation suggests that prosodic rather than temporal salience captures attention in unattended speech sounds.