Neural retrieval processes occur more rapidly for visual mental images that were previously encoded with high-vividness.

Visual mental imagery refers to our ability to experience visual images in the absence of sensory stimulation. Studies have shown that visual mental imagery can improve episodic memory. However, we have limited understanding of the neural mechanisms underlying this improvement. Using electroencephalography, we examined the neural processes associated with the retrieval of previously generated visual mental images, focusing on how the vividness at generation can modulate retrieval processes. Participants viewed word stimuli referring to common objects, forming a visual mental image of each word and rating the vividness of the mental image. This was followed by a surprise old/new recognition task. We compared retrieval performance for items rated as high- versus low-vividness at encoding. High-vividness items were retrieved with faster reaction times and higher confidence ratings in the memory judgment. While controlling for confidence, neural measures indicated that high-vividness items produced an earlier decrease in alpha-band activity at retrieval compared with low-vividness items, suggesting an earlier memory reinstatement. Even when low-vividness items were remembered with high confidence, they were not retrieved as quickly as high-vividness items. These results indicate that when highly vivid mental images are encoded, the speed of their retrieval occurs more rapidly, relative to low-vivid items.

The influence of imagery vividness and internally-directed attention on the neural mechanisms underlying the encoding of visual mental images into episodic memory.

Attention can be directed externally toward sensory information or internally toward self-generated information. Using electroencephalography (EEG), we investigated the attentional processes underlying the formation and encoding of self-generated mental images into episodic memory. Participants viewed flickering words referring to common objects and were tasked with forming visual mental images of the objects and rating their vividness. Subsequent memory for the presented object words was assessed using an old-new recognition task. Internally-directed attention during image generation was indexed as a reduction in steady-state visual evoked potentials (SSVEPs), oscillatory EEG responses at the frequency of a flickering stimulus. The results yielded 3 main findings. First, SSVEP power driven by the flickering word stimuli decreased as subjects directed attention internally to form the corresponding mental image. Second, SSVEP power returned to pre-imagery baseline more slowly for low- than high-vividness later remembered items, suggesting that longer internally-directed attention is required to generate subsequently remembered low-vividness images. Finally, the event-related-potential difference due to memory was more sustained for subsequently remembered low- versus high-vividness items, suggesting that additional conceptual processing may have been needed to remember the low-vividness visual images. Taken together, the results clarify the neural mechanisms supporting the encoding of self-generated information.

Reward magnitude enhances early attentional processing of auditory stimuli.

Reward associations are known to shape the brain's processing of visual stimuli, but relatively less is known about how reward associations impact the processing of auditory stimuli. We leveraged the high-temporal resolution of electroencephalography (EEG) and event-related potentials (ERPs) to investigate the influence of low- and high-magnitude stimulus-reward associations in an auditory oddball task. We associated fast, correct detection of certain auditory target stimuli with larger monetary rewards, and other auditory targets with smaller rewards. We found enhanced attentional processing of the more highly rewarded target stimuli, as evidenced by faster behavioral detection of those stimuli compared with lower-rewarded stimuli. Neurally, higher-reward associations enhanced the early sensory processing of auditory targets. Targets associated with higher-magnitude rewards had higher amplitude N1 and mismatch negativity (MMN) ERP components than targets associated with lower-magnitude rewards. Reward did not impact the latency of these early components. Higher-reward magnitude also decreased the latency and increased the amplitude of the longer-latency P3 component, suggesting that reward also can enhance the final processing stages of auditory target stimuli. These results provide insight into how the sensory and attentional neural processing of auditory stimuli is modulated by stimulus-reward associations and the magnitude of those associations, with higher-magnitude reward associations yielding enhanced auditory processing at both early and late stages compared with lower-magnitude reward associations.

The effects of ongoing distraction on the neural processes underlying signal detection.

Distraction can impede our ability to detect and effectively process task-relevant stimuli in our environment. Here we leveraged the high temporal resolution of event-related potentials (ERPs) to study the neural consequences of a global, continuous distractor on signal-detection processes. Healthy, young adults performed the dSAT task, a translational sustained-attention task that has been used across different species and in clinical groups, in the presence and absence of ongoing distracting stimulation. We found the presence of distracting stimuli impaired participants' ability to behaviorally detect task-relevant signal stimuli and greatly affected the neural cascade of processes underlying signal detection. Specifically, we found distraction reduced an anterior and a posterior early-latency N2 ERP component (~140-220ms) and modulated long-latency, detection-related P3 components (P3a: ~200-330ms, P3b: 300-700ms), even to correctly detected targets. These data provide evidence that distraction can induce powerful alterations in the neural processes related to signal detection, even when stimuli are behaviorally detected.

Rapid Context-based Identification of Target Sounds in an Auditory Scene.

To make sense of our dynamic and complex auditory environment, we must be able to parse the sensory input into usable parts and pick out relevant sounds from all the potentially distracting auditory information. Although it is unclear exactly how we accomplish this difficult task, Gamble and Woldorff [Gamble, M. L., & Woldorff, M. G. The temporal cascade of neural processes underlying target detection and attentional processing during auditory search. Cerebral Cortex (New York, N.Y.: 1991), 2014] recently reported an ERP study of an auditory target-search task in a temporally and spatially distributed, rapidly presented, auditory scene. They reported an early, differential, bilateral activation (beginning at 60 msec) between feature-deviating target stimuli and physically equivalent feature-deviating nontargets, reflecting a rapid target detection process. This was followed shortly later (at 130 msec) by the lateralized N2ac ERP activation, that reflects the focusing of auditory spatial attention toward the target sound and parallels the attentional-shifting processes widely studied in vision. Here we directly examined the early, bilateral, target-selective effect to better understand its nature and functional role. Participants listened to midline-presented sounds that included target and nontarget stimuli that were randomly either embedded in a brief rapid stream or presented alone. The results indicate that this early bilateral effect results from a template for the target that utilizes its feature deviancy within a stream to enable rapid identification. Moreover, individual-differences analysis showed that the size of this effect was larger for participants with faster RTs. The findings support the hypothesis that our auditory attentional systems can implement and utilize a context-based relational template for a target sound, making use of additional auditory information in the environment when needing to rapidly detect a relevant sound.

The Temporal Cascade of Neural Processes Underlying Target Detection and Attentional Processing During Auditory Search.

The posterior visual event-related potential (ERP) component, the N2pc, has been widely used to study lateralized shifts of attention within visual arrays. Recently, Gamble and Luck (2011) reported an auditory analog of this activity (the fronto-central "N2ac"), reflecting the lateralized focusing of attention toward a Target sound among 2 simultaneous auditory stimuli. Here, we directed an electrophysiological approach toward understanding auditory Target search within a more complex auditory environment in which rapidly occurring sounds were distributed across both time and space. Trials consisted of ten 40-ms monaural sounds rapidly presented to the 2 ears: 8 medium-pitch tones and 2 deviant sounds (one high and one low). For each block, one deviant type was designated as the Target, which participants needed to identify within each trial to discriminate its tonal quality. The extracted electrophysiological results included a very early enhancement, starting at approximately 50 ms, of a bilateral negative-polarity auditory brain response to the designated Target Deviant (compared with the Nontarget Deviant), followed at approximately 130 ms by the N2ac activity reflecting the lateralized focusing of attention toward that Target. The results delineate the tightly orchestrated sequence of neural processes underlying the detection of, and focusing of attention toward, Target sounds in complex auditory scenes.

The rapid distraction of attentional resources toward the source of incongruent stimulus input during multisensory conflict.

Neuroimaging work on multisensory conflict suggests that the relevant modality receives enhanced processing in the face of incongruency. However, the degree of stimulus processing in the irrelevant modality and the temporal cascade of the attentional modulations in either the relevant or irrelevant modalities are unknown. Here, we employed an audiovisual conflict paradigm with a sensory probe in the task-irrelevant modality (vision) to gauge the attentional allocation to that modality. ERPs were recorded as participants attended to and discriminated spoken auditory letters while ignoring simultaneous bilateral visual letter stimuli that were either fully congruent, fully incongruent, or partially incongruent (one side incongruent, one congruent) with the auditory stimulation. Half of the audiovisual letter stimuli were followed 500-700 msec later by a bilateral visual probe stimulus. As expected, ERPs to the audiovisual stimuli showed an incongruency ERP effect (fully incongruent versus fully congruent) of an enhanced, centrally distributed, negative-polarity wave starting ∼250 msec. More critically here, the sensory ERP components to the visual probes were larger when they followed fully incongruent versus fully congruent multisensory stimuli, with these enhancements greatest on fully incongruent trials with the slowest RTs. In addition, on the slowest-response partially incongruent trials, the P2 sensory component to the visual probes was larger contralateral to the preceding incongruent visual stimulus. These data suggest that, in response to conflicting multisensory stimulus input, the initial cognitive effect is a capture of attention by the incongruent irrelevant-modality input, pulling neural processing resources toward that modality, resulting in rapid enhancement, rather than rapid suppression, of that input.

Is conflict monitoring supramodal? Spatiotemporal dynamics of cognitive control processes in an auditory Stroop task.

The electrophysiological correlates of conflict processing and cognitive control have been well characterized for the visual modality in paradigms such as the Stroop task. Much less is known about corresponding processes in the auditory modality. Here, electroencephalographic recordings of brain activity were measured during an auditory Stroop task, using three different forms of behavioral response (overt verbal, covert verbal, and manual), that closely paralleled our previous visual Stroop study. As was expected, behavioral responses were slower and less accurate for incongruent than for congruent trials. Neurally, incongruent trials showed an enhanced fronto-central negative polarity wave (N(inc)), similar to the N450 in visual Stroop tasks, with similar variations as a function of behavioral response mode, but peaking ~150 ms earlier, followed by an enhanced positive posterior wave. In addition, sequential behavioral and neural effects were observed that supported the conflict-monitoring and cognitive adjustment hypothesis. Thus, while some aspects of the conflict detection processes, such as timing, may be modality dependent, the general mechanisms would appear to be supramodal.

The involvement of the dopaminergic midbrain and cortico-striatal-thalamic circuits in the integration of reward prospect and attentional task demands.

Reward has been shown to promote human performance in multiple task domains. However, an important debate has developed about the uniqueness of reward-related neural signatures associated with such facilitation, as similar neural patterns can be triggered by increased attentional focus independent of reward. Here, we used functional magnetic resonance imaging to directly investigate the neural commonalities and interactions between the anticipation of both reward and task difficulty, by independently manipulating these factors in a cued-attention paradigm. In preparation for the target stimulus, both factors increased activity within the midbrain, dorsal striatum, and fronto-parietal areas, while inducing deactivations in default-mode regions. Additionally, reward engaged the ventral striatum, posterior cingulate, and occipital cortex, while difficulty engaged medial and dorsolateral frontal regions. Importantly, a network comprising the midbrain, caudate nucleus, thalamus, and anterior midcingulate cortex exhibited an interaction between reward and difficulty, presumably reflecting additional resource recruitment for demanding tasks with profitable outcome. This notion was consistent with a negative correlation between cue-related midbrain activity and difficulty-induced performance detriments in reward-predictive trials. Together, the data demonstrate that expected value and attentional demands are integrated in cortico-striatal-thalamic circuits in coordination with the dopaminergic midbrain to flexibly modulate resource allocation for an effective pursuit of behavioral goals.

The cross-modal spread of attention reveals differential constraints for the temporal and spatial linking of visual and auditory stimulus events.

The integration of multisensory information has been shown to be guided by spatial and temporal proximity, as well as to be influenced by attention. Here we used neural measures of the multisensory spread of attention to investigate the spatial and temporal linking of synchronous versus near-synchronous auditory and visual events. Human participants attended selectively to one of two lateralized visual-stimulus streams while task-irrelevant tones were presented centrally. Electrophysiological measures of brain activity showed that tones occurring simultaneously or delayed by 100 ms were temporally linked to an attended visual stimulus, as reflected by robust cross-modal spreading-of-attention activity, but not when delayed by 300 ms. The neural data also indicated a ventriloquist-like spatial linking of the auditory to the attended visual stimuli, but only when occurring simultaneously. These neurophysiological results thus provide unique insight into the temporal and spatial principles of multisensory feature integration and the fundamental role attention plays in such integration.

Video game players show more precise multisensory temporal processing abilities.

Recent research has demonstrated enhanced visual attention and visual perception in individuals with extensive experience playing action video games. These benefits manifest in several realms, but much remains unknown about the ways in which video game experience alters perception and cognition. In the present study, we examined whether video game players' benefits generalize beyond vision to multisensory processing by presenting auditory and visual stimuli within a short temporal window to video game players and non-video game players. Participants performed two discrimination tasks, both of which revealed benefits for video game players: In a simultaneity judgment task, video game players were better able to distinguish whether simple visual and auditory stimuli occurred at the same moment or slightly offset in time, and in a temporal-order judgment task, they revealed an enhanced ability to determine the temporal sequence of multisensory stimuli. These results suggest that people with extensive experience playing video games display benefits that extend beyond the visual modality to also impact multisensory processing.

Intermodal attention affects the processing of the temporal alignment of audiovisual stimuli.

The temporal asynchrony between inputs to different sensory modalities has been shown to be a critical factor influencing the interaction between such inputs. We used scalp-recorded event-related potentials (ERPs) to investigate the effects of attention on the processing of audiovisual multisensory stimuli as the temporal asynchrony between the auditory and visual inputs varied across the audiovisual integration window (i.e., up to 125 ms). Randomized streams of unisensory auditory stimuli, unisensory visual stimuli, and audiovisual stimuli (consisting of the temporally proximal presentation of the visual and auditory stimulus components) were presented centrally while participants attended to either the auditory or the visual modality to detect occasional target stimuli in that modality. ERPs elicited by each of the contributing sensory modalities were extracted by signal processing techniques from the combined ERP waveforms elicited by the multisensory stimuli. This was done for each of the five different 50-ms subranges of stimulus onset asynchrony (SOA: e.g., V precedes A by 125-75 ms, by 75-25 ms, etc.). The extracted ERPs for the visual inputs of the multisensory stimuli were compared among each other and with the ERPs to the unisensory visual control stimuli, separately when attention was directed to the visual or to the auditory modality. The results showed that the attention effects on the right-hemisphere visual P1 was largest when auditory and visual stimuli were temporally aligned. In contrast, the N1 attention effect was smallest at this latency, suggesting that attention may play a role in the processing of the relative temporal alignment of the constituent parts of multisensory stimuli. At longer latencies an occipital selection negativity for the attended versus unattended visual stimuli was also observed, but this effect did not vary as a function of SOA, suggesting that by that latency a stable representation of the auditory and visual stimulus components has been established.

Electrophysiological measures of time processing in infant and adult brains: Weber's Law holds.

Behavioral studies have demonstrated that time perception in adults, children, and nonhuman animals is subject to Weber's Law. More specifically, as with discriminations of other features, it has been found that it is the ratio between two durations rather than their absolute difference that controls the ability of an animal to discriminate them. Here, we show that scalp-recorded event-related electrical brain potentials (ERPs) in both adults and 10-month-old human infants, in response to changes in interstimulus interval (ISI), appear to obey the scalar property found in time perception in adults, children, and nonhuman animals. Using a timing-interval oddball paradigm, we tested adults and infants in conditions where the ratio between the standard and deviant interval in a train of homogeneous auditory stimuli varied such that there was a 1:4 (only for the infants), 1:3, 1:2, and 2:3 ratio between the standard and deviant intervals. We found that the amplitude of the deviant-triggered mismatch negativity ERP component (deviant-ISI ERP minus standard-ISI ERP) varied as a function of the ratio of the standard to deviant interval. Moreover, when absolute values were varied and ratio was held constant, the mismatch negativity did not vary.

Good times for multisensory integration: Effects of the precision of temporal synchrony as revealed by gamma-band oscillations.

The synchronous occurrence of the unisensory components of a multisensory stimulus contributes to their successful merging into a coherent perceptual representation. Oscillatory gamma-band responses (GBRs, 30-80 Hz) have been linked to feature integration mechanisms and to multisensory processing, suggesting they may also be sensitive to the temporal alignment of multisensory stimulus components. Here we examined the effects on early oscillatory GBR brain activity of varying the precision of the temporal synchrony of the unisensory components of an audio-visual stimulus. Audio-visual stimuli were presented with stimulus onset asynchronies ranging from -125 to +125 ms. Randomized streams of auditory (A), visual (V), and audio-visual (AV) stimuli were presented centrally while subjects attended to either the auditory or visual modality to detect occasional targets. GBRs to auditory and visual components of multisensory AV stimuli were extracted for five subranges of asynchrony (e.g., A preceded by V by 100+/-25 ms, by 50+/-25 ms, etc.) and compared with GBRs to unisensory control stimuli. Robust multisensory interactions were observed in the early GBRs when the auditory and visual stimuli were presented with the closest synchrony. These effects were found over medial-frontal brain areas after 30-80 ms and over occipital brain areas after 60-120 ms. A second integration effect, possibly reflecting the perceptual separation of the two sensory inputs, was found over occipital areas when auditory inputs preceded visual by 100+/-25 ms. No significant interactions were observed for the other subranges of asynchrony. These results show that the precision of temporal synchrony can have an impact on early cross-modal interactions in human cortex.

Selective attention and audiovisual integration: is attending to both modalities a prerequisite for early integration?

Interactions between multisensory integration and attention were studied using a combined audiovisual streaming design and a rapid serial visual presentation paradigm. Event-related potentials (ERPs) following audiovisual objects (AV) were compared with the sum of the ERPs following auditory (A) and visual objects (V). Integration processes were expressed as the difference between these AV and (A + V) responses and were studied while attention was directed to one or both modalities or directed elsewhere. Results show that multisensory integration effects depend on the multisensory objects being fully attended--that is, when both the visual and auditory senses were attended. In this condition, a superadditive audiovisual integration effect was observed on the P50 component. When unattended, this effect was reversed; the P50 components of multisensory ERPs were smaller than the unisensory sum. Additionally, we found an enhanced late frontal negativity when subjects attended the visual component of a multisensory object. This effect, bearing a strong resemblance to the auditory processing negativity, appeared to reflect late attention-related processing that had spread to encompass the auditory component of the multisensory object. In conclusion, our results shed new light on how the brain processes multisensory auditory and visual information, including how attention modulates multisensory integration processes.

Attentional capacity for processing concurrent stimuli is larger across sensory modalities than within a modality.

One finding in attention research is that visual and auditory attention mechanisms are linked together. Such a link would predict a central, amodal capacity limit in processing visual and auditory stimuli. Here we show that this is not the case. Letter streams were accompanied by asynchronously presented streams of auditory, visual, and audiovisual objects. Either the letter streams or the visual, auditory, or audiovisual parts of the object streams were attended. Attending to various aspects of the objects resulted in modulations of the letter-stream-elicited steady-state evoked potentials (SSVEPs). SSVEPs were larger when auditory objects were attended than when either visual objects alone or when auditory and visual object stimuli were attended together. SSVEP amplitudes were the same in the latter conditions, indicating that attentional capacity between modalities is larger than attentional capacity within one and the same modality.

The spread of attention across modalities and space in a multisensory object.

Attending to a stimulus is known to enhance the neural responses to that stimulus. Recent experiments on visual attention have shown that this modulation can have object-based characteristics, such that, when certain parts of a visual object are attended, other parts automatically also receive enhanced processing. Here, we investigated whether visual attention can modulate neural responses to other components of a multisensory object defined by synchronous, but spatially disparate, auditory and visual stimuli. The audiovisual integration of such multisensory stimuli typically leads to mislocalization of the sound toward the visual stimulus (ventriloquism illusion). Using event-related potentials and functional MRI, we found that the brain's response to task-irrelevant sounds occurring synchronously with a visual stimulus from a different location was larger when that accompanying visual stimulus was attended versus unattended. The event-related potential effect consisted of sustained, frontally distributed, brain activity that emerged relatively late in processing, an effect resembling attention-related enhancements seen at earlier latencies during intramodal auditory attention. Moreover, the functional MRI data confirmed that the effect included specific enhancement of activity in auditory cortex. These findings indicate that attention to one sensory modality can spread to encompass simultaneous signals from another modality, even when they are task-irrelevant and from a different location. This cross-modal attentional spread appears to reflect an object-based, late selection process wherein spatially discrepant auditory stimulation is grouped with synchronous attended visual input into a multisensory object, resulting in the auditory information being pulled into the attentional spotlight and bestowed with enhanced processing.

Multisensory processing and oscillatory gamma responses: effects of spatial selective attention.

Here we describe an EEG study investigating the interactions between multisensory (audio-visual) integration and spatial attention, using oscillatory gamma-band responses (GBRs). The results include a comparison with previously reported event-related potential (ERP) findings from the same paradigm. Unisensory-auditory (A), unisensory-visual (V), and multisensory (AV) stimuli were presented to the left and right hemispaces while subjects attended to a designated side to detect deviant target stimuli in either sensory modality. For attended multisensory stimuli we observed larger evoked GBRs approximately 40-50 ms post-stimulus over medial-frontal brain areas compared with those same multisensory stimuli when unattended. Further analysis indicated that the integration effect and its attentional enhancement may be caused in part by a stimulus-triggered phase resetting of ongoing gamma-band responses. Interestingly, no such early interaction effects (<90 ms) could be found in the ERP waveforms, suggesting that oscillatory GBRs may be more sensitive than ERPs to these early latency attention effects. Moreover, no GBR attention effects could be found for the unisensory auditory or unisensory visual stimuli, suggesting that attention particularly affects the integrative processing of audiovisual stimuli at these early latencies.

The ERP omitted stimulus response to "no-stim" events and its implications for fast-rate event-related fMRI designs.

A major difficulty in fast-rate event-related fMRI experiments is the extensive overlap from adjacent trials in the stimulus sequence. One approach to address this problem is to include "no-stim" or "null" events as a trial type. These are randomized as if they were true stimulus events but no stimulus actually occurs. Assuming that no response is elicited by the null events, their time-locked average reflects only the averaged overlap. Thus, contrasting the averages for the other trial types versus the null-event average subtracts out the overlap, enabling the extraction of the response functions for these other trial types. ERP studies, however, have indicated that an endogenous brain response, the omitted stimulus response (OSR), can be evoked by a missing event in a stream of regularly occurring stimuli. To the extent that this response is elicited by null events in an event-related fMRI experiment, the null-event subtraction or contrast would falsely introduce the inverse of the OSR into the averaged responses to the other trial types. Using high-density ERP recordings, we investigated the effect of different percentages of omitted stimuli (11, 22, and 33%) on the auditory OSR at stimulus rates of one event per second or one event per 2 s. Significant OSRs were found for each percentage in the 1-s condition as well as in the 11% 2-s condition. The responses consisted of an early posterior negative wave (180-280 ms) followed by a larger anterior positive wave. These results have important implications for fast-rate fMRI designs, while also providing new data on the brain's response to omitted stimuli.