Feeling senseless sensations: a crossmodal EEG study of mismatched tactile and visual experiences in virtual reality.

To create highly immersive experiences in virtual reality (VR) it is important to not only include the visual sense but also to involve multimodal sensory input. To achieve optimal results, the temporal and spatial synchronization of these multimodal inputs is critical. It is therefore necessary to find methods to objectively evaluate the synchronization of VR experiences with a continuous tracking of the user. In this study a passive touch experience was incorporated in a visual-tactile VR setup using VR glasses and tactile sensations in mid-air. Inconsistencies of multimodal perception were intentionally integrated into a discrimination task. The participants' electroencephalogram (EEG) was recorded to obtain neural correlates of visual-tactile mismatch situations. The results showed significant differences in the event-related potentials (ERP) between match and mismatch situations. A biphasic ERP configuration consisting of a positivity at 120 ms and a later negativity at 370 ms was observed following a visual-tactile mismatch. This late negativity could be related to the N400 that is associated with semantic incongruency. These results provide a promising approach towards the objective evaluation of visual-tactile synchronization in virtual experiences.

The time course of feature-selective attention inside and outside the focus of spatial attention.

Research on attentional selection of stimulus features has yielded seemingly contradictory results. On the one hand, many experiments in humans and animals have observed a "global" facilitation of attended features across the entire visual field, even when spatial attention is focused on a single location. On the other hand, several event-related potential studies in humans reported that attended features are enhanced at the attended location only. The present experiment demonstrates that these conflicting results can be explained by differences in the timing of attentional allocation inside and outside the spatial focus of attention. Participants attended to fields of either red or blue randomly moving dots on either the left or right side of fixation with the task of detecting brief coherent motion targets. Recordings of steady-state visual evoked potentials elicited by the flickering stimuli allowed concurrent measurement of the time course of feature-selective attention in visual cortex on both the attended and the unattended sides. The onset of feature-selective attentional modulation on the attended side occurred around 150 ms earlier than on the unattended side. This finding that feature-selective attention is not spatially global from the outset but extends to unattended locations after a temporal delay resolves previous contradictions between studies finding global versus hierarchical selection of features and provides insight into the fundamental relationship between feature-based and location-based (spatial) attention mechanisms.

Isolating the Neural Substrates of Visually Guided Attention Orienting in Humans.

The neural processes that enable healthy humans to orient attention to sudden visual events are poorly understood because they are tightly intertwined with purely sensory processes. Here we isolated visually guided orienting activity from sensory activity using event-related potentials (ERPs). By recording ERPs to a lateral stimulus and comparing waveforms obtained under conditions of attention and inattention, we identified an early positive deflection over the ipsilateral visual cortex that was associated with the covert orienting of visual attention to the stimulus. Across five experiments with male and female adult participants, this ipsilateral visual orienting activity (VOA) could be distinguished from purely sensory-evoked activity and from other top-down spatial attention effects. The VOA was linked with behavioral measures of orienting, being significantly larger when the stimulus was detected rapidly than when it was detected more slowly, and its presence was independent of saccadic eye movements toward the targets. The VOA appears to be a specific neural index of the visually guided orienting of attention to a stimulus that appears abruptly in an otherwise uncluttered visual field.SIGNIFICANCE STATEMENT The study of visual attention orienting has been an important impetus for the field of cognitive neuroscience. Seminal reaction-time studies demonstrated that a suddenly appearing visual stimulus attracts attention involuntarily, but the neural processes associated with visually guided attention orienting have been difficult to isolate because they are intertwined with sensory processes that trigger the orienting. Here, we disentangled orienting activity from sensory activity using scalp recordings of event-related electrical activity in the human brain. A specific neural index of visually guided attention orienting was identified. Surprisingly, whereas peripheral sensory stimulation is processed initially and predominantly by the contralateral visual cortex, this electrophysiological index of visual orienting was recorded over the cerebral hemisphere that was ipsilateral to the attention-capturing stimulus.

A simple metric to study the mechanisms generating event-related potentials.

BACKGROUND: There is an active debate about the mechanism underlying the generation of event-related potentials, and, particularly, whether these are generated by additive components, independent of the background EEG, or the phase-resetting of ongoing oscillations. METHOD: We present a new metric to evaluate trial-by-trial covariations of successive ERP components. Our main assumption is that if two successive ERP components are generated by phase-resetting of a unitary oscillation, they should be time-locked to each other and their single-trial latencies should covary. In contrast, if the components are generated by independent additive components, single-trial latency covariations should not be observed. To quantify the covariance between the single-trial latencies, we define a metric based on latency-corrected averages, which we applied to both simulated and real ERPs. RESULTS: For the simulated data, there was a clear distinction in latency covariation between the ERPs generated with unitary phase-resetting versus additive models. For real visual and auditory ERPs, we observed a lack of latency covariation of successive components. COMPARISON WITH EXISTING METHODS: The new metric is complementary to other approaches to study the mechanisms underlying ERP generation, and does not suffer from potential caveats due to filtering artifacts. Moreover, the method proved to be more sensitive than another estimation of single-trial latency covariations using the cross-correlation function. CONCLUSION: The observed lack of latency covariation shows the presence of parallel, independent processing within each cortical sensory pathway.

Split-Brain: What We Know Now and Why This is Important for Understanding Consciousness.

Recently, the discussion regarding the consequences of cutting the corpus callosum ("split-brain") has regained momentum (Corballis, Corballis, Berlucchi, & Marzi, Brain, 141(6), e46, 2018; Pinto et al., Brain, 140(5), 1231-1237, 2017a; Pinto, Lamme, & de Haan, Brain, 140(11), e68, 2017; Volz & Gazzaniga, Brain, 140(7), 2051-2060, 2017; Volz, Hillyard, Miller, & Gazzaniga, Brain, 141(3), e15, 2018). This collective review paper aims to summarize the empirical common ground, to delineate the different interpretations, and to identify the remaining questions. In short, callosotomy leads to a broad breakdown of functional integration ranging from perception to attention. However, the breakdown is not absolute as several processes, such as action control, seem to remain unified. Disagreement exists about the responsible mechanisms for this remaining unity. The main issue concerns the first-person perspective of a split-brain patient. Does a split-brain harbor a split consciousness or is consciousness unified? The current consensus is that the body of evidence is insufficient to answer this question, and different suggestions are made with respect to how future studies might address this paucity. In addition, it is suggested that the answers might not be a simple yes or no but that intermediate conceptualizations need to be considered.

Parallel attentional facilitation of features and objects in early visual cortex.

Selective attention can enhance the processing of attended features across the entire visual field. Attention also spreads within objects, enhancing all internal locations and task-irrelevant features of selected objects. Here, we examine the extent to which attentional enhancement of a feature spreads across attended and unattended objects. Two fully overlapping counter-rotating bicolored surfaces of light and dark random dots were presented on a gray background of intermediate luminance. This stimulus creates a percept of two separate semitransparent surfaces and allows the measurement of feature- and object-based selections while controlling spatial attention. On each trial, human participants attended to a subset of dots defined by feature (luminance polarity) and object (surface) in order to detect brief episodes of radial motion while ignoring any events in the unattended groups of dots. Attentional selection was assessed by means of steady-state visual evoked potentials (SSVEPs) and behavioral measures. SSVEP amplitudes recorded at medial occipital electrode sites were modulated both by surface-based and luminance polarity-based selection in a manner consistent with independent multiplicative enhancement of attentional effects in different dimensions in early visual cortex. This finding supports the view that feature-based attention spreads across object boundaries, at least at an early stage of processing. However, SSVEPs elicited at more lateral electrode sites showed a hierarchical pattern of selection, potentially reflecting the binding of surface-defining features with luminance features to enable surface-based attention.

Involuntary orienting of attention to sight or sound relies on similar neural biasing mechanisms in early visual processing.

A sudden visual or acoustic change in the environment can capture attention involuntarily and facilitate perceptual processing of a subsequent visual target at the same location. The behavioral consequences of this involuntary (exogenous) cueing of attention have been well documented, but the underlying neural mechanisms and how they may differ depending on the modality of the cue remain unknown. We here report the effects of a spatially uninformative visual cue on the processing of a subsequent visual target and neural activity elicited by the cue itself and compare these results to the effects of an auditory cue. The results reveal that both visual and auditory cues enhanced the perceived brightness contrast of the subsequent co-localized target and boosted early cortical processing of the target beginning at about 100 ms post-target onset. Furthermore, both visual and auditory cues elicited a slow positive deflection (visible on target-absent trials) that was larger over contralateral relative to ipsilateral occipital scalp regions and was hypothesized to reflect the biasing of visual sensitivity for potential targets at that location. Overall, the data suggest that sudden events in the environment - regardless of sensory modality - initiate involuntary shifts of attention to the event's location and that the visual-perceptual consequences and neural mechanisms of these involuntary shifts are qualitatively similar for auditory and visual events.

Electrophysiological correlates of visual singleton detection.

Identifying a fixed-feature singleton that pops out from an otherwise uniform array of distractors elicits an ERP component called the N2pc over the posterior scalp. The N2pc has been used to track attention with millisecond accuracy, inform theories of visual selection, and test for specific attention deficits in clinical populations, yet it is still unclear what neuro-cognitive process gives rise to the component. One hypothesis is that the N2pc reflects a spatial filtering process that suppresses irrelevant distractors. In support of this hypothesis, Luck and Hillyard (1994a) showed that the N2pc is eliminated when the features of the target and distractors switch unpredictably across trials (so that participants cannot prepare to filter out irrelevant items). The present study aimed to replicate Luck and Hillyard's singleton detection experiment but with modifications to enhance the N2pc signal and to gain statistical power. We show that orientation singletons do, in fact, elicit the N2pc as well as an earlier-onset and longer-lasting singleton detection positivity over the occipital scalp when the target and distractor orientations swap randomly across trials. We conclude that spatial filtering might not play a major role in the generation of the N2pc and that the selection processes required to search for fixed-feature targets (in feature-search mode) are also engaged in the detection of variable-feature singletons (in singleton detection mode).

Neural Correlates of Enhanced Visual Attentional Control in Action Video Game Players: An Event-Related Potential Study.

Action video game players (AVGPs) outperform non-action video game players (NAVGPs) on a range of perceptual and attentional tasks. Although several studies have reported neuroplastic changes within the frontoparietal networks of attention in AVGPs, little is known about possible changes in attentional modulation in low-level visual areas. To assess the contribution of these different levels of neural processing to the perceptual and attentional enhancements noted in AVGPs, visual event-related potentials (ERPs) were recorded from 14 AVGPs and 14 NAVGPs during a target discrimination task that required participants to attend to rapid sequences of Gabor patches under either focused or divided attention conditions. AVGPs responded faster to target Gabors in the focused attention condition compared with the NAVGPs. Correspondingly, ERPs to standard Gabors revealed a more pronounced negativity in the time range of the parietally generated anterior N1 component in AVGPs compared with NAVGPs during focused attention. In addition, the P2 component of the visual ERP was more pronounced in AVGPs than in NAVGPs over the hemisphere contralateral to the stimulus position in response to standard Gabors. Contrary to predictions, however, attention-modulated occipital components generated in the low-level extrastriate visual pathways, including the P1 and posterior N1, showed no significant group differences. Thus, the main neural signature of enhanced perceptual and attentional control functions in AVGPs appears linked to an attention-dependent parietal process, indexed by the anterior N1 component, and possibly to more efficient higher-order perceptual processing, indexed by the P2 component.

Impaired Motion Processing in Schizophrenia and the Attenuated Psychosis Syndrome: Etiological and Clinical Implications.

OBJECTIVE: The ability to perceive the motion of biological objects, such as faces, is a critical component of daily function and correlates with the ability to successfully navigate social situations (social cognition). Deficits in motion perception in schizophrenia were first demonstrated about 20 years ago but remain understudied, especially in the early, potentially prodromal, stages of the illness. The authors examined the neural bases of visual sensory processing impairments, including motion, in patients with schizophrenia (N=63) and attenuated psychosis (clinical high risk) (N=32) compared with age-matched healthy control subjects (N=67). METHOD: Electrophysiological recordings during stimulus and motion processing were analyzed using oscillatory (time frequency) approaches that differentiated motion-onset-evoked activity from stimulus-onset sensory-evoked responses. These were compared with functional MRI (fMRI) measures of motion processing. RESULTS: Significant deficits in motion processing were observed across the two patient groups, and these deficits predicted impairments in both face-emotion recognition and cognitive function. In contrast to motion processing, sensory-evoked stimulus-onset responses were intact in patients with attenuated psychosis, and, further, the relative deficit in motion-onset responses compared with stimulus-onset responses predicted transition to schizophrenia. In patients with schizophrenia, motion detection deficits mapped to impaired activation in motion-sensitive visual cortex during fMRI. Additional visual impairments in patients with schizophrenia, not present in patients with attenuated psychosis, implicated other visual regions, including the middle occipital gyrus and pulvinar thalamic nucleus. CONCLUSIONS: The study findings emphasize the importance of sensory-level visual dysfunction in the etiology of schizophrenia and in the personal experience of individuals with the disorder and demonstrate that motion-processing deficits may predate illness onset and contribute to impaired function even in patients with attenuated psychosis.

The relationship between attention and consciousness: an expanded taxonomy and implications for 'no-report' paradigms.

Tensions between global neuronal workspace theory and recurrent processing theory have sparked much debate in the field of consciousness research. Here, we focus on one of the key distinctions between these theories: the proposed relationship between attention and consciousness. By reviewing recent empirical evidence, we argue that both theories contain key insights and that certain aspects of each theory can be reconciled into a novel framework that may help guide future research. Alternative theories are also considered, including attended intermediate-level representations theory, integrated information theory and higher order thought theory. With the aim of offering a fresh and nuanced perspective to current theoretical debates, an updated taxonomy of conscious and non-conscious states is proposed. This framework maps a wider spectrum of conscious states by incorporating contemporary views from cognitive neuroscience regarding the variety of attentional mechanisms that are known to interact with sensory processing. Whether certain types of attention are necessary for phenomenal and access consciousness is considered and incorporated into this extended taxonomy. To navigate this expanded space, we review recent 'no-report' paradigms and address several methodological misunderstandings in order to pave a clear path forward for identifying the neural basis of perceptual awareness.This article is part of the theme issue 'Perceptual consciousness and cognitive access'.

Neural mechanisms of divided feature-selective attention to colour.

Feature-based attentional selection of colour is challenging to investigate due to the multidimensional nature of colour-space. When attending concurrently to features from different feature dimensions (e.g. red and horizontal), the attentional selections of the separate dimensions are largely independent. Therefore, if colour constitutes multiple independent feature dimensions for attentional purposes, concurrently attending to two colours should be effective and independent of the specific configuration of target and distractor colours. Here, observers attended concurrently to two out of four fully overlapping random dot kinematograms of different colours, and the allocation of attention to each colour was assessed separately by recordings of steady-state visual evoked potentials. The magnitude of attention effects depended on colour proximity and was well described by a simple model which suggested that colour space is rescaled in an adaptive manner to achieve attentional selection. In conclusion, different spatially overlaid colours can be attended concurrently with an efficiency that is determined by their configuration in colour space, supporting the idea that (at least in terms of hue) colour acts as a single dimension for attentional purposes.

Does spatial attention modulate the earliest component of the visual evoked potential?

Whether visual spatial attention can modulate feedforward input to human primary visual cortex (V1) is debated. A prominent and long-standing hypothesis is that visual spatial attention can influence processing in V1, but only at delayed latencies suggesting a feedback-mediated mechanism and a lack of modulation during the initial afferent volley. The most promising challenge to this hypothesis comes from an event-related potential (ERP) study that showed an amplitude enhancement of the earliest visual ERP component, called the 'C1', in response to spatially attended relative to spatially unattended stimuli. In the Kelly et al. study, several important experimental design modifications were introduced, including tailoring the stimulus locations and recording electrodes to each individual subject. In the current study, we employed the same methodological procedures and tested for attentional enhancements of the C1 component in each quadrant of the visual field. Using the same analysis strategies as Kelly et al., we found no evidence for an attention-based modulation of the C1 (measured from 50-80 ms). Attention-based amplitude enhancements were clear and robust for the subsequent P1 component (90-140 ms). Thus, despite using methods specifically designed to reveal C1 attention effects, the current study provided no confirmatory evidence for such effects.

Still wanted: a reproducible demonstration of a genuine C1 attention effect.

Slotnick (this issue) has specified a number of experimental parameters that appear critical for enabling an attention-related modulation of the C1 component. These include stimulus presentation in the upper visual field, the presence of distractors, a high perceptual or attentional load, and measurements at midline occipito-parietal sites. While we agree with many of these recommendations, we would modify others and even dispute a few. Despite the employment of these parameters in a few existing studies, there has not yet been a convincing, reproducible demonstration of a modulation of the C1 component by spatial attention that can be localized to primary visual cortex.

Does spatial attention modulate the C1 component? The jury continues to deliberate.

The thoughful comments on our study (Baumgartner et al., this issue) that failed to replicate the C1 attention effect reported by a previous study roughly fall into three broad categories. First, the commentators identified specific differences between the two studies that may have contributed to the discrepant results. Second, they highlighted some of the theoretical and methodological problems that are encountered when trying to demonstrate attention effects on the initial evoked response in primary visual cortex. Third, they offered a number of proposals for optimizing experimental designs and analysis methods that may increase the likelihood of observing attention-related modulations of the C1. We consider each of these topics in turn.

Involuntary orienting of attention to a sound desynchronizes the occipital alpha rhythm and improves visual perception.

Directing attention voluntarily to the location of a visual target results in an amplitude reduction (desynchronization) of the occipital alpha rhythm (8-14Hz), which is predictive of improved perceptual processing of the target. Here we investigated whether modulations of the occipital alpha rhythm triggered by the involuntary orienting of attention to a salient but spatially non-predictive sound would similarly influence perception of a subsequent visual target. Target discrimination was more accurate when a sound preceded the target at the same location (validly cued trials) than when the sound was on the side opposite to the target (invalidly cued trials). This behavioral effect was accompanied by a sound-induced desynchronization of the alpha rhythm over the lateral occipital scalp. The magnitude of alpha desynchronization over the hemisphere contralateral to the sound predicted correct discriminations of validly cued targets but not of invalidly cued targets. These results support the conclusion that cue-induced alpha desynchronization over the occipital cortex is a manifestation of a general priming mechanism that improves visual processing and that this mechanism can be activated either by the voluntary or involuntary orienting of attention. Further, the observed pattern of alpha modulations preceding correct and incorrect discriminations of valid and invalid targets suggests that involuntary orienting to the non-predictive sound has a rapid and purely facilitatory influence on processing targets on the cued side, with no inhibitory influence on targets on the opposite side.

Perceptual Learning Induces Persistent Attentional Capture by Nonsalient Shapes.

Visual attention can be attracted automatically by salient simple features, but whether and how nonsalient complex stimuli such as shapes may capture attention in humans remains unclear. Here, we present strong electrophysiological evidence that a nonsalient shape presented among similar shapes can provoke a robust and persistent capture of attention as a consequence of extensive training in visual search (VS) for that shape. Strikingly, this attentional capture that followed perceptual learning (PL) was evident even when the trained shape was task-irrelevant, was presented outside the focus of top-down spatial attention, and was undetected by the observer. Moreover, this attentional capture persisted for at least 3-5 months after training had been terminated. This involuntary capture of attention was indexed by electrophysiological recordings of the N2pc component of the event-related brain potential, which was localized to ventral extrastriate visual cortex, and was highly predictive of stimulus-specific improvement in VS ability following PL. These findings provide the first evidence that nonsalient shapes can capture visual attention automatically following PL and challenge the prominent view that detection of feature conjunctions requires top-down focal attention.

Cross-modal orienting of visual attention.

This article reviews a series of experiments that combined behavioral and electrophysiological recording techniques to explore the hypothesis that salient sounds attract attention automatically and facilitate the processing of visual stimuli at the sound's location. This cross-modal capture of visual attention was found to occur even when the attracting sound was irrelevant to the ongoing task and was non-predictive of subsequent events. A slow positive component in the event-related potential (ERP) that was localized to the visual cortex was found to be closely coupled with the orienting of visual attention to a sound's location. This neural sign of visual cortex activation was predictive of enhanced perceptual processing and was paralleled by a desynchronization (blocking) of the ongoing occipital alpha rhythm. Further research is needed to determine the nature of the relationship between the slow positive ERP evoked by the sound and the alpha desynchronization and to understand how these electrophysiological processes contribute to improved visual-perceptual processing.