Perception of self-generated and externally-generated visual stimuli: Evidence from EEG and behavior.

Efference copy-based forward model mechanisms may help us to distinguish between self-generated and externally-generated sensory consequences. Previous studies have shown that self-initiation modulates neural and perceptual responses to identical stimulation. For example, event-related potentials (ERPs) elicited by tones that follow a button press are reduced in amplitude relative to ERPs elicited by passively attended tones. However, previous EEG studies investigating visual stimuli in this context are rare, provide inconclusive results, and lack adequate control conditions with passive movements. Furthermore, although self-initiation is known to modulate behavioral responses, it is not known whether differences in the amplitude of ERPs also reflect differences in perception of sensory outcomes. In this study, we presented to participants visual stimuli consisting of gray discs following either active button presses, or passive button presses, in which an electromagnet moved the participant's finger. Two discs presented visually 500-1250 ms apart followed each button press, and participants judged which of the two was more intense. Early components of the primary visual response (N1 and P2) over the occipital electrodes were suppressed in the active condition. Interestingly, suppression in the intensity judgment task was only correlated with suppression of the visual P2 component. These data support the notion of efference copy-based forward model predictions in the visual sensory modality, but especially later processes (P2) seem to be perceptually relevant. Taken together, the results challenge the assumption that N1 differences reflect perceptual suppression and emphasize the relevance of the P2 ERP component.

Predictive coding in a multisensory path integration task: An fMRI study.

During self-motion through an environment, our sensory systems are confronted with a constant flow of information from different modalities. To successfully navigate, self-induced sensory signals have to be dissociated from externally induced sensory signals. Previous studies have suggested that the processing of self-induced sensory information is modulated by means of predictive coding mechanisms. However, the neural correlates of processing self-induced sensory information from different modalities during self-motion are largely unknown. Here, we asked if and how the processing of visually simulated self-motion and/or associated auditory stimuli is modulated by self-controlled action. Participants were asked to actively reproduce a previously observed simulated self-displacement (path integration). Blood oxygen level-dependent (BOLD) activation during this path integration was compared with BOLD activation during a condition in which we passively replayed the exact sensory stimulus that had been produced by the participants in previous trials. We found supramodal BOLD suppression in parietal and frontal regions. Remarkably, BOLD contrast in sensory areas was enhanced in a modality-specific manner. We conclude that the effect of action on sensory processing is strictly dependent on the respective behavioral task and its relevance.

Spatial-temporal dynamics of gesture-speech integration: a simultaneous EEG-fMRI study.

The semantic integration between gesture and speech (GSI) is mediated by the left posterior temporal sulcus/middle temporal gyrus (pSTS/MTG) and the left inferior frontal gyrus (IFG). Evidence from electroencephalography (EEG) suggests that oscillations in the alpha and beta bands may support processes at different stages of GSI. In the present study, we investigated the relationship between electrophysiological oscillations and blood-oxygen-level-dependent (BOLD) activity during GSI. In a simultaneous EEG-fMRI study, German participants (n = 19) were presented with videos of an actor either performing meaningful gestures in the context of a comprehensible German (GG) or incomprehensible Russian sentence (GR), or just speaking a German sentence (SG). EEG results revealed reduced alpha and beta power for the GG vs. SG conditions, while fMRI analyses showed BOLD increase in the left pSTS/MTG for GG > GR ∩ GG > SG. In time-window-based EEG-informed fMRI analyses, we further found a positive correlation between single-trial alpha power and BOLD signal in the left pSTS/MTG, the left IFG, and several sub-cortical regions. Moreover, the alpha-pSTS/MTG correlation was observed in an earlier time window in comparison to the alpha-IFG correlation, thus supporting a two-stage processing model of GSI. Our study shows that EEG-informed fMRI implies multiple roles of alpha oscillations during GSI, and that the method is a best candidate for multidimensional investigations on complex cognitive functions such as GSI.

The angular gyrus is a supramodal comparator area in action-outcome monitoring.

Predicting and processing the sensory consequences of one's own actions is essential to enable successful interactions with the environment. Previous studies have suggested that the angular gyrus detects discrepancies between predicted and actual action consequences, at least for unimodal feedback. However, most actions lead to multisensory consequences, raising the question whether previous models can sufficiently explain action-outcome processing. Here, we investigated neural comparator processes during detection of delays between action and unimodal or bimodal consequences in human subjects with fMRI, using parametric and connectivity analyses. Participants had to perform button presses, which led to the presentation of either a dot on the screen, a tone, or both, presented with a variable delay after the button press. Participants were asked to judge whether there was a delay between action and feedback. Activity in the angular gyrus correlated positively with delay for both visual, auditory, and audio-visual action consequences. Furthermore, the angular gyrus was functionally connected with midline structures such as the posterior cingulate cortex and precuneus in all conditions. Our results show that the angular gyrus is (1) a supramodal area, sensitive to delays in multiple modalities, and (2) functionally connected with self-referential areas during delay detection of both unimodal and bimodal action consequences. Overall, our results suggest that the angular gyrus functions as a mediator between perception and interpretation, and that this process is remarkably similar for unimodal and bimodal action consequences.

Predicting the Multisensory Consequences of One's Own Action: BOLD Suppression in Auditory and Visual Cortices.

Predictive mechanisms are essential to successfully interact with the environment and to compensate for delays in the transmission of neural signals. However, whether and how we predict multisensory action outcomes remains largely unknown. Here we investigated the existence of multisensory predictive mechanisms in a context where actions have outcomes in different modalities. During fMRI data acquisition auditory, visual and auditory-visual stimuli were presented in active and passive conditions. In the active condition, a self-initiated button press elicited the stimuli with variable short delays (0-417ms) between action and outcome, and participants had to detect the presence of a delay for auditory or visual outcome (task modality). In the passive condition, stimuli appeared automatically, and participants had to detect the number of stimulus modalities (unimodal/bimodal). For action consequences compared to identical but unpredictable control stimuli we observed suppression of the blood oxygen level depended (BOLD) response in a broad network including bilateral auditory and visual cortices. This effect was independent of task modality or stimulus modality and strongest for trials where no delay was detected (undetected

Modulating the assessment of semantic speech-gesture relatedness via transcranial direct current stimulation of the left frontal cortex.

BACKGROUND: Co-verbal gestures are crucial for communication. Neuroimaging studies suggest that the left frontal lobe may be especially important for processing metaphoric co-verbal gestures. However, so far, the specific functional relevance of the left frontal lobe in metaphoric (abstract sentence content) co-verbal gesture processing compared to iconic (concrete sentence content) co-verbal gesture processing has not been demonstrated. OBJECTIVE: We investigated the functional relevance of the left frontal lobe for processing metaphoric co-verbal gestures using transcranial direct current stimulation (tDCS). We hypothesised a polarisation dependent effect of left frontal tDCS on reaction times and ratings in a speech-gesture semantic relatedness assessment task. METHODS: We applied anodal, cathodal and sham stimulation to the frontal (F3/F4), parietal (CP3/CP4) and frontoparietal (F3/CP4) areas. During stimulation, seventeen subjects were presented with videos of an actor saying concrete or abstract sentences accompanied by related or unrelated iconic or metaphoric gestures and rated to what extent gestures were related to the sentence content. RESULTS: We found electrode localisation- and polarisation-dependent changes in reaction times and ratings for metaphoric co-verbal gestures compared to iconic gestures. Post-hoc tests revealed a specific polarisation effect for frontoparietal stimulation sites: compared to cathodal stimulation, anodal stimulation of the left frontal lobe decreased reaction times and relatedness assessments for metaphoric conditions only. CONCLUSION: Using tDCS, we demonstrated the functional relevance of the left frontal lobe for processing metaphoric co-verbal gestures. Thus, tDCS may possibly constitute an approach to facilitate metaphoric co-verbal gesture-processing in patients with specific deficits.

Predicting the sensory consequences of one's own action: First evidence for multisensory facilitation.

Predicting the sensory consequences of our own actions contributes to efficient sensory processing and might help distinguish the consequences of self- versus externally generated actions. Previous research using unimodal stimuli has provided evidence for the existence of a forward model, which explains how such sensory predictions are generated and used to guide behavior. However, whether and how we predict multisensory action outcomes remains largely unknown. Here, we investigated this question in two behavioral experiments. In Experiment 1, we presented unimodal (visual or auditory) and bimodal (visual and auditory) sensory feedback with various delays after a self-initiated buttonpress. Participants had to report whether they detected a delay between their buttonpress and the stimulus in the predefined task modality. In Experiment 2, the sensory feedback and task were the same as in Experiment 1, but in half of the trials the action was externally generated. We observed enhanced delay detection for bimodal relative to unimodal trials, with better performance in general for actively generated actions. Furthermore, in the active condition, the bimodal advantage was largest when the stimulus in the task-irrelevant modality was not delayed-that is, when it was time-contiguous with the action-as compared to when both the task-relevant and task-irrelevant modalities were delayed. This specific enhancement for trials with a nondelayed task-irrelevant modality was absent in the passive condition. These results suggest that a forward model creates predictions for multiple modalities, and consequently contributes to multisensory interactions in the context of action.

Neural correlates of semantic associations in patients with schizophrenia.

Patients with schizophrenia have semantic processing disturbances leading to expressive language deficits (formal thought disorder). The underlying pathology has been related to alterations in the semantic network and its neural correlates. Moreover, crossmodal processing, an important aspect of communication, is impaired in schizophrenia. Here we investigated specific processing abnormalities in patients with schizophrenia with regard to modality and semantic distance in a semantic priming paradigm. Fourteen patients with schizophrenia and fourteen demographically matched controls made visual lexical decisions on successively presented word-pairs (SOA = 350 ms) with direct or indirect relations, unrelated word-pairs, and pseudoword-target stimuli during fMRI measurement. Stimuli were presented in a unimodal (visual) or crossmodal (auditory-visual) fashion. On the neural level, the effect of semantic relation indicated differences (patients > controls) within the right angular gyrus and precuneus. The effect of modality revealed differences (controls > patients) within the left superior frontal, middle temporal, inferior occipital, right angular gyri, and anterior cingulate cortex. Semantic distance (direct vs. indirect) induced distinct activations within the left middle temporal, fusiform gyrus, right precuneus, and thalamus with patients showing fewer differences between direct and indirect word-pairs. The results highlight aberrant priming-related brain responses in patients with schizophrenia. Enhanced activation for patients possibly reflects deficits in semantic processes that might be caused by a delayed and enhanced spread of activation within the semantic network. Modality-specific decreases of activation in patients might be related to impaired perceptual integration. Those deficits could induce and increase the prominent symptoms of schizophrenia like impaired speech processing.

An overview of the neuro-cognitive processes involved in the encoding, consolidation, and retrieval of true and false memories.

Perception and memory are imperfect reconstructions of reality. These reconstructions are prone to be influenced by several factors, which may result in false memories. A false memory is the recollection of an event, or details of an episode, that did not actually occur. Memory formation comprises at least three different sub-processes: encoding, consolidation and the retrieval of the learned material. All of these sub-processes are vulnerable for specific errors and consequently may result in false memories. Whereas, processes like imagery, self-referential encoding or spreading activation can lead to the formation of false memories at encoding, semantic generalization during sleep and updating processes due to misleading post event information, in particular, are relevant at the consolidation stage. Finally at the retrieval stage, monitoring processes, which are assumed to be essential to reject false memories, are of specific importance. Different neuro-cognitive processes have been linked to the formation of true and false memories. Most consistently the medial temporal lobe and the medial and lateral prefrontal cortex have been reported with regard to the formation of true and false memories. Despite the fact that all phases entailing memory formation, consolidation of stored information and retrieval processes, are relevant for the forming of false memories, most studies focused on either memory encoding or retrieval. Thus, future studies should try to integrate data from all phases to give a more comprehensive view on systematic memory distortions. An initial outline is developed within this review to connect the different memory stages and research strategies.

Memory effects of speech and gesture binding: cortical and hippocampal activation in relation to subsequent memory performance.

In human face-to-face communication, the content of speech is often illustrated by coverbal gestures. Behavioral evidence suggests that gestures provide advantages in the comprehension and memory of speech. Yet, how the human brain integrates abstract auditory and visual information into a common representation is not known. Our study investigates the neural basis of memory for bimodal speech and gesture representations. In this fMRI study, 12 participants were presented with video clips showing an actor performing meaningful metaphoric gestures (MG), unrelated, free gestures (FG), and no arm and hand movements (NG) accompanying sentences with an abstract content. After the fMRI session, the participants performed a recognition task. Behaviorally, the participants showed the highest hit rate for sentences accompanied by meaningful metaphoric gestures. Despite comparable old/new discrimination performances (d') for the three conditions, we obtained distinct memory-related left-hemispheric activations in the inferior frontal gyrus (IFG), the premotor cortex (BA 6), and the middle temporal gyrus (MTG), as well as significant correlations between hippocampal activation and memory performance in the metaphoric gesture condition. In contrast, unrelated speech and gesture information (FG) was processed in areas of the left occipito-temporal and cerebellar region and the right IFG just like the no-gesture condition (NG). We propose that the specific left-lateralized activation pattern for the metaphoric speech-gesture sentences reflects semantic integration of speech and gestures. These results provide novel evidence about the neural integration of abstract speech and gestures as it contributes to subsequent memory performance.

Neural interaction of speech and gesture: differential activations of metaphoric co-verbal gestures.

Gestures are an important part of human communication. However, little is known about the neural correlates of gestures accompanying speech comprehension. The goal of this study is to investigate the neural basis of speech-gesture interaction as reflected in activation increase and decrease during observation of natural communication. Fourteen German participants watched video clips of 5 s duration depicting an actor who performed metaphoric gestures to illustrate the abstract content of spoken sentences. Furthermore, video clips of isolated gestures (without speech), isolated spoken sentences (without gestures) and gestures in the context of an unknown language (Russian) were additionally presented while functional magnetic resonance imaging (fMRI) data were acquired. Bimodal speech and gesture processing led to left hemispheric activation increases of the posterior middle temporal gyrus, the premotor cortex, the inferior frontal gyrus, and the right superior temporal sulcus. Activation reductions during the bimodal condition were located in the left superior temporal gyrus and the left posterior insula. Gesture related activation increases and decreases were dependent on language semantics and were not found in the unknown-language condition. Our results suggest that semantic integration processes for bimodal speech plus gesture comprehension are reflected in activation increases in the classical left hemispheric language areas. Speech related gestures seem to enhance language comprehension during the face-to-face communication.