Perception of self-generated and externally-generated visual stimuli: Evidence from EEG and behavior.

Efference copy-based forward model mechanisms may help us to distinguish between self-generated and externally-generated sensory consequences. Previous studies have shown that self-initiation modulates neural and perceptual responses to identical stimulation. For example, event-related potentials (ERPs) elicited by tones that follow a button press are reduced in amplitude relative to ERPs elicited by passively attended tones. However, previous EEG studies investigating visual stimuli in this context are rare, provide inconclusive results, and lack adequate control conditions with passive movements. Furthermore, although self-initiation is known to modulate behavioral responses, it is not known whether differences in the amplitude of ERPs also reflect differences in perception of sensory outcomes. In this study, we presented to participants visual stimuli consisting of gray discs following either active button presses, or passive button presses, in which an electromagnet moved the participant's finger. Two discs presented visually 500-1250 ms apart followed each button press, and participants judged which of the two was more intense. Early components of the primary visual response (N1 and P2) over the occipital electrodes were suppressed in the active condition. Interestingly, suppression in the intensity judgment task was only correlated with suppression of the visual P2 component. These data support the notion of efference copy-based forward model predictions in the visual sensory modality, but especially later processes (P2) seem to be perceptually relevant. Taken together, the results challenge the assumption that N1 differences reflect perceptual suppression and emphasize the relevance of the P2 ERP component.

Subcortical shape alterations in major depressive disorder: Findings from the ENIGMA major depressive disorder working group.

Alterations in regional subcortical brain volumes have been investigated as part of the efforts of an international consortium, ENIGMA, to identify reliable neural correlates of major depressive disorder (MDD). Given that subcortical structures are comprised of distinct subfields, we sought to build significantly from prior work by precisely mapping localized MDD-related differences in subcortical regions using shape analysis. In this meta-analysis of subcortical shape from the ENIGMA-MDD working group, we compared 1,781 patients with MDD and 2,953 healthy controls (CTL) on individual measures of shape metrics (thickness and surface area) on the surface of seven bilateral subcortical structures: nucleus accumbens, amygdala, caudate, hippocampus, pallidum, putamen, and thalamus. Harmonized data processing and statistical analyses were conducted locally at each site, and findings were aggregated by meta-analysis. Relative to CTL, patients with adolescent-onset MDD (≤ 21 years) had lower thickness and surface area of the subiculum, cornu ammonis (CA) 1 of the hippocampus and basolateral amygdala (Cohen's d = -0.164 to -0.180). Relative to first-episode MDD, recurrent MDD patients had lower thickness and surface area in the CA1 of the hippocampus and the basolateral amygdala (Cohen's d = -0.173 to -0.184). Our results suggest that previously reported MDD-associated volumetric differences may be localized to specific subfields of these structures that have been shown to be sensitive to the effects of stress, with important implications for mapping treatments to patients based on specific neural targets and key clinical features.

Replication of a hippocampus specific effect of the tescalcin regulating variant rs7294919 on gray matter structure.

While the hippocampus remains a region of high interest for neuropsychiatric research, the precise contributors to hippocampal morphometry are still not well understood. We and others previously reported a hippocampus specific effect of a tescalcin gene (TESC) regulating single nucleotide polymorphism (rs7294919) on gray matter volume. Here we aimed to replicate and extend these findings. Two complementary morphometric approaches (voxel based morphometry (VBM) and automated volumetric segmentation) were applied in a well-powered cohort from the Marburg-Münster Affective Disorder Cohort Study (MACS) including N=1137 participants (n=636 healthy controls, n=501 depressed patients). rs7294919 homozygous T-allele genotype was significantly associated with lower hippocampal gray matter density as well as with reduced hippocampal volume. Exploratory whole brain VBM analyses revealed no further associations with gray matter volume outside the hippocampus. No interaction effects of rs7294919 with depression nor with childhood trauma on hippocampal morphometry could be detected. Hippocampal subfield analyses revealed similar effects of rs7294919 in all hippocampal subfields. In sum, our results replicate a hippocampus specific effect of rs7294919 on brain structure. Due to the robust evidence for a pronounced association between the reported polymorphism and hippocampal morphometry, future research should consider investigating the potential clinical and functional relevance of the reported association.

Predictive coding in a multisensory path integration task: An fMRI study.

During self-motion through an environment, our sensory systems are confronted with a constant flow of information from different modalities. To successfully navigate, self-induced sensory signals have to be dissociated from externally induced sensory signals. Previous studies have suggested that the processing of self-induced sensory information is modulated by means of predictive coding mechanisms. However, the neural correlates of processing self-induced sensory information from different modalities during self-motion are largely unknown. Here, we asked if and how the processing of visually simulated self-motion and/or associated auditory stimuli is modulated by self-controlled action. Participants were asked to actively reproduce a previously observed simulated self-displacement (path integration). Blood oxygen level-dependent (BOLD) activation during this path integration was compared with BOLD activation during a condition in which we passively replayed the exact sensory stimulus that had been produced by the participants in previous trials. We found supramodal BOLD suppression in parietal and frontal regions. Remarkably, BOLD contrast in sensory areas was enhanced in a modality-specific manner. We conclude that the effect of action on sensory processing is strictly dependent on the respective behavioral task and its relevance.

Longitudinal brain volume changes in major depressive disorder.

Patients with major depressive disorder (MDD) exhibit gray matter volume (GMV) reductions in limbic regions. Clinical variables-such as the number of depressive episodes-seem to affect volume alterations. It is unclear whether the observed cross-sectional GMV abnormalities in MDD change over time, and whether there is a longitudinal relationship between GMV changes and the course of disorder. We investigated T1 structural MRI images of 54 healthy control (HC) and 37 MDD patients in a 3-Tesla-MRI with a follow-up interval of 3 years. The Cat12 toolbox was used to analyze longitudinal data (p < 0.05, FWE-corrected, whole-brain analysis; flexible factorial design). Interaction effects indicated increasing GMV in MDD in the bilateral amygdala, and decreasing GMV in the right thalamus between T1 and T2. Further analyses comparing patients with a mild course of disorder (MCD; 0-1 depressive episode during the follow-up) to patients with a severe course of disorder (SCD; > 1 depressive episode during the follow-up) revealed increasing amygdalar volume in MCD. Our study confirms structural alterations in limbic regions in MDD patients and an association between these impairments and the course of disorder. Thus, we assume that the reported volumetric alterations in the left amygdala (i.e. volumetric normalization) are reversible and apparently driven by the clinical phenotype. Hence, these results support the assumption that the severity and progression of disease influences amygdalar GMV changes in MDD or vice versa.

Spatial-temporal dynamics of gesture-speech integration: a simultaneous EEG-fMRI study.

The semantic integration between gesture and speech (GSI) is mediated by the left posterior temporal sulcus/middle temporal gyrus (pSTS/MTG) and the left inferior frontal gyrus (IFG). Evidence from electroencephalography (EEG) suggests that oscillations in the alpha and beta bands may support processes at different stages of GSI. In the present study, we investigated the relationship between electrophysiological oscillations and blood-oxygen-level-dependent (BOLD) activity during GSI. In a simultaneous EEG-fMRI study, German participants (n = 19) were presented with videos of an actor either performing meaningful gestures in the context of a comprehensible German (GG) or incomprehensible Russian sentence (GR), or just speaking a German sentence (SG). EEG results revealed reduced alpha and beta power for the GG vs. SG conditions, while fMRI analyses showed BOLD increase in the left pSTS/MTG for GG > GR ∩ GG > SG. In time-window-based EEG-informed fMRI analyses, we further found a positive correlation between single-trial alpha power and BOLD signal in the left pSTS/MTG, the left IFG, and several sub-cortical regions. Moreover, the alpha-pSTS/MTG correlation was observed in an earlier time window in comparison to the alpha-IFG correlation, thus supporting a two-stage processing model of GSI. Our study shows that EEG-informed fMRI implies multiple roles of alpha oscillations during GSI, and that the method is a best candidate for multidimensional investigations on complex cognitive functions such as GSI.

The angular gyrus is a supramodal comparator area in action-outcome monitoring.

Predicting and processing the sensory consequences of one's own actions is essential to enable successful interactions with the environment. Previous studies have suggested that the angular gyrus detects discrepancies between predicted and actual action consequences, at least for unimodal feedback. However, most actions lead to multisensory consequences, raising the question whether previous models can sufficiently explain action-outcome processing. Here, we investigated neural comparator processes during detection of delays between action and unimodal or bimodal consequences in human subjects with fMRI, using parametric and connectivity analyses. Participants had to perform button presses, which led to the presentation of either a dot on the screen, a tone, or both, presented with a variable delay after the button press. Participants were asked to judge whether there was a delay between action and feedback. Activity in the angular gyrus correlated positively with delay for both visual, auditory, and audio-visual action consequences. Furthermore, the angular gyrus was functionally connected with midline structures such as the posterior cingulate cortex and precuneus in all conditions. Our results show that the angular gyrus is (1) a supramodal area, sensitive to delays in multiple modalities, and (2) functionally connected with self-referential areas during delay detection of both unimodal and bimodal action consequences. Overall, our results suggest that the angular gyrus functions as a mediator between perception and interpretation, and that this process is remarkably similar for unimodal and bimodal action consequences.

Predicting the Multisensory Consequences of One's Own Action: BOLD Suppression in Auditory and Visual Cortices.

Predictive mechanisms are essential to successfully interact with the environment and to compensate for delays in the transmission of neural signals. However, whether and how we predict multisensory action outcomes remains largely unknown. Here we investigated the existence of multisensory predictive mechanisms in a context where actions have outcomes in different modalities. During fMRI data acquisition auditory, visual and auditory-visual stimuli were presented in active and passive conditions. In the active condition, a self-initiated button press elicited the stimuli with variable short delays (0-417ms) between action and outcome, and participants had to detect the presence of a delay for auditory or visual outcome (task modality). In the passive condition, stimuli appeared automatically, and participants had to detect the number of stimulus modalities (unimodal/bimodal). For action consequences compared to identical but unpredictable control stimuli we observed suppression of the blood oxygen level depended (BOLD) response in a broad network including bilateral auditory and visual cortices. This effect was independent of task modality or stimulus modality and strongest for trials where no delay was detected (undetected

Predicting the sensory consequences of one's own action: First evidence for multisensory facilitation.

Predicting the sensory consequences of our own actions contributes to efficient sensory processing and might help distinguish the consequences of self- versus externally generated actions. Previous research using unimodal stimuli has provided evidence for the existence of a forward model, which explains how such sensory predictions are generated and used to guide behavior. However, whether and how we predict multisensory action outcomes remains largely unknown. Here, we investigated this question in two behavioral experiments. In Experiment 1, we presented unimodal (visual or auditory) and bimodal (visual and auditory) sensory feedback with various delays after a self-initiated buttonpress. Participants had to report whether they detected a delay between their buttonpress and the stimulus in the predefined task modality. In Experiment 2, the sensory feedback and task were the same as in Experiment 1, but in half of the trials the action was externally generated. We observed enhanced delay detection for bimodal relative to unimodal trials, with better performance in general for actively generated actions. Furthermore, in the active condition, the bimodal advantage was largest when the stimulus in the task-irrelevant modality was not delayed-that is, when it was time-contiguous with the action-as compared to when both the task-relevant and task-irrelevant modalities were delayed. This specific enhancement for trials with a nondelayed task-irrelevant modality was absent in the passive condition. These results suggest that a forward model creates predictions for multiple modalities, and consequently contributes to multisensory interactions in the context of action.

Target-based biomarker selection - Mineralocorticoid receptor-related biomarkers and treatment outcome in major depression.

Aldosterone and mineralocorticoid receptor (MR)-function have been related to depression. We examined central and peripheral parameters of MR-function in order to characterize their relationship to clinical treatment outcome after six weeks in patients with acute depression. 30 patients with a diagnosis of major depression were examined 3 times over a 6 week period. Aldosterone and cortisol salvia samples were taken at 7.00 a.m. before patients got out of bed. Easy to use e-devices were used to measure markers of central MR function, i.e. slow wave sleep (SWS) and heart-rate variability (HRV). Salt-taste intensity (STI) and salt pleasantness (SP) of a 0.9% salt solution were determined by a newly developed scale. In addition, systolic blood pressure (SBP) and plasma electrolytes were determined as markers for peripheral MR activity. The relationship between the levels of these biomarkers at baseline and the change in clinical outcome parameters (Hamilton depression rating scale (HDRS)-21, anxiety, QIDS and BDI) after 6 weeks of treatment was investigated. A higher aldosterone/cortisol ratio (Aldo/Cort) (n = 17 due to missing values; p < 0.05) and lower SBP (n = 24; p < 0.05) at baseline predicted poor outcome, as measured with the HDRS, independent of gender. Only in male patients higher STI, lower SP, lower SWS (all n = 13) and higher HRV (n = 11) at baseline predicted good outcome p < 0.05). Likewise, in male patients low baseline sodium appears to be predictive for a poor outcome (n = 12; p = 0.05; based on HDRS-6). In conclusion, correlates of higher central MR-activation are associated with poorer clinical improvement, particularly in men. This contrasts with the finding of a peripheral MR-desensitization in more refractory patients. As one potential mechanism to consider, sodium loss on the basis of dysfunctional peripheral MR function and additional environmental factors may trigger increased aldosterone secretion and consequently worse outcome. These markers deserve further study as potential biological correlates for therapy refractory depression.

Association of rs1006737 in CACNA1C with alterations in prefrontal activation and fronto-hippocampal connectivity.

BACKGROUND: Genome-wide association studies have identified the rs1006737 single nucleotide polymorphism (SNP) in the CACNA1C gene as a susceptibility locus for schizophrenia and bipolar disorder. On the neural systems level this association is explained by altered functioning of the dorsolateral prefrontal cortex (DLPFC) and the hippocampal formation (HF), brain regions also affected by mental illness. In the present study we investigated the association of rs1006737 genotype with prefrontal activation and fronto-hippocampal connectivity. METHODS: We used functional magnetic resonance imaging to measure neural activation during an n-back working memory task in 94 healthy subjects. All subjects were genotyped for the SNP rs1006737. We tested associations of the rs1006737 genotype with changes in working-memory-related DLPFC activation and functional integration using a seed region functional connectivity approach. RESULTS: Rs1006737 genotype was associated with altered right-hemispheric DLPFC activation. The homozygous A (risk) group showed decreased activation compared to G-allele carriers. Further, the functional connectivity analysis revealed a positive association of fronto-hippocampal connectivity with rs1006737 A alleles. CONCLUSIONS: We did not replicate the previous findings of increased right DLPFC activation in CACNA1C rs1006737 A homozygotes. In fact, we found the opposite effect, thus questioning prefrontal inefficiency as rs1006737 genotype-related intermediate phenotype. On the other hand, our results indicate that alterations in the functional coupling between the prefrontal cortex and the medial temporal lobe could represent a neural system phenotype that is mediated by CACNA1C rs1006737 and other genetic susceptibility loci for schizophrenia and bipolar disorder.

Neural correlates of semantic associations in patients with schizophrenia.

Patients with schizophrenia have semantic processing disturbances leading to expressive language deficits (formal thought disorder). The underlying pathology has been related to alterations in the semantic network and its neural correlates. Moreover, crossmodal processing, an important aspect of communication, is impaired in schizophrenia. Here we investigated specific processing abnormalities in patients with schizophrenia with regard to modality and semantic distance in a semantic priming paradigm. Fourteen patients with schizophrenia and fourteen demographically matched controls made visual lexical decisions on successively presented word-pairs (SOA = 350 ms) with direct or indirect relations, unrelated word-pairs, and pseudoword-target stimuli during fMRI measurement. Stimuli were presented in a unimodal (visual) or crossmodal (auditory-visual) fashion. On the neural level, the effect of semantic relation indicated differences (patients > controls) within the right angular gyrus and precuneus. The effect of modality revealed differences (controls > patients) within the left superior frontal, middle temporal, inferior occipital, right angular gyri, and anterior cingulate cortex. Semantic distance (direct vs. indirect) induced distinct activations within the left middle temporal, fusiform gyrus, right precuneus, and thalamus with patients showing fewer differences between direct and indirect word-pairs. The results highlight aberrant priming-related brain responses in patients with schizophrenia. Enhanced activation for patients possibly reflects deficits in semantic processes that might be caused by a delayed and enhanced spread of activation within the semantic network. Modality-specific decreases of activation in patients might be related to impaired perceptual integration. Those deficits could induce and increase the prominent symptoms of schizophrenia like impaired speech processing.

Baseline activity predicts working memory load of preceding task condition.

The conceptual notion of the so-called resting state of the brain has been recently challenged by studies indicating a continuing effect of cognitive processes on subsequent rest. In particular, activity in posterior parietal and medial prefrontal areas has been found to be modulated by preceding experimental conditions. In this study, we investigated which brain areas show working memory dependent patterns in subsequent baseline periods and how specific they are for the preceding experimental condition. During functional magnetic resonance imaging, 94 subjects performed a letter-version of the n-back task with the conditions 0-back and 2-back followed by a low-level baseline in which subjects had to passively observe the letters appearing. In a univariate analysis, 2-back served as control condition while 0-back, baseline after 0-back and baseline after 2-back were modeled as regressors to test for activity changes between both baseline conditions. Additionally, we tested, using Gaussian process classifiers, the recognition of task condition from functional images acquired during baseline. Besides the expected activity changes in the precuneus and medial prefrontal cortex, we found differential activity in the thalamus, putamen, and postcentral gyrus that were affected by the preceding task. The multivariate analysis revealed that images of the subsequent baseline block contain task related patterns that yield a recognition rate of 70%. The results suggest that the influence of a cognitive task on subsequent baseline is strong and specific for some areas but not restricted to areas of the so-called default mode network.

Acquired self-control of insula cortex modulates emotion recognition and brain network connectivity in schizophrenia.

Real-time functional magnetic resonance imaging (rtfMRI) is a novel technique that has allowed subjects to achieve self-regulation of circumscribed brain regions. Despite its anticipated therapeutic benefits, there is no report on successful application of this technique in psychiatric populations. The objectives of the present study were to train schizophrenia patients to achieve volitional control of bilateral anterior insula cortex on multiple days, and to explore the effect of learned self-regulation on face emotion recognition (an extensively studied deficit in schizophrenia) and on brain network connectivity. Nine patients with schizophrenia were trained to regulate the hemodynamic response in bilateral anterior insula with contingent rtfMRI neurofeedback, through a 2-weeks training. At the end of the training stage, patients performed a face emotion recognition task to explore behavioral effects of learned self-regulation. A learning effect in self-regulation was found for bilateral anterior insula, which persisted through the training. Following successful self-regulation, patients recognized disgust faces more accurately and happy faces less accurately. Improvements in disgust recognition were correlated with levels of self-activation of right insula. RtfMRI training led to an increase in the number of the incoming and outgoing effective connections of the anterior insula. This study shows for the first time that patients with schizophrenia can learn volitional brain regulation by rtfMRI feedback training leading to changes in the perception of emotions and modulations of the brain network connectivity. These findings open the door for further studies of rtfMRI in severely ill psychiatric populations, and possible therapeutic applications.

Supramodal representation of emotions.

Supramodal representation of emotion and its neural substrates have recently attracted attention as a marker of social cognition. However, the question whether perceptual integration of facial and vocal emotions takes place in primary sensory areas, multimodal cortices, or in affective structures remains unanswered yet. Using novel computer-generated stimuli, we combined emotional faces and voices in congruent and incongruent ways and assessed functional brain data (fMRI) during an emotional classification task. Both congruent and incongruent audiovisual stimuli evoked larger responses in thalamus and superior temporal regions compared with unimodal conditions. Congruent emotions were characterized by activation in amygdala, insula, ventral posterior cingulate (vPCC), temporo-occipital, and auditory cortices; incongruent emotions activated a frontoparietal network and bilateral caudate nucleus, indicating a greater processing load in working memory and emotion-encoding areas. The vPCC alone exhibited differential reactions to congruency and incongruency for all emotion categories and can thus be considered a central structure for supramodal representation of complex emotional information. Moreover, the left amygdala reflected supramodal representation of happy stimuli. These findings document that emotional information does not merge at the perceptual audiovisual integration level in unimodal or multimodal areas, but in vPCC and amygdala.

Neuromagnetic oscillations and hemodynamic correlates of P50 suppression in schizophrenia.

Behavioral and electrophysiological data indicate compromised stimulus suppression in schizophrenia. The physiological basis of this effect and its contributions to the etiology of the disease are poorly understood. We examined neural and metabolic measures of P50 suppression in 12 patients with schizophrenia and controls. First, whole-head magnetoencephalography (MEG) assessed amplitudes of left- and right-hemispheric evoked responses and induced oscillations. Secondly, functional magnetic resonance imaging (fMRI) measured the hemodynamic responses to pairs of beeps with a short interval (500ms) as compared with those with a long interval (1500ms). The suppression of alpha power (8-13Hz) time-locked to the stimuli was negatively correlated with the suppression of evoked components and the hemodynamic measures. Remarkably, the suppression of alpha power was reduced in the patients already prior to stimulus onset. Conceivably, alpha oscillations play a central role in stimulus adaptation of neuronal networks and reflect an active mechanism for sensory suppression. The reduced stimulus suppression in schizophrenia seems to be in part due to impaired generation of alpha oscillations in the auditory cortex, resulting in higher metabolic demand as detected by fMRI. Delayed recovery of alpha rhythm may reflect an impaired gating function and contribute to sensory and cognitive deficits in schizophrenia.

Impact of plant extracts tested in attention-deficit/hyperactivity disorder treatment on cell survival and energy metabolism in human neuroblastoma SH-SY5Y cells.

Plant extracts such as Hypericum perforatum and Pycnogenol have been tested as alternatives to the classical ADHD drugs. It has been possible to describe neuroprotective effects of such plant extracts. A reduction of ADHD symptoms could be shown in clinical studies after the application of Pycnogenol, which is a pine bark extract. The impacts of the standardized herbal extracts Hypericum perforatum, Pycnogenol and Enzogenol up to a concentration of 5000 ng/mL on cell survival and energy metabolism in human SH-SY5Y neuroblastoma cells has been investigated in the present examination. Hypericum perforatum significantly decreased the survival of cells after treatment with a concentration of 5000 ng/mL, whereas lower concentrations exerted no significant effects. Pycnogenol( induced a significant increase of cell survival after incubation with a concentration of 32.25 ng/mL and a concentration of 250 ng/mL. Other applied concentrations of Pycnogenol failed to exert significant effects. Treatment with Enzogenol did not lead to significant changes in cell survival.Concerning energy metabolism, the treatment of cells with a concentration of 5000 ng/mL Hypericum perforatum led to a significant increase of ATP levels, whereas treatment with a concentration of 500 ng/mL had no significant effect. Incubation of cells with Pycnogenol and Enzogenol exerted no significant effects.None of the tested substances caused any cytotoxic effect when used in therapeutically relevant concentrations.

Why the leash constrains the dog: the impact of semantic associations on sentence production.

The production of language is one of the most complex and amazing skills in humans. Increasing evidence demonstrated that associative relations (e.g., car - garage) play an important role during concept formation but during speech production the effects and processing of associations are highly debated. Hence, the present study investigated the impact of associations and different SOAs on the production of sentences (Experiment 1) and on naming objects (Experiment 2). In an adapted version of the picture-word interference task, participants were asked to name two pictures using a standardized sentence (e.g., "The car is to the left of the trousers"). Thereby, a simultaneous (SOA is 0 ms) or slightly preceding (SOA is -150 ms) auditory or visual distractor had to be ignored. Distractors were related to the first noun (for example: "The car is to the left to the trousers", distractor: "garage") or to the second noun (distractor: "belt") or unrelated to both nouns (distractor: "bottle") of the sentence. At simultaneous presentation, visual and auditory distractors related to the first noun of the sentence prolonged naming responses (i.e., interference). For slightly preceding distractors, only visual presentation induced interference for the first noun of the sentence. During no condition, longer naming responses were found for the second noun of the sentence. These effects suggest that associatively related concepts are active during speech production and can be competitors, i.e., they lead to semantic interference. In Experiment 2, subjects had to name an object (e.g., car) while ignoring a visually presented distractor (e.g., motor). The stimulus set was the same as in Experiment 1. The results showed a facilitation effect if the distractor and the target were associatively related. Overall, the current results provide new insight in the models of speech production: while during single word production, associations facilitate naming, they interfere during sentence production. Hence, associations have an important influence on producing speech but the impact is varied by the context, i.e., single word or sentential.

Memory effects of speech and gesture binding: cortical and hippocampal activation in relation to subsequent memory performance.

In human face-to-face communication, the content of speech is often illustrated by coverbal gestures. Behavioral evidence suggests that gestures provide advantages in the comprehension and memory of speech. Yet, how the human brain integrates abstract auditory and visual information into a common representation is not known. Our study investigates the neural basis of memory for bimodal speech and gesture representations. In this fMRI study, 12 participants were presented with video clips showing an actor performing meaningful metaphoric gestures (MG), unrelated, free gestures (FG), and no arm and hand movements (NG) accompanying sentences with an abstract content. After the fMRI session, the participants performed a recognition task. Behaviorally, the participants showed the highest hit rate for sentences accompanied by meaningful metaphoric gestures. Despite comparable old/new discrimination performances (d') for the three conditions, we obtained distinct memory-related left-hemispheric activations in the inferior frontal gyrus (IFG), the premotor cortex (BA 6), and the middle temporal gyrus (MTG), as well as significant correlations between hippocampal activation and memory performance in the metaphoric gesture condition. In contrast, unrelated speech and gesture information (FG) was processed in areas of the left occipito-temporal and cerebellar region and the right IFG just like the no-gesture condition (NG). We propose that the specific left-lateralized activation pattern for the metaphoric speech-gesture sentences reflects semantic integration of speech and gestures. These results provide novel evidence about the neural integration of abstract speech and gestures as it contributes to subsequent memory performance.

Neural interaction of speech and gesture: differential activations of metaphoric co-verbal gestures.

Gestures are an important part of human communication. However, little is known about the neural correlates of gestures accompanying speech comprehension. The goal of this study is to investigate the neural basis of speech-gesture interaction as reflected in activation increase and decrease during observation of natural communication. Fourteen German participants watched video clips of 5 s duration depicting an actor who performed metaphoric gestures to illustrate the abstract content of spoken sentences. Furthermore, video clips of isolated gestures (without speech), isolated spoken sentences (without gestures) and gestures in the context of an unknown language (Russian) were additionally presented while functional magnetic resonance imaging (fMRI) data were acquired. Bimodal speech and gesture processing led to left hemispheric activation increases of the posterior middle temporal gyrus, the premotor cortex, the inferior frontal gyrus, and the right superior temporal sulcus. Activation reductions during the bimodal condition were located in the left superior temporal gyrus and the left posterior insula. Gesture related activation increases and decreases were dependent on language semantics and were not found in the unknown-language condition. Our results suggest that semantic integration processes for bimodal speech plus gesture comprehension are reflected in activation increases in the classical left hemispheric language areas. Speech related gestures seem to enhance language comprehension during the face-to-face communication.