The role of the left and right inferior frontal gyrus in processing metaphoric and unrelated co-speech gestures.

Gestures are an integral part of in-person conversations and complement the meaning of the speech they accompany. The neural processing of co-speech gestures is supported by a mostly left-lateralized network of fronto-temporal regions. However, in contrast to iconic gestures, metaphoric as well as unrelated gestures have been found to more strongly engage the left and right inferior frontal gyrus (IFG), respectively. With this study, we conducted the first systematic comparison of all three types of gestures and resulting potential laterality effects. During collection of functional imaging data, 74 subjects were presented with 5 s videos of abstract speech with related metaphoric gestures, concrete speech with related iconic gestures and concrete speech with unrelated gestures. They were asked to judge whether the content of the speech and gesture matched or not. Differential contrasts revealed that both abstract related and concrete unrelated compared to concrete related stimuli elicited stronger activation of the bilateral IFG. Analyses of lateralization indices for IFG activation further showed a left hemispheric dominance for metaphoric gestures and a right hemispheric dominance for unrelated gestures. Our results give support to the hypothesis that the bilateral IFG is activated specifically when processing load for speech-gesture combinations is high. In addition, laterality effects indicate a stronger involvement of the right IFG in mismatch detection and conflict processing, whereas the left IFG performs the actual integration of information from speech and gesture.

Feeling addressed! The neural processing of social communicative cues in patients with major depression.

The feeling of being addressed is the first step in a complex processing stream enabling successful social communication. Social impairments are a relevant characteristic of patients with major depressive disorder (MDD). Here, we investigated a mechanism which-if impaired-might contribute to withdrawal or isolation in MDD, namely, the neural processing of social cues such as body orientation and gesture. During funtional magnetic resonance imaging (fMRI) data acquisition, 33 patients with MDD and 43 healthy control subjects watched video clips of a speaking actor: one version with a gesture accompanying the speech and one without gesture. Videos were filmed simultaneously from two different viewpoints: one with the actor facing the viewer head-on (frontal) and one side-view (lateral). After every clip, the participants were instructed to evaluate whether they felt addressed or not. Despite overall comparable addressment ratings and a large overlap in activation patterns in MDD and healthy subjects for gesture processing, the anterior cingulate cortex, bilateral superior/middle frontal cortex, and right angular gyrus were more strongly activated in patients than in healthy subjects for the frontal conditions. Our analyses revealed that patients showed specifically higher activation than healthy subjects for the frontal condition without gesture in regions including the posterior cingulate cortex, left prefrontal cortex, and the left hippocampus. We conclude that MDD patients can recognize and interpret social cues such as gesture or body orientation; however, they seem to require more neural resources. This additional effort might affect successful communication and contribute to social isolation in MDD.

The facilitative effect of gestures on the neural processing of semantic complexity in a continuous narrative.

Gestures are elemental components of social communication and aid comprehension of verbal messages; however, little is known about the potential role of gestures in facilitating processing of semantic complexity in an ecologically valid setting. The goal of this study was to investigate whether cognitive load, as indexed by semantic complexity, is modulated by the presentation of gestures accompanying speech. Twenty healthy participants watched 16 video clips of a short narrative while instructed to carefully listen to and watch the narrator while functional magnetic resonance imaging (fMRI) data were acquired. The videos contained passages with and without various co-speech gestures, as well as passages where the semantic complexity was either low or high, as measured by the metric of idea density. Increasing semantic complexity led to reduced activation within the default mode network (DMN); whereas, presents of gestures decreased activation in language-related regions (left middle temporal gyrus and left inferior frontal gyrus) and increased activation in high-level visual and multimodal regions of occipitotemporal cortex. Most interestingly, an interaction between semantic complexity and gestures was observed in a language-related area in left anterior temporal cortex; specifically, increasing gestures led to a greater drop in activation with high vs. Low semantic complexity. These results provide evidence that the facilitation of gestures on semantic processing, particularly for complex narratives, is reflected in the neural substrates of language processing.

Intergroup empathy: Enhanced neural resonance for ingroup facial emotion in a shared neural production-perception network.

INTRODUCTION: Empathic behavior and related neural processing are strongly modified by group membership. Shared neural circuits for the production and perception of facial emotional expressions represent mirror neuron mechanisms which play a pivotal role for empathy. In this study, we investigate the influence of group membership on mirror neuron mechanisms for emotional facial expressions. METHODS: In a functional magnetic resonance imaging task, 178 healthy subjects perceived emotional and neutral facial expressions of artificial ingroup and outgroup members, displayed as 5 s video clips, and produced these facial expressions themselves. Before scanning, artificial group membership was manipulated ad-hoc through a minimal group paradigm. RESULTS: Shared neural activity for emotional facial expression production and perception was revealed in a large network with right-hemispheric preponderance encompassing motor mirror neuron regions, i.e., inferior frontal gyrus, supplementary motor area and middle temporal gyrus, in addition to limbic regions, i.e., amygdala, hippocampus, para-hippocampus, and insula. Within this network there was greater neural activation for ingroup compared to outgroup members in temporal poles, amygdalae, the left insula, the left inferior frontal gyrus, and the inferior and middle temporal gyrus, the right hippocampus and parahippocampus. DISCUSSION: We validate and extend knowledge on brain regions with mirror neuron properties. Most crucially, we provide evidence for the influence of group membership on regions within the mirror neuron system, indicating more neural resonance (mirroring) for ingroup facial emotional expressions.

Action-Related Speech Modulates Beta Oscillations During Observation of Tool-Use Gestures.

Language and action have been thought of as closely related. Comprehending words or phrases that are related to actions commonly activates motor and premotor areas, and this comprehension process interacts with action preparation and/or execution. However, it remains unclear whether comprehending action-related language interacts with action observation. In the current study, we examined whether the observation of tool-use gesture subjects to interaction with language. In an electroencephalography (EEG) study (n = 20), participants were presented with video clips of an actor performing tool-use (TU, e.g., hammering with a fist) and emblematic (EM, e.g., the thumb up sign for 'good job') gestures accompanied by either comprehensible German (G) or incomprehensible Russian sentences (R). Participants performed a semantic judging task, evaluating whether the co-speech gestures were object- or socially-related. Behavioral results from the semantic task showed faster response for the TU versus EM gestures only in the German condition. For EEG, we found that TU elicited beta power decrease (~ 20 Hz) when compared to EM gestures, however this effect was reduced when gestures were accompanied by German instead of Russian sentences. We concluded that the processing of action-related sentences might facilitate gesture observation, in the sense that motor simulation required for TU gestures, as indexed by reduced beta power, was modulated when accompanied by comprehensible German speech. Our results corroborate the functional role of the beta oscillations during perception of hand gestures, and provide novel evidence concerning language-motor interaction.

Feeling addressed! The role of body orientation and co-speech gesture in social communication.

During face-to-face communication, body orientation and coverbal gestures influence how information is conveyed. The neural pathways underpinning the comprehension of such nonverbal social cues in everyday interaction are to some part still unknown. During fMRI data acquisition, 37 participants were presented with video clips showing an actor speaking short sentences. The actor produced speech-associated iconic gestures (IC) or no gestures (NG) while he was visible either from an egocentric (ego) or from an allocentric (allo) position. Participants were asked to indicate via button press whether they felt addressed or not. We found a significant interaction of body orientation and gesture in addressment evaluations, indicating that participants evaluated IC-ego conditions as most addressing. The anterior cingulate cortex (ACC) and left fusiform gyrus were stronger activated for egocentric versus allocentric actor position in gesture context. Activation increase in the ACC for IC-ego>IC-allo further correlated positively with increased addressment ratings in the egocentric gesture condition. Gesture-related activation increase in the supplementary motor area, left inferior frontal gyrus and right insula correlated positively with gesture-related increase of addressment evaluations in the egocentric context. Results indicate that gesture use and body-orientation contribute to the feeling of being addressed and together influence neural processing in brain regions involved in motor simulation, empathy and mentalizing.

Neural Basis of Speech-Gesture Mismatch Detection in Schizophrenia Spectrum Disorders.

Patients with schizophrenia spectrum disorders (SSD) exhibit an aberrant perception and comprehension of abstract speech-gesture combinations associated with dysfunctional activation of the left inferior frontal gyrus (IFG). Recently, a significant deficit of speech-gesture mismatch detection was identified in SSD, but the underlying neural mechanisms have not yet been examined. A novel mismatch-detection fMRI paradigm was implemented manipulating speech-gesture abstractness (abstract/concrete) and relatedness (related/unrelated). During fMRI data acquisition, 42 SSD patients (schizophrenia, schizoaffective disorder, or other non-organic psychotic disorder [ICD-10: F20, F25, F28; DSM-IV: 295.X]) and 36 healthy controls were presented with short video clips of an actor reciting abstract or concrete sentences accompanied by either a semantically related or unrelated gesture. Participants indicated via button press whether they perceived each gesture as matching the speech content or not. Speech-gesture mismatch detection performance was significantly impaired in patients compared to controls. fMRI data analysis revealed that patients showed lower activation in bilateral frontal areas, including the IFG for all abstract > concrete speech-gesture pairs. In addition, they exhibited reduced engagement of the right supplementary motor area (SMA) and bilateral anterior cingulate cortices (ACC) for unrelated > related stimuli. We provide first evidence that impaired speech-gesture mismatch detection in SSD could be the result of dysfunctional activation of the SMA and ACC. Failure to activate the left IFG disrupts the integration of abstract speech-gesture combinations in particular. Future investigations should focus on brain stimulation of the SMA, ACC, and the IFG to improve communication and social functioning in SSD.

Modality-specific dysfunctional neural processing of social-abstract and non-social-concrete information in schizophrenia.

Schizophrenia is characterized by marked communication dysfunctions encompassing potential impairments in the processing of social-abstract and non-social-concrete information, especially in everyday situations where multiple modalities are present in the form of speech and gesture. To date, the neurobiological basis of these deficits remains elusive. In a functional magnetic resonance imaging (fMRI) study, 17 patients with schizophrenia or schizoaffective disorder, and 18 matched controls watched videos of an actor speaking, gesturing (unimodal), and both speaking and gesturing (bimodal) about social or non-social events in a naturalistic way. Participants were asked to judge whether each video contains person-related (social) or object-related (non-social) information. When processing social-abstract content, patients showed reduced activation in the medial prefrontal cortex (mPFC) only in the gesture but not in the speech condition. For non-social-concrete content, remarkably, reduced neural activation for patients in the left postcentral gyrus and the right insula was observed only in the speech condition. Moreover, in the bimodal conditions, patients displayed improved task performance and comparable activation to controls in both social and non-social content. To conclude, patients with schizophrenia displayed modality-specific aberrant neural processing of social and non-social information, which is not present for the bimodal conditions. This finding provides novel insights into dysfunctional multimodal communication in schizophrenia, and may have potential therapeutic implications.

Conflicting group memberships modulate neural activation in an emotional production-perception network.

Social group membership modulates the neural processing of emotional facial expressions, which, in turn, recruits part of the neural production system. However, little is known about how mixed - and potentially conflicting - social identity cues affect this mechanism. In this study, we tested the hypothesis that incongruent cues of two group memberships (ethnic and experimentally created minimal groups) elicit conflict processing for neutral and, in particular, angry facial expressions. We further expected this interaction of ethnic group, minimal group and emotion to also modulate activation in an emotional production-perception network. Twenty-two healthy German subjects saw dynamic angry and neutral facial expressions, presented in short video clips during functional MRI scanning. All depicted actors belonged to an ethnic in- or outgroup (German or Turkish descent) as well as an ad hoc experimentally created minimal in- or outgroup. Additionally, subjects produced angry or neutral expressions themselves. The whole-brain interaction of ethnic group, minimal group and emotion revealed activity in the right parietal lobule and left cerebellum. Both showed strongest activation for angry faces with conflicting group memberships (e.g., 'ethnic outgroup/minimal ingroup'). In addition, a sub-region of the left cerebellum cluster was also activated for both perceiving and producing angry versus neutral expressions. These results suggest that incongruent group members displaying angry facial expressions elicit conflict processing. Group interaction effects in an emotional production-perception network further indicate stronger neural resonance for incongruent group members.

Emotion specific neural activation for the production and perception of facial expressions.

The distinction between different facial emotions is crucial for interpersonal communication. Shared neural circuits for facial emotion production and perception are considered to facilitate the ability to understand other's emotional state via mirror neuron mechanisms. Little is known about how diverse emotions differentially activate the Mirror Neuron System (MNS) during facial expression processing. In a functional magnetic resonance imaging (fMRI) task, 178 healthy subjects perceived and produced emotional (happy vs angry) and non-emotional (lip-protrusion vs no movement) facial expressions. Dynamic facial expressions were displayed as 5 sec video clips. We identified three overlapping networks of neural activation for happy, angry, and non-emotional (lip-protrusion) facial expression production and perception. Importantly, this overlap was largely due to the common motor component of facial expressions. However, for happy facial expressions, we found specific MNS activation in the right temporal pole. For angry facial expressions we found such activation in the left inferior frontal gyrus, pars orbitalis, and the cerebellum (lobules VII and VIII). We extend knowledge on mirror neuron mechanisms as our results provide evidence for emotion specific shared neural activation for the production and perception of facial emotions. This emotion specific representation of other's emotion in one's own neural production system might facilitate understanding of other's mental or emotional states.

Evidence for gesture-speech mismatch detection impairments in schizophrenia.

Patients with schizophrenia suffer from impairments in the perception and production of gestures. The extent to which patients can access the semantic association between speech and co-verbal gestures in concrete or abstract/metaphorical meaning contexts is unknown. We investigated 1) how patients differ from controls in gesture matching performance, 2) how performance differs in the context of abstract versus concrete meaning, and 3) whether formal thought disorder (FTD) symptom severity predicts task impairment. Forty-five patients with schizophrenia spectrum disorders (two subgroups, "mild" and "severe") took part in this study. Participants were presented with video clips, each showing an actor saying a sentence while producing a gesture. Sentences contained either concrete or abstract/metaphorical information, and the accompanying gesture was either semantically related or unrelated to the sentence. Participants indicated via button press whether the gesture matched the content of the verbal utterance. Both patient subgroups demonstrated reduced performance in all comparisons. A significant interaction was found between patient subgroup and sentence abstractness. Task performance was worst for patients with severe positive FTD symptomatology in the abstract condition, while there were no patient subgroup differences in the concrete condition. These data shed new light on gesture-speech mismatch detection impairments in schizophrenia.

Spatial-temporal dynamics of gesture-speech integration: a simultaneous EEG-fMRI study.

The semantic integration between gesture and speech (GSI) is mediated by the left posterior temporal sulcus/middle temporal gyrus (pSTS/MTG) and the left inferior frontal gyrus (IFG). Evidence from electroencephalography (EEG) suggests that oscillations in the alpha and beta bands may support processes at different stages of GSI. In the present study, we investigated the relationship between electrophysiological oscillations and blood-oxygen-level-dependent (BOLD) activity during GSI. In a simultaneous EEG-fMRI study, German participants (n = 19) were presented with videos of an actor either performing meaningful gestures in the context of a comprehensible German (GG) or incomprehensible Russian sentence (GR), or just speaking a German sentence (SG). EEG results revealed reduced alpha and beta power for the GG vs. SG conditions, while fMRI analyses showed BOLD increase in the left pSTS/MTG for GG > GR ∩ GG > SG. In time-window-based EEG-informed fMRI analyses, we further found a positive correlation between single-trial alpha power and BOLD signal in the left pSTS/MTG, the left IFG, and several sub-cortical regions. Moreover, the alpha-pSTS/MTG correlation was observed in an earlier time window in comparison to the alpha-IFG correlation, thus supporting a two-stage processing model of GSI. Our study shows that EEG-informed fMRI implies multiple roles of alpha oscillations during GSI, and that the method is a best candidate for multidimensional investigations on complex cognitive functions such as GSI.

The EEG and fMRI signatures of neural integration: An investigation of meaningful gestures and corresponding speech.

One of the key features of human interpersonal communication is our ability to integrate information communicated by speech and accompanying gestures. However, it is still not fully understood how this essential combinatory process is represented in the human brain. Functional magnetic resonance imaging (fMRI) studies have unanimously attested the relevance of activation in the posterior superior temporal sulcus/middle temporal gyrus (pSTS/MTG), while electroencephalography (EEG) studies have shown oscillatory activity in specific frequency bands to be associated with multisensory integration. In the current study, we used fMRI and EEG to separately investigate the anatomical and oscillatory neural signature of integrating intrinsically meaningful gestures (IMG; e.g. "Thumbs-up gesture") and corresponding speech (e.g., "The actor did a good job"). In both the fMRI (n =2 0) and EEG (n = 20) study, participants were presented with videos of an actor either: performing IMG in the context of a German sentence (GG), IMG in the context of a Russian (as a foreign language) sentence (GR), or speaking an isolated German sentence without gesture (SG). The results of the fMRI experiment confirmed that gesture-speech processing of IMG activates the posterior MTG (GG>GR∩GG>SG). In the EEG experiment we found that the identical integration process (GG>GR∩GG>SG) is related to a centrally-distributed alpha (7-13 Hz) power decrease within 700-1400 ms post-onset of the critical word. These new findings suggest that BOLD response increase in the pMTG and alpha power decrease represent the neural correlates of integrating intrinsically meaningful gestures with their corresponding speech.

Supramodal neural processing of abstract information conveyed by speech and gesture.

ness and modality of interpersonal communication have a considerable impact on comprehension. They are relevant for determining thoughts and constituting internal models of the environment. Whereas concrete object-related information can be represented in mind irrespective of language, abstract concepts require a representation in speech. Consequently, modality-independent processing of abstract information can be expected. Here we investigated the neural correlates of abstractness (abstract vs. concrete) and modality (speech vs. gestures), to identify an abstractness-specific supramodal neural network. During fMRI data acquisition 20 participants were presented with videos of an actor either speaking sentences with an abstract-social [AS] or concrete-object-related content [CS], or performing meaningful abstract-social emblematic [AG] or concrete-object-related tool-use gestures [CG]. Gestures were accompanied by a foreign language to increase the comparability between conditions and to frame the communication context of the gesture videos. Participants performed a content judgment task referring to the person vs. object-relatedness of the utterances. The behavioral data suggest a comparable comprehension of contents communicated by speech or gesture. Furthermore, we found common neural processing for abstract information independent of modality (AS > CS ∩ AG > CG) in a left hemispheric network including the left inferior frontal gyrus (IFG), temporal pole, and medial frontal cortex. Modality specific activations were found in bilateral occipital, parietal, and temporal as well as right inferior frontal brain regions for gesture (G > S) and in left anterior temporal regions and the left angular gyrus for the processing of speech semantics (S > G). These data support the idea that abstract concepts are represented in a supramodal manner. Consequently, gestures referring to abstract concepts are processed in a predominantly left hemispheric language related neural network.