An investigation of the modulatory effects of empathic and autistic traits on emotional and facial motor responses during live social interactions.

A close relationship between emotional contagion and spontaneous facial mimicry has been theoretically proposed and is supported by empirical data. Facial expressions are essential in terms of both emotional and motor synchrony. Previous studies have demonstrated that trait emotional empathy enhanced spontaneous facial mimicry, but the relationship between autistic traits and spontaneous mimicry remained controversial. Moreover, previous studies presented faces that were static or videotaped, which may lack the "liveliness" of real-life social interactions. We addressed this limitation by using an image relay system to present live performances and pre-recorded videos of smiling or frowning dynamic facial expressions to 94 healthy female participants. We assessed their subjective experiential valence and arousal ratings to infer the amplitude of emotional contagion. We measured the electromyographic activities of the zygomaticus major and corrugator supercilii muscles to estimate spontaneous facial mimicry. Individual differences measures included trait emotional empathy (empathic concern) and the autism-spectrum quotient. We did not find that live performances enhanced the modulatory effect of trait differences on emotional contagion or spontaneous facial mimicry. However, we found that a high trait empathic concern was associated with stronger emotional contagion and corrugator mimicry. We found no two-way interaction between the autism spectrum quotient and emotional condition, suggesting that autistic traits did not modulate emotional contagion or spontaneous facial mimicry. Our findings imply that previous findings regarding the relationship between emotional empathy and emotional contagion/spontaneous facial mimicry using videos and photos could be generalized to real-life interactions.

Asynchrony enhances uncanniness in human, android, and virtual dynamic facial expressions.

OBJECTIVE: Uncanniness plays a vital role in interactions with humans and artificial agents. Previous studies have shown that uncanniness is caused by a higher sensitivity to deviation or atypicality in specialized categories, such as faces or facial expressions, marked by configural processing. We hypothesized that asynchrony, understood as a temporal deviation in facial expression, could cause uncanniness in the facial expression. We also hypothesized that the effect of asynchrony could be disrupted through inversion. RESULTS: Sixty-four participants rated the uncanniness of synchronous or asynchronous dynamic face emotion expressions of human, android, or computer-generated (CG) actors, presented either upright or inverted. Asynchrony vs. synchrony expressions increased uncanniness for all upright expressions except for CG angry expressions. Inverted compared with upright presentations produced less evident asynchrony effects for human angry and android happy expressions. These results suggest that asynchrony can cause dynamic expressions to appear uncanny, which is related to configural processing but different across agents.

Electromyographic Validation of Spontaneous Facial Mimicry Detection Using Automated Facial Action Coding.

Although electromyography (EMG) remains the standard, researchers have begun using automated facial action coding system (FACS) software to evaluate spontaneous facial mimicry despite the lack of evidence of its validity. Using the facial EMG of the zygomaticus major (ZM) as a standard, we confirmed the detection of spontaneous facial mimicry in action unit 12 (AU12, lip corner puller) via an automated FACS. Participants were alternately presented with real-time model performance and prerecorded videos of dynamic facial expressions, while simultaneous ZM signal and frontal facial videos were acquired. Facial videos were estimated for AU12 using FaceReader, Py-Feat, and OpenFace. The automated FACS is less sensitive and less accurate than facial EMG, but AU12 mimicking responses were significantly correlated with ZM responses. All three software programs detected enhanced facial mimicry by live performances. The AU12 time series showed a roughly 100 to 300 ms latency relative to the ZM. Our results suggested that while the automated FACS could not replace facial EMG in mimicry detection, it could serve a purpose for large effect sizes. Researchers should be cautious with the automated FACS outputs, especially when studying clinical populations. In addition, developers should consider the EMG validation of AU estimation as a benchmark.

Differences in configural processing for human versus android dynamic facial expressions.

Humanlike androids can function as social agents in social situations and in experimental research. While some androids can imitate facial emotion expressions, it is unclear whether their expressions tap the same processing mechanisms utilized in human expression processing, for example configural processing. In this study, the effects of global inversion and asynchrony between facial features as configuration manipulations were compared in android and human dynamic emotion expressions. Seventy-five participants rated (1) angry and happy emotion recognition and (2) arousal and valence ratings of upright or inverted, synchronous or asynchronous, android or human agent dynamic emotion expressions. Asynchrony in dynamic expressions significantly decreased all ratings (except valence in angry expressions) in all human expressions, but did not affect android expressions. Inversion did not affect any measures regardless of agent type. These results suggest that dynamic facial expressions are processed in a synchrony-based configural manner for humans, but not for androids.

The inversion effect on the cubic humanness-uncanniness relation in humanlike agents.

The uncanny valley describes the typically nonlinear relation between the esthetic appeal of artificial entities and their human likeness. The effect has been attributed to specialized (configural) processing that increases sensitivity to deviations from human norms. We investigate this effect in computer-generated, humanlike android and human faces using dynamic facial expressions. Angry and happy expressions with varying degrees of synchrony were presented upright and inverted and rated on their eeriness, strangeness, and human likeness. A sigmoidal function of human likeness and uncanniness ("uncanny slope") was found for upright expressions and a linear relation for inverted faces. While the function is not indicative of an uncanny valley, the results support the view that configural processing moderates the effect of human likeness on uncanniness and extend its role to dynamic facial expressions.

Enhanced mirror neuron network activity and effective connectivity during live interaction among female subjects.

Facial expressions are indispensable in daily human communication. Previous neuroimaging studies investigating facial expression processing have presented pre-recorded stimuli and lacked live face-to-face interaction. Our paradigm alternated between presentations of real-time model performance and pre-recorded videos of dynamic facial expressions to participants. Simultaneous functional magnetic resonance imaging (fMRI) and facial electromyography activity recordings, as well as post-scan valence and arousal ratings were acquired from 44 female participants. Live facial expressions enhanced the subjective valence and arousal ratings as well as facial muscular responses. Live performances showed greater engagement of the right posterior superior temporal sulcus (pSTS), right inferior frontal gyrus (IFG), right amygdala and right fusiform gyrus, and modulated the effective connectivity within the right mirror neuron system (IFG, pSTS, and right inferior parietal lobule). A support vector machine algorithm could classify multivoxel activation patterns in brain regions involved in dynamic facial expression processing in the mentalizing networks (anterior and posterior cingulate cortex). These results indicate that live social interaction modulates the activity and connectivity of the right mirror neuron system and enhances spontaneous mimicry, further facilitating emotional contagion.

Enhanced emotional and motor responses to live versus videotaped dynamic facial expressions.

Facial expression is an integral aspect of non-verbal communication of affective information. Earlier psychological studies have reported that the presentation of prerecorded photographs or videos of emotional facial expressions automatically elicits divergent responses, such as emotions and facial mimicry. However, such highly controlled experimental procedures may lack the vividness of real-life social interactions. This study incorporated a live image relay system that delivered models' real-time performance of positive (smiling) and negative (frowning) dynamic facial expressions or their prerecorded videos to participants. We measured subjective ratings of valence and arousal and facial electromyography (EMG) activity in the zygomaticus major and corrugator supercilii muscles. Subjective ratings showed that the live facial expressions were rated to elicit higher valence and more arousing than the corresponding videos for positive emotion conditions. Facial EMG data showed that compared with the video, live facial expressions more effectively elicited facial muscular activity congruent with the models' positive facial expressions. The findings indicate that emotional facial expressions in live social interactions are more evocative of emotional reactions and facial mimicry than earlier experimental data have suggested.

Author Correction: Neurocognitive Signatures of Naturalistic Reading of Scientific Texts: A Fixation-Related fMRI Study.

An amendment to this paper has been published and can be accessed via a link at the top of the paper.

Affective iconic words benefit from additional sound-meaning integration in the left amygdala.

Recent studies have shown that a similarity between sound and meaning of a word (i.e., iconicity) can help more readily access the meaning of that word, but the neural mechanisms underlying this beneficial role of iconicity in semantic processing remain largely unknown. In an fMRI study, we focused on the affective domain and examined whether affective iconic words (e.g., high arousal in both sound and meaning) activate additional brain regions that integrate emotional information from different domains (i.e., sound and meaning). In line with our hypothesis, affective iconic words, compared to their non-iconic counterparts, elicited additional BOLD responses in the left amygdala known for its role in multimodal representation of emotions. Functional connectivity analyses revealed that the observed amygdalar activity was modulated by an interaction of iconic condition and activations in two hubs representative for processing sound (left superior temporal gyrus) and meaning (left inferior frontal gyrus) of words. These results provide a neural explanation for the facilitative role of iconicity in language processing and indicate that language users are sensitive to the interaction between sound and meaning aspect of words, suggesting the existence of iconicity as a general property of human language.

Neurocognitive Signatures of Naturalistic Reading of Scientific Texts: A Fixation-Related fMRI Study.

How do students gain scientific knowledge while reading expository text? This study examines the underlying neurocognitive basis of textual knowledge structure and individual readers' cognitive differences and reading habits, including the influence of text and reader characteristics, on outcomes of scientific text comprehension. By combining fixation-related fMRI and multiband data acquisition, the study is among the first to consider self-paced naturalistic reading inside the MRI scanner. Our results revealed the underlying neurocognitive patterns associated with information integration of different time scales during text reading, and significant individual differences due to the interaction between text characteristics (e.g., optimality of the textual knowledge structure) and reader characteristics (e.g., electronic device use habits). Individual differences impacted the amount of neural resources deployed for multitasking and information integration for constructing the underlying scientific mental models based on the text being read. Our findings have significant implications for understanding science reading in a population that is increasingly dependent on electronic devices.

Atypical Reward-Driven Modulation of Mimicry-Related Neural Activity in Autism.

Autism spectrum disorder (ASD) is characterized by deficits in social functioning and difficulties in forming social bonds. According to the social motivation theory of ASD, people with ASD fail to attend social stimuli because they do not experience them as rewarding, resulting in deficits in social cognition. In neurotypical (NT) individuals, more rewarding faces have been shown to elicit greater spontaneous facial mimicry. This association between reward and mimicry is reduced in people with high autistic traits, suggesting that altered reward processing might explain the deficits in spontaneous facial mimicry observed in individuals with ASD. In a previous study, we observed that learned reward value of a face modulates mimicry-related neural response to it and that this modulation is reduced in people with high autistic traits. Using an identical evaluative conditioning paradigm where neutral faces were conditioned with high and low rewards, we tested the modulating effect of reward value on mimicry-related brain activity in a group of adults with and without ASD. We focused on the activity in a cluster within the inferior frontal gyrus (IFG) identified through an independent meta-analysis of 139 neuroimaging studies of mimicry, in response to passively viewing videos of the conditioned faces. The blood oxygen level dependent (BOLD) response contrast of high- vs. low-reward faces was reduced in participants with ASD compared to NT controls. The extent of reward-driven modulation was negatively correlated with autistic traits across the whole sample. Our results indicate that the mimicry-related brain response is less modulated by learned reward value in individuals with ASD when compared to NT controls. In previous studies, we found in a similar sample that being mimicked by faces was associated with less reward-related brain response in individuals ASD compared to an NT sample, suggesting that the link between reward and mimicry is affected in both directions in ASD. Together, this reduced bidirectional link between reward and mimicry can point to a potential mechanism underlying some of the social cognitive features of ASD.

Idiomatic expressions evoke stronger emotional responses in the brain than literal sentences.

Recent neuroscientific research shows that metaphors engage readers at the emotional level more strongly than literal expressions. What still remains unclear is what makes metaphors more engaging, and whether this generalises to all figurative expressions, no matter how conventionalised they are. This fMRI study aimed to investigate whether idiomatic expressions - the least creative part of figurative language - indeed trigger a higher affective resonance than literal expressions, and to explore possible interactions between activation in emotion-relevant neural structures and regions associated with figurative language processing. Participants silently read for comprehension a set of emotionally positive, negative and neutral idioms embedded in short sentences, and similarly valenced literal sentences. As in studies on metaphors, we found enhanced activation of the left inferior frontal gyrus and left amygdala in response to idioms, indexing stronger recruitment of executive control functions and enhanced emotional engagement, respectively. This suggests that the comprehension of even highly conventionalised and familiar figurative expressions, namely idioms, recruits regions involved in emotional processing. Furthermore, increased activation of the IFG interacted positively with activation in the amygdala, suggesting that the stronger cognitive engagement driven by idioms may in turn be coupled with stronger involvement at the emotional level.

The Sound of Words Evokes Affective Brain Responses.

The long history of poetry and the arts, as well as recent empirical results suggest that the way a word sounds (e.g., soft vs. harsh) can convey affective information related to emotional responses (e.g., pleasantness vs. harshness). However, the neural correlates of the affective potential of the sound of words remain unknown. In an fMRI study involving passive listening, we focused on the affective dimension of arousal and presented words organized in two discrete groups of sublexical (i.e., sound) arousal (high vs. low), while controlling for lexical (i.e., semantic) arousal. Words sounding high arousing, compared to their low arousing counterparts, resulted in an enhanced BOLD signal in bilateral posterior insula, the right auditory and premotor cortex, and the right supramarginal gyrus. This finding provides first evidence on the neural correlates of affectivity in the sound of words. Given the similarity of this neural network to that of nonverbal emotional expressions and affective prosody, our results support a unifying view that suggests a core neural network underlying any type of affective sound processing.

How mimicry influences the neural correlates of reward: An fMRI study.

Mimicry has been suggested to function as a "social glue", a key mechanism that helps to build social rapport. It leads to increased feeling of closeness toward the mimicker as well as greater liking, suggesting close bidirectional links with reward. In recent work using eye-gaze tracking, we have demonstrated that the reward value of being mimicked, measured using a preferential looking paradigm, is directly proportional to trait empathy (Neufeld and Chakrabarti, 2016). In the current manuscript, we investigated the reward value of the act of mimicking, using a simple task manipulation that involved allowing or inhibiting spontaneous facial mimicry in response to dynamic expressions of positive emotion. We found greater reward-related neural activity in response to the condition where mimicry was allowed compared to that where mimicry was inhibited. The magnitude of this link from mimicry to reward response was positively correlated to trait empathy.

Reduced reward-related neural response to mimicry in individuals with autism.

Mimicry is a facilitator of social bonds in humans, from infancy. This facilitation is made possible through changing the reward value of social stimuli; for example, we like and affiliate more with people who mimic us. Autism spectrum disorders (ASD) are marked by difficulties in forming social bonds. In this study, we investigate whether the reward-related neural response to being mimicked is altered in individuals with ASD, using a simple conditioning paradigm. Multiple studies in humans and nonhuman primates have established a crucial role for the ventral striatal (VS) region in responding to rewards. In this study, adults with ASD and matched controls first underwent a conditioning task outside the scanner, where they were mimicked by one face and 'anti-mimicked' by another. In the second part, participants passively viewed the conditioned faces in a 3T MRI scanner using a multi-echo sequence. The differential neural response towards mimicking vs. anti-mimicking faces in the VS was tested for group differences as well as an association with self-reported autistic traits. Multiple regression analysis revealed lower left VS response to mimicry (mimicking > anti-mimicking faces) in the ASD group compared to controls. The VS response to mimicry was negatively correlated with autistic traits across the whole sample. Our results suggest that for individuals with ASD and high autistic traits, being mimicked is associated with lower reward-related neural response. This result points to a potential mechanism underlying the difficulties reported by many of individuals with ASD in building social rapport.

Thinking about others and the future: Neural correlates of perspective taking relate to preferences for delayed rewards.

We infer the thoughts and feelings of others by taking their perspectives. Similar processes could be used to understand how we will be affected by future events, by allowing us to take the perspective of our future self. In this paper, we test this idea using a previously presented framework for guiding predictions. The framework proposes that a shared neural mechanism is involved in controlling egocentric bias, both while shifting our perspective away from self and towards others, and while shifting our perspective from immediate to future perspectives. To test this framework, 36 adults performed an intertemporal choice task. They were then scanned using 3T functional magnetic resonance imaging while completing a false-belief "localizer" task, which requires egocentric bias control. A positive correlation was observed between the right temporoparietal junction (rTPJ) response during the false-belief task, and preferences for delayed rewards in intertemporal choices. A subset of participants performed the intertemporal choice task again in the scanner, which revealed that the response of the same rTPJ cluster, individually localized during the false-belief task, was higher during delayed over immediate reward choices. In addition, functional connectivity between the rTPJ and ventromedial prefrontal cortex was found to differ between immediate and delayed choices. The current results indicate an overlap in processes of egocentric bias control and those that determine preferences in intertemporal choices, offering a social cognitive explanation for why rewards are devalued with delay in temporal discounting.

The magical activation of left amygdala when reading Harry Potter: an fMRI study on how descriptions of supra-natural events entertain and enchant.

Literature containing supra-natural, or magical events has enchanted generations of readers. When reading narratives describing such events, readers mentally simulate a text world different from the real one. The corresponding violation of world-knowledge during this simulation likely increases cognitive processing demands for ongoing discourse integration, catches readers' attention, and might thus contribute to the pleasure and deep emotional experience associated with ludic immersive reading. In the present study, we presented participants in an MR scanner with passages selected from the Harry Potter book series, half of which described magical events, while the other half served as control condition. Passages in both conditions were closely matched for relevant psycholinguistic variables including, e.g., emotional valence and arousal, passage-wise mean word imageability and frequency, and syntactic complexity. Post-hoc ratings showed that readers considered supra-natural contents more surprising and more strongly associated with reading pleasure than control passages. In the fMRI data, we found stronger neural activation for the supra-natural than the control condition in bilateral inferior frontal gyri, bilateral inferior parietal lobules, left fusiform gyrus, and left amygdala. The increased activation in the amygdala (part of the salience and emotion processing network) appears to be associated with feelings of surprise and the reading pleasure, which supra-natural events, full of novelty and unexpectedness, brought about. The involvement of bilateral inferior frontal gyri likely reflects higher cognitive processing demand due to world knowledge violations, whereas increased attention to supra-natural events is reflected in inferior frontal gyri and inferior parietal lobules that are part of the fronto-parietal attention network.

The emotion potential of words and passages in reading Harry Potter--an fMRI study.

Previous studies suggested that the emotional connotation of single words automatically recruits attention. We investigated the potential of words to induce emotional engagement when reading texts. In an fMRI experiment, we presented 120 text passages from the Harry Potter book series. Results showed significant correlations between affective word (lexical) ratings and passage ratings. Furthermore, affective lexical ratings correlated with activity in regions associated with emotion, situation model building, multi-modal semantic integration, and Theory of Mind. We distinguished differential influences of affective lexical, inter-lexical, and supra-lexical variables: differential effects of lexical valence were significant in the left amygdala, while effects of arousal-span (the dynamic range of arousal across a passage) were significant in the left amygdala and insula. However, we found no differential effect of passage ratings in emotion-associated regions. Our results support the hypothesis that the emotion potential of short texts can be predicted by lexical and inter-lexical affective variables.

Can Harry Potter still put a spell on us in a second language? An fMRI study on reading emotion-laden literature in late bilinguals.

In this fMRI study we contrasted emotional responses to literary reading in late bilinguals' first or second language. German participants with adequate English proficiency in their second language (L2) English read short text passages from Harry Potter books characterized by a "negative" or "positive" versus "neutral" emotional valence manipulation. Previous studies have suggested that given sufficient L2 proficiency, neural substrates involved in L1 versus L2 do not differ (Fabbro, 2001). On the other hand, the question of attenuated emotionality of L2 language processing is still an open debate (see Conrad, Recio, & Jacobs, 2011). Our results revealed a set of neural structures involved in the processing of emotion-laden literature, including emotion-related amygdala and a set of lateral prefrontal, anterior temporal, and temporo-parietal regions associated with discourse comprehension, high-level semantic integration, and Theory-of-Mind processing. Yet, consistent with post-scan emotion ratings of text passages, factorial fMRI analyses revealed stronger hemodynamic responses to "happy" than to "neutral" in bilateral amygdala and the left precentral cortex that were restricted to L1 reading. Furthermore, multivariate pattern analyses (MVPA) demonstrated better classifiability of differential patterns of brain activity elicited by passages of different emotional content in L1 than in L2 for the whole brain level. Overall, our results suggest that reading emotion-laden texts in our native language provides a stronger and more differentiated emotional experience than reading in a second language.

Fiction feelings in Harry Potter: haemodynamic response in the mid-cingulate cortex correlates with immersive reading experience.

Immersion in reading, described as a feeling of 'getting lost in a book', is a ubiquitous phenomenon widely appreciated by readers. However, it has been largely ignored in cognitive neuroscience. According to the fiction feeling hypothesis, narratives with emotional contents invite readers more to be empathic with the protagonists and thus engage the affective empathy network of the brain, the anterior insula and mid-cingulate cortex, than do stories with neutral contents. To test the hypothesis, we presented participants with text passages from the Harry Potter series in a functional MRI experiment and collected post-hoc immersion ratings, comparing the neural correlates of passage mean immersion ratings when reading fear-inducing versus neutral contents. Results for the conjunction contrast of baseline brain activity of reading irrespective of emotional content against baseline were in line with previous studies on text comprehension. In line with the fiction feeling hypothesis, immersion ratings were significantly higher for fear-inducing than for neutral passages, and activity in the mid-cingulate cortex correlated more strongly with immersion ratings of fear-inducing than of neutral passages. Descriptions of protagonists' pain or personal distress featured in the fear-inducing passages apparently caused increasing involvement of the core structure of pain and affective empathy the more readers immersed in the text. The predominant locus of effects in the mid-cingulate cortex seems to reflect that the immersive experience was particularly facilitated by the motor component of affective empathy for our stimuli from the Harry Potter series featuring particularly vivid descriptions of the behavioural aspects of emotion.