Mimicry of partially occluded emotional faces: do we mimic what we see or what we know?

Facial electromyography (EMG) was used to investigate patterns of facial mimicry in response to partial facial expressions in two contexts that differ in how naturalistic and socially significant the faces are. Experiment 1 presented participants with either the upper- or lower-half of facial expressions and used a forced-choice emotion categorisation task. This task emphasises cognition at the expense of ecological and social validity. Experiment 2 presented whole heads and expressions were occluded by clothing. Additionally, the emotion recognition task is more open-ended. This context has greater social validity. We found mimicry in both experiments, however mimicry differed in terms of which emotions were mimicked and the extent to which the mimicry involved muscle sites that were not observed. In the more cognitive context, there was relatively more motor matching (i.e. mimicking only what was seen). In the more socially valid context, participants were less likely to mimic only what they saw - and instead mimicked what they knew. Additionally, participants mimicked anger in the cognitive context but not the social context. These findings suggest that mimicry involves multiple mechanisms and that the more social the context, the more likely it is to reflect a mechanism of social regulation.

Verbal working memory and co-speech gesture processing.

Multimodal discourse requires an assembly of cognitive processes that are uniquely recruited for language comprehension in social contexts. In this study, we investigated the role of verbal working memory for the online integration of speech and iconic gestures. Participants memorized and rehearsed a series of auditorily presented digits in low (one digit) or high (four digits) memory load conditions. To observe how verbal working memory load impacts online discourse comprehension, ERPs were recorded while participants watched discourse videos containing either congruent or incongruent speech-gesture combinations during the maintenance portion of the memory task. While expected speech-gesture congruity effects were found in the low memory load condition, high memory load trials elicited enhanced frontal positivities that indicated a unique interaction between online speech-gesture integration and the availability of verbal working memory resources. This work contributes to an understanding of discourse comprehension by demonstrating that language processing in a multimodal context is subject to the relationship between cognitive resource availability and the degree of controlled processing required for task performance. We suggest that verbal working memory is less important for speech-gesture integration than it is for mediating speech processing under high task demands.

Sensorimotor simulation and emotion processing: Impairing facial action increases semantic retrieval demands.

Sensorimotor models suggest that understanding the emotional content of a face recruits a simulation process in which a viewer partially reproduces the facial expression in their own sensorimotor system. An important prediction of these models is that disrupting simulation should make emotion recognition more difficult. Here we used electroencephalogram (EEG) and facial electromyogram (EMG) to investigate how interfering with sensorimotor signals from the face influences the real-time processing of emotional faces. EEG and EMG were recorded as healthy adults viewed emotional faces and rated their valence. During control blocks, participants held a conjoined pair of chopsticks loosely between their lips. During interference blocks, participants held the chopsticks horizontally between their teeth and lips to generate motor noise on the lower part of the face. This noise was confirmed by EMG at the zygomaticus. Analysis of EEG indicated that faces expressing happiness or disgust-lower face expressions-elicited larger amplitude N400 when they were presented during the interference than the control blocks, suggesting interference led to greater semantic retrieval demands. The selective impact of facial motor interference on the brain response to lower face expressions supports sensorimotor models of emotion understanding.

Facial Action and Emotional Language: ERP Evidence that Blocking Facial Feedback Selectively Impairs Sentence Comprehension.

There is a lively and theoretically important debate about whether, how, and when embodiment contributes to language comprehension. This study addressed these questions by testing how interference with facial action impacts the brain's real-time response to emotional language. Participants read sentences about positive and negative events (e.g., "She reached inside the pocket of her coat from last winter and found some (cash/bugs) inside it.") while ERPs were recorded. Facial action was manipulated within participants by asking participants to hold chopsticks in their mouths using a position that allowed or blocked smiling, as confirmed by EMG. Blocking smiling did not influence ERPs to the valenced words (e.g., cash, bugs) but did influence ERPs to final words of sentences describing positive events. Results show that affectively positive sentences can evoke smiles and that such facial action can facilitate the semantic processing indexed by the N400 component. Overall, this study offers causal evidence that embodiment impacts some aspects of high-level comprehension, presumably involving the construction of the situation model.

Iconic Gestures Facilitate Discourse Comprehension in Individuals With Superior Immediate Memory for Body Configurations.

To understand a speaker's gestures, people may draw on kinesthetic working memory (KWM)-a system for temporarily remembering body movements. The present study explored whether sensitivity to gesture meaning was related to differences in KWM capacity. KWM was evaluated through sequences of novel movements that participants viewed and reproduced with their own bodies. Gesture sensitivity was assessed through a priming paradigm. Participants judged whether multimodal utterances containing congruent, incongruent, or no gestures were related to subsequent picture probes depicting the referents of those utterances. Individuals with low KWM were primarily inhibited by incongruent speech-gesture primes, whereas those with high KWM showed facilitation-that is, they were able to identify picture probes more quickly when preceded by congruent speech and gestures than by speech alone. Group differences were most apparent for discourse with weakly congruent speech and gestures. Overall, speech-gesture congruency effects were positively correlated with KWM abilities, which may help listeners match spatial properties of gestures to concepts evoked by speech.

Brain responses to a lab-evolved artificial language with space-time metaphors.

What is the connection between the cultural evolution of a language and the rapid processing response to that language in the brains of individual learners? In an iterated communication study that was conducted previously, participants were asked to communicate temporal concepts such as "tomorrow," "day after," "year," and "past" using vertical movements recorded on a touch screen. Over time, participants developed simple artificial 'languages' that used space metaphorically to communicate in nuanced ways about time. Some conventions appeared rapidly and universally (e.g., using larger vertical movements to convey greater temporal durations). Other conventions required extensive social interaction and exhibited idiosyncratic variation (e.g., using vertical location to convey past or future). Here we investigate whether the brain's response during acquisition of such a language reflects the process by which the language's conventions originally evolved. We recorded participants' EEG as they learned one of these artificial space-time languages. Overall, the brain response to this artificial communication system was language-like, with, for instance, violations to the system's conventions eliciting an N400-like component. Over the course of learning, participants' brain responses developed in ways that paralleled the process by which the language had originally evolved, with early neural sensitivity to violations of a rapidly-evolving universal convention, and slowly developing neural sensitivity to an idiosyncratic convention that required slow social negotiation to emerge. This study opens up exciting avenues of future work to disentangle how neural biases influence learning and transmission in the emergence of structure in language.

Network resonance and the auditory steady state response.

The auditory steady state response (ASSR) arises when periodic sounds evoke stable responses in auditory networks that reflect the acoustic characteristics of the stimuli, such as the amplitude of the sound envelope. Larger for some stimulus rates than others, the ASSR in the human electroencephalogram (EEG) is notably maximal for sounds modulated in amplitude at 40 Hz. To investigate the local circuit underpinnings of the large ASSR to 40 Hz amplitude-modulated (AM) sounds, we acquired skull EEG and local field potential (LFP) recordings from primary auditory cortex (A1) in the rat during the presentation of 20, 30, 40, 50, and 80 Hz AM tones. 40 Hz AM tones elicited the largest ASSR from the EEG acquired above auditory cortex and the LFP acquired from each cortical layer in A1. The large ASSR in the EEG to 40 Hz AM tones was not due to larger instantaneous amplitude of the signals or to greater phase alignment of the LFP across the cortical layers. Instead, it resulted from decreased latency variability (or enhanced temporal consistency) of the 40 Hz response. Statistical models indicate the EEG signal was best predicted by LFPs in either the most superficial or deep cortical layers, suggesting deep layer coordinators of the ASSR. Overall, our results indicate that the recruitment of non-uniform but more temporally consistent responses across A1 layers underlie the larger ASSR to amplitude-modulated tones at 40 Hz.

Strong Prediction: Language Model Surprisal Explains Multiple N400 Effects.

Theoretical accounts of the N400 are divided as to whether the amplitude of the N400 response to a stimulus reflects the extent to which the stimulus was predicted, the extent to which the stimulus is semantically similar to its preceding context, or both. We use state-of-the-art machine learning tools to investigate which of these three accounts is best supported by the evidence. GPT-3, a neural language model trained to compute the conditional probability of any word based on the words that precede it, was used to operationalize contextual predictability. In particular, we used an information-theoretic construct known as surprisal (the negative logarithm of the conditional probability). Contextual semantic similarity was operationalized by using two high-quality co-occurrence-derived vector-based meaning representations for words: GloVe and fastText. The cosine between the vector representation of the sentence frame and final word was used to derive contextual cosine similarity estimates. A series of regression models were constructed, where these variables, along with cloze probability and plausibility ratings, were used to predict single trial N400 amplitudes recorded from healthy adults as they read sentences whose final word varied in its predictability, plausibility, and semantic relationship to the likeliest sentence completion. Statistical model comparison indicated GPT-3 surprisal provided the best account of N400 amplitude and suggested that apparently disparate N400 effects of expectancy, plausibility, and contextual semantic similarity can be reduced to variation in the predictability of words. The results are argued to support predictive coding in the human language network.

Enhanced mental rotation ability in time-space synesthesia.

Time-space synesthesia is a variant of sequence-space synesthesia and involves the involuntary association of months of the year with 2D and 3D spatial forms, such as arcs, circles, and ellipses. Previous studies have revealed conflicting results regarding the association between time-space synesthesia and enhanced spatial processing ability. Here, we tested 15 time-space synesthetes, and 15 non-synesthetic controls matched for age, education, and gender on standard tests of mental rotation ability, spatial working memory, and verbal working memory. Synesthetes performed better than controls on our test of mental rotation, but similarly to controls on tests of spatial and verbal working memory. Results support a dissociation between visuo-spatial imagery and spatial working memory capacity, and suggest time-space synesthesia is associated only with enhanced visuo-spatial imagery. These data are consistent with the time-space connectivity thesis that time-space synesthesia results from enhanced connectivity in the parietal lobe between regions supporting the representation of temporal sequences and those underlying visuo-spatial imagery.

Contextual priming in grapheme-color synesthetes and yoked controls: 400 msec in the life of a synesthete.

Grapheme-color synesthesia is a heritable trait where graphemes ("2") elicit the concurrent perception of specific colors (red). Researchers have questioned whether synesthetic experiences are meaningful or simply arbitrary associations and whether these associations are perceptual or conceptual. To address these fundamental questions, ERPs were recorded as 12 synesthetes read statements such as "The Coca-Cola logo is white and 2," in which the final grapheme induced a color that was either contextually congruous (red) or incongruous ("...white and 7," for a synesthetes who experienced 7 as green). Grapheme congruity was found to modulate the amplitude of the N1, P2, N300, and N400 components in synesthetes, suggesting that synesthesia impacts perceptual as well as conceptual aspects of processing. To evaluate whether observed ERP effects required the experience of colored graphemes versus knowledge of grapheme-color pairings, we ran three separate groups of controls on a similar task. Controls trained to a synesthete's associations elicited N400 modulation, indicating that knowledge of grapheme-color mappings was sufficient to modulate this component. Controls trained to synesthetic associations and given explicit visualization instructions elicited both N300 and N400 modulations. Lastly, untrained controls who viewed physically colored graphemes ("2" printed in red) elicited N1 and N400 modulations. The N1 grapheme congruity effect began earlier in synesthetes than colored grapheme controls but had similar scalp topography. Data suggest that, in synesthetes, achromatic graphemes engage similar visual processing networks as colored graphemes in nonsynesthetes and are in keeping with models of synesthesia that posit early feed-forward connections between form and color processing areas in extrastriate cortex. The P2 modulation was unique to the synesthetes and may reflect neural activity that underlies the conscious experience of the synesthetic induction.

Event related spectral perturbations of gesture congruity: Visuospatial resources are recruited for multimodal discourse comprehension.

Here we examine the role of visuospatial working memory (WM) during the comprehension of multimodal discourse with co-speech iconic gestures. EEG was recorded as healthy adults encoded either a sequence of one (low load) or four (high load) dot locations on a grid and rehearsed them until a free recall response was collected later in the trial. During the rehearsal period of the WM task, participants observed videos of a speaker describing objects in which half of the trials included semantically related co-speech gestures (congruent), and the other half included semantically unrelated gestures (incongruent). Discourse processing was indexed by oscillatory EEG activity in the alpha and beta bands during the videos. Across all participants, effects of speech and gesture incongruity were more evident in low load trials than in high load trials. Effects were also modulated by individual differences in visuospatial WM capacity. These data suggest visuospatial WM resources are recruited in the comprehension of multimodal discourse.

Spatial cueing in time-space synesthetes: An event-related brain potential study.

Some people report that they consistently and involuntarily associate time events, such as months of the year, with specific spatial locations; a condition referred to as time-space synesthesia. The present study investigated the manner in which such synesthetic time-space associations affect visuo-spatial attention via an endogenous cuing paradigm. Reaction times and ERPs were recorded as 12 time-space synesthetes and 12 control participants did a peripheral target detection task, cued by three different types of centrally presented cues: arrows pointing left or right, direction words "left" or "right", and month names associated with either the left or the right side of the synesthete's mental calendar (e.g., "October" or "May"). Cues were followed by probes on the left or right side of the screen, and participants responded to the probes with button presses. Behavioral and ERP data suggested that for synesthetes, month words functioned more effectively as cues to direct attention in space. In synesthetes but not controls, a comparison of ERPs to probes cued by months revealed effects of cue validity on the P3b component peaking 370 ms post-onset and on the subsequent positive slow wave (pSW) observed 600-900 ms post-onset (both larger for invalid probes). No effects of cue validity were observed on early visual potentials (N1) for probes cued by months. The findings suggest that in these time-space synesthetes cue validity influenced post-perceptual processes, such as stimulus evaluation and categorization, with no evidence for enhanced visual processing.

Temporal sequences, synesthetic mappings, and cultural biases: the geography of time.

Time-space synesthetes report that they experience the months of the year as having a spatial layout. In Study 1, we characterize the phenomenology of calendar sequences produced by synesthetes and non-synesthetes, and show a conservative estimate of time-space synesthesia at 2.2% of the population. We demonstrate that synesthetes most commonly experience the months in a circular path, while non-synesthetes default to linear rows or rectangles. Study 2 compared synesthetes' and non-synesthetes' ability to memorize a novel spatial calendar, and revealed better performance in synesthetes. The capacity to learn mappings between arbitrary spatial forms and temporal sequences is present in all individuals, and time-space synesthetes' enhanced visuo-spatial memory abilities may underlie their creation of idiosyncratic spatial calendar forms.

Congruity Effects in Time and Space: Behavioral and ERP Measures.

Two experiments investigated whether motion metaphors for time affected the perception of spatial motion. Participants read sentences either about literal motion through space or metaphorical motion through time written from either the ego-moving or object-moving perspective. Each sentence was followed by a cartoon clip. Smiley-moving clips showed an iconic happy face moving toward a polygon, and shape-moving clips showed a polygon moving toward a happy face. In Experiment 1, using an explicit judgment task, participants judged smiley-moving cartoons as related to ego-moving sentences about space and about time, and shape-moving cartoons as related to object-moving sentences. In Experiment 2, participants viewed the same stimuli, but the cartoons were task-irrelevant. Event-related brain potentials revealed an early attentional effect of congruity on cartoons following sentences about space, and a later semantic effect on cartoons following sentences about time. Results are most consistent with accounts that posit differences in the processing of novel and conventional metaphors.

Dynamic grounding of emotion concepts.

Emotion concepts are important. They help us to understand, experience and predict human behaviour. Emotion concepts also link the realm of the abstract with the realm of bodily experience and actions. Accordingly, the key question is how such concepts are created, represented and used. Embodied cognition theories hold that concepts are grounded in neural systems that produce experiential and motor states. Concepts are also contextually situated and thus engage sensorimotor resources in a dynamic, flexible way. Finally, on that framework, conceptual understanding unfolds in time, reflecting embodied as well as linguistic and cultural influences. In this article, we review empirical work on emotion concepts and show how it highlights their grounded, yet dynamic and context-sensitive nature. The conclusions are consistent with recent developments in embodied cognition that allow concepts to be linked to sensorimotor systems, yet be flexibly sensitive to current representational and action needs.This article is part of the theme issue 'Varieties of abstract concepts: development, use and representation in the brain'.

A special role for the right hemisphere in metaphor comprehension? ERP evidence from hemifield presentation.

It has been suggested that the right hemisphere (RH) has a privileged role in the processing of figurative language, including metaphors, idioms, and verbal humor. Previous experiments using hemifield visual presentation combined with human electrophysiology support the idea that the RH plays a special role in joke comprehension. The current study examines metaphoric language. Event-related potentials (ERPs) were recorded as healthy adults read English sentences that ended predictably (High-cloze Literals), or with a plausible but unexpected word (Low-cloze Literals and Low-cloze Metaphoricals). Sentence final words were presented in either the left or the right visual hemifield. Relative to High-cloze Literals, Low-cloze Literals elicited a larger N400 component after presentation to both the left and the right hemifield. Low-cloze Literals also elicited a larger frontal positivity following the N400, but only with presentation to the right hemifield (left hemisphere). These data suggest both cerebral hemispheres can benefit from supportive sentence context, but may suggest an important role for anterior regions of the left hemisphere in the selection of semantic information in the face of competing alternatives. Relative to Low-cloze Literals, Low-cloze Metaphoricals elicited more negative ERPs during the timeframe of the N400 and afterwards. However, ERP metaphoricity effects were very similar across hemifields, suggesting that the integration of metaphoric meanings was similarly taxing for the two hemispheres, contrary to the predictions of the right hemisphere theory of metaphor.

Iconic gestures prime related concepts: an ERP study.

To assess priming by iconic gestures, we recorded EEG (at 29 scalp sites) in two experiments while adults watched short, soundless videos of spontaneously produced, cospeech iconic gestures followed by related or unrelated probe words. In Experiment 1, participants classified the relatedness between gestures and words. In Experiment 2, they attended to stimuli, and performed an incidental recognition memory test on words presented during the EEG recording session. Event-related potentials (ERPs) time-locked to the onset of probe words were measured, along with response latencies and word recognition rates. Although word relatedness did not affect reaction times or recognition rates, contextually related probe words elicited less-negative ERPs than did unrelated ones between 300 and 500 msec after stimulus onset (N400) in both experiments. These findings demonstrate sensitivity to semantic relations between iconic gestures and words in brain activity engendered during word comprehension.

Hemispheric asymmetry and pun comprehension: when cowboys have sore calves.

Event-related potentials (ERPs) were recorded as healthy participants listened to puns such as "During branding, cowboys have sore calves." To assess hemispheric differences in pun comprehension, visually presented probes that were either highly related (COW), moderately related (LEG), or unrelated, were presented in either the left or right visual half field (LVF/RVF). The sensitivity of each hemisphere to the different meanings evoked by the pun was assessed by ERP relatedness effects with presentation to the LVF and the RVF. In Experiment 1, the inter-stimulus interval between the pun and the onset of the visual probe was 0 ms; in Experiment 2, this value was 500 ms. In Experiment 1, both highly and moderately related probes elicited similar priming effects with RVF presentation. Relative to their unrelated counterparts, related probes elicited less negative ERPs in the N400 interval (300-600 ms post-onset), and more positive ERPs 600-900 ms post-onset, suggesting both meanings of the pun were equally active in the left hemisphere. LVF presentation yielded similar priming effects (less negative N400 and a larger positivity thereafter) for the highly related probes, but no effects for moderately related probes. In Experiment 2, similar N400 priming effects were observed for highly and moderately related probes presented to both visual fields. Compared to unrelated probes 600-900 ms post-onset, related probes elicited a centro-parietal positivity with RVF presentation, but a fronto-polar positivity with LVF presentation. Results suggest that initially, the different meanings evoked by a pun are both active in the left hemisphere, but only the most highly related meaning is active in the right hemisphere. By 500 ms, both meanings are active in both hemispheres.

How iconic gestures enhance communication: an ERP study.

EEG was recorded as adults watched short segments of spontaneous discourse in which the speaker's gestures and utterances contained complementary information. Videos were followed by one of four types of picture probes: cross-modal related probes were congruent with both speech and gestures; speech-only related probes were congruent with information in the speech, but not the gesture; and two sorts of unrelated probes were created by pairing each related probe with a different discourse prime. Event-related potentials (ERPs) elicited by picture probes were measured within the time windows of the N300 (250-350 ms post-stimulus) and N400 (350-550 ms post-stimulus). Cross-modal related probes elicited smaller N300 and N400 than speech-only related ones, indicating that pictures were easier to interpret when they corresponded with gestures. N300 and N400 effects were not due to differences in the visual complexity of each probe type, since the same cross-modal and speech-only picture probes elicited N300 and N400 with similar amplitudes when they appeared as unrelated items. These findings extend previous research on gesture comprehension by revealing how iconic co-speech gestures modulate conceptualization, enabling listeners to better represent visuo-spatial aspects of the speaker's meaning.

Projectors, associators, visual imagery, and the time course of visual processing in grapheme-color synesthesia.

In grapheme-color synesthesia, seeing particular letters or numbers evokes the experience of specific colors. We investigate the brain's real-time processing of words in this population by recording event-related brain potentials (ERPs) from 15 grapheme-color synesthetes and 15 controls as they judged the validity of word pairs ('yellow banana' vs. 'blue banana') presented under high and low visual contrast. Low contrast words elicited delayed P1/N170 visual ERP components in both groups, relative to high contrast. When color concepts were conveyed to synesthetes by individually tailored achromatic grapheme strings ('55555 banana'), visual contrast effects were like those in color words: P1/N170 components were delayed but unchanged in amplitude. When controls saw equivalent colored grapheme strings, visual contrast modulated P1/N170 amplitude but not latency. Color induction in synesthetes thus differs from color perception in controls. Independent from experimental effects, all orthographic stimuli elicited larger N170 and P2 in synesthetes than controls. While P2 (150-250ms) enhancement was similar in all synesthetes, N170 (130-210ms) amplitude varied with individual differences in synesthesia and visual imagery. Results suggest immediate cross-activation in visual areas processing color and shape is most pronounced in so-called projector synesthetes whose concurrent colors are experienced as originating in external space.