Network resonance and the auditory steady state response.

The auditory steady state response (ASSR) arises when periodic sounds evoke stable responses in auditory networks that reflect the acoustic characteristics of the stimuli, such as the amplitude of the sound envelope. Larger for some stimulus rates than others, the ASSR in the human electroencephalogram (EEG) is notably maximal for sounds modulated in amplitude at 40 Hz. To investigate the local circuit underpinnings of the large ASSR to 40 Hz amplitude-modulated (AM) sounds, we acquired skull EEG and local field potential (LFP) recordings from primary auditory cortex (A1) in the rat during the presentation of 20, 30, 40, 50, and 80 Hz AM tones. 40 Hz AM tones elicited the largest ASSR from the EEG acquired above auditory cortex and the LFP acquired from each cortical layer in A1. The large ASSR in the EEG to 40 Hz AM tones was not due to larger instantaneous amplitude of the signals or to greater phase alignment of the LFP across the cortical layers. Instead, it resulted from decreased latency variability (or enhanced temporal consistency) of the 40 Hz response. Statistical models indicate the EEG signal was best predicted by LFPs in either the most superficial or deep cortical layers, suggesting deep layer coordinators of the ASSR. Overall, our results indicate that the recruitment of non-uniform but more temporally consistent responses across A1 layers underlie the larger ASSR to amplitude-modulated tones at 40 Hz.

Strong Prediction: Language Model Surprisal Explains Multiple N400 Effects.

Theoretical accounts of the N400 are divided as to whether the amplitude of the N400 response to a stimulus reflects the extent to which the stimulus was predicted, the extent to which the stimulus is semantically similar to its preceding context, or both. We use state-of-the-art machine learning tools to investigate which of these three accounts is best supported by the evidence. GPT-3, a neural language model trained to compute the conditional probability of any word based on the words that precede it, was used to operationalize contextual predictability. In particular, we used an information-theoretic construct known as surprisal (the negative logarithm of the conditional probability). Contextual semantic similarity was operationalized by using two high-quality co-occurrence-derived vector-based meaning representations for words: GloVe and fastText. The cosine between the vector representation of the sentence frame and final word was used to derive contextual cosine similarity estimates. A series of regression models were constructed, where these variables, along with cloze probability and plausibility ratings, were used to predict single trial N400 amplitudes recorded from healthy adults as they read sentences whose final word varied in its predictability, plausibility, and semantic relationship to the likeliest sentence completion. Statistical model comparison indicated GPT-3 surprisal provided the best account of N400 amplitude and suggested that apparently disparate N400 effects of expectancy, plausibility, and contextual semantic similarity can be reduced to variation in the predictability of words. The results are argued to support predictive coding in the human language network.

Brain responses to a lab-evolved artificial language with space-time metaphors.

What is the connection between the cultural evolution of a language and the rapid processing response to that language in the brains of individual learners? In an iterated communication study that was conducted previously, participants were asked to communicate temporal concepts such as "tomorrow," "day after," "year," and "past" using vertical movements recorded on a touch screen. Over time, participants developed simple artificial 'languages' that used space metaphorically to communicate in nuanced ways about time. Some conventions appeared rapidly and universally (e.g., using larger vertical movements to convey greater temporal durations). Other conventions required extensive social interaction and exhibited idiosyncratic variation (e.g., using vertical location to convey past or future). Here we investigate whether the brain's response during acquisition of such a language reflects the process by which the language's conventions originally evolved. We recorded participants' EEG as they learned one of these artificial space-time languages. Overall, the brain response to this artificial communication system was language-like, with, for instance, violations to the system's conventions eliciting an N400-like component. Over the course of learning, participants' brain responses developed in ways that paralleled the process by which the language had originally evolved, with early neural sensitivity to violations of a rapidly-evolving universal convention, and slowly developing neural sensitivity to an idiosyncratic convention that required slow social negotiation to emerge. This study opens up exciting avenues of future work to disentangle how neural biases influence learning and transmission in the emergence of structure in language.

Mimicry of partially occluded emotional faces: do we mimic what we see or what we know?

Facial electromyography (EMG) was used to investigate patterns of facial mimicry in response to partial facial expressions in two contexts that differ in how naturalistic and socially significant the faces are. Experiment 1 presented participants with either the upper- or lower-half of facial expressions and used a forced-choice emotion categorisation task. This task emphasises cognition at the expense of ecological and social validity. Experiment 2 presented whole heads and expressions were occluded by clothing. Additionally, the emotion recognition task is more open-ended. This context has greater social validity. We found mimicry in both experiments, however mimicry differed in terms of which emotions were mimicked and the extent to which the mimicry involved muscle sites that were not observed. In the more cognitive context, there was relatively more motor matching (i.e. mimicking only what was seen). In the more socially valid context, participants were less likely to mimic only what they saw - and instead mimicked what they knew. Additionally, participants mimicked anger in the cognitive context but not the social context. These findings suggest that mimicry involves multiple mechanisms and that the more social the context, the more likely it is to reflect a mechanism of social regulation.

Event related spectral perturbations of gesture congruity: Visuospatial resources are recruited for multimodal discourse comprehension.

Here we examine the role of visuospatial working memory (WM) during the comprehension of multimodal discourse with co-speech iconic gestures. EEG was recorded as healthy adults encoded either a sequence of one (low load) or four (high load) dot locations on a grid and rehearsed them until a free recall response was collected later in the trial. During the rehearsal period of the WM task, participants observed videos of a speaker describing objects in which half of the trials included semantically related co-speech gestures (congruent), and the other half included semantically unrelated gestures (incongruent). Discourse processing was indexed by oscillatory EEG activity in the alpha and beta bands during the videos. Across all participants, effects of speech and gesture incongruity were more evident in low load trials than in high load trials. Effects were also modulated by individual differences in visuospatial WM capacity. These data suggest visuospatial WM resources are recruited in the comprehension of multimodal discourse.

Verbal working memory and co-speech gesture processing.

Multimodal discourse requires an assembly of cognitive processes that are uniquely recruited for language comprehension in social contexts. In this study, we investigated the role of verbal working memory for the online integration of speech and iconic gestures. Participants memorized and rehearsed a series of auditorily presented digits in low (one digit) or high (four digits) memory load conditions. To observe how verbal working memory load impacts online discourse comprehension, ERPs were recorded while participants watched discourse videos containing either congruent or incongruent speech-gesture combinations during the maintenance portion of the memory task. While expected speech-gesture congruity effects were found in the low memory load condition, high memory load trials elicited enhanced frontal positivities that indicated a unique interaction between online speech-gesture integration and the availability of verbal working memory resources. This work contributes to an understanding of discourse comprehension by demonstrating that language processing in a multimodal context is subject to the relationship between cognitive resource availability and the degree of controlled processing required for task performance. We suggest that verbal working memory is less important for speech-gesture integration than it is for mediating speech processing under high task demands.

Dynamic grounding of emotion concepts.

Emotion concepts are important. They help us to understand, experience and predict human behaviour. Emotion concepts also link the realm of the abstract with the realm of bodily experience and actions. Accordingly, the key question is how such concepts are created, represented and used. Embodied cognition theories hold that concepts are grounded in neural systems that produce experiential and motor states. Concepts are also contextually situated and thus engage sensorimotor resources in a dynamic, flexible way. Finally, on that framework, conceptual understanding unfolds in time, reflecting embodied as well as linguistic and cultural influences. In this article, we review empirical work on emotion concepts and show how it highlights their grounded, yet dynamic and context-sensitive nature. The conclusions are consistent with recent developments in embodied cognition that allow concepts to be linked to sensorimotor systems, yet be flexibly sensitive to current representational and action needs.This article is part of the theme issue 'Varieties of abstract concepts: development, use and representation in the brain'.

Projectors, associators, visual imagery, and the time course of visual processing in grapheme-color synesthesia.

In grapheme-color synesthesia, seeing particular letters or numbers evokes the experience of specific colors. We investigate the brain's real-time processing of words in this population by recording event-related brain potentials (ERPs) from 15 grapheme-color synesthetes and 15 controls as they judged the validity of word pairs ('yellow banana' vs. 'blue banana') presented under high and low visual contrast. Low contrast words elicited delayed P1/N170 visual ERP components in both groups, relative to high contrast. When color concepts were conveyed to synesthetes by individually tailored achromatic grapheme strings ('55555 banana'), visual contrast effects were like those in color words: P1/N170 components were delayed but unchanged in amplitude. When controls saw equivalent colored grapheme strings, visual contrast modulated P1/N170 amplitude but not latency. Color induction in synesthetes thus differs from color perception in controls. Independent from experimental effects, all orthographic stimuli elicited larger N170 and P2 in synesthetes than controls. While P2 (150-250ms) enhancement was similar in all synesthetes, N170 (130-210ms) amplitude varied with individual differences in synesthesia and visual imagery. Results suggest immediate cross-activation in visual areas processing color and shape is most pronounced in so-called projector synesthetes whose concurrent colors are experienced as originating in external space.

Sensorimotor simulation and emotion processing: Impairing facial action increases semantic retrieval demands.

Sensorimotor models suggest that understanding the emotional content of a face recruits a simulation process in which a viewer partially reproduces the facial expression in their own sensorimotor system. An important prediction of these models is that disrupting simulation should make emotion recognition more difficult. Here we used electroencephalogram (EEG) and facial electromyogram (EMG) to investigate how interfering with sensorimotor signals from the face influences the real-time processing of emotional faces. EEG and EMG were recorded as healthy adults viewed emotional faces and rated their valence. During control blocks, participants held a conjoined pair of chopsticks loosely between their lips. During interference blocks, participants held the chopsticks horizontally between their teeth and lips to generate motor noise on the lower part of the face. This noise was confirmed by EMG at the zygomaticus. Analysis of EEG indicated that faces expressing happiness or disgust-lower face expressions-elicited larger amplitude N400 when they were presented during the interference than the control blocks, suggesting interference led to greater semantic retrieval demands. The selective impact of facial motor interference on the brain response to lower face expressions supports sensorimotor models of emotion understanding.

Iconic Gestures Facilitate Discourse Comprehension in Individuals With Superior Immediate Memory for Body Configurations.

To understand a speaker's gestures, people may draw on kinesthetic working memory (KWM)-a system for temporarily remembering body movements. The present study explored whether sensitivity to gesture meaning was related to differences in KWM capacity. KWM was evaluated through sequences of novel movements that participants viewed and reproduced with their own bodies. Gesture sensitivity was assessed through a priming paradigm. Participants judged whether multimodal utterances containing congruent, incongruent, or no gestures were related to subsequent picture probes depicting the referents of those utterances. Individuals with low KWM were primarily inhibited by incongruent speech-gesture primes, whereas those with high KWM showed facilitation-that is, they were able to identify picture probes more quickly when preceded by congruent speech and gestures than by speech alone. Group differences were most apparent for discourse with weakly congruent speech and gestures. Overall, speech-gesture congruency effects were positively correlated with KWM abilities, which may help listeners match spatial properties of gestures to concepts evoked by speech.

Facial Action and Emotional Language: ERP Evidence that Blocking Facial Feedback Selectively Impairs Sentence Comprehension.

There is a lively and theoretically important debate about whether, how, and when embodiment contributes to language comprehension. This study addressed these questions by testing how interference with facial action impacts the brain's real-time response to emotional language. Participants read sentences about positive and negative events (e.g., "She reached inside the pocket of her coat from last winter and found some (cash/bugs) inside it.") while ERPs were recorded. Facial action was manipulated within participants by asking participants to hold chopsticks in their mouths using a position that allowed or blocked smiling, as confirmed by EMG. Blocking smiling did not influence ERPs to the valenced words (e.g., cash, bugs) but did influence ERPs to final words of sentences describing positive events. Results show that affectively positive sentences can evoke smiles and that such facial action can facilitate the semantic processing indexed by the N400 component. Overall, this study offers causal evidence that embodiment impacts some aspects of high-level comprehension, presumably involving the construction of the situation model.

Co-speech iconic gestures and visuo-spatial working memory.

Three experiments tested the role of verbal versus visuo-spatial working memory in the comprehension of co-speech iconic gestures. In Experiment 1, participants viewed congruent discourse primes in which the speaker's gestures matched the information conveyed by his speech, and incongruent ones in which the semantic content of the speaker's gestures diverged from that in his speech. Discourse primes were followed by picture probes that participants judged as being either related or unrelated to the preceding clip. Performance on this picture probe classification task was faster and more accurate after congruent than incongruent discourse primes. The effect of discourse congruency on response times was linearly related to measures of visuo-spatial, but not verbal, working memory capacity, as participants with greater visuo-spatial WM capacity benefited more from congruent gestures. In Experiments 2 and 3, participants performed the same picture probe classification task under conditions of high and low loads on concurrent visuo-spatial (Experiment 2) and verbal (Experiment 3) memory tasks. Effects of discourse congruency and verbal WM load were additive, while effects of discourse congruency and visuo-spatial WM load were interactive. Results suggest that congruent co-speech gestures facilitate multi-modal language comprehension, and indicate an important role for visuo-spatial WM in these speech-gesture integration processes.

A psychometric measure of working memory capacity for configured body movement.

Working memory (WM) models have traditionally assumed at least two domain-specific storage systems for verbal and visuo-spatial information. We review data that suggest the existence of an additional slave system devoted to the temporary storage of body movements, and present a novel instrument for its assessment: the movement span task. The movement span task assesses individuals' ability to remember and reproduce meaningless configurations of the body. During the encoding phase of a trial, participants watch short videos of meaningless movements presented in sets varying in size from one to five items. Immediately after encoding, they are prompted to reenact as many items as possible. The movement span task was administered to 90 participants along with standard tests of verbal WM, visuo-spatial WM, and a gesture classification test in which participants judged whether a speaker's gestures were congruent or incongruent with his accompanying speech. Performance on the gesture classification task was not related to standard measures of verbal or visuo-spatial working memory capacity, but was predicted by scores on the movement span task. Results suggest the movement span task can serve as an assessment of individual differences in WM capacity for body-centric information.

How vertical hand movements impact brain activity elicited by literally and metaphorically related words: an ERP study of embodied metaphor.

Embodied metaphor theory suggests abstract concepts are metaphorically linked to more experientially basic ones and recruit sensorimotor cortex for their comprehension. To test whether words associated with spatial attributes reactivate traces in sensorimotor cortex, we recorded EEG from the scalp of healthy adults as they read words while performing a concurrent task involving either upward- or downward- directed arm movements. ERPs were time-locked to words associated with vertical space-either literally (ascend, descend) or metaphorically (inspire, defeat)-as participants made vertical movements that were either congruent or incongruent with the words. Congruency effects emerged 200-300 ms after word onset for literal words, but not until after 500 ms post-onset for metaphorically related words. Results argue against a strong version of embodied metaphor theory, but support a role for sensorimotor simulation in concrete language.

Grammatical number agreement processing using the visual half-field paradigm: an event-related brain potential study.

Despite indications in the split-brain and lesion literatures that the right hemisphere is capable of some syntactic analysis, few studies have investigated right hemisphere contributions to syntactic processing in people with intact brains. Here we used the visual half-field paradigm in healthy adults to examine each hemisphere's processing of correct and incorrect grammatical number agreement marked either lexically, e.g., antecedent/reflexive pronoun ("The grateful niece asked herself/*themselves…") or morphologically, e.g., subject/verb ("Industrial scientists develop/*develops…"). For reflexives, response times and accuracy of grammaticality decisions suggested similar processing regardless of visual field of presentation. In the subject/verb condition, we observed similar response times and accuracies for central and right visual field (RVF) presentations. For left visual field (LVF) presentation, response times were longer and accuracy rates were reduced relative to RVF presentation. An event-related brain potential (ERP) study using the same materials revealed similar ERP responses to the reflexive pronouns in the two visual fields, but very different ERP effects to the subject/verb violations. For lexically marked violations on reflexives, P600 was elicited by stimuli in both the LVF and RVF; for morphologically marked violations on verbs, P600 was elicited only by RVF stimuli. These data suggest that both hemispheres can process lexically marked pronoun agreement violations, and do so in a similar fashion. Morphologically marked subject/verb agreement errors, however, showed a distinct LH advantage.

Enhanced mental rotation ability in time-space synesthesia.

Time-space synesthesia is a variant of sequence-space synesthesia and involves the involuntary association of months of the year with 2D and 3D spatial forms, such as arcs, circles, and ellipses. Previous studies have revealed conflicting results regarding the association between time-space synesthesia and enhanced spatial processing ability. Here, we tested 15 time-space synesthetes, and 15 non-synesthetic controls matched for age, education, and gender on standard tests of mental rotation ability, spatial working memory, and verbal working memory. Synesthetes performed better than controls on our test of mental rotation, but similarly to controls on tests of spatial and verbal working memory. Results support a dissociation between visuo-spatial imagery and spatial working memory capacity, and suggest time-space synesthesia is associated only with enhanced visuo-spatial imagery. These data are consistent with the time-space connectivity thesis that time-space synesthesia results from enhanced connectivity in the parietal lobe between regions supporting the representation of temporal sequences and those underlying visuo-spatial imagery.

Hemispheric asymmetry in interpreting novel literal language: an event-related potential study.

Conceptual mapping, or making connections between conceptual structure in different domains, is a key mechanism of creative language use whose neural underpinnings are not well understood. The present study involved the combination of event-related potentials (ERPs) with the divided visual field presentation technique to explore the relative contributions of the left and right hemispheres (LH and RH) to the construction of novel meanings in fully literal language. Electroencephalogram (EEG) was recorded as healthy adults read sentences that supported either a conventional literal reading of the sentence final word ("His main method of transportation is a boat,"), or a novel literal meaning derived from conceptual mapping ("The clever boys used a cardboard box as a boat,"). The novel and conventional conditions were matched for cloze probability (a measure of predictability based on the sentence context), lexical association between the sentence frame and the final word (using latent semantic analysis), and other factors known to influence ERPs to language stimuli. To compare effects of novelty to previously reported effects of predictability, a high-cloze conventional condition ("The only way to get around Venice is to navigate the canals in a boat.") was included. ERPs were time-locked to sentence final words ("boat") presented in either the left visual field, to preferentially stimulate the RH (lvf/RH), or in the right visual field, targeting the LH (rvf/LH). The N400 component of the ERP was affected by predictability in both presentation sides, but by novelty only in rvf/LH. Two distinct late frontal positive effects were observed. Word predictability modulated a frontal positivity with a LH focus, but semantic novelty modulated a frontal positivity focused in RH. This is the first demonstration that the frontal positivity may be composed of multiple overlapping components with distinct functional and anatomical characteristics. Extending contemporary accounts of the frontal positivity, we suggest that both frontal positivities reflect learning mechanisms involving prediction based on statistical regularities in language (LH) and world knowledge (RH).

Handedness and calendar orientations in time-space synaesthesia.

In one common variant of time-space synaesthesia, individuals report the consistent experience of months bound to a spatial arrangement, commonly described as a circle extending outside of the body. Whereas the layout of these calendars has previously been thought to be relatively random and to differ greatly between synaesthetes, Study 1 provides the first evidence suggesting one critical aspect of these calendars is mediated by handedness: clockwise versus counter-clockwise orientation. A study of 34 time-space synaesthetes revealed a strong association between handedness and the orientation of circular calendars. That is, left-handed time-space synaesthetes tended to report counter-clockwise arrangements and right-handed synaesthetes clockwise. Study 2 tested whether a similar bias was present in non-synaesthetes whose task was to memorize and recall the spatial configuration of a clockwise and counter-clockwise calendar. Non-synaesthetes' relative performance on these two sorts of calendars was significantly correlated with their handedness scores in a pattern similar to synaesthetes. Specifically, left-handed controls performed better on counter-clockwise calendars compared to clockwise, and right-handed controls on clockwise over counter-clockwise. We suggest that the implicit biases seen in controls are mediated by similar mechanisms as in synaesthesia, highlighting the graded nature of synaesthetic associations.

Predictability and novelty in literal language comprehension: an ERP study.

Linguists have suggested that one mechanism for the creative extension of meaning in language involves mapping, or constructing correspondences between conceptual domains. For example, the sentence, "The clever boys used a cardboard box as a boat," sets up a novel mapping between the concepts cardboard box and boat, while "His main method of transportation is a boat," relies on a more conventional mapping between method of transportation and boat. To examine the electrophysiological signature of this mapping process, electroencephalogram (EEG) was recorded from the scalp as healthy adults read three sorts of sentences: low-cloze (unpredictable) conventional ("His main method of transportation is a boat,"), low-cloze novel mapp'ing ("The clever boys used a cardboard box as a boat,"), and high-cloze (predictable) conventional ("The only way to get around Venice is to navigate the canals in a boat,"). Event-related brain potentials (ERPs) were time-locked to sentence final words. The novel and conventional conditions were matched for cloze probability (a measure of predictability based on the sentence context), lexical association between the sentence frame and the final word (using latent semantic analysis), and other factors known to influence ERPs to language stimuli. The high-cloze conventional control condition was included to compare the effects of mapping conventionality to those of predictability. The N400 component of the ERPs was affected by predictability but not by conventionality. By contrast, a late positivity was affected both by the predictability of sentence final words, being larger for words in low-cloze contexts that made target words difficult to predict, and by novelty, as words in the novel condition elicited a larger positivity 700-900ms than the same words in the (cloze-matched) conventional condition.