Inhibitory TMS over Visual Area V5/MT Disrupts Visual Speech Recognition.

During face-to-face communication, the perception and recognition of facial movements can facilitate individuals' understanding of what is said. Facial movements are a form of complex biological motion. Separate neural pathways are thought to processing (1) simple, nonbiological motion with an obligatory waypoint in the motion-sensitive visual middle temporal area (V5/MT); and (2) complex biological motion. Here, we present findings that challenge this dichotomy. Neuronavigated offline transcranial magnetic stimulation (TMS) over V5/MT on 24 participants (17 females and 7 males) led to increased response times in the recognition of simple, nonbiological motion as well as visual speech recognition compared with TMS over the vertex, an active control region. TMS of area V5/MT also reduced practice effects on response times, that are typically observed in both visual speech and motion recognition tasks over time. Our findings provide the first indication that area V5/MT causally influences the recognition of visual speech.SIGNIFICANCE STATEMENT In everyday face-to-face communication, speech comprehension is often facilitated by viewing a speaker's facial movements. Several brain areas contribute to the recognition of visual speech. One area of interest is the motion-sensitive visual medial temporal area (V5/MT), which has been associated with the perception of simple, nonbiological motion such as moving dots, as well as more complex, biological motion such as visual speech. Here, we demonstrate using noninvasive brain stimulation that area V5/MT is causally relevant in recognizing visual speech. This finding provides new insights into the neural mechanisms that support the perception of human communication signals, which will help guide future research in typically developed individuals and populations with communication difficulties.

Motor Cortex Causally Contributes to Vocabulary Translation following Sensorimotor-Enriched Training.

The role of the motor cortex in perceptual and cognitive functions is highly controversial. Here, we investigated the hypothesis that the motor cortex can be instrumental for translating foreign language vocabulary. Human participants of both sexes were trained on foreign language (L2) words and their native language translations over 4 consecutive days. L2 words were accompanied by complementary gestures (sensorimotor enrichment) or pictures (sensory enrichment). Following training, participants translated the auditorily presented L2 words that they had learned. During translation, repetitive transcranial magnetic stimulation was applied bilaterally to a site within the primary motor cortex (Brodmann area 4) located in the vicinity of the arm functional compartment. Responses within the stimulated motor region have previously been found to correlate with behavioral benefits of sensorimotor-enriched L2 vocabulary learning. Compared to sham stimulation, effective perturbation by repetitive transcranial magnetic stimulation slowed down the translation of sensorimotor-enriched L2 words, but not sensory-enriched L2 words. This finding suggests that sensorimotor-enriched training induced changes in L2 representations within the motor cortex, which in turn facilitated the translation of L2 words. The motor cortex may play a causal role in precipitating sensorimotor-based learning benefits, and may directly aid in remembering the native language translations of foreign language words following sensorimotor-enriched training. These findings support multisensory theories of learning while challenging reactivation-based theories.SIGNIFICANCE STATEMENT Despite the potential for sensorimotor enrichment to serve as a powerful tool for learning in many domains, its underlying brain mechanisms remain largely unexplored. Using transcranial magnetic stimulation and a foreign language (L2) learning paradigm, we found that sensorimotor-enriched training can induce changes in L2 representations within the motor cortex, which in turn causally facilitate the translation of L2 words. The translation of recently acquired L2 words may therefore rely not only on auditory information stored in memory or on modality-independent L2 representations, but also on the sensorimotor context in which the words have been experienced.

Visual Sensory Cortices Causally Contribute to Auditory Word Recognition Following Sensorimotor-Enriched Vocabulary Training.

Despite a rise in the use of "learning by doing" pedagogical methods in praxis, little is known as to how the brain benefits from these methods. Learning by doing strategies that utilize complementary information ("enrichment") such as gestures have been shown to optimize learning outcomes in several domains including foreign language (L2) training. Here we tested the hypothesis that behavioral benefits of gesture-based enrichment are critically supported by integrity of the biological motion visual cortices (bmSTS). Prior functional neuroimaging work has implicated the visual motion cortices in L2 translation following sensorimotor-enriched training; the current study is the first to investigate the causal relevance of these structures in learning by doing contexts. Using neuronavigated transcranial magnetic stimulation and a gesture-enriched L2 vocabulary learning paradigm, we found that the bmSTS causally contributed to behavioral benefits of gesture-enriched learning. Visual motion cortex integrity benefitted both short- and long-term learning outcomes, as well as the learning of concrete and abstract words. These results adjudicate between opposing predictions of two neuroscientific learning theories: While reactivation-based theories predict no functional role of specialized sensory cortices in vocabulary learning outcomes, the current study supports the predictive coding theory view that these cortices precipitate sensorimotor-based learning benefits.

Rhythm Complexity Modulates Behavioral and Neural Dynamics During Auditory-Motor Synchronization.

We addressed how rhythm complexity influences auditory-motor synchronization in musically trained individuals who perceived and produced complex rhythms while EEG was recorded. Participants first listened to two-part auditory sequences (Listen condition). Each part featured a single pitch presented at a fixed rate; the integer ratio formed between the two rates varied in rhythmic complexity from low (1:1) to moderate (1:2) to high (3:2). One of the two parts occurred at a constant rate across conditions. Then, participants heard the same rhythms as they synchronized their tapping at a fixed rate (Synchronize condition). Finally, they tapped at the same fixed rate (Motor condition). Auditory feedback from their taps was present in all conditions. Behavioral effects of rhythmic complexity were evidenced in all tasks; detection of missing beats (Listen) worsened in the most complex (3:2) rhythm condition, and tap durations (Synchronize) were most variable and least synchronous with stimulus onsets in the 3:2 condition. EEG power spectral density was lowest at the fixed rate during the 3:2 rhythm and greatest during the 1:1 rhythm (Listen and Synchronize). ERP amplitudes corresponding to an N1 time window were smallest for the 3:2 rhythm and greatest for the 1:1 rhythm (Listen). Finally, synchronization accuracy (Synchronize) decreased as amplitudes in the N1 time window became more positive during the high rhythmic complexity condition (3:2). Thus, measures of neural entrainment corresponded to synchronization accuracy, and rhythmic complexity modulated the behavioral and neural measures similarly.

Sensory, Cognitive, and Sensorimotor Learning Effects in Recognition Memory for Music.

Recent research suggests that perception and action are strongly interrelated and that motor experience may aid memory recognition. We investigated the role of motor experience in auditory memory recognition processes by musicians using behavioral, ERP, and neural source current density measures. Skilled pianists learned one set of novel melodies by producing them and another set by perception only. Pianists then completed an auditory memory recognition test during which the previously learned melodies were presented with or without an out-of-key pitch alteration while the EEG was recorded. Pianists indicated whether each melody was altered from or identical to one of the original melodies. Altered pitches elicited a larger N2 ERP component than original pitches, and pitches within previously produced melodies elicited a larger N2 than pitches in previously perceived melodies. Cortical motor planning regions were more strongly activated within the time frame of the N2 following altered pitches in previously produced melodies compared with previously perceived melodies, and larger N2 amplitudes were associated with greater detection accuracy following production learning than perception learning. Early sensory (N1) and later cognitive (P3a) components elicited by pitch alterations correlated with predictions of sensory echoic and schematic tonality models, respectively, but only for the perception learning condition, suggesting that production experience alters the extent to which performers rely on sensory and tonal recognition cues. These findings provide evidence for distinct time courses of sensory, schematic, and motoric influences within the same recognition task and suggest that learned auditory-motor associations influence responses to out-of-key pitches.

Sensorimotor Learning Enhances Expectations During Auditory Perception.

Sounds that have been produced with one's own motor system tend to be remembered better than sounds that have only been perceived, suggesting a role of motor information in memory for auditory stimuli. To address potential contributions of the motor network to the recognition of previously produced sounds, we used event-related potential, electric current density, and behavioral measures to investigate memory for produced and perceived melodies. Musicians performed or listened to novel melodies, and then heard the melodies either in their original version or with single pitch alterations. Production learning enhanced subsequent recognition accuracy and increased amplitudes of N200, P300, and N400 responses to pitch alterations. Premotor and supplementary motor regions showed greater current density during the initial detection of alterations in previously produced melodies than in previously perceived melodies, associated with the N200. Primary motor cortex was more strongly engaged by alterations in previously produced melodies within the P300 and N400 timeframes. Motor memory traces may therefore interface with auditory pitch percepts in premotor regions as early as 200 ms following perceived pitch onsets. Outcomes suggest that auditory-motor interactions contribute to memory benefits conferred by production experience, and support a role of motor prediction mechanisms in the production effect.

Reduction in physical activity during Covid-19 lockdowns predicts individual differences in cognitive performance several months after the end of the safety measures.

Prior studies suggest that the reductions in physical activity during Covid-19-related lockdowns impacted physical and mental health. Whether reductions in physical activity that occurred during lockdowns also relate to cognitive functions such as memory and attention is less explored. Here, we investigated whether changes in physical activity (PA) that occurred during and following Covid-19-related lockdowns could predict a variety of measures of cognitive performance in 318 young adults. Participants were assessed on their engagement in PA before, during, and after lockdowns. They also completed tests of cognitive control, working memory, and short-term memory following lockdown(s). As expected, engagement in PA decreased during lockdown and returned to near baseline levels thereafter. Decreases in PA during lockdown predicted individual differences in cognitive performance following lockdown. Greater reductions in PA during lockdown were associated with lower scores on the go/no-go task, a measure of cognitive control ability, and the n-back task, a measure of working memory performance. Larger post-lockdown increases in PA were associated with higher scores on the same tasks. Individual differences in pandemic-related stress and insomnia also predicted cognitive outcomes. These findings suggest that reductions of PA can predict cognitive performance, and underscore the importance of maintaining PA for cognitive health, especially in situations such as lockdowns.

Prior multisensory learning can facilitate auditory-only voice-identity and speech recognition in noise.

Seeing the visual articulatory movements of a speaker, while hearing their voice, helps with understanding what is said. This multisensory enhancement is particularly evident in noisy listening conditions. Multisensory enhancement also occurs even in auditory-only conditions: auditory-only speech and voice-identity recognition are superior for speakers previously learned with their face, compared to control learning; an effect termed the "face-benefit." Whether the face-benefit can assist in maintaining robust perception in increasingly noisy listening conditions, similar to concurrent multisensory input, is unknown. Here, in two behavioural experiments, we examined this hypothesis. In each experiment, participants learned a series of speakers' voices together with their dynamic face or control image. Following learning, participants listened to auditory-only sentences spoken by the same speakers and recognised the content of the sentences (speech recognition, Experiment 1) or the voice-identity of the speaker (Experiment 2) in increasing levels of auditory noise. For speech recognition, we observed that 14 of 30 participants (47%) showed a face-benefit. 19 of 25 participants (76%) showed a face-benefit for voice-identity recognition. For those participants who demonstrated a face-benefit, the face-benefit increased with auditory noise levels. Taken together, the results support an audio-visual model of auditory communication and suggest that the brain can develop a flexible system in which learned facial characteristics are used to deal with varying auditory uncertainty.

Enriched learning: behavior, brain, and computation.

The presence of complementary information across multiple sensory or motor modalities during learning, referred to as multimodal enrichment, can markedly benefit learning outcomes. Why is this? Here, we integrate cognitive, neuroscientific, and computational approaches to understanding the effectiveness of enrichment and discuss recent neuroscience findings indicating that crossmodal responses in sensory and motor brain regions causally contribute to the behavioral benefits of enrichment. The findings provide novel evidence for multimodal theories of enriched learning, challenge assumptions of longstanding cognitive theories, and provide counterevidence to unimodal neurobiologically inspired theories. Enriched educational methods are likely effective not only because they may engage greater levels of attention or deeper levels of processing, but also because multimodal interactions in the brain can enhance learning and memory.

Benefits of Enacting and Observing Gestures on Foreign Language Vocabulary Learning: A Systematic Review and Meta-Analysis.

The integration of physical movements, such as gestures, into learning holds potential for enhancing foreign language (L2) education. Uncovering whether actively performing gestures during L2 learning is more, or equally, effective compared to simply observing such movements is central to deepening our understanding of the efficacy of movement-based learning strategies. Here, we present a meta-analysis of seven studies containing 309 participants that compares the effects of gesture self-enactment and observation on L2 vocabulary learning. The results showed that gesture observation was just as effective for L2 learning as gesture enactment, based on free recall, cued L2 recognition, and cued native language recognition performance, with a large dispersion of true effect across studies. Gesture observation may be sufficient for inducing embodied L2 learning benefits, in support of theories positing shared mechanisms underlying enactment and observation. Future studies should examine the effects of gesture-based learning over longer time periods with larger sample sizes and more diverse word classes.

Visual recognition of words learned with gestures induces motor resonance in the forearm muscles.

According to theories of Embodied Cognition, memory for words is related to sensorimotor experiences collected during learning. At a neural level, words encoded with self-performed gestures are represented in distributed sensorimotor networks that resonate during word recognition. Here, we ask whether muscles involved in gesture execution also resonate during word recognition. Native German speakers encoded words by reading them (baseline condition) or by reading them in tandem with picture observation, gesture observation, or gesture observation and execution. Surface electromyogram (EMG) activity from both arms was recorded during the word recognition task and responses were detected using eye-tracking. The recognition of words encoded with self-performed gestures coincided with an increase in arm muscle EMG activity compared to the recognition of words learned under other conditions. This finding suggests that sensorimotor networks resonate into the periphery and provides new evidence for a strongly embodied view of recognition memory.

Electrical Brain Responses Reveal Sequential Constraints on Planning during Music Performance.

Elements in speech and music unfold sequentially over time. To produce sentences and melodies quickly and accurately, individuals must plan upcoming sequence events, as well as monitor outcomes via auditory feedback. We investigated the neural correlates of sequential planning and monitoring processes by manipulating auditory feedback during music performance. Pianists performed isochronous melodies from memory at an initially cued rate while their electroencephalogram was recorded. Pitch feedback was occasionally altered to match either an immediately upcoming Near-Future pitch (next sequence event) or a more distant Far-Future pitch (two events ahead of the current event). Near-Future, but not Far-Future altered feedback perturbed the timing of pianists' performances, suggesting greater interference of Near-Future sequential events with current planning processes. Near-Future feedback triggered a greater reduction in auditory sensory suppression (enhanced response) than Far-Future feedback, reflected in the P2 component elicited by the pitch event following the unexpected pitch change. Greater timing perturbations were associated with enhanced cortical sensory processing of the pitch event following the Near-Future altered feedback. Both types of feedback alterations elicited feedback-related negativity (FRN) and P3a potentials and amplified spectral power in the theta frequency range. These findings suggest similar constraints on producers' sequential planning to those reported in speech production.

Auditory N1 reveals planning and monitoring processes during music performance.

The current study investigated the relationship between planning processes and feedback monitoring during music performance, a complex task in which performers prepare upcoming events while monitoring their sensory outcomes. Theories of action planning in auditory-motor production tasks propose that the planning of future events co-occurs with the perception of auditory feedback. This study investigated the neural correlates of planning and feedback monitoring by manipulating the contents of auditory feedback during music performance. Pianists memorized and performed melodies at a cued tempo in a synchronization-continuation task while the EEG was recorded. During performance, auditory feedback associated with single melody tones was occasionally substituted with tones corresponding to future (next), present (current), or past (previous) melody tones. Only future-oriented altered feedback disrupted behavior: Future-oriented feedback caused pianists to slow down on the subsequent tone more than past-oriented feedback, and amplitudes of the auditory N1 potential elicited by the tone immediately following the altered feedback were larger for future-oriented than for past-oriented or noncontextual (unrelated) altered feedback; larger N1 amplitudes were associated with greater slowing following altered feedback in the future condition only. Feedback-related negativities were elicited in all altered feedback conditions. In sum, behavioral and neural evidence suggests that future-oriented feedback disrupts performance more than past-oriented feedback, consistent with planning theories that posit similarity-based interference between feedback and planning contents. Neural sensory processing of auditory feedback, reflected in the N1 ERP, may serve as a marker for temporal disruption caused by altered auditory feedback in auditory-motor production tasks.

Electrical Brain Responses to Beat Irregularities in Two Cases of Beat Deafness.

Beat deafness, a recently documented form of congenital amusia, provides a unique window into functional specialization of neural circuitry for the processing of musical stimuli: Beat-deaf individuals exhibit deficits that are specific to the detection of a regular beat in music and the ability to move along with a beat. Studies on the neural underpinnings of beat processing in the general population suggest that the auditory system is capable of pre-attentively generating a predictive model of upcoming sounds in a rhythmic pattern, subserved largely within auditory cortex and reflected in mismatch negativity (MMN) and P3 event-related potential (ERP) components. The current study examined these neural correlates of beat perception in two beat-deaf individuals, Mathieu and Marjorie, and a group of control participants under conditions in which auditory stimuli were either attended or ignored. Compared to control participants, Mathieu demonstrated reduced behavioral sensitivity to beat omissions in metrical patterns, and Marjorie showed a bias to identify irregular patterns as regular. ERP responses to beat omissions reveal an intact pre-attentive system for processing beat irregularities in cases of beat deafness, reflected in the MMN component, and provide partial support for abnormalities in later cognitive stages of beat processing, reflected in an unreliable P3b component exhibited by Mathieu-but not Marjorie-compared to control participants. P3 abnormalities observed in the current study resemble P3 abnormalities exhibited by individuals with pitch-based amusia, and are consistent with attention or auditory-motor coupling accounts of deficits in beat perception.

Context and meter enhance long-range planning in music performance.

Neural responses demonstrate evidence of resonance, or oscillation, during the production of periodic auditory events. Music contains periodic auditory events that give rise to a sense of beat, which in turn generates a sense of meter on the basis of multiple periodicities. Metrical hierarchies may aid memory for music by facilitating similarity-based associations among sequence events at different periodic distances that unfold in longer contexts. A fundamental question is how metrical associations arising from a musical context influence memory during music performance. Longer contexts may facilitate metrical associations at higher hierarchical levels more than shorter contexts, a prediction of the range model, a formal model of planning processes in music performance (Palmer and Pfordresher, 2003; Pfordresher et al., 2007). Serial ordering errors, in which intended sequence events are produced in incorrect sequence positions, were measured as skilled pianists performed musical pieces that contained excerpts embedded in long or short musical contexts. Pitch errors arose from metrically similar positions and further sequential distances more often when the excerpt was embedded in long contexts compared to short contexts. Musicians' keystroke intensities and error rates also revealed influences of metrical hierarchies, which differed for performances in long and short contexts. The range model accounted for contextual effects and provided better fits to empirical findings when metrical associations between sequence events were included. Longer sequence contexts may facilitate planning during sequence production by increasing conceptual similarity between hierarchically associated events. These findings are consistent with the notion that neural oscillations at multiple periodicities may strengthen metrical associations across sequence events during planning.

Sensorimotor mechanisms in music performance: actions that go partially wrong.

Even expert musicians make errors occasionally, and overt responses that are correct may be accompanied by partial-error behavior that can be indicative of online error detection processes. We compare pianists' production of correct pitches, pitch errors, and partial errors (correct pitches with incorrect force or duration) by examining events prior to errors. Errors tended to be produced with slower durations and softer intensities (associated with force reduction) than correct events. In addition, pre-error events tended to have durations and intensities that fell between those of errors and correct responses, presumably due to response competition with upcoming errors that resulted in partial-error outcomes. These findings support the inference that partial information about upcoming (planned) sequence events is used to guide current responses, consistent with cascade models of activation during sequence production.