White matter plasticity during second language learning within and across hemispheres.

Adult second language (L2) learning is a challenging enterprise inducing neuroplastic changes in the human brain. However, it remains unclear how the structural language connectome and its subnetworks change during adult L2 learning. The current study investigated longitudinal changes in white matter (WM) language networks in each hemisphere, as well as their interconnection, in a large group of Arabic-speaking adults who learned German intensively for 6 mo. We found a significant increase in WM-connectivity within bilateral temporal-parietal semantic and phonological subnetworks and right temporal-frontal pathways mainly in the second half of the learning period. At the same time, WM-connectivity between the two hemispheres decreased significantly. Crucially, these changes in WM-connectivity are correlated with L2 performance. The observed changes in subnetworks of the two hemispheres suggest a network reconfiguration due to lexical learning. The reduced interhemispheric connectivity may indicate a key role of the corpus callosum in L2 learning by reducing the inhibition of the language-dominant left hemisphere. Our study highlights the dynamic changes within and across hemispheres in adult language-related networks driven by L2 learning.

Causal evidence for a coordinated temporal interplay within the language network.

Recent neurobiological models on language suggest that auditory sentence comprehension is supported by a coordinated temporal interplay within a left-dominant brain network, including the posterior inferior frontal gyrus (pIFG), posterior superior temporal gyrus and sulcus (pSTG/STS), and angular gyrus (AG). Here, we probed the timing and causal relevance of the interplay between these regions by means of concurrent transcranial magnetic stimulation and electroencephalography (TMS-EEG). Our TMS-EEG experiments reveal region- and time-specific causal evidence for a bidirectional information flow from left pSTG/STS to left pIFG and back during auditory sentence processing. Adapting a condition-and-perturb approach, our findings further suggest that the left pSTG/STS can be supported by the left AG in a state-dependent manner.

Left Motor δ Oscillations Reflect Asynchrony Detection in Multisensory Speech Perception.

During multisensory speech perception, slow δ oscillations (∼1-3 Hz) in the listener's brain synchronize with the speech signal, likely engaging in speech signal decomposition. Notable fluctuations in the speech amplitude envelope, resounding speaker prosody, temporally align with articulatory and body gestures and both provide complementary sensations that temporally structure speech. Further, δ oscillations in the left motor cortex seem to align with speech and musical beats, suggesting their possible role in the temporal structuring of (quasi)-rhythmic stimulation. We extended the role of δ oscillations to audiovisual asynchrony detection as a test case of the temporal analysis of multisensory prosody fluctuations in speech. We recorded Electroencephalograph (EEG) responses in an audiovisual asynchrony detection task while participants watched videos of a speaker. We filtered the speech signal to remove verbal content and examined how visual and auditory prosodic features temporally (mis-)align. Results confirm (1) that participants accurately detected audiovisual asynchrony, and (2) increased δ power in the left motor cortex in response to audiovisual asynchrony. The difference of δ power between asynchronous and synchronous conditions predicted behavioral performance, and (3) decreased δ-ß coupling in the left motor cortex when listeners could not accurately map visual and auditory prosodies. Finally, both behavioral and neurophysiological evidence was altered when a speaker's face was degraded by a visual mask. Together, these findings suggest that motor δ oscillations support asynchrony detection of multisensory prosodic fluctuation in speech.SIGNIFICANCE STATEMENT Speech perception is facilitated by regular prosodic fluctuations that temporally structure the auditory signal. Auditory speech processing involves the left motor cortex and associated δ oscillations. However, visual prosody (i.e., a speaker's body movements) complements auditory prosody, and it is unclear how the brain temporally analyses different prosodic features in multisensory speech perception. We combined an audiovisual asynchrony detection task with electroencephalographic (EEG) recordings to investigate how δ oscillations support the temporal analysis of multisensory speech. Results confirmed that asynchrony detection of visual and auditory prosodies leads to increased δ power in left motor cortex and correlates with performance. We conclude that δ oscillations are invoked in an effort to resolve denoted temporal asynchrony in multisensory speech perception.

The time course of speaker-specific language processing.

Listeners are sensitive to a speaker's individual language use and generate expectations for particular speakers. It is unclear, however, how such expectations affect online language processing. In the present EEG study, we presented thirty-two participants with auditory sentence stimuli of two speakers. Speakers differed in their use of two particular syntactic structures, easy subject-initial SOV structures and more difficult object-initial OSV structures. One speaker, the SOV-Speaker, had a high proportion of SOV sentences (75%) and a low proportion of OSV sentences (25%), and vice-versa for the OSV-Speaker. Participants were exposed to the speakers' individual language use in a training session followed by a test session on the consecutive day. ERP-results show that early stages of sentence processing are driven by syntactic processing only and are unaffected by speaker-specific expectations. In a late stage, however, an interaction between speaker and syntax information was observed. For the SOV-Speaker condition, the classical P600-effect reflected the effort of processing difficult and unexpected sentence structures. For the OSV-Speaker condition, both structures elicited different responses on frontal electrodes, possibly indexing effort to switch from a local speaker model to a global model of language use. Overall, the study identifies distinct neural mechanisms related to speaker-specific expectations.

Musical rhythm effects on visual attention are non-rhythmical: evidence against metrical entrainment.

The idea that external rhythms synchronize attention cross-modally has attracted much interest and scientific inquiry. Yet, whether associated attentional modulations are indeed rhythmical in that they spring from and map onto an underlying meter has not been clearly established. Here we tested this idea while addressing the shortcomings of previous work associated with confounding (i) metricality and regularity, (ii) rhythmic and temporal expectations or (iii) global and local temporal effects. We designed sound sequences that varied orthogonally (high/low) in metricality and regularity and presented them as task-irrelevant auditory background in four separate blocks. The participants' task was to detect rare visual targets occurring at a silent metrically aligned or misaligned temporal position. We found that target timing was irrelevant for reaction times and visual event-related potentials. High background regularity and to a lesser extent metricality facilitated target processing across metrically aligned and misaligned positions. Additionally, high regularity modulated auditory background frequencies in the EEG recorded over occipital cortex. We conclude that external rhythms, rather than synchronizing attention cross-modally, confer general, nontemporal benefits. Their predictability conserves processing resources that then benefit stimulus representations in other modalities.

Distinct Neural Networks Relate to Common and Speaker-Specific Language Priors.

Effective natural communication requires listeners to incorporate not only very general linguistic principles which evolved during a lifetime but also other information like the specific individual language use of a particular interlocutor. Traditionally, research has focused on the general linguistic rules, and brain science has shown a left hemispheric fronto-temporal brain network related to this processing. The present fMRI research explores speaker-specific individual language use because it is unknown whether this processing is supported by similar or distinct neural structures. Twenty-eight participants listened to sentences of persons who used more easy or difficult language. This was done by manipulating the proportion of easy SOV vs. complex OSV sentences for each speaker. Furthermore, ambiguous probe sentences were included to test top-down influences of speaker information in the absence of syntactic structure information. We observed distinct neural processing for syntactic complexity and speaker-specific language use. Syntactic complexity correlated with left frontal and posterior temporal regions. Speaker-specific processing correlated with bilateral (right-dominant) fronto-parietal brain regions. Finally, the top-down influence of speaker information was found in frontal and striatal brain regions, suggesting a mechanism for controlled syntactic processing. These findings show distinct neural networks related to general language principles as well as speaker-specific individual language use.

Contributions of left frontal and temporal cortex to sentence comprehension: Evidence from simultaneous TMS-EEG.

Sentence comprehension requires the rapid analysis of semantic and syntactic information. These processes are supported by a left hemispheric dominant fronto-temporal network, including left posterior inferior frontal gyrus (pIFG) and posterior superior temporal gyrus/sulcus (pSTG/STS). Previous electroencephalography (EEG) studies have associated semantic expectancy within a sentence with a modulation of the N400 and syntactic gender violations with increases in the LAN and P600. Here, we combined focal perturbations of neural activity by means of short bursts of transcranial magnetic stimulation (TMS) with simultaneous EEG recordings to probe the functional relevance of pIFG and pSTG/STS for sentence comprehension. We applied 10 Hz TMS bursts of three pulses at verb onset during auditory presentation of short sentences. Verb-based semantic expectancy and article-based syntactic gender requirement were manipulated for the sentence final noun. We did not find any TMS effect at the noun. However, TMS had a short-lasting impact at the mid-sentence verb that differed for the two stimulation sites. Specifically, TMS over pIFG elicited a frontal positivity in the first 200 msec post verb onset whereas TMS over pSTG/STS was limited to a parietal negativity at 200-400 msec post verb onset. This indicates that during verb processing in sentential context, frontal brain areas play an earlier role than temporal areas in predicting the upcoming noun. The short-living perturbation effects at the mid-sentence verb suggest a high degree of online compensation within the language system since the sentence final noun processing was unaffected.

Young children's sentence comprehension: Neural correlates of syntax-semantic competition.

Sentence comprehension requires the assignment of thematic relations between the verb and its noun arguments in order to determine who is doing what to whom. In some languages, such as English, word order is the primary syntactic cue. In other languages, such as German, case-marking is additionally used to assign thematic roles. During development children have to acquire the thematic relevance of these syntactic cues and weigh them against semantic cues. Here we investigated the processing of syntactic cues and semantic cues in 2- and 3-year-old children by analyzing their behavioral and neurophysiological responses. Case-marked subject-first and object-first sentences (syntactic cue) including animate and inanimate nouns (semantic cue) were presented auditorily. The semantic animacy cue either conflicted with or supported the thematic roles assigned by syntactic case-marking. In contrast to adults, for whom semantics did not interfere with case-marking, children attended to both syntactic and to semantic cues with a stronger reliance on semantic cues in early development. Children's event-related brain potentials indicated sensitivity to syntactic information but increased processing costs when case-marking and animacy assigned conflicting thematic roles. These results demonstrate an early developmental sensitivity and ongoing shift towards the use of syntactic cues during sentence comprehension.

Communicative predictions can overrule linguistic priors.

Predictions allow for efficient human communication. To be efficient, listeners' predictions need to be adapted to the communicative context. Here we show that during speech processing this adaptation is a highly flexible and selective process that is able to fine-tune itself to individual language styles of specific interlocutors. In a newly developed paradigm, speakers differed in the probabilities by which they used particular sentence structures. Probe trials were applied to infer participants' syntactic expectations for a given speaker and to track changes of these expectations over time. The results show that listeners fine-tune their linguistic expectations according to the individual language style of a speaker. Strikingly, nine months after the initial experiment these highly specific expectations could be rapidly reactivated when confronted with the particular language style of a speaker but not merely on the basis of an association with speaker identity per se. These findings highlight that communicative interaction fine-tunes and consolidates interlocutor specific communicative predictions which can overrule strong linguistic priors.

The right touch: Stroking of CT-innervated skin promotes vocal emotion processing.

Research has revealed a special mechanoreceptor, called C-tactile (CT) afferent, that is situated in hairy skin and that seems relevant for the processing of social touch. We pursued a possible role of this receptor in the perception of other social signals such as a person's voice. Participants completed three sessions in which they heard surprised and neutral vocal and nonvocal sounds and detected rare sound repetitions. In a given session, participants received no touch or soft brushstrokes to the arm (CT innervated) or palm (CT free). Event-related potentials elicited to sounds revealed that stroking to the arm facilitated the integration of vocal and emotional information. The late positive potential was greater for surprised vocal relative to neutral vocal and nonvocal sounds, and this effect was greater for arm touch relative to both palm touch and no touch. Together, these results indicate that stroking to the arm facilitates the allocation of processing resources to emotional voices, thus supporting the possibility that CT stimulation benefits social perception cross-modally.

Don't Get Me Wrong: ERP Evidence from Cueing Communicative Intentions.

How to make sure that one's utterances are understood as intended when not facing each other? In order to convey communicative intentions, in digital communication emoticons and pragmatic cues are frequently used. Such cueing becomes even more crucial for implied interpretations (e.g., irony) that cannot be understood literally, but require extra information. Sentences, such as 'That's fantastic,' may achieve either a literal or ironic meaning depending on the contextual constraints. In two experiments using event-related brain potentials (ERPs), we examined the effects of cueing communicative intentions (i.e., by means of quotation marks) on ironic and literal language comprehension. An impact of cueing on language processing was seen as early as 200 ms post-stimulus onset by the emergence of a P300 preceding a sustained positivity for cued irony relative to literal language, while for uncued irony a P200-P600 pattern was obtained. In presence of additional information for ironic intentions, pragmatic reanalysis allowing inferences on the message level may have occured immediately. Moreover, by examining the way of cueing (i.e., ambiguous vs. unambiguous cueing) this type of information for communicative intentions appeared to be only effective when the cues were unambiguous by matching pragmatic conventions. The findings suggest that cueing communicative intentions may immediately affect language comprehension, albeit depending on pragmatic conventions of the cues' usage.

When to Take a Gesture Seriously: On How We Use and Prioritize Communicative Cues.

When people talk, their speech is often accompanied by gestures. Although it is known that co-speech gestures can influence face-to-face communication, it is currently unclear to what extent they are actively used and under which premises they are prioritized to facilitate communication. We investigated these open questions in two experiments that varied how pointing gestures disambiguate the utterances of an interlocutor. Participants, whose event-related brain responses were measured, watched a video, where an actress was interviewed about, for instance, classical literature (e.g., Goethe and Shakespeare). While responding, the actress pointed systematically to the left side to refer to, for example, Goethe, or to the right to refer to Shakespeare. Her final statement was ambiguous and combined with a pointing gesture. The P600 pattern found in Experiment 1 revealed that, when pointing was unreliable, gestures were only monitored for their cue validity and not used for reference tracking related to the ambiguity. However, when pointing was a valid cue (Experiment 2), it was used for reference tracking, as indicated by a reduced N400 for pointing. In summary, these findings suggest that a general prioritization mechanism is in use that constantly monitors and evaluates the use of communicative cues against communicative priors on the basis of accumulated error information.

Temporal signatures of processing voiceness and emotion in sound.

This study explored the temporal course of vocal and emotional sound processing. Participants detected rare repetitions in a stimulus stream comprising neutral and surprised non-verbal exclamations and spectrally rotated control sounds. Spectral rotation preserved some acoustic and emotional properties of the vocal originals. Event-related potentials elicited to unrepeated sounds revealed effects of voiceness and emotion. Relative to non-vocal sounds, vocal sounds elicited a larger centro-parietally distributed N1. This effect was followed by greater positivity to vocal relative to non-vocal sounds beginning with the P2 and extending throughout the recording epoch (N4, late positive potential) with larger amplitudes in female than in male listeners. Emotion effects overlapped with the voiceness effects but were smaller and differed topographically. Voiceness and emotion interacted only for the late positive potential, which was greater for vocal-emotional as compared with all other sounds. Taken together, these results point to a multi-stage process in which voiceness and emotionality are represented independently before being integrated in a manner that biases responses to stimuli with socio-emotional relevance.

Dyslexia risk gene relates to representation of sound in the auditory brainstem.

Dyslexia is a reading disorder with strong associations with KIAA0319 and DCDC2. Both genes play a functional role in spike time precision of neurons. Strikingly, poor readers show an imprecise encoding of fast transients of speech in the auditory brainstem. Whether dyslexia risk genes are related to the quality of sound encoding in the auditory brainstem remains to be investigated. Here, we quantified the response consistency of speech-evoked brainstem responses to the acoustically presented syllable [da] in 159 genotyped, literate and preliterate children. When controlling for age, sex, familial risk and intelligence, partial correlation analyses associated a higher dyslexia risk loading with KIAA0319 with noisier responses. In contrast, a higher risk loading with DCDC2 was associated with a trend towards more stable responses. These results suggest that unstable representation of sound, and thus, reduced neural discrimination ability of stop consonants, occurred in genotypes carrying a higher amount of KIAA0319 risk alleles. Current data provide the first evidence that the dyslexia-associated gene KIAA0319 can alter brainstem responses and impair phoneme processing in the auditory brainstem. This brain-gene relationship provides insight into the complex relationships between phenotype and genotype thereby improving the understanding of the dyslexia-inherent complex multifactorial condition.

Can rhythmic auditory cuing remediate language-related deficits in Parkinson's disease?

Neurodegenerative changes of the basal ganglia in idiopathic Parkinson's disease (IPD) lead to motor deficits as well as general cognitive decline. Given these impairments, the question arises as to whether motor and nonmotor deficits can be ameliorated similarly. We reason that a domain-general sensorimotor circuit involved in temporal processing may support the remediation of such deficits. Following findings that auditory cuing benefits gait kinematics, we explored whether reported language-processing deficits in IPD can also be remediated via auditory cuing. During continuous EEG measurement, an individual diagnosed with IPD heard two types of temporally predictable but metrically different auditory beat-based cues: a march, which metrically aligned with the speech accent structure, a waltz that did not metrically align, or no cue before listening to naturally spoken sentences that were either grammatically well formed or were semantically or syntactically incorrect. Results confirmed that only the cuing with a march led to improved computation of syntactic and semantic information. We infer that a marching rhythm may lead to a stronger engagement of the cerebello-thalamo-cortical circuit that compensates dysfunctional striato-cortical timing. Reinforcing temporal realignment, in turn, may lead to the timely processing of linguistic information embedded in the temporally variable speech signal.

A speaker's gesture style can affect language comprehension: ERP evidence from gesture-speech integration.

In face-to-face communication, speech is typically enriched by gestures. Clearly, not all people gesture in the same way, and the present study explores whether such individual differences in gesture style are taken into account during the perception of gestures that accompany speech. Participants were presented with one speaker that gestured in a straightforward way and another that also produced self-touch movements. Adding trials with such grooming movements makes the gesture information a much weaker cue compared with the gestures of the non-grooming speaker. The Electroencephalogram was recorded as participants watched videos of the individual speakers. Event-related potentials elicited by the speech signal revealed that adding grooming movements attenuated the impact of gesture for this particular speaker. Thus, these data suggest that there is sensitivity to the personal communication style of a speaker and that affects the extent to which gesture and speech are integrated during language comprehension.

Inconsistent use of gesture space during abstract pointing impairs language comprehension.

Pointing toward concrete objects is a well-known and efficient communicative strategy. Much less is known about the communicative effectiveness of abstract pointing where the pointing gestures are directed to "empty space." McNeill's (2003) observations suggest that abstract pointing can be used to establish referents in gesture space, without the referents being physically present. Recently, however, it has been shown that abstract pointing typically provides redundant information to the uttered speech thereby suggesting a very limited communicative value (So et al., 2009). In a first approach to tackle this issue we were interested to know whether perceivers are sensitive at all to this gesture cue or whether it is completely discarded as irrelevant add-on information. Sensitivity to for instance a gesture-speech mismatch would suggest a potential communicative function of abstract pointing. Therefore, we devised a mismatch paradigm in which participants watched a video where a female was interviewed on various topics. During her responses, she established two concepts in space using abstract pointing (e.g., pointing to the left when saying Donald, and pointing to the right when saying Mickey). In the last response to each topic, the pointing gesture accompanying a target word (e.g., Donald) was either consistent or inconsistent with the previously established location. Event related brain potentials showed an increased N400 and P600 when gesture and speech referred to different referents, indicating that inconsistent use of gesture space impairs language comprehension. Abstract pointing was found to influence comprehension even though gesture was not crucial to understanding the sentences or conducting the experimental task. These data suggest that a referent was retrieved via abstract pointing and that abstract pointing can potentially be used for referent indication in a discourse. We conclude that abstract pointing has a potential communicative function.

Multisensory integration: the case of a time window of gesture-speech integration.

This experiment investigates the integration of gesture and speech from a multisensory perspective. In a disambiguation paradigm, participants were presented with short videos of an actress uttering sentences like "She was impressed by the BALL, because the GAME/DANCE...." The ambiguous noun (BALL) was accompanied by an iconic gesture fragment containing information to disambiguate the noun toward its dominant or subordinate meaning. We used four different temporal alignments between noun and gesture fragment: the identification point (IP) of the noun was either prior to (+120 msec), synchronous with (0 msec), or lagging behind the end of the gesture fragment (-200 and -600 msec). ERPs triggered to the IP of the noun showed significant differences for the integration of dominant and subordinate gesture fragments in the -200, 0, and +120 msec conditions. The outcome of this integration was revealed at the target words. These data suggest a time window for direct semantic gesture-speech integration ranging from at least -200 up to +120 msec. Although the -600 msec condition did not show any signs of direct integration at the homonym, significant disambiguation was found at the target word. An explorative analysis suggested that gesture information was directly integrated at the verb, indicating that there are multiple positions in a sentence where direct gesture-speech integration takes place. Ultimately, this would implicate that in natural communication, where a gesture lasts for some time, several aspects of that gesture will have their specific and possibly distinct impact on different positions in an utterance.

Auditory Discrimination Between Function Words in Children and Adults: A Mismatch Negativity Study.

Previous behavioral studies showed that it is not until around the age of seven that German children reliably use case markers for the interpretation of complex sentences. Some explanations of this late development suggested that children might have difficulties in perceptual differentiation between function words that carry case information. We tested this hypothesis by using the neurophysiological index of pre-attentive discrimination, the mismatch negativity (MMN). Our data showed that children at the age of 3 years are able to automatically discriminate between the two determiner forms der and den when presented out of sentential context. The determiner form der elicited a more mature MMN response in children than the form den. In adults, the MMN pattern also differed with der showing an earlier peak than den. These findings indicate that der is easier to process than den, which in turn is related to the occurrence frequency of the determiner forms in language.

Distinguishing neurocognitive processes reflected by P600 effects: evidence from ERPs and neural oscillations.

Research on language comprehension using event-related potentials (ERPs) reported distinct ERP components reliably related to the processing of semantic (N400) and syntactic information (P600). Recent ERP studies have challenged this well-defined distinction by showing P600 effects for semantic and pragmatic anomalies. So far, it is still unresolved whether the P600 reflects specific or rather common processes. The present study addresses this question by investigating ERPs in response to a syntactic and pragmatic (irony) manipulation, as well as a combined syntactic and pragmatic manipulation. For the syntactic condition, a morphosyntactic violation was applied, whereas for the pragmatic condition, such as "That is rich", either an ironic or literal interpretation was achieved, depending on the prior context. The ERPs at the critical word showed a LAN-P600 pattern for syntactically incorrect sentences relative to correct ones. For ironic compared to literal sentences, ERPs showed a P200 effect followed by a P600 component. In comparison of the syntax-related P600 to the irony-related P600, distributional differences were found. Moreover, for the P600 time window (i.e., 500-900 ms), different changes in theta power between the syntax and pragmatics effects were found, suggesting that different patterns of neural activity contributed to each respective effect. Thus, both late positivities seem to be differently sensitive to these two types of linguistic information, and might reflect distinct neurocognitive processes, such as reanalysis of the sentence structure versus pragmatic reanalysis.