Localizing Syntactic Composition with Left-Corner Recurrent Neural Network Grammars.

In computational neurolinguistics, it has been demonstrated that hierarchical models such as recurrent neural network grammars (RNNGs), which jointly generate word sequences and their syntactic structures via the syntactic composition, better explained human brain activity than sequential models such as long short-term memory networks (LSTMs). However, the vanilla RNNG has employed the top-down parsing strategy, which has been pointed out in the psycholinguistics literature as suboptimal especially for head-final/left-branching languages, and alternatively the left-corner parsing strategy has been proposed as the psychologically plausible parsing strategy. In this article, building on this line of inquiry, we investigate not only whether hierarchical models like RNNGs better explain human brain activity than sequential models like LSTMs, but also which parsing strategy is more neurobiologically plausible, by developing a novel fMRI corpus where participants read newspaper articles in a head-final/left-branching language, namely Japanese, through the naturalistic fMRI experiment. The results revealed that left-corner RNNGs outperformed both LSTMs and top-down RNNGs in the left inferior frontal and temporal-parietal regions, suggesting that there are certain brain regions that localize the syntactic composition with the left-corner parsing strategy.

Expectations modulate retrieval interference during ellipsis resolution.

Memory operations during language comprehension are subject to interference: retrieval is harder when items are linguistically similar to each other. We test how such interference effects might be modulated by linguistic expectations. Theories differ in how these factors might interact; we consider three possibilities: (i) predictability determines the need for retrieval, (ii) predictability affects cue-preference during retrieval, or (iii) word predictability moderates the effect of noise in memory during retrieval. We first demonstrate that expectations for a target word modulate retrieval interference in Mandarin noun-phrase ellipsis in an electroencephalography (EEG) experiment. This result obtains in globally ungrammatical sentences - termed "facilitatory interference." Such a pattern is inconsistent with theories that focus only on the need for retrieval. To tease apart cue-preferences from noisy-memory representations, we operationalize the latter using a Transformer neural network language model. Confronting the model with our stimuli reveals an interference effect, consistent with prior work, but that effect does not interact with predictability in contrast to human EEG results. Together, these data are most consistent with the hypothesis that the predictability of target items affects cue-preferences during retrieval.

Modeling Structure-Building in the Brain With CCG Parsing and Large Language Models.

To model behavioral and neural correlates of language comprehension in naturalistic environments, researchers have turned to broad-coverage tools from natural-language processing and machine learning. Where syntactic structure is explicitly modeled, prior work has relied predominantly on context-free grammars (CFGs), yet such formalisms are not sufficiently expressive for human languages. Combinatory categorial grammars (CCGs) are sufficiently expressive directly compositional models of grammar with flexible constituency that affords incremental interpretation. In this work, we evaluate whether a more expressive CCG provides a better model than a CFG for human neural signals collected with functional magnetic resonance imaging (fMRI) while participants listen to an audiobook story. We further test between variants of CCG that differ in how they handle optional adjuncts. These evaluations are carried out against a baseline that includes estimates of next-word predictability from a transformer neural network language model. Such a comparison reveals unique contributions of CCG structure-building predominantly in the left posterior temporal lobe: CCG-derived measures offer a superior fit to neural signals compared to those derived from a CFG. These effects are spatially distinct from bilateral superior temporal effects that are unique to predictability. Neural effects for structure-building are thus separable from predictability during naturalistic listening, and those effects are best characterized by a grammar whose expressive power is motivated on independent linguistic grounds.

Le Petit Prince multilingual naturalistic fMRI corpus.

Neuroimaging using more ecologically valid stimuli such as audiobooks has advanced our understanding of natural language comprehension in the brain. However, prior naturalistic stimuli have typically been restricted to a single language, which limited generalizability beyond small typological domains. Here we present the Le Petit Prince fMRI Corpus (LPPC-fMRI), a multilingual resource for research in the cognitive neuroscience of speech and language during naturalistic listening (OpenNeuro: ds003643). 49 English speakers, 35 Chinese speakers and 28 French speakers listened to the same audiobook The Little Prince in their native language while multi-echo functional magnetic resonance imaging was acquired. We also provide time-aligned speech annotation and word-by-word predictors obtained using natural language processing tools. The resulting timeseries data are shown to be of high quality with good temporal signal-to-noise ratio and high inter-subject correlation. Data-driven functional analyses provide further evidence of data quality. This annotated, multilingual fMRI dataset facilitates future re-analysis that addresses cross-linguistic commonalities and differences in the neural substrate of language processing on multiple perceptual and linguistic levels.

Hierarchy, Not Lexical Regularity, Modulates Low-Frequency Neural Synchrony During Language Comprehension.

Neural responses appear to synchronize with sentence structure. However, researchers have debated whether this response in the delta band (0.5-3 Hz) really reflects hierarchical information or simply lexical regularities. Computational simulations in which sentences are represented simply as sequences of high-dimensional numeric vectors that encode lexical information seem to give rise to power spectra similar to those observed for sentence synchronization, suggesting that sentence-level cortical tracking findings may reflect sequential lexical or part-of-speech information, and not necessarily hierarchical syntactic information. Using electroencephalography (EEG) data and the frequency-tagging paradigm, we develop a novel experimental condition to tease apart the predictions of the lexical and the hierarchical accounts of the attested low-frequency synchronization. Under a lexical model, synchronization should be observed even when words are reversed within their phrases (e.g., "sheep white grass eat" instead of "white sheep eat grass"), because the same lexical items are preserved at the same regular intervals. Critically, such stimuli are not syntactically well-formed; thus a hierarchical model does not predict synchronization of phrase- and sentence-level structure in the reversed phrase condition. Computational simulations confirm these diverging predictions. EEG data from N = 31 native speakers of Mandarin show robust delta synchronization to syntactically well-formed isochronous speech. Importantly, no such pattern is observed for reversed phrases, consistent with the hierarchical, but not the lexical, accounts.

EEG Correlates of Long-Distance Dependency Formation in Mandarin Wh-Questions.

Event-related potential components are sensitive to the processes underlying how questions are understood. We use so-called "covert" wh-questions in Mandarin to probe how such components generalize across different kinds of constructions. This study shows that covert Mandarin wh-questions do not elicit anterior negativities associated with memory maintenance, even when such a dependency is unambiguously cued. N = 37 native speakers of Mandarin Chinese read Chinese questions and declarative sentences word-by-word during EEG recording. In contrast to prior studies, no sustained anterior negativity (SAN) was observed between the cue word, such as the question-embedding verb "wonder," and the in-situ wh-filler. SANs have been linked with working memory maintenance, suggesting that grammatical features may not impose the same maintenance demands as the content words used in prior work.

Predictive Processing during a Naturalistic Statistical Learning Task in ASD.

Children's sensitivity to regularities within the linguistic stream, such as the likelihood that syllables co-occur, is foundational to speech segmentation and language acquisition. Yet, little is known about the neurocognitive mechanisms underlying speech segmentation in typical development and in neurodevelopmental disorders that impact language acquisition such as autism spectrum disorder (ASD). Here, we investigate the neural signals of statistical learning in 15 human participants (children ages 8-12) with a clinical diagnosis of ASD and 14 age-matched and gender-matched typically developing peers. We tracked the evoked neural responses to syllable sequences in a naturalistic statistical learning corpus using magnetoencephalography (MEG) in the left primary auditory cortex, posterior superior temporal gyrus (pSTG), and inferior frontal gyrus (IFG), across three repetitions of the passage. In typically developing children, we observed a neural index of learning in all three regions of interest (ROIs), measured by the change in evoked response amplitude as a function of syllable surprisal across passage repetitions. As surprisal increased, the amplitude of the neural response increased; this sensitivity emerged after repeated exposure to the corpus. Children with ASD did not show this pattern of learning in all three regions. We discuss two possible hypotheses related to children's sensitivity to bottom-up sensory deficits and difficulty with top-down incremental processing.

Localizing syntactic predictions using recurrent neural network grammars.

Brain activity in numerous perisylvian brain regions is modulated by the expectedness of linguistic stimuli. We leverage recent advances in computational parsing models to test what representations guide the processes reflected by this activity. Recurrent Neural Network Grammars (RNNGs) are generative models of (tree, string) pairs that use neural networks to drive derivational choices. Parsing with them yields a variety of incremental complexity metrics that we evaluate against a publicly available fMRI data-set recorded while participants simply listen to an audiobook story. Surprisal, which captures a word's un-expectedness, correlates with a wide range of temporal and frontal regions when it is calculated based on word-sequence information using a top-performing LSTM neural network language model. The explicit encoding of hierarchy afforded by the RNNG additionally captures activity in left posterior temporal areas. A separate metric tracking the number of derivational steps taken between words correlates with activity in the left temporal lobe and inferior frontal gyrus. This pattern of results narrows down the kinds of linguistic representations at play during predictive processing across the brain's language network.

Phase synchronization varies systematically with linguistic structure composition.

Computation in neuronal assemblies is putatively reflected in the excitatory and inhibitory cycles of activation distributed throughout the brain. In speech and language processing, coordination of these cycles resulting in phase synchronization has been argued to reflect the integration of information on different timescales (e.g. segmenting acoustics signals to phonemic and syllabic representations; (Giraud and Poeppel 2012 Nat. Neurosci.15, 511 (doi:10.1038/nn.3063)). A natural extension of this claim is that phase synchronization functions similarly to support the inference of more abstract higher-level linguistic structures (Martin 2016 Front. Psychol.7, 120; Martin and Doumas 2017 PLoS Biol. 15, e2000663 (doi:10.1371/journal.pbio.2000663); Martin and Doumas. 2019 Curr. Opin. Behav. Sci.29, 77-83 (doi:10.1016/j.cobeha.2019.04.008)). Hale et al. (Hale et al. 2018 Finding syntax in human encephalography with beam search. arXiv 1806.04127 (http://arxiv.org/abs/1806.04127)) showed that syntactically driven parsing decisions predict electroencephalography (EEG) responses in the time domain; here we ask whether phase synchronization in the form of either inter-trial phrase coherence or cross-frequency coupling (CFC) between high-frequency (i.e. gamma) bursts and lower-frequency carrier signals (i.e. delta, theta), changes as the linguistic structures of compositional meaning (viz., bracket completions, as denoted by the onset of words that complete phrases) accrue. We use a naturalistic story-listening EEG dataset from Hale et al. to assess the relationship between linguistic structure and phase alignment. We observe increased phase synchronization as a function of phrase counts in the delta, theta, and gamma bands, especially for function words. A more complex pattern emerged for CFC as phrase count changed, possibly related to the lack of a one-to-one mapping between 'size' of linguistic structure and frequency band-an assumption that is tacit in recent frameworks. These results emphasize the important role that phase synchronization, desynchronization, and thus, inhibition, play in the construction of compositional meaning by distributed neural networks in the brain. This article is part of the theme issue 'Towards mechanistic models of meaning composition'.

Hierarchical structure guides rapid linguistic predictions during naturalistic listening.

The grammar, or syntax, of human language is typically understood in terms of abstract hierarchical structures. However, theories of language processing that emphasize sequential information, not hierarchy, successfully model diverse phenomena. Recent work probing brain signals has shown mixed evidence for hierarchical information in some tasks. We ask whether sequential or hierarchical information guides the expectations that a human listener forms about a word's part-of-speech when simply listening to every-day language. We compare the predictions of three computational models against electroencephalography signals recorded from human participants who listen passively to an audiobook story. We find that predictions based on hierarchical structure correlate with the human brain response above-and-beyond predictions based only on sequential information. This establishes a link between hierarchical linguistic structure and neural signals that generalizes across the range of syntactic structures found in every-day language.

Mapping Meanings.

In a 2016 paper, Huth and colleagues probed, in a general way, how word meanings map onto cortical locations. By comparing the fit between alternative maps, this methodology offered a means to evaluate what sorts of meaning representations the brain handles under ecologically realistic conditions.

Patterns of altered neural synchrony in the default mode network in autism spectrum disorder revealed with magnetoencephalography (MEG): Relationship to clinical symptomatology.

Disrupted neural synchrony may be a primary electrophysiological abnormality in autism spectrum disorders (ASD), altering communication between discrete brain regions and contributing to abnormalities in patterns of connectivity within identified neural networks. Studies exploring brain dynamics to comprehensively characterize and link connectivity to large-scale cortical networks and clinical symptoms are lagging considerably. Patterns of neural coherence within the Default Mode Network (DMN) and Salience Network (SN) during resting state were investigated in 12 children with ASD (MAge = 9.2) and 13 age and gender-matched neurotypicals (NT) (MAge = 9.3) with magnetoencephalography. Coherence between 231 brain region pairs within four frequency bands (theta (4-7 Hz), alpha, (8-12 Hz), beta (13-30 Hz), and gamma (30-80 Hz)) was calculated. Relationships between neural coherence and social functioning were examined. ASD was characterized by lower synchronization across all frequencies, reaching clinical significance in the gamma band. Lower gamma synchrony between fronto-temporo-parietal regions was observed, partially consistent with diminished default mode network (DMN) connectivity. Lower gamma coherence in ASD was evident in cross-hemispheric connections between: angular with inferior/middle frontal; middle temporal with middle/inferior frontal; and within right-hemispheric connections between angular, middle temporal, and inferior/middle frontal cortices. Lower gamma coherence between left angular and left superior frontal, right inferior/middle frontal, and right precuneus and between right angular and inferior/middle frontal cortices was related to lower social/social-communication functioning. Results suggest a pattern of lower gamma band coherence in a subset of regions within the DMN in ASD (angular and middle temporal cortical areas) related to lower social/social-communicative functioning. Autism Res 2018, 11: 434-449. © 2017 International Society for Autism Research, Wiley Periodicals, Inc. LAY SUMMARY: Communication between different areas of the brain was observed in children with ASD and neurotypical children while awake, but not working on a task. Magnetoencephalography was used to measure tiny magnetic fields naturally generated via brain activity. The brains of children with ASD showed less communication between areas that are important for social information processing compared to the brains of neurotypical children. The amount of communication between these areas was associated with social and social communication difficulties.

MEG Evidence for Incremental Sentence Composition in the Anterior Temporal Lobe.

Research investigating the brain basis of language comprehension has associated the left anterior temporal lobe (ATL) with sentence-level combinatorics. Using magnetoencephalography (MEG), we test the parsing strategy implemented in this brain region. The number of incremental parse steps from a predictive left-corner parsing strategy that is supported by psycholinguistic research is compared with those from a less-predictive strategy. We test for a correlation between parse steps and source-localized MEG activity recorded while participants read a story. Left-corner parse steps correlated with activity in the left ATL around 350-500 ms after word onset. No other correlations specific to sentence comprehension were observed. These data indicate that the left ATL engages in combinatoric processing that is well characterized by a predictive left-corner parsing strategy.

Magnetoencephalography shows atypical sensitivity to linguistic sound sequences in autism spectrum disorder.

Neuroscientific evidence points toward atypical auditory processing in individuals with autism spectrum disorders (ASD), and yet, the consequences of this for receptive language remain unclear. Using magnetoencephalography and a passive listening task, we test for cascading effects on speech sound processing. Children with ASD and age-matched control participants (8-12 years old) listened to nonce linguistic stimuli that either did or did not conform to the phonological rules that govern consonant sequences in English (e.g. legal 'vimp' vs. illegal 'vimk'). Beamformer source analysis was used to isolate evoked responses (0.1-30 Hz) to these stimuli in the left and the right auditory cortex. Right auditory responses from participants with ASD, but not control participants, showed an attenuated response to illegal sequences relative to legal sequences that emerged around 330 ms after the onset of the critical phoneme. These results suggest that phonological processing is impacted in ASD, perhaps because of cascading effects from disrupted initial acoustic processing.

Abstract linguistic structure correlates with temporal activity during naturalistic comprehension.

Neurolinguistic accounts of sentence comprehension identify a network of relevant brain regions, but do not detail the information flowing through them. We investigate syntactic information. Does brain activity implicate a computation over hierarchical grammars or does it simply reflect linear order, as in a Markov chain? To address this question, we quantify the cognitive states implied by alternative parsing models. We compare processing-complexity predictions from these states against fMRI timecourses from regions that have been implicated in sentence comprehension. We find that hierarchical grammars independently predict timecourses from left anterior and posterior temporal lobe. Markov models are predictive in these regions and across a broader network that includes the inferior frontal gyrus. These results suggest that while linear effects are wide-spread across the language network, certain areas in the left temporal lobe deal with abstract, hierarchical syntactic representations.

Multimodal imaging of temporal processing in typical and atypical language development.

New approaches to understanding language and reading acquisition propose that the human brain's ability to synchronize its neural firing rate to syllable-length linguistic units may be important to children's ability to acquire human language. Yet, little evidence from brain imaging studies has been available to support this proposal. Here, we summarize three recent brain imaging (functional near-infrared spectroscopy (fNIRS), functional magnetic resonance imaging (fMRI), and magnetoencephalography (MEG)) studies from our laboratories with young English-speaking children (aged 6-12 years). In the first study (fNIRS), we used an auditory beat perception task to show that, in children, the left superior temporal gyrus (STG) responds preferentially to rhythmic beats at 1.5 Hz. In the second study (fMRI), we found correlations between children's amplitude rise-time sensitivity, phonological awareness, and brain activation in the left STG. In the third study (MEG), typically developing children outperformed children with autism spectrum disorder in extracting words from rhythmically rich foreign speech and displayed different brain activation during the learning phase. The overall findings suggest that the efficiency with which left temporal regions process slow temporal (rhythmic) information may be important for gains in language and reading proficiency. These findings carry implications for better understanding of the brain's mechanisms that support language and reading acquisition during both typical and atypical development.

Spectro-temporal correlates of lexical access during auditory lexical decision.

Lexical access during speech comprehension comprises numerous computations, including activation, competition, and selection. The spatio-temporal profile of these processes involves neural activity in peri-auditory cortices at least as early as 200 ms after stimulation. Their oscillatory dynamics are less well understood, although reports link alpha band de-synchronization with lexical processing. We used magnetoencephalography (MEG) to examine whether these alpha-related oscillations reflect the speed of lexical access, as would be predicted if they index lexical activation. In an auditory semantic priming protocol, monosyllabic nouns were presented while participants performed a lexical decision task. Spatially-localizing beamforming was used to examine spectro-temporal effects in left and right auditory cortex time-locked to target word onset. Alpha and beta de-synchronization (10-20 Hz ERD) was attenuated for words following a related prime compared to an unrelated prime beginning about 270 ms after stimulus onset. This timing is consistent with how information about word identity unfolds incrementally in speech, quantified in information-theoretic terms. These findings suggest that alpha de-synchronization during auditory word processing is associated with early stages of lexical access.

The time-course and spatial distribution of brain activity associated with sentence processing.

Sentence comprehension involves a host of highly interrelated processes, including syntactic parsing, semantic composition, and pragmatic inferencing. In neuroimaging, a primary paradigm for examining the brain bases of sentence processing has been to compare brain activity elicited by sentences versus unstructured lists of words. These studies commonly find an effect of increased activity for sentences in the anterior temporal lobes (aTL). Together with neuropsychological data, these findings have motivated the hypothesis that the aTL is engaged in sentence level combinatorics. Combinatoric processing during language comprehension, however, occurs within tens and hundreds of milliseconds, i.e., at a time-scale much faster than the temporal resolution of hemodynamic measures. Here, we examined the time-course of sentence-level processing using magnetoencephalography (MEG) to better understand the temporal profile of activation in this common paradigm and to test a key prediction of the combinatoric hypothesis: because sentences are interpreted incrementally, word-by-word, activity associated with basic linguistic combinatorics should be time-locked to word-presentation. Our results reveal increased anterior temporal activity for sentences compared to word lists beginning approximately 250 ms after word onset. We also observed increased activation in a network of other brain areas, extending across posterior temporal, inferior frontal, and ventral medial areas. These findings confirm a key prediction of the combinatoric hypothesis for the aTL and further elucidate the spatio-temporal characteristics of sentence-level computations in the brain.

Syntactic structure building in the anterior temporal lobe during natural story listening.

The neural basis of syntax is a matter of substantial debate. In particular, the inferior frontal gyrus (IFG), or Broca's area, has been prominently linked to syntactic processing, but the anterior temporal lobe has been reported to be activated instead of IFG when manipulating the presence of syntactic structure. These findings are difficult to reconcile because they rely on different laboratory tasks which tap into distinct computations, and may only indirectly relate to natural sentence processing. Here we assessed neural correlates of syntactic structure building in natural language comprehension, free from artificial task demands. Subjects passively listened to Alice in Wonderland during functional magnetic resonance imaging and we correlated brain activity with a word-by-word measure of the amount syntactic structure analyzed. Syntactic structure building correlated with activity in the left anterior temporal lobe, but there was no evidence for a correlation between syntactic structure building and activity in inferior frontal areas. Our results suggest that the anterior temporal lobe computes syntactic structure under natural conditions.

Processing events: behavioral and neuromagnetic correlates of Aspectual Coercion.

Much recent psycho- and neuro-linguistic work has aimed to elucidate the mechanisms by which sentence meanings are composed by investigating the processing of semantic mismatch. One controversial case for theories of semantic composition is expressions such as the clown jumped for ten minutes, in which the aspectual properties of a punctual verb clash with those of a durative modifier. Such sentences have been proposed to involve a coercion operation which shifts the punctual meaning of the verb to an iterative one. However, processing studies addressing this hypothesis have yielded mixed results. In this study, we tested four hypotheses of how aspectual mismatch is resolved with self-paced reading and magnetoencephalography. Using a set of verbs normed for punctuality, we identified an immediate behavioral cost of mismatch. The neural correlates of this processing were found to match effects in midline prefrontal regions previously implicated in the resolution of complement coercion. We also identified earlier effects in right-lateral frontal and temporal sites. We suggest that of the representational hypotheses currently in the literature, these data are most consistent with an account where aspectual mismatch initially involves the composition of an anomalous meaning that is later repaired via coercion.