The neural architecture of language: Integrative modeling converges on predictive processing.

The neuroscience of perception has recently been revolutionized with an integrative modeling approach in which computation, brain function, and behavior are linked across many datasets and many computational models. By revealing trends across models, this approach yields novel insights into cognitive and neural mechanisms in the target domain. We here present a systematic study taking this approach to higher-level cognition: human language processing, our species' signature cognitive skill. We find that the most powerful "transformer" models predict nearly 100% of explainable variance in neural responses to sentences and generalize across different datasets and imaging modalities (functional MRI and electrocorticography). Models' neural fits ("brain score") and fits to behavioral responses are both strongly correlated with model accuracy on the next-word prediction task (but not other language tasks). Model architecture appears to substantially contribute to neural fit. These results provide computationally explicit evidence that predictive processing fundamentally shapes the language comprehension mechanisms in the human brain.

Incremental Language Comprehension Difficulty Predicts Activity in the Language Network but Not the Multiple Demand Network.

What role do domain-general executive functions play in human language comprehension? To address this question, we examine the relationship between behavioral measures of comprehension and neural activity in the domain-general "multiple demand" (MD) network, which has been linked to constructs like attention, working memory, inhibitory control, and selection, and implicated in diverse goal-directed behaviors. Specifically, functional magnetic resonance imaging data collected during naturalistic story listening are compared with theory-neutral measures of online comprehension difficulty and incremental processing load (reading times and eye-fixation durations). Critically, to ensure that variance in these measures is driven by features of the linguistic stimulus rather than reflecting participant- or trial-level variability, the neuroimaging and behavioral datasets were collected in nonoverlapping samples. We find no behavioral-neural link in functionally localized MD regions; instead, this link is found in the domain-specific, fronto-temporal "core language network," in both left-hemispheric areas and their right hemispheric homotopic areas. These results argue against strong involvement of domain-general executive circuits in language comprehension.

Functional Network Dynamics of the Language System.

During linguistic processing, a set of brain regions on the lateral surfaces of the left frontal, temporal, and parietal cortices exhibit robust responses. These areas display highly correlated activity while a subject rests or performs a naturalistic language comprehension task, suggesting that they form an integrated functional system. Evidence suggests that this system is spatially and functionally distinct from other systems that support high-level cognition in humans. Yet, how different regions within this system might be recruited dynamically during task performance is not well understood. Here we use network methods, applied to fMRI data collected from 22 human subjects performing a language comprehension task, to reveal the dynamic nature of the language system. We observe the presence of a stable core of brain regions, predominantly located in the left hemisphere, that consistently coactivate with one another. We also observe the presence of a more flexible periphery of brain regions, predominantly located in the right hemisphere, that coactivate with different regions at different times. However, the language functional ROIs in the angular gyrus and the anterior temporal lobe were notable exceptions to this trend. By highlighting the temporal dimension of language processing, these results suggest a trade-off between a region's specialization and its capacity for flexible network reconfiguration.

Semantic projection recovers rich human knowledge of multiple object features from word embeddings.

How is knowledge about word meaning represented in the mental lexicon? Current computational models infer word meanings from lexical co-occurrence patterns. They learn to represent words as vectors in a multidimensional space, wherein words that are used in more similar linguistic contexts-that is, are more semantically related-are located closer together. However, whereas inter-word proximity captures only overall relatedness, human judgements are highly context dependent. For example, dolphins and alligators are similar in size but differ in dangerousness. Here, we use a domain-general method to extract context-dependent relationships from word embeddings: 'semantic projection' of word-vectors onto lines that represent features such as size (the line connecting the words 'small' and 'big') or danger ('safe' to 'dangerous'), analogous to 'mental scales'. This method recovers human judgements across various object categories and properties. Thus, the geometry of word embeddings explicitly represents a wealth of context-dependent world knowledge.

Lack of selectivity for syntax relative to word meanings throughout the language network.

To understand what you are reading now, your mind retrieves the meanings of words and constructions from a linguistic knowledge store (lexico-semantic processing) and identifies the relationships among them to construct a complex meaning (syntactic or combinatorial processing). Do these two sets of processes rely on distinct, specialized mechanisms or, rather, share a common pool of resources? Linguistic theorizing, empirical evidence from language acquisition and processing, and computational modeling have jointly painted a picture whereby lexico-semantic and syntactic processing are deeply inter-connected and perhaps not separable. In contrast, many current proposals of the neural architecture of language continue to endorse a view whereby certain brain regions selectively support syntactic/combinatorial processing, although the locus of such "syntactic hub", and its nature, vary across proposals. Here, we searched for selectivity for syntactic over lexico-semantic processing using a powerful individual-subjects fMRI approach across three sentence comprehension paradigms that have been used in prior work to argue for such selectivity: responses to lexico-semantic vs. morpho-syntactic violations (Experiment 1); recovery from neural suppression across pairs of sentences differing in only lexical items vs. only syntactic structure (Experiment 2); and same/different meaning judgments on such sentence pairs (Experiment 3). Across experiments, both lexico-semantic and syntactic conditions elicited robust responses throughout the left fronto-temporal language network. Critically, however, no regions were more strongly engaged by syntactic than lexico-semantic processing, although some regions showed the opposite pattern. Thus, contra many current proposals of the neural architecture of language, syntactic/combinatorial processing is not separable from lexico-semantic processing at the level of brain regions-or even voxel subsets-within the language network, in line with strong integration between these two processes that has been consistently observed in behavioral and computational language research. The results further suggest that the language network may be generally more strongly concerned with meaning than syntactic form, in line with the primary function of language-to share meanings across minds.

fMRI reveals language-specific predictive coding during naturalistic sentence comprehension.

Much research in cognitive neuroscience supports prediction as a canonical computation of cognition across domains. Is such predictive coding implemented by feedback from higher-order domain-general circuits, or is it locally implemented in domain-specific circuits? What information sources are used to generate these predictions? This study addresses these two questions in the context of language processing. We present fMRI evidence from a naturalistic comprehension paradigm (1) that predictive coding in the brain's response to language is domain-specific, and (2) that these predictions are sensitive both to local word co-occurrence patterns and to hierarchical structure. Using a recently developed continuous-time deconvolutional regression technique that supports data-driven hemodynamic response function discovery from continuous BOLD signal fluctuations in response to naturalistic stimuli, we found effects of prediction measures in the language network but not in the domain-general multiple-demand network, which supports executive control processes and has been previously implicated in language comprehension. Moreover, within the language network, surface-level and structural prediction effects were separable. The predictability effects in the language network were substantial, with the model capturing over 37% of explainable variance on held-out data. These findings indicate that human sentence processing mechanisms generate predictions about upcoming words using cognitive processes that are sensitive to hierarchical structure and specialized for language processing, rather than via feedback from high-level executive control mechanisms.

A robust dissociation among the language, multiple demand, and default mode networks: Evidence from inter-region correlations in effect size.

Complex cognitive processes, including language, rely on multiple mental operations that are carried out by several large-scale functional networks in the frontal, temporal, and parietal association cortices of the human brain. The central division of cognitive labor is between two fronto-parietal bilateral networks: (a) the multiple demand (MD) network, which supports executive processes, such as working memory and cognitive control, and is engaged by diverse task domains, including language, especially when comprehension gets difficult; and (b) the default mode network (DMN), which supports introspective processes, such as mind wandering, and is active when we are not engaged in processing external stimuli. These two networks are strongly dissociated in both their functional profiles and their patterns of activity fluctuations during naturalistic cognition. Here, we focus on the functional relationship between these two networks and a third network: (c) the fronto-temporal left-lateralized "core" language network, which is selectively recruited by linguistic processing. Is the language network distinct and dissociated from both the MD network and the DMN, or is it synchronized and integrated with one or both of them? Recent work has provided evidence for a dissociation between the language network and the MD network. However, the relationship between the language network and the DMN is less clear, with some evidence for coordinated activity patterns and similar response profiles, perhaps due to the role of both in semantic processing. Here we use a novel fMRI approach to examine the relationship among the three networks: we measure the strength of activations in different language, MD, and DMN regions to functional contrasts typically used to identify each network, and then test which regions co-vary in their contrast effect sizes across 60 individuals. We find that effect sizes correlate strongly within each network (e.g., one language region and another language region, or one DMN region and another DMN region), but show little or no correlation for region pairs across networks (e.g., a language region and a DMN region). Thus, using our novel method, we replicate the language/MD network dissociation discovered previously with other approaches, and also show that the language network is robustly dissociated from the DMN, overall suggesting that these three networks contribute to high-level cognition in different ways and, perhaps, support distinct computations. Inter-individual differences in effect sizes therefore do not simply reflect general differences in vascularization or attention, but exhibit sensitivity to the functional architecture of the brain. The strength of activation in each network can thus be probed separately in studies that attempt to link neural variability to behavioral or genetic variability.