Deep Artificial Neural Networks Reveal a Distributed Cortical Network Encoding Propositional Sentence-Level Meaning.

Understanding how and where in the brain sentence-level meaning is constructed from words presents a major scientific challenge. Recent advances have begun to explain brain activation elicited by sentences using vector models of word meaning derived from patterns of word co-occurrence in text corpora. These studies have helped map out semantic representation across a distributed brain network spanning temporal, parietal, and frontal cortex. However, it remains unclear whether activation patterns within regions reflect unified representations of sentence-level meaning, as opposed to superpositions of context-independent component words. This is because models have typically represented sentences as "bags-of-words" that neglect sentence-level structure. To address this issue, we interrogated fMRI activation elicited as 240 sentences were read by 14 participants (9 female, 5 male), using sentences encoded by a recurrent deep artificial neural-network trained on a sentence inference task (InferSent). Recurrent connections and nonlinear filters enable InferSent to transform sequences of word vectors into unified "propositional" sentence representations suitable for evaluating intersentence entailment relations. Using voxelwise encoding modeling, we demonstrate that InferSent predicts elements of fMRI activation that cannot be predicted by bag-of-words models and sentence models using grammatical rules to assemble word vectors. This effect occurs throughout a distributed network, which suggests that propositional sentence-level meaning is represented within and across multiple cortical regions rather than at any single site. In follow-up analyses, we place results in the context of other deep network approaches (ELMo and BERT) and estimate the degree of unpredicted neural signal using an "experiential" semantic model and cross-participant encoding.SIGNIFICANCE STATEMENT A modern-day scientific challenge is to understand how the human brain transforms word sequences into representations of sentence meaning. A recent approach, emerging from advances in functional neuroimaging, big data, and machine learning, is to computationally model meaning, and use models to predict brain activity. Such models have helped map a cortical semantic information-processing network. However, how unified sentence-level information, as opposed to word-level units, is represented throughout this network remains unclear. This is because models have typically represented sentences as unordered "bags-of-words." Using a deep artificial neural network that recurrently and nonlinearly combines word representations into unified propositional sentence representations, we provide evidence that sentence-level information is encoded throughout a cortical network, rather than in a single region.

An Integrated Neural Decoder of Linguistic and Experiential Meaning.

The brain is thought to combine linguistic knowledge of words and nonlinguistic knowledge of their referents to encode sentence meaning. However, functional neuroimaging studies aiming at decoding language meaning from neural activity have mostly relied on distributional models of word semantics, which are based on patterns of word co-occurrence in text corpora. Here, we present initial evidence that modeling nonlinguistic "experiential" knowledge contributes to decoding neural representations of sentence meaning. We model attributes of peoples' sensory, motor, social, emotional, and cognitive experiences with words using behavioral ratings. We demonstrate that fMRI activation elicited in sentence reading is more accurately decoded when this experiential attribute model is integrated with a text-based model than when either model is applied in isolation (participants were 5 males and 9 females). Our decoding approach exploits a representation-similarity-based framework, which benefits from being parameter free, while performing at accuracy levels comparable with those from parameter fitting approaches, such as ridge regression. We find that the text-based model contributes particularly to the decoding of sentences containing linguistically oriented "abstract" words and reveal tentative evidence that the experiential model improves decoding of more concrete sentences. Finally, we introduce a cross-participant decoding method to estimate an upper bound on model-based decoding accuracy. We demonstrate that a substantial fraction of neural signal remains unexplained, and leverage this gap to pinpoint characteristics of weakly decoded sentences and hence identify model weaknesses to guide future model development.SIGNIFICANCE STATEMENT Language gives humans the unique ability to communicate about historical events, theoretical concepts, and fiction. Although words are learned through language and defined by their relations to other words in dictionaries, our understanding of word meaning presumably draws heavily on our nonlinguistic sensory, motor, interoceptive, and emotional experiences with words and their referents. Behavioral experiments lend support to the intuition that word meaning integrates aspects of linguistic and nonlinguistic "experiential" knowledge. However, behavioral measures do not provide a window on how meaning is represented in the brain and tend to necessitate artificial experimental paradigms. We present a model-based approach that reveals early evidence that experiential and linguistically acquired knowledge can be detected in brain activity elicited in reading natural sentences.

Longitudinal Functional Brain Mapping in Supernormals.

Prevention of age-related cognitive decline is an increasingly important topic. Recently, increased attention is being directed at understanding biological models of successful cognitive aging. Here, we examined resting-state brain regional low-frequency oscillations using functional magnetic resonance imaging in 19 older adults with excellent cognitive abilities (Supernormals), 28 older adults with normative cognition, 57 older adults with amnestic mild cognitive impairment, and 26 with Alzheimer's disease. We identified a "Supernormal map", a set of regions whose oscillations were resistant to the aging-associated neurodegenerative process, including the right fusiform gyrus, right middle frontal gyrus, right anterior cingulate cortex, left middle temporal gyrus, left precentral gyrus, and left orbitofrontal cortex. The map was unique to the Supernormals, differentiated this group from cognitive average-ager comparisons, and predicted a 1-year change in global cognition (indexed by the Montreal Cognitive Assessment scores, adjusted R2 = 0.68). The map was also correlated to Alzheimer's pathophysiological features (beta-amyloid/pTau ratio, adjusted R2 = 0.66) in participants with and without cognitive impairment. These findings in phenotypically successful cognitive agers suggest a divergent pattern of brain regions that may either reflect inherent neural integrity that contributes to Supernormals' cognitive success, or alternatively indicate adaptive reorganization to the demands of aging.

Multiple Regions of a Cortical Network Commonly Encode the Meaning of Words in Multiple Grammatical Positions of Read Sentences.

Deciphering how sentence meaning is represented in the brain remains a major challenge to science. Semantically related neural activity has recently been shown to arise concurrently in distributed brain regions as successive words in a sentence are read. However, what semantic content is represented by different regions, what is common across them, and how this relates to words in different grammatical positions of sentences is weakly understood. To address these questions, we apply a semantic model of word meaning to interpret brain activation patterns elicited in sentence reading. The model is based on human ratings of 65 sensory/motor/emotional and cognitive features of experience with words (and their referents). Through a process of mapping functional Magnetic Resonance Imaging activation back into model space we test: which brain regions semantically encode content words in different grammatical positions (e.g., subject/verb/object); and what semantic features are encoded by different regions. In left temporal, inferior parietal, and inferior/superior frontal regions we detect the semantic encoding of words in all grammatical positions tested and reveal multiple common components of semantic representation. This suggests that sentence comprehension involves a common core representation of multiple words' meaning being encoded in a network of regions distributed across the brain.

Predicting Neural Activity Patterns Associated with Sentences Using a Neurobiologically Motivated Model of Semantic Representation.

We introduce an approach that predicts neural representations of word meanings contained in sentences then superposes these to predict neural representations of new sentences. A neurobiological semantic model based on sensory, motor, social, emotional, and cognitive attributes was used as a foundation to define semantic content. Previous studies have predominantly predicted neural patterns for isolated words, using models that lack neurobiological interpretation. Fourteen participants read 240 sentences describing everyday situations while undergoing fMRI. To connect sentence-level fMRI activation patterns to the word-level semantic model, we devised methods to decompose the fMRI data into individual words. Activation patterns associated with each attribute in the model were then estimated using multiple-regression. This enabled synthesis of activation patterns for trained and new words, which were subsequently averaged to predict new sentences. Region-of-interest analyses revealed that prediction accuracy was highest using voxels in the left temporal and inferior parietal cortex, although a broad range of regions returned statistically significant results, showing that semantic information is widely distributed across the brain. The results show how a neurobiologically motivated semantic model can decompose sentence-level fMRI data into activation features for component words, which can be recombined to predict activation patterns for new sentences.

Semantic Structural Alignment of Neural Representational Spaces Enables Translation between English and Chinese Words.

Two sets of items can share the same underlying conceptual structure, while appearing unrelated at a surface level. Humans excel at recognizing and using alignments between such underlying structures in many domains of cognition, most notably in analogical reasoning. Here we show that structural alignment reveals how different people's neural representations of word meaning are preserved across different languages, such that patterns of brain activation can be used to translate words from one language to another. Groups of Chinese and English speakers underwent fMRI scanning while reading words in their respective native languages. Simply by aligning structures representing the two groups' neural semantic spaces, we successfully infer all seven Chinese-English word translations. Beyond language translation, conceptual structural alignment underlies many aspects of high-level cognition, and this work opens the door to deriving many such alignments directly from neural representational content.

Representational similarity encoding for fMRI: Pattern-based synthesis to predict brain activity using stimulus-model-similarities.

Patterns of neural activity are systematically elicited as the brain experiences categorical stimuli and a major challenge is to understand what these patterns represent. Two influential approaches, hitherto treated as separate analyses, have targeted this problem by using model-representations of stimuli to interpret the corresponding neural activity patterns. Stimulus-model-based-encoding synthesizes neural activity patterns by first training weights to map between stimulus-model features and voxels. This allows novel model-stimuli to be mapped into voxel space, and hence the strength of the model to be assessed by comparing predicted against observed neural activity. Representational Similarity Analysis (RSA) assesses models by testing how well the grand structure of pattern-similarities measured between all pairs of model-stimuli aligns with the same structure computed from neural activity patterns. RSA does not require model fitting, but also does not allow synthesis of neural activity patterns, thereby limiting its applicability. We introduce a new approach, representational similarity-encoding, that builds on the strengths of RSA and robustly enables stimulus-model-based neural encoding without model fitting. The approach therefore sidesteps problems associated with overfitting that notoriously confront any approach requiring parameter estimation (and is consequently low cost computationally), and importantly enables encoding analyses to be incorporated within the wider Representational Similarity Analysis framework. We illustrate this new approach by using it to synthesize and decode fMRI patterns representing the meanings of words, and discuss its potential biological relevance to encoding in semantic memory. Our new similarity-based encoding approach unites the two previously disparate methods of encoding models and RSA, capturing the strengths of both, and enabling similarity-based synthesis of predicted fMRI patterns.

Categorical speech processing in Broca's area: an fMRI study using multivariate pattern-based analysis.

Although much effort has been directed toward understanding the neural basis of speech processing, the neural processes involved in the categorical perception of speech have been relatively less studied, and many questions remain open. In this functional magnetic resonance imaging (fMRI) study, we probed the cortical regions mediating categorical speech perception using an advanced brain-mapping technique, whole-brain multivariate pattern-based analysis (MVPA). Normal healthy human subjects (native English speakers) were scanned while they listened to 10 consonant-vowel syllables along the /ba/-/da/ continuum. Outside of the scanner, individuals' own category boundaries were measured to divide the fMRI data into /ba/ and /da/ conditions per subject. The whole-brain MVPA revealed that Broca's area and the left pre-supplementary motor area evoked distinct neural activity patterns between the two perceptual categories (/ba/ vs /da/). Broca's area was also found when the same analysis was applied to another dataset (Raizada and Poldrack, 2007), which previously yielded the supramarginal gyrus using a univariate adaptation-fMRI paradigm. The consistent MVPA findings from two independent datasets strongly indicate that Broca's area participates in categorical speech perception, with a possible role of translating speech signals into articulatory codes. The difference in results between univariate and multivariate pattern-based analyses of the same data suggest that processes in different cortical areas along the dorsal speech perception stream are distributed on different spatial scales.

What makes different people's representations alike: neural similarity space solves the problem of across-subject fMRI decoding.

A central goal in neuroscience is to interpret neural activation and, moreover, to do so in a way that captures universal principles by generalizing across individuals. Recent research in multivoxel pattern-based fMRI analysis has led to considerable success at decoding within individual subjects. However, the goal of being able to decode across subjects is still challenging: It has remained unclear what population-level regularities of neural representation there might be. Here, we present a novel and highly accurate solution to this problem, which decodes across subjects between eight different stimulus conditions. The key to finding this solution was questioning the seemingly obvious idea that neural decoding should work directly on neural activation patterns. On the contrary, to decode across subjects, it is beneficial to abstract away from subject-specific patterns of neural activity and, instead, to operate on the similarity relations between those patterns: Our new approach performs decoding purely within similarity space. These results demonstrate a hitherto unknown population-level regularity in neural representation and also reveal a striking convergence between our empirical findings in fMRI and discussions in the philosophy of mind addressing the problem of conceptual similarity across neural diversity.

Analogy-Related Information Can Be Accessed by Simple Addition and Subtraction of fMRI Activation Patterns, Without Participants Performing any Analogy Task.

Analogical reasoning, for example, inferring that teacher is to chalk as mechanic is to wrench, plays a fundamental role in human cognition. However, whether brain activity patterns of individual words are encoded in a way that could facilitate analogical reasoning is unclear. Recent advances in computational linguistics have shown that information about analogical problems can be accessed by simple addition and subtraction of word embeddings (e.g., wrench = mechanic + chalk - teacher). Critically, this property emerges in artificial neural networks that were not trained to produce analogies but instead were trained to produce general-purpose semantic representations. Here, we test whether such emergent property can be observed in representations in human brains, as well as in artificial neural networks. fMRI activation patterns were recorded while participants viewed isolated words but did not perform analogical reasoning tasks. Analogy relations were constructed from word pairs that were categorically or thematically related, and we tested whether the predicted fMRI pattern calculated with simple arithmetic was more correlated with the pattern of the target word than other words. We observed that the predicted fMRI patterns contain information about not only the identity of the target word but also its category and theme (e.g., teaching-related). In summary, this study demonstrated that information about analogy questions can be reliably accessed with the addition and subtraction of fMRI patterns, and that, similar to word embeddings, this property holds for task-general patterns elicited when participants were not explicitly told to perform analogical reasoning.

Selective amplification of stimulus differences during categorical processing of speech.

The brain's perceptual stimuli are constantly changing: some of these changes are treated as invariances and are suppressed, whereas others are selectively amplified, giving emphasis to the distinctions that matter most. The starkest form of such amplification is categorical perception. In speech, for example, a continuum of phonetic stimuli gets carved into perceptually distinct categories. We used fMRI to measure the degree to which this process of selective amplification takes place. The most categorically processing area was the left supramarginal gyrus: stimuli from different phonetic categories, when presented together in a contrasting pair, were neurally amplified more than two-fold. Low-level auditory cortical areas, however, showed comparatively little amplification of changes that crossed category boundaries. Selective amplification serves to emphasize key stimulus differences, thereby shaping perceptual categories. The approach presented here provides a quantitative way to measure the degree to which such processing is taking place.

Quantifying the adequacy of neural representations for a cross-language phonetic discrimination task: prediction of individual differences.

In order for stimuli to be perceptually discriminable, their representations in the brain must be distinct. Investigating the task of discriminating the syllables /ra/ and /la/, we hypothesized that the more distinct a person's neural representations of those sounds were, the better their behavioral ability to discriminate them would be. Standard neuroimaging approaches are ill-suited to testing this hypothesis as they have problems differentiating between neural representations spatially intermingled within the same brain area. We therefore performed multi-voxel pattern-based analysis of the functional magnetic resonance imaging (fMRI) activity elicited by these syllables, in native speakers of English and Japanese. In right primary auditory cortex, the statistical separability of these fMRI patterns predicted subjects' behavioral ability to tell the sounds apart, not only across groups but also across individuals. This opens up a new approach for identifying neural representations and for quantifying their task suitability.

Linking brain-wide multivoxel activation patterns to behaviour: Examples from language and math.

A key goal of cognitive neuroscience is to find simple and direct connections between brain and behaviour. However, fMRI analysis typically involves choices between many possible options, with each choice potentially biasing any brain-behaviour correlations that emerge. Standard methods of fMRI analysis assess each voxel individually, but then face the problem of selection bias when combining those voxels into a region-of-interest, or ROI. Multivariate pattern-based fMRI analysis methods use classifiers to analyse multiple voxels together, but can also introduce selection bias via data-reduction steps as feature selection of voxels, pre-selecting activated regions, or principal components analysis. We show here that strong brain-behaviour links can be revealed without any voxel selection or data reduction, using just plain linear regression as a classifier applied to the whole brain at once, i.e. treating each entire brain volume as a single multi-voxel pattern. The brain-behaviour correlations emerged despite the fact that the classifier was not provided with any information at all about subjects' behaviour, but instead was given only the neural data and its condition-labels. Surprisingly, more powerful classifiers such as a linear SVM and regularised logistic regression produce very similar results. We discuss some possible reasons why the very simple brain-wide linear regression model is able to find correlations with behaviour that are as strong as those obtained on the one hand from a specific ROI and on the other hand from more complex classifiers. In a manner which is unencumbered by arbitrary choices, our approach offers a method for investigating connections between brain and behaviour which is simple, rigorous and direct.

Identify a shared neural circuit linking multiple neuropsychiatric symptoms with Alzheimer's pathology.

Neuropsychiatric symptoms (NPS) are common in Alzheimer's disease (AD)-associated neurodegeneration. However, NPS lack a consistent relationship with AD pathology. It is unknown whether any common neural circuits can link these clinically disparate while mechanistically similar features with AD pathology. Here, we explored the neural circuits of NPS in AD-associated neurodegeneration using multivariate pattern analysis (MVPA) of resting-state functional MRI data. Data from 98 subjects (70 amnestic mild cognitive impairment and 28 AD subjects) were obtained. The top 10 regions differentiating symptom presence across NPS were identified, which were mostly the fronto-limbic regions (medial prefrontal cortex, caudate, etc.). These 10 regions' functional connectivity classified symptomatic subjects across individual NPS at 69.46-81.27%, and predicted multiple NPS (indexed by Neuropsychiatric Symptom Questionnaire-Inventory) and AD pathology (indexed by baseline and change of beta-amyloid/pTau ratio) all above 70%. Our findings suggest a fronto-limbic dominated neural circuit that links multiple NPS and AD pathology. With further examination of the structural and pathological changes within the circuit, the circuit may shed light on linking behavioral disturbances with AD-associated neurodegeneration.

Decoding semantic representations from functional near-infrared spectroscopy signals.

This study uses representational similarity-based neural decoding to test whether semantic information elicited by words and pictures is encoded in functional near-infrared spectroscopy (fNIRS) data. In experiment 1, subjects passively viewed eight audiovisual word and picture stimuli for 15 min. Blood oxygen levels were measured using the Hitachi ETG-4000 fNIRS system with a posterior array over the occipital lobe and a left lateral array over the temporal lobe. Each participant's response patterns were abstracted to representational similarity space and compared to the group average (excluding that subject, i.e., leave-one-out cross-validation) and to a distributional model of semantic representation. Mean accuracy for both decoding tasks significantly exceeded chance. In experiment 2, we compared three group-level models by averaging the similarity structures from sets of eight participants in each group. In these models, the posterior array was accurately decoded by the semantic model, while the lateral array was accurately decoded in the between-groups comparison. Our findings indicate that semantic representations are encoded in the fNIRS data, preserved across subjects, and decodable by an extrinsic representational model. These results are the first attempt to link the functional response pattern measured by fNIRS to higher-level representations of how words are related to each other.

Decoding the infant mind: Multivariate pattern analysis (MVPA) using fNIRS.

The MRI environment restricts the types of populations and tasks that can be studied by cognitive neuroscientists (e.g., young infants, face-to-face communication). FNIRS is a neuroimaging modality that records the same physiological signal as fMRI but without the constraints of MRI, and with better spatial localization than EEG. However, research in the fNIRS community largely lacks the analytic sophistication of analogous fMRI work, restricting the application of this imaging technology. The current paper presents a method of multivariate pattern analysis for fNIRS that allows the authors to decode the infant mind (a key fNIRS population). Specifically, multivariate pattern analysis (MVPA) employs a correlation-based decoding method where a group model is constructed for all infants except one; both average patterns (i.e., infant-level) and single trial patterns (i.e., trial-level) of activation are decoded. Between subjects decoding is a particularly difficult task, because each infant has their own somewhat idiosyncratic patterns of neural activation. The fact that our method succeeds at across-subject decoding demonstrates the presence of group-level multi-channel regularities across infants. The code for implementing these analyses has been made readily available online to facilitate the quick adoption of this method to advance the methodological tools available to the fNIRS researcher.

Smoothness without smoothing: why Gaussian naive Bayes is not naive for multi-subject searchlight studies.

Spatial smoothness is helpful when averaging fMRI signals across multiple subjects, as it allows different subjects' corresponding brain areas to be pooled together even if they are slightly misaligned. However, smoothing is usually not applied when performing multivoxel pattern-based analyses (MVPA), as it runs the risk of blurring away the information that fine-grained spatial patterns contain. It would therefore be desirable, if possible, to carry out pattern-based analyses which take unsmoothed data as their input but which produce smooth images as output. We show here that the Gaussian Naive Bayes (GNB) classifier does precisely this, when it is used in "searchlight" pattern-based analyses. We explain why this occurs, and illustrate the effect in real fMRI data. Moreover, we show that analyses using GNBs produce results at the multi-subject level which are statistically robust, neurally plausible, and which replicate across two independent data sets. By contrast, SVM classifiers applied to the same data do not generate a replication, even if the SVM-derived searchlight maps have smoothing applied to them. An additional advantage of GNB classifiers for searchlight analyses is that they are orders of magnitude faster to compute than more complex alternatives such as SVMs. Collectively, these results suggest that Gaussian Naive Bayes classifiers may be a highly non-naive choice for multi-subject pattern-based fMRI studies.

Effects of socioeconomic status on brain development, and how cognitive neuroscience may contribute to levelling the playing field.

THE STUDY OF SOCIOECONOMIC STATUS (SES) AND THE BRAIN FINDS ITSELF IN A CIRCUMSTANCE UNUSUAL FOR COGNITIVE NEUROSCIENCE: large numbers of questions with both practical and scientific importance exist, but they are currently under-researched and ripe for investigation. This review aims to highlight these questions, to outline their potential significance, and to suggest routes by which they might be approached. Although remarkably few neural studies have been carried out so far, there exists a large literature of previous behavioural work. This behavioural research provides an invaluable guide for future neuroimaging work, but also poses an important challenge for it: how can we ensure that the neural data contributes predictive or diagnostic power over and above what can be derived from behaviour alone? We discuss some of the open mechanistic questions which Cognitive Neuroscience may have the power to illuminate, spanning areas including language, numerical cognition, stress, memory, and social influences on learning. These questions have obvious practical and societal significance, but they also bear directly on a set of longstanding questions in basic science: what are the environmental and neural factors which affect the acquisition and retention of declarative and nondeclarative skills? Perhaps the best opportunity for practical and theoretical interests to converge is in the study of interventions. Many interventions aimed at improving the cognitive development of low SES children are currently underway, but almost all are operating without either input from, or study by, the Cognitive Neuroscience community. Given that longitudinal intervention studies are very hard to set up, but can, with proper designs, be ideal tests of causal mechanisms, this area promises exciting opportunities for future research.

Socioeconomic status predicts hemispheric specialisation of the left inferior frontal gyrus in young children.

Reading is a complex skill that is not mastered by all children. At the age of 5, on the cusp of prereading development, many factors combine to influence a child's future reading success, including neural and behavioural factors such as phonological awareness and the auditory processing of phonetic input, and environmental factors, such as socioeconomic status (SES). We investigated the interactions between these factors in 5-year-old children by administering a battery of standardised cognitive and linguistic tests, measuring SES with a standardised scale, and using fMRI to record neural activity during a behavioral task, rhyming, that is predictive of reading skills. Correlation tests were performed, and then corrected for multiple comparisons using the false discovery rate (FDR) procedure. It emerged that only one relationship linking neural with behavioural or environmental factors survived as significant after FDR correction: a correlation between SES and the degree of hemispheric specialisation in the left inferior frontal gyrus (IFG), a region which includes Broca's area. This neural-environmental link remained significant even after controlling for the children's scores on the standardised language and cognition tests. In order to investigate possible environmental influences on the left IFG further, grey and white matter volumes were calculated. Marginally significant correlations with SES were found, indicating that environmental effects may manifest themselves in the brain anatomically as well as functionally. Collectively, these findings suggest that the weaker language skills of low-SES children are related to reduced underlying neural specialisation, and that these neural problems go beyond what is revealed by behavioural tests alone.

Challenge-driven attention: interacting frontal and brainstem systems.

The world is an unpredictable place, presenting challenges that fluctuate from moment to moment. However, the neural systems for responding to such challenges are far from fully understood. Using fMRI, we studied an audiovisual task in which the trials' difficulty and onset times varied unpredictably. Two regions were found to increase their activation for challenging trials, with their activities strongly correlated: right frontal cortex and the brainstem. The frontal area matched regions found in previous human studies of cognitive control, and activated in a graded manner with increasing task difficulty. The brainstem responded only to the most difficult trials, showing a phasic activity pattern paralleling locus coeruleus recordings in monkeys. These results reveal a bridge between animal and human studies, and suggest interacting roles for the brainstem and right frontal cortex: the brainstem may signal that an attentional challenge is occurring, while right frontal cortex allocates cognitive resources in response.