Dimensionality and Ramping: Signatures of Sentence Integration in the Dynamics of Brains and Deep Language Models.

A sentence is more than the sum of its words: its meaning depends on how they combine with one another. The brain mechanisms underlying such semantic composition remain poorly understood. To shed light on the neural vector code underlying semantic composition, we introduce two hypotheses: (1) the intrinsic dimensionality of the space of neural representations should increase as a sentence unfolds, paralleling the growing complexity of its semantic representation; and (2) this progressive integration should be reflected in ramping and sentence-final signals. To test these predictions, we designed a dataset of closely matched normal and jabberwocky sentences (composed of meaningless pseudo words) and displayed them to deep language models and to 11 human participants (5 men and 6 women) monitored with simultaneous MEG and intracranial EEG. In both deep language models and electrophysiological data, we found that representational dimensionality was higher for meaningful sentences than jabberwocky. Furthermore, multivariate decoding of normal versus jabberwocky confirmed three dynamic patterns: (1) a phasic pattern following each word, peaking in temporal and parietal areas; (2) a ramping pattern, characteristic of bilateral inferior and middle frontal gyri; and (3) a sentence-final pattern in left superior frontal gyrus and right orbitofrontal cortex. These results provide a first glimpse into the neural geometry of semantic integration and constrain the search for a neural code of linguistic composition.SIGNIFICANCE STATEMENT Starting from general linguistic concepts, we make two sets of predictions in neural signals evoked by reading multiword sentences. First, the intrinsic dimensionality of the representation should grow with additional meaningful words. Second, the neural dynamics should exhibit signatures of encoding, maintaining, and resolving semantic composition. We successfully validated these hypotheses in deep neural language models, artificial neural networks trained on text and performing very well on many natural language processing tasks. Then, using a unique combination of MEG and intracranial electrodes, we recorded high-resolution brain data from human participants while they read a controlled set of sentences. Time-resolved dimensionality analysis showed increasing dimensionality with meaning, and multivariate decoding allowed us to isolate the three dynamical patterns we had hypothesized.

Top-down activation of the visuo-orthographic system during spoken sentence processing.

The left ventral occipitotemporal cortex (vOT) is considered the key area of the visuo-orthographic system. However, some studies reported that the area is also involved in speech processing tasks, especially those that require activation of orthographic knowledge. These findings suggest the existence of a top-down activation mechanism allowing such cross-modal activation. Yet, little is known about the involvement of the vOT in more natural speech processing situations like spoken sentence processing. Here, we addressed this issue in a functional Magnetic Resonance Imaging (fMRI) study while manipulating the impacts of two factors, i.e., task demands (semantic vs. low-level perceptual task) and the quality of speech signals (sentences presented against clear vs. noisy background). Analyses were performed at the levels of whole brain and region-of-interest (ROI) focusing on the vOT voxels individually identified through a reading task. Whole brain analysis showed that processing spoken sentences induced activity in a large network including the regions typically involved in phonological, articulatory, semantic and orthographic processing. ROI analysis further specified that a significant part of the vOT voxels that responded to written words also responded to spoken sentences, thus, suggesting that the same area within the left occipitotemporal pathway contributes to both reading and speech processing. Interestingly, both analyses provided converging evidence that vOT responses to speech were sensitive to both task demands and quality of speech signals: Compared to the low-level perceptual task, activity of the area increased when efforts on comprehension were required. The impact of background noise depended on task demands. It led to a decrease of vOT activity in the semantic task but not in the low-level perceptual task. Our results provide new insights into the function of this key area of the reading network, notably by showing that its speech-induced top-down activation also generalizes to ecological speech processing situations.

Automaticity of phonological and semantic processing during visual word recognition.

Reading involves activation of phonological and semantic knowledge. Yet, the automaticity of the activation of these representations remains subject to debate. The present study addressed this issue by examining how different brain areas involved in language processing responded to a manipulation of bottom-up (level of visibility) and top-down information (task demands) applied to written words. The analyses showed that the same brain areas were activated in response to written words whether the task was symbol detection, rime detection, or semantic judgment. This network included posterior, temporal and prefrontal regions, which clearly suggests the involvement of orthographic, semantic and phonological/articulatory processing in all tasks. However, we also found interactions between task and stimulus visibility, which reflected the fact that the strength of the neural responses to written words in several high-level language areas varied across tasks. Together, our findings suggest that the involvement of phonological and semantic processing in reading is supported by two complementary mechanisms. First, an automatic mechanism that results from a task-independent spread of activation throughout a network in which orthography is linked to phonology and semantics. Second, a mechanism that further fine-tunes the sensitivity of high-level language areas to the sensory input in a task-dependent manner.

Spatiotemporal Dynamics of Morphological Processing in Visual Word Recognition.

The spatiotemporal dynamics of morphological, orthographic, and semantic processing were investigated in a primed lexical decision task in French using magnetoencephalography (MEG). The goal was to investigate orthographic and semantic contributions to morphological priming and compare these effects with pure orthographic and semantic priming. The time course of these effects was analyzed in anatomically defined ROIs that were selected according to previous MEG and fMRI findings. The results showed that morphological processing was not localized in one specific area but distributed over a vast network that involved left inferior temporal gyrus, left superior temporal gyrus, left inferior frontal gyrus, and left orbitofrontal gyrus. Second, all morphological effects were specific, that is, in none of the ROIs could morphology effects be explained by pure orthographic or pure semantic overlap. Third, the ventral route was sensitive to both the orthographic and semantic "part" of the morphological priming effect in the M350 time window. Fourth, the earliest effects of morphology occurred in left superior temporal gyrus around 250 msec and reflected the semantic contribution to morphological facilitation. Together then, the present results show that morphological processing is not just an emergent property of processing form or meaning and that semantic contributions to morphological facilitation can occur as early as 250 msec in the left superior temporal gyrus.

Multisite longitudinal reliability of tract-based spatial statistics in diffusion tensor imaging of healthy elderly subjects.

Large-scale longitudinal neuroimaging studies with diffusion imaging techniques are necessary to test and validate models of white matter neurophysiological processes that change in time, both in healthy and diseased brains. The predictive power of such longitudinal models will always be limited by the reproducibility of repeated measures acquired during different sessions. At present, there is limited quantitative knowledge about the across-session reproducibility of standard diffusion metrics in 3T multi-centric studies on subjects in stable conditions, in particular when using tract based spatial statistics and with elderly people. In this study we implemented a multi-site brain diffusion protocol in 10 clinical 3T MRI sites distributed across 4 countries in Europe (Italy, Germany, France and Greece) using vendor provided sequences from Siemens (Allegra, Trio Tim, Verio, Skyra, Biograph mMR), Philips (Achieva) and GE (HDxt) scanners. We acquired DTI data (2 × 2 × 2 mm(3), b = 700 s/mm(2), 5 b0 and 30 diffusion weighted volumes) of a group of healthy stable elderly subjects (5 subjects per site) in two separate sessions at least a week apart. For each subject and session four scalar diffusion metrics were considered: fractional anisotropy (FA), mean diffusivity (MD), radial diffusivity (RD) and axial (AD) diffusivity. The diffusion metrics from multiple subjects and sessions at each site were aligned to their common white matter skeleton using tract-based spatial statistics. The reproducibility at each MRI site was examined by looking at group averages of absolute changes relative to the mean (%) on various parameters: i) reproducibility of the signal-to-noise ratio (SNR) of the b0 images in centrum semiovale, ii) full brain test-retest differences of the diffusion metric maps on the white matter skeleton, iii) reproducibility of the diffusion metrics on atlas-based white matter ROIs on the white matter skeleton. Despite the differences of MRI scanner configurations across sites (vendors, models, RF coils and acquisition sequences) we found good and consistent test-retest reproducibility. White matter b0 SNR reproducibility was on average 7 ± 1% with no significant MRI site effects. Whole brain analysis resulted in no significant test-retest differences at any of the sites with any of the DTI metrics. The atlas-based ROI analysis showed that the mean reproducibility errors largely remained in the 2-4% range for FA and AD and 2-6% for MD and RD, averaged across ROIs. Our results show reproducibility values comparable to those reported in studies using a smaller number of MRI scanners, slightly different DTI protocols and mostly younger populations. We therefore show that the acquisition and analysis protocols used are appropriate for multi-site experimental scenarios.

Functional connectivity changes differ in early and late-onset Alzheimer's disease.

At a similar stage, patients with early onset Alzheimer's disease (EOAD) have greater neocortical but less medial temporal lobe dysfunction and atrophy than the late-onset form of the disease (LOAD). Whether the organization of neural networks also differs has never been investigated. This study aims at characterizing basal functional connectivity (FC) patterns of EOAD and LOAD in two groups of 14 patients matched for disease duration and severity, relative to age-matched controls. All subjects underwent an extensive neuropsychological assessment. Magnetic resonance imaging was used to quantify atrophy and resting-state FC focusing on : the default mode network (DMN), found impaired in earlier studies on AD, and the anterior temporal network (ATN) and dorso-lateral prefrontal network (DLPFN), respectively involved in declarative memory and executive functions. Patterns of atrophy and cognitive impairment in EOAD and LOAD were in accordance with previous reports. FC within the DMN was similarly decreased in both EOAD and LOAD relative to controls. However, a double-dissociated pattern of FC changes in ATN and DLPFN was found. EOAD exhibited decreased FC in the DLPFN and increased FC in the ATN relative to controls, while the reverse pattern was found in LOAD. In addition, ATN and DLPFN connectivity correlated respectively with memory and executive performances, suggesting that increased FC is here likely to reflect compensatory mechanisms. Thus, large-scale neural network changes in EOAD and LOAD endorse both common features and differences, probably related to a distinct distribution of pathological changes.

Cortico-Cerebellar Monitoring of Speech Sequence Production.

In a functional magnetic resonance imaging study, we examined speech error monitoring in a cortico-cerebellar network for two contrasts: (a) correct trials with high versus low articulatory error probability and (b) overtly committed errors versus correct trials. Engagement of the cognitive cerebellar region Crus I in both contrasts suggests that this region is involved in overarching performance monitoring. The activation of cerebellar motor regions (superior medial cerebellum, lobules VI and VIII) indicates the additional presence of a sensorimotor driven implementation of control. The combined pattern of pre-supplementary motor area (active across contrasts) and anterior cingulate cortex (only active in the contrast involving overt errors) activations suggests sensorimotor driven feedback monitoring in the medial frontal cortex, making use of proprioception and auditory feedback through overt errors. Differential temporal and parietal cortex activation across contrasts indicates involvement beyond sensorimotor driven feedback in line with speech production models that link these regions to auditory target processing and internal modeling-like mechanisms. These results highlight the presence of multiple, possibly hierarchically interdependent, mechanisms that support the optimizing of speech production.

Brain morphometry reproducibility in multi-center 3T MRI studies: a comparison of cross-sectional and longitudinal segmentations.

Large-scale longitudinal multi-site MRI brain morphometry studies are becoming increasingly crucial to characterize both normal and clinical population groups using fully automated segmentation tools. The test-retest reproducibility of morphometry data acquired across multiple scanning sessions, and for different MR vendors, is an important reliability indicator since it defines the sensitivity of a protocol to detect longitudinal effects in a consortium. There is very limited knowledge about how across-session reliability of morphometry estimates might be affected by different 3T MRI systems. Moreover, there is a need for optimal acquisition and analysis protocols in order to reduce sample sizes. A recent study has shown that the longitudinal FreeSurfer segmentation offers improved within session test-retest reproducibility relative to the cross-sectional segmentation at one 3T site using a nonstandard multi-echo MPRAGE sequence. In this study we implement a multi-site 3T MRI morphometry protocol based on vendor provided T1 structural sequences from different vendors (3D MPRAGE on Siemens and Philips, 3D IR-SPGR on GE) implemented in 8 sites located in 4 European countries. The protocols used mild acceleration factors (1.5-2) when possible. We acquired across-session test-retest structural data of a group of healthy elderly subjects (5 subjects per site) and compared the across-session reproducibility of two full-brain automated segmentation methods based on either longitudinal or cross-sectional FreeSurfer processing. The segmentations include cortical thickness, intracranial, ventricle and subcortical volumes. Reproducibility is evaluated as absolute changes relative to the mean (%), Dice coefficient for volume overlap and intraclass correlation coefficients across two sessions. We found that this acquisition and analysis protocol gives comparable reproducibility results to previous studies that used longer acquisitions without acceleration. We also show that the longitudinal processing is systematically more reliable across sites regardless of MRI system differences. The reproducibility errors of the longitudinal segmentations are on average approximately half of those obtained with the cross sectional analysis for all volume segmentations and for entorhinal cortical thickness. No significant differences in reliability are found between the segmentation methods for the other cortical thickness estimates. The average of two MPRAGE volumes acquired within each test-retest session did not systematically improve the across-session reproducibility of morphometry estimates. Our results extend those from previous studies that showed improved reliability of the longitudinal analysis at single sites and/or with non-standard acquisition methods. The multi-site acquisition and analysis protocol presented here is promising for clinical applications since it allows for smaller sample sizes per MRI site or shorter trials in studies evaluating the role of potential biomarkers to predict disease progression or treatment effects.

Is there evidence for a noisy computation deficit in developmental dyslexia?

The noisy computation hypothesis of developmental dyslexia (DD) is particularly appealing because it can explain deficits across a variety of domains, such as temporal, auditory, phonological, visual and attentional processes. A key prediction is that noisy computations lead to more variable and less stable word representations. A way to test this hypothesis is through repetition of words, that is, when there is noise in the system, the neural signature of repeated stimuli should be more variable. The hypothesis was tested in an functional magnetic resonance imaging experiment with dyslexic and typical readers by repeating words twelve times. Variability measures were computed both at the behavioral and neural levels. At the behavioral level, we compared the standard deviation of reaction time distributions of repeated words. At the neural level, in addition to standard univariate analyses and measures of intra-item variability, we also used multivariate pattern analyses (representational similarity and classification) to find out whether there was evidence for noisier representations in dyslexic readers compared to typical readers. Results showed that there were no significant differences between the two groups in any of the analyses despite robust results within each group (i.e., high representational similarity between repeated words, good classification of words vs. non-words). In summary, there was no evidence in favor of the idea that dyslexic readers would have noisier neural representations than typical readers.

Graph theoretical analysis reveals the functional role of the left ventral occipito-temporal cortex in speech processing.

The left ventral occipito-temporal cortex (left-vOT) plays a key role in reading. Interestingly, the area also responds to speech input, suggesting that it may have other functions beyond written word recognition. Here, we adopt graph theoretical analysis to investigate the left-vOT's functional role in the whole-brain network while participants process spoken sentences in different contexts. Overall, different connectivity measures indicate that the left-vOT acts as an interface enabling the communication between distributed brain regions and sub-networks. During simple speech perception, the left-vOT is systematically part of the visual network and contributes to the communication between neighboring areas, remote areas, and sub-networks, by acting as a local bridge, a global bridge, and a connector, respectively. However, when speech comprehension is explicitly required, the specific functional role of the area and the sub-network to which the left-vOT belongs change and vary with the quality of speech signal and task difficulty. These connectivity patterns provide insightful information on the contribution of the left-vOT in various contexts of language processing beyond its role in reading. They advance our general understanding of the neural mechanisms underlying the flexibility of the language network that adjusts itself according to the processing context.

Cerebellar and Cortical Correlates of Internal and External Speech Error Monitoring.

An event-related functional magnetic resonance imaging study examined how speakers inspect their own speech for errors. Concretely, we sought to assess 1) the role of the temporal cortex in monitoring speech errors, linked with comprehension-based monitoring; 2) the involvement of the cerebellum in internal and external monitoring, linked with forward modeling; and 3) the role of the medial frontal cortex for internal monitoring, linked with conflict-based monitoring. In a word production task priming speech errors, we observed enhanced involvement of the right posterior cerebellum for trials that were correct, but on which participants were more likely to make a word as compared with a nonword error (contrast of internal monitoring). Furthermore, comparing errors to correct utterances (contrast of external monitoring), we observed increased activation of the same cerebellar region, of the superior medial cerebellum, and of regions in temporal and medial frontal cortex. The presence of the cerebellum for both internal and external monitoring indicates the use of forward modeling across the planning and articulation of speech. Dissociations across internal and external monitoring in temporal and medial frontal cortex indicate that monitoring of overt errors is more reliant on vocal feedback control.

Frequency-tagged visual evoked responses track syllable effects in visual word recognition.

The processing of syllables in visual word recognition was investigated using a novel paradigm based on steady-state visual evoked potentials (SSVEPs). French words were presented to proficient readers in a delayed naming task. Words were split into two segments, the first of which was flickered at 18.75 Hz and the second at 25 Hz. The first segment either matched (congruent condition) or did not match (incongruent condition) the first syllable. The SSVEP responses in the congruent condition showed increased power compared to the responses in the incongruent condition, providing new evidence that syllables are important sublexical units in visual word recognition and reading aloud. With respect to the neural correlates of the effect, syllables elicited an early activation of a right hemisphere network. This network is typically associated with the programming of complex motor sequences, cognitive control and timing. Subsequently, responses were obtained in left hemisphere areas related to phonological processing.

Steady state visual evoked potentials in reading aloud: Effects of lexicality, frequency and orthographic familiarity.

The present study explored the possibility to use Steady-State Visual Evoked Potentials (SSVEPs) as a tool to investigate the core mechanisms in visual word recognition. In particular, we investigated three benchmark effects of reading aloud: lexicality (words vs. pseudowords), frequency (high-frequency vs. low-frequency words), and orthographic familiarity ('familiar' versus 'unfamiliar' pseudowords). We found that words and pseudowords elicited robust SSVEPs. Words showed larger SSVEPs than pseudowords and high-frequency words showed larger SSVEPs than low-frequency words. SSVEPs were not sensitive to orthographic familiarity. We further localized the neural generators of the SSVEP effects. The lexicality effect was located in areas associated with early level of visual processing, i.e. in the right occipital lobe and in the right precuneus. Pseudowords produced more activation than words in left sensorimotor areas, rolandic operculum, insula, supramarginal gyrus and in the right temporal gyrus. These areas are devoted to speech processing and/or spelling-to-sound conversion. The frequency effect involved the left temporal pole and orbitofrontal cortex, areas previously implicated in semantic processing and stimulus-response associations respectively, and the right postcentral and parietal inferior gyri, possibly indicating the involvement of the right attentional network.

Grammatical class modulates the (left) inferior frontal gyrus within 100 milliseconds when syntactic context is predictive.

The current study set out to examine the spatiotemporal dynamics of predictive processing during syntactic processing. To do so, we conducted an MEG experiment in which we contrasted MRI-constrained sources elicited by nouns and verbs when they were preceded by a predictive syntactic context (i.e., possessive pronouns for nouns, and personal pronouns for verbs) versus a non-predictive syntactic context (visually matched symbols). The results showed rapid (from ~80 ms onwards) noun-verb differences in the left and (to a lesser extent) right inferior frontal gyri (IFG), but only when those nouns and verbs were preceded by the syntactically predictive context (i.e. their corresponding pronoun). Furthermore, the contrast between possessive and personal pronouns that preceded the rapid noun-verb modulations in the (L)IFG also produced differences in source activation in various regions of the prefrontal cortex (the superior frontal and orbitofrontal cortex). We suggest the data show that syntactic unification manifests very early on during processing in the LIFG. The speed of such syntactic unification operations is hypothesized to be driven by predictive top-down activations stemming from a domain-general network in the prefrontal cortex.

Spatiotemporal reorganization of the reading network in adult dyslexia.

Developmental dyslexia is characterized by impairments in reading fluency and spelling that persist into adulthood. Here, we hypothesized that high-achieving adult dyslexics (i.e., university students with a history of dyslexia) manage to cope with these deficits by relying to a greater extent on morphological information than do non-impaired adult readers. We used magnetoencephalography (MEG) in a primed lexical decision task, in which we contrasted orthographic, morphological and semantic processing. Behavioral results confirmed that adult dyslexics did indeed rely to a greater extent on the semantic properties of morphemes than controls. In line with this, MEG results showed early morphological effects (100-200 msec) in a frontal network, which reflected the contribution of semantic processing. The same effects occurred much later in controls (∼400 msec). In contrast, controls showed early orthographic priming effects in posterior left inferior temporal gyrus (LITG) at around 130 msec, which were not seen in dyslexics. In the LITG, dyslexics showed only late activation of semantic and orthographic information. The present results suggest a spatiotemporal reorganization of the reading network, in which morphological information located in frontal regions is activated earlier in high-achieving adults dyslexics than controls.