Negation mitigates rather than inverts the neural representations of adjectives.

Combinatoric linguistic operations underpin human language processes, but how meaning is composed and refined in the mind of the reader is not well understood. We address this puzzle by exploiting the ubiquitous function of negation. We track the online effects of negation ("not") and intensifiers ("really") on the representation of scalar adjectives (e.g., "good") in parametrically designed behavioral and neurophysiological (MEG) experiments. The behavioral data show that participants first interpret negated adjectives as affirmative and later modify their interpretation towards, but never exactly as, the opposite meaning. Decoding analyses of neural activity further reveal significant above chance decoding accuracy for negated adjectives within 600 ms from adjective onset, suggesting that negation does not invert the representation of adjectives (i.e., "not bad" represented as "good"); furthermore, decoding accuracy for negated adjectives is found to be significantly lower than that for affirmative adjectives. Overall, these results suggest that negation mitigates rather than inverts the neural representations of adjectives. This putative suppression mechanism of negation is supported by increased synchronization of beta-band neural activity in sensorimotor areas. The analysis of negation provides a steppingstone to understand how the human brain represents changes of meaning over time.

Dimensionality and Ramping: Signatures of Sentence Integration in the Dynamics of Brains and Deep Language Models.

A sentence is more than the sum of its words: its meaning depends on how they combine with one another. The brain mechanisms underlying such semantic composition remain poorly understood. To shed light on the neural vector code underlying semantic composition, we introduce two hypotheses: (1) the intrinsic dimensionality of the space of neural representations should increase as a sentence unfolds, paralleling the growing complexity of its semantic representation; and (2) this progressive integration should be reflected in ramping and sentence-final signals. To test these predictions, we designed a dataset of closely matched normal and jabberwocky sentences (composed of meaningless pseudo words) and displayed them to deep language models and to 11 human participants (5 men and 6 women) monitored with simultaneous MEG and intracranial EEG. In both deep language models and electrophysiological data, we found that representational dimensionality was higher for meaningful sentences than jabberwocky. Furthermore, multivariate decoding of normal versus jabberwocky confirmed three dynamic patterns: (1) a phasic pattern following each word, peaking in temporal and parietal areas; (2) a ramping pattern, characteristic of bilateral inferior and middle frontal gyri; and (3) a sentence-final pattern in left superior frontal gyrus and right orbitofrontal cortex. These results provide a first glimpse into the neural geometry of semantic integration and constrain the search for a neural code of linguistic composition.SIGNIFICANCE STATEMENT Starting from general linguistic concepts, we make two sets of predictions in neural signals evoked by reading multiword sentences. First, the intrinsic dimensionality of the representation should grow with additional meaningful words. Second, the neural dynamics should exhibit signatures of encoding, maintaining, and resolving semantic composition. We successfully validated these hypotheses in deep neural language models, artificial neural networks trained on text and performing very well on many natural language processing tasks. Then, using a unique combination of MEG and intracranial electrodes, we recorded high-resolution brain data from human participants while they read a controlled set of sentences. Time-resolved dimensionality analysis showed increasing dimensionality with meaning, and multivariate decoding allowed us to isolate the three dynamical patterns we had hypothesized.

The Human Brain Encodes a Chronicle of Visual Events at Each Instant of Time Through the Multiplexing of Traveling Waves.

The human brain continuously processes streams of visual input. Yet, a single image typically triggers neural responses that extend beyond 1s. To understand how the brain encodes and maintains successive images, we analyzed with electroencephalography the brain activity of human subjects while they watched ∼5000 visual stimuli presented in fast sequences. First, we confirm that each stimulus can be decoded from brain activity for ∼1s, and we demonstrate that the brain simultaneously represents multiple images at each time instant. Second, we source localize the corresponding brain responses in the expected visual hierarchy and show that distinct brain regions represent, at each time instant, different snapshots of past stimulations. Third, we propose a simple framework to further characterize the dynamical system of these traveling waves. Our results show that a chain of neural circuits, which each consist of (1) a hidden maintenance mechanism and (2) an observable update mechanism, accounts for the dynamics of macroscopic brain representations elicited by visual sequences. Together, these results detail a simple architecture explaining how successive visual events and their respective timings can be simultaneously represented in the brain.SIGNIFICANCE STATEMENT Our retinas are continuously bombarded with a rich flux of visual input. Yet, how our brain continuously processes such visual streams is a major challenge to neuroscience. Here, we developed techniques to decode and track, from human brain activity, multiple images flashed in rapid succession. Our results show that the brain simultaneously represents multiple successive images at each time instant by multiplexing them along a neural cascade. Dynamical modeling shows that these results can be explained by a hierarchy of neural assemblies that continuously propagate multiple visual contents. Overall, this study sheds new light on the biological basis of our visual experience.

Ongoing neural oscillations influence behavior and sensory representations by suppressing neuronal excitability.

The ability to process and respond to external input is critical for adaptive behavior. Why, then, do neural and behavioral responses vary across repeated presentations of the same sensory input? Ongoing fluctuations of neuronal excitability are currently hypothesized to underlie the trial-by-trial variability in sensory processing. To test this, we capitalized on intracranial electrophysiology in neurosurgical patients performing an auditory discrimination task with visual cues: specifically, we examined the interaction between prestimulus alpha oscillations, excitability, task performance, and decoded neural stimulus representations. We found that strong prestimulus oscillations in the alpha+ band (i.e., alpha and neighboring frequencies), rather than the aperiodic signal, correlated with a low excitability state, indexed by reduced broadband high-frequency activity. This state was related to slower reaction times and reduced neural stimulus encoding strength. We propose that the alpha+ rhythm modulates excitability, thereby resulting in variability in behavior and sensory representations despite identical input.

Detection of Brain Activation in Unresponsive Patients with Acute Brain Injury.

BACKGROUND: Brain activation in response to spoken motor commands can be detected by electroencephalography (EEG) in clinically unresponsive patients. The prevalence and prognostic importance of a dissociation between commanded motor behavior and brain activation in the first few days after brain injury are not well understood. METHODS: We studied a prospective, consecutive series of patients in a single intensive care unit who had acute brain injury from a variety of causes and who were unresponsive to spoken commands, including some patients with the ability to localize painful stimuli or to fixate on or track visual stimuli. Machine learning was applied to EEG recordings to detect brain activation in response to commands that patients move their hands. The functional outcome at 12 months was determined with the Glasgow Outcome Scale-Extended (GOS-E; levels range from 1 to 8, with higher levels indicating better outcomes). RESULTS: A total of 16 of 104 unresponsive patients (15%) had brain activation detected by EEG at a median of 4 days after injury. The condition in 8 of these 16 patients (50%) and in 23 of 88 patients (26%) without brain activation improved such that they were able to follow commands before discharge. At 12 months, 7 of 16 patients (44%) with brain activation and 12 of 84 patients (14%) without brain activation had a GOS-E level of 4 or higher, denoting the ability to function independently for 8 hours (odds ratio, 4.6; 95% confidence interval, 1.2 to 17.1). CONCLUSIONS: A dissociation between the absence of behavioral responses to motor commands and the evidence of brain activation in response to these commands in EEG recordings was found in 15% of patients in a consecutive series of patients with acute brain injury. (Supported by the Dana Foundation and the James S. McDonnell Foundation.).

Differential Brain Mechanisms of Selection and Maintenance of Information during Working Memory.

Working memory is our ability to select and temporarily hold information as needed for complex cognitive operations. The temporal dynamics of sustained and transient neural activity supporting the selection and holding of memory content is not known. To address this problem, we recorded magnetoencephalography in healthy participants performing a retro-cue working memory task in which the selection rule and the memory content varied independently. Multivariate decoding and source analyses showed that selecting the memory content relies on prefrontal and parieto-occipital persistent oscillatory neural activity. By contrast, the memory content was reactivated in a distributed occipitotemporal posterior network, preceding the working memory decision and in a different format than during the visual encoding. These results identify a neural signature of content selection and characterize differentiated spatiotemporal constraints for subprocesses of working memory.SIGNIFICANCE STATEMENT Our brain selects and maintains information during short time windows in a way that is essential to reasoning and learning. Recent advances in multivariate analysis of brain activity allowed the characterization of brain regions that stores the memory. We applied multivariate analysis to time-resolved brain signals to characterize the spatiotemporal signature underlying these subprocesses. The selection of information relies on sustained oscillatory activity in a network that includes the ventrolateral prefrontal cortex while memory content is transiently replayed in an occipitotemporal network that differs from encoding. Our results characterized differentiated spatiotemporal activity underlying encoding, selection, and maintenance of information during working memory.

Back-to-back regression: Disentangling the influence of correlated factors from multivariate observations.

Identifying causes solely from observations can be particularly challenging when i) the factors under investigation are difficult to manipulate independently from one-another and ii) observations are high-dimensional. To address this issue, we introduce ''Back-to-Back'' regression (B2B), a linear method designed to efficiently estimate, from a set of correlated factors, those that most plausibly account for multidimensional observations. First, we prove the consistency of B2B, its links to other linear approaches, and show how it can provide a robust, unbiased and interpretable scalar estimate for each factor. Second, we use a variety of simulated data to show that B2B can outperform forward modeling ("encoding"), backward modeling ("decoding") as well as cross-decomposition modeling (i.e. canonical correlation analysis and partial least squares) on causal identification when the factors and the observations are not orthogonal. Finally, we apply B2B to a hundred magneto-encephalography recordings and to a hundred functional Magnetic Resonance Imaging recordings acquired while subjects performed a 1 â€‹h reading task. B2B successfully disentangles the respective contribution of collinear factors such as word length, word frequency in the early visual and late associative cortical responses respectively. B2B compared favorably to other standard techniques on this disentanglement. We discuss how the speed and the generality of B2B sets promising foundations to help identify the causal contributions of covarying factors from high-dimensional observations.

What Limits Our Capacity to Process Nested Long-Range Dependencies in Sentence Comprehension?

Sentence comprehension requires inferring, from a sequence of words, the structure of syntactic relationships that bind these words into a semantic representation. Our limited ability to build some specific syntactic structures, such as nested center-embedded clauses (e.g., "The dog that the cat that the mouse bit chased ran away"), suggests a striking capacity limitation of sentence processing, and thus offers a window to understand how the human brain processes sentences. Here, we review the main hypotheses proposed in psycholinguistics to explain such capacity limitation. We then introduce an alternative approach, derived from our recent work on artificial neural networks optimized for language modeling, and predict that capacity limitation derives from the emergence of sparse and feature-specific syntactic units. Unlike psycholinguistic theories, our neural network-based framework provides precise capacity-limit predictions without making any a priori assumptions about the form of the grammar or parser. Finally, we discuss how our framework may clarify the mechanistic underpinning of language processing and its limitations in the human brain.

Robust EEG-based cross-site and cross-protocol classification of states of consciousness.

Determining the state of consciousness in patients with disorders of consciousness is a challenging practical and theoretical problem. Recent findings suggest that multiple markers of brain activity extracted from the EEG may index the state of consciousness in the human brain. Furthermore, machine learning has been found to optimize their capacity to discriminate different states of consciousness in clinical practice. However, it is unknown how dependable these EEG markers are in the face of signal variability because of different EEG configurations, EEG protocols and subpopulations from different centres encountered in practice. In this study we analysed 327 recordings of patients with disorders of consciousness (148 unresponsive wakefulness syndrome and 179 minimally conscious state) and 66 healthy controls obtained in two independent research centres (Paris Pitié-Salpêtrière and Liège). We first show that a non-parametric classifier based on ensembles of decision trees provides robust out-of-sample performance on unseen data with a predictive area under the curve (AUC) of ~0.77 that was only marginally affected when using alternative EEG configurations (different numbers and positions of sensors, numbers of epochs, average AUC = 0.750 ± 0.014). In a second step, we observed that classifiers based on multiple as well as single EEG features generalize to recordings obtained from different patient cohorts, EEG protocols and different centres. However, the multivariate model always performed best with a predictive AUC of 0.73 for generalization from Paris 1 to Paris 2 datasets, and an AUC of 0.78 from Paris to Liège datasets. Using simulations, we subsequently demonstrate that multivariate pattern classification has a decisive performance advantage over univariate classification as the stability of EEG features decreases, as different EEG configurations are used for feature-extraction or as noise is added. Moreover, we show that the generalization performance from Paris to Liège remains stable even if up to 20% of the diagnostic labels are randomly flipped. Finally, consistent with recent literature, analysis of the learned decision rules of our classifier suggested that markers related to dynamic fluctuations in theta and alpha frequency bands carried independent information and were most influential. Our findings demonstrate that EEG markers of consciousness can be reliably, economically and automatically identified with machine learning in various clinical and acquisition contexts.

Electrophysiological Responses in the Ventral Temporal Cortex During Reading of Numerals and Calculation.

Recent evidence suggests that specific neuronal populations in the ventral temporal cortex show larger electrophysiological responses to visual numerals compared with morphologically similar stimuli. This study investigates how these responses change from simple reading of numerals to the active use of numerals in an arithmetic context. We recorded high-frequency broadband (HFB) signals, a reliable measure for local neuronal population activity, while 10 epilepsy patients implanted with subdural electrodes performed separate numeral reading and calculation tasks. We found that calculation increased activity in the posterior inferior temporal gyrus (ITG) with a factor of approximately 1.5 over the first 500 ms of calculation, whereas no such increase was noted for reading numerals without calculation or reading and judging memory statements. In a second experiment conducted in 2 of the same subjects, we show that HFB responses increase in a systematic manner when the single numerals were presented successively in a calculation context: The HFB response in the ITG, to the second and third numerals (i.e., b and c in a + b = c), was approximately 1.5 times larger than the responses to the first numeral (a). These results provide electrophysiological evidence for modulation of local neuronal population responses to visual stimuli based on increasing task demands.

Disruption of hierarchical predictive coding during sleep.

When presented with an auditory sequence, the brain acts as a predictive-coding device that extracts regularities in the transition probabilities between sounds and detects unexpected deviations from these regularities. Does such prediction require conscious vigilance, or does it continue to unfold automatically in the sleeping brain? The mismatch negativity and P300 components of the auditory event-related potential, reflecting two steps of auditory novelty detection, have been inconsistently observed in the various sleep stages. To clarify whether these steps remain during sleep, we recorded simultaneous electroencephalographic and magnetoencephalographic signals during wakefulness and during sleep in normal subjects listening to a hierarchical auditory paradigm including short-term (local) and long-term (global) regularities. The global response, reflected in the P300, vanished during sleep, in line with the hypothesis that it is a correlate of high-level conscious error detection. The local mismatch response remained across all sleep stages (N1, N2, and REM sleep), but with an incomplete structure; compared with wakefulness, a specific peak reflecting prediction error vanished during sleep. Those results indicate that sleep leaves initial auditory processing and passive sensory response adaptation intact, but specifically disrupts both short-term and long-term auditory predictive coding.

Event-Related Potential, Time-frequency, and Functional Connectivity Facets of Local and Global Auditory Novelty Processing: An Intracranial Study in Humans.

Auditory novelty detection has been associated with different cognitive processes. Bekinschtein et al. (2009) developed an experimental paradigm to dissociate these processes, using local and global novelty, which were associated, respectively, with automatic versus strategic perceptual processing. They have mostly been studied using event-related potentials (ERPs), but local spiking activity as indexed by gamma (60-120 Hz) power and interactions between brain regions as indexed by modulations in beta-band (13-25 Hz) power and functional connectivity have not been explored. We thus recorded 9 epileptic patients with intracranial electrodes to compare the precise dynamics of the responses to local and global novelty. Local novelty triggered an early response observed as an intracranial mismatch negativity (MMN) contemporary with a strong power increase in the gamma band and an increase in connectivity in the beta band. Importantly, all these responses were strictly confined to the temporal auditory cortex. In contrast, global novelty gave rise to a late ERP response distributed across brain areas, contemporary with a sustained power decrease in the beta band (13-25 Hz) and an increase in connectivity in the alpha band (8-13 Hz) within the frontal lobe. We discuss these multi-facet signatures in terms of conscious access to perceptual information.

Decoding the dynamics of action, intention, and error detection for conscious and subliminal stimuli.

How do we detect our own errors, even before we receive any external feedback? One model hypothesizes that error detection results from the confrontation of two signals: a fast and unconscious motor code, based on a direct sensory-motor pathway; and a slower conscious intention code that computes the required response given the stimulus and task instructions. To test this theory and assess how the chain of cognitive processes leading to error detection is modulated by consciousness, we applied multivariate decoding methods to single-trial magnetoencephalography and electroencephalography data. Human participants performed a fast bimanual number comparison task on masked digits presented at threshold, such that about half of them remained unseen. By using both erroneous and correct trials, we designed orthogonal decoders for the actual response (left or right), the required response (left or right), and the response accuracy (correct or incorrect). While perceptual stimulus information and the actual response hand could be decoded on both conscious and non-conscious trials, the required response could only be decoded on conscious trials. Moreover, whether the current response was correct or incorrect could be decoded only when the target digits were conscious, at a time and with a certainty that varied with the amount of evidence in favor of the correct response. These results are in accordance with the proposed dual-route model of conscious versus nonconscious evidence accumulation, and suggest that explicit error detection is possible only when the brain computes a conscious representation of the desired response, distinct from the ongoing motor program.

Large scale screening of neural signatures of consciousness in patients in a vegetative or minimally conscious state.

In recent years, numerous electrophysiological signatures of consciousness have been proposed. Here, we perform a systematic analysis of these electroencephalography markers by quantifying their efficiency in differentiating patients in a vegetative state from those in a minimally conscious or conscious state. Capitalizing on a review of previous experiments and current theories, we identify a series of measures that can be organized into four dimensions: (i) event-related potentials versus ongoing electroencephalography activity; (ii) local dynamics versus inter-electrode information exchange; (iii) spectral patterns versus information complexity; and (iv) average versus fluctuations over the recording session. We analysed a large set of 181 high-density electroencephalography recordings acquired in a 30 minutes protocol. We show that low-frequency power, electroencephalography complexity, and information exchange constitute the most reliable signatures of the conscious state. When combined, these measures synergize to allow an automatic classification of patients' state of consciousness.

Tracking the neural codes for words and phrases during semantic composition, working-memory storage, and retrieval.

The ability to compose successive words into a meaningful phrase is a characteristic feature of human cognition, yet its neural mechanisms remain incompletely understood. Here, we analyze the cortical mechanisms of semantic composition using magnetoencephalography (MEG) while participants read one-word, two-word, and five-word noun phrases and compared them with a subsequent image. Decoding of MEG signals revealed three processing stages. During phrase comprehension, the representation of individual words was sustained for a variable duration depending on phrasal context. During the delay period, the word code was replaced by a working-memory code whose activation increased with semantic complexity. Finally, the speed and accuracy of retrieval depended on semantic complexity and was faster for surface than for deep semantic properties. In conclusion, we propose that the brain initially encodes phrases using factorized dimensions for successive words but later compresses them in working memory and requires a period of decompression to access them.

Hierarchical dynamic coding coordinates speech comprehension in the brain.

Speech comprehension requires the human brain to transform an acoustic waveform into meaning. To do so, the brain generates a hierarchy of features that converts the sensory input into increasingly abstract language properties. However, little is known about how these hierarchical features are generated and continuously coordinated. Here, we propose that each linguistic feature is dynamically represented in the brain to simultaneously represent successive events. To test this 'Hierarchical Dynamic Coding' (HDC) hypothesis, we use time-resolved decoding of brain activity to track the construction, maintenance, and integration of a comprehensive hierarchy of language features spanning acoustic, phonetic, sub-lexical, lexical, syntactic and semantic representations. For this, we recorded 21 participants with magnetoencephalography (MEG), while they listened to two hours of short stories. Our analyses reveal three main findings. First, the brain incrementally represents and simultaneously maintains successive features. Second, the duration of these representations depend on their level in the language hierarchy. Third, each representation is maintained by a dynamic neural code, which evolves at a speed commensurate with its corresponding linguistic level. This HDC preserves the maintenance of information over time while limiting the interference between successive features. Overall, HDC reveals how the human brain continuously builds and maintains a language hierarchy during natural speech comprehension, thereby anchoring linguistic theories to their biological implementations.

Differential cortical layer engagement during seizure initiation and spread in humans.

Despite decades of research, we still do not understand how spontaneous human seizures start and spread - especially at the level of neuronal microcircuits. In this study, we used laminar arrays of micro-electrodes to simultaneously record the local field potentials and multi-unit neural activities across the six layers of the neocortex during focal seizures in humans. We found that, within the ictal onset zone, the discharges generated during a seizure consisted of current sinks and sources only within the infra-granular and granular layers. Outside of the seizure onset zone, ictal discharges reflected current flow in the supra-granular layers. Interestingly, these patterns of current flow evolved during the course of the seizure - especially outside the seizure onset zone where superficial sinks and sources extended into the deeper layers. Based on these observations, a framework describing cortical-cortical dynamics of seizures is proposed with implications for seizure localization, surgical targeting, and neuromodulation techniques to block the generation and propagation of seizures.

Evidence of a predictive coding hierarchy in the human brain listening to speech.

Considerable progress has recently been made in natural language processing: deep learning algorithms are increasingly able to generate, summarize, translate and classify texts. Yet, these language models still fail to match the language abilities of humans. Predictive coding theory offers a tentative explanation to this discrepancy: while language models are optimized to predict nearby words, the human brain would continuously predict a hierarchy of representations that spans multiple timescales. To test this hypothesis, we analysed the functional magnetic resonance imaging brain signals of 304 participants listening to short stories. First, we confirmed that the activations of modern language models linearly map onto the brain responses to speech. Second, we showed that enhancing these algorithms with predictions that span multiple timescales improves this brain mapping. Finally, we showed that these predictions are organized hierarchically: frontoparietal cortices predict higher-level, longer-range and more contextual representations than temporal cortices. Overall, these results strengthen the role of hierarchical predictive coding in language processing and illustrate how the synergy between neuroscience and artificial intelligence can unravel the computational bases of human cognition.

Introducing MEG-MASC a high-quality magneto-encephalography dataset for evaluating natural speech processing.

The "MEG-MASC" dataset provides a curated set of raw magnetoencephalography (MEG) recordings of 27 English speakers who listened to two hours of naturalistic stories. Each participant performed two identical sessions, involving listening to four fictional stories from the Manually Annotated Sub-Corpus (MASC) intermixed with random word lists and comprehension questions. We time-stamp the onset and offset of each word and phoneme in the metadata of the recording, and organize the dataset according to the 'Brain Imaging Data Structure' (BIDS). This data collection provides a suitable benchmark to large-scale encoding and decoding analyses of temporally-resolved brain responses to speech. We provide the Python code to replicate several validations analyses of the MEG evoked responses such as the temporal decoding of phonetic features and word frequency. All code and MEG, audio and text data are publicly available to keep with best practices in transparent and reproducible research.