A shared model-based linguistic space for transmitting our thoughts from brain to brain in natural conversations.

Effective communication hinges on a mutual understanding of word meaning in different contexts. We recorded brain activity using electrocorticography during spontaneous, face-to-face conversations in five pairs of epilepsy patients. We developed a model-based coupling framework that aligns brain activity in both speaker and listener to a shared embedding space from a large language model (LLM). The context-sensitive LLM embeddings allow us to track the exchange of linguistic information, word by word, from one brain to another in natural conversations. Linguistic content emerges in the speaker's brain before word articulation and rapidly re-emerges in the listener's brain after word articulation. The contextual embeddings better capture word-by-word neural alignment between speaker and listener than syntactic and articulatory models. Our findings indicate that the contextual embeddings learned by LLMs can serve as an explicit numerical model of the shared, context-rich meaning space humans use to communicate their thoughts to one another.

Scale matters: Large language models with billions (rather than millions) of parameters better match neural representations of natural language.

Recent research has used large language models (LLMs) to study the neural basis of naturalistic language processing in the human brain. LLMs have rapidly grown in complexity, leading to improved language processing capabilities. However, neuroscience researchers haven't kept up with the quick progress in LLM development. Here, we utilized several families of transformer-based LLMs to investigate the relationship between model size and their ability to capture linguistic information in the human brain. Crucially, a subset of LLMs were trained on a fixed training set, enabling us to dissociate model size from architecture and training set size. We used electrocorticography (ECoG) to measure neural activity in epilepsy patients while they listened to a 30-minute naturalistic audio story. We fit electrode-wise encoding models using contextual embeddings extracted from each hidden layer of the LLMs to predict word-level neural signals. In line with prior work, we found that larger LLMs better capture the structure of natural language and better predict neural activity. We also found a log-linear relationship where the encoding performance peaks in relatively earlier layers as model size increases. We also observed variations in the best-performing layer across different brain regions, corresponding to an organized language processing hierarchy.

Shared functional specialization in transformer-based language models and the human brain.

When processing language, the brain is thought to deploy specialized computations to construct meaning from complex linguistic structures. Recently, artificial neural networks based on the Transformer architecture have revolutionized the field of natural language processing. Transformers integrate contextual information across words via structured circuit computations. Prior work has focused on the internal representations ("embeddings") generated by these circuits. In this paper, we instead analyze the circuit computations directly: we deconstruct these computations into the functionally-specialized "transformations" that integrate contextual information across words. Using functional MRI data acquired while participants listened to naturalistic stories, we first verify that the transformations account for considerable variance in brain activity across the cortical language network. We then demonstrate that the emergent computations performed by individual, functionally-specialized "attention heads" differentially predict brain activity in specific cortical regions. These heads fall along gradients corresponding to different layers and context lengths in a low-dimensional cortical space.

Alignment of brain embeddings and artificial contextual embeddings in natural language points to common geometric patterns.

Contextual embeddings, derived from deep language models (DLMs), provide a continuous vectorial representation of language. This embedding space differs fundamentally from the symbolic representations posited by traditional psycholinguistics. We hypothesize that language areas in the human brain, similar to DLMs, rely on a continuous embedding space to represent language. To test this hypothesis, we densely record the neural activity patterns in the inferior frontal gyrus (IFG) of three participants using dense intracranial arrays while they listened to a 30-minute podcast. From these fine-grained spatiotemporal neural recordings, we derive a continuous vectorial representation for each word (i.e., a brain embedding) in each patient. Using stringent zero-shot mapping we demonstrate that brain embeddings in the IFG and the DLM contextual embedding space have common geometric patterns. The common geometric patterns allow us to predict the brain embedding in IFG of a given left-out word based solely on its geometrical relationship to other non-overlapping words in the podcast. Furthermore, we show that contextual embeddings capture the geometry of IFG embeddings better than static word embeddings. The continuous brain embedding space exposes a vector-based neural code for natural language processing in the human brain.

Bayesian Surprise Predicts Human Event Segmentation in Story Listening.

Event segmentation theory posits that people segment continuous experience into discrete events and that event boundaries occur when there are large transient increases in prediction error. Here, we set out to test this theory in the context of story listening, by using a deep learning language model (GPT-2) to compute the predicted probability distribution of the next word, at each point in the story. For three stories, we used the probability distributions generated by GPT-2 to compute the time series of prediction error. We also asked participants to listen to these stories while marking event boundaries. We used regression models to relate the GPT-2 measures to the human segmentation data. We found that event boundaries are associated with transient increases in Bayesian surprise but not with a simpler measure of prediction error (surprisal) that tracks, for each word in the story, how strongly that word was predicted at the previous time point. These results support the hypothesis that prediction error serves as a control mechanism governing event segmentation and point to important differences between operational definitions of prediction error.

A shared linguistic space for transmitting our thoughts from brain to brain in natural conversations.

Effective communication hinges on a mutual understanding of word meaning in different contexts. The embedding space learned by large language models can serve as an explicit model of the shared, context-rich meaning space humans use to communicate their thoughts. We recorded brain activity using electrocorticography during spontaneous, face-to-face conversations in five pairs of epilepsy patients. We demonstrate that the linguistic embedding space can capture the linguistic content of word-by-word neural alignment between speaker and listener. Linguistic content emerged in the speaker's brain before word articulation, and the same linguistic content rapidly reemerged in the listener's brain after word articulation. These findings establish a computational framework to study how human brains transmit their thoughts to one another in real-world contexts.

Perspective changes in human listeners are aligned with the contextual transformation of the word embedding space.

Word embedding representations have been shown to be effective in predicting human neural responses to lingual stimuli. While these representations are sensitive to the textual context, they lack the extratextual sources of context such as prior knowledge, thoughts, and beliefs, all of which constitute the listener's perspective. In this study, we propose conceptualizing the listeners' perspective as a source that induces changes in the embedding space. We relied on functional magnetic resonance imaging data collected by Yeshurun Y, Swanson S, Simony E, Chen J, Lazaridi C, Honey CJ, Hasson U. Same story, different story: the neural representation of interpretive frameworks. Psychol Sci. 2017:28(3):307-319, in which two groups of human listeners (n = 40) were listening to the same story but with different perspectives. Using a dedicated fine-tuning process, we created two modified versions of a word embedding space, corresponding to the two groups of listeners. We found that each transformed space was better fitted with neural responses of the corresponding group, and that the spatial distances between these spaces reflect both interpretational differences between the perspectives and the group-level neural differences. Together, our results demonstrate how aligning a continuous embedding space to a specific context can provide a novel way of modeling listeners' intrinsic perspectives.

Shared computational principles for language processing in humans and deep language models.

Departing from traditional linguistic models, advances in deep learning have resulted in a new type of predictive (autoregressive) deep language models (DLMs). Using a self-supervised next-word prediction task, these models generate appropriate linguistic responses in a given context. In the current study, nine participants listened to a 30-min podcast while their brain responses were recorded using electrocorticography (ECoG). We provide empirical evidence that the human brain and autoregressive DLMs share three fundamental computational principles as they process the same natural narrative: (1) both are engaged in continuous next-word prediction before word onset; (2) both match their pre-onset predictions to the incoming word to calculate post-onset surprise; (3) both rely on contextual embeddings to represent words in natural contexts. Together, our findings suggest that autoregressive DLMs provide a new and biologically feasible computational framework for studying the neural basis of language.

Regression to the Mean Does Not Explain Away Nonconscious Processing.

In studies that use subliminal presentations, participants may become aware of stimuli that are intended to remain subliminal. A common solution to this problem is to analyze the results of the group of participants for whom the stimuli remained subliminal. A recent article (Shanks, 2017) argued that this method leads to a regression to the mean artifact, which may account for many of the observed effects. However, conceptual and statistical characteristics of the original publication lead to overestimation of the influence of the artifact. Using simulations, we demonstrate that this overestimation leads to the mistaken conclusion that regression to the mean accounts for nonconscious effects. We conclude by briefly outlining a new description of the influence of the artifact and how it should be statistically addressed.

The "Narratives" fMRI dataset for evaluating models of naturalistic language comprehension.

The "Narratives" collection aggregates a variety of functional MRI datasets collected while human subjects listened to naturalistic spoken stories. The current release includes 345 subjects, 891 functional scans, and 27 diverse stories of varying duration totaling ~4.6 hours of unique stimuli (~43,000 words). This data collection is well-suited for naturalistic neuroimaging analysis, and is intended to serve as a benchmark for models of language and narrative comprehension. We provide standardized MRI data accompanied by rich metadata, preprocessed versions of the data ready for immediate use, and the spoken story stimuli with time-stamped phoneme- and word-level transcripts. All code and data are publicly available with full provenance in keeping with current best practices in transparent and reproducible neuroimaging.

Did you see it? Robust individual differences in the speed with which meaningful visual stimuli break suppression.

Perceptual conscious experiences result from non-conscious processes that precede them. We document a new characteristic of the cognitive system: the speed with which visual meaningful stimuli are prioritized to consciousness over competing noise in visual masking paradigms. In ten experiments (N = 399) we find that an individual's non-conscious visual prioritization speed (NVPS) is ubiquitous across a wide variety of stimuli, and generalizes across visual masks, suppression tasks, and time. We also find that variation in NVPS is unique, in that it cannot be explained by variation in general speed, perceptual decision thresholds, short-term visual memory, or three networks of attention (alerting, orienting and executive). Finally, we find that NVPS is correlated with subjective measures of sensitivity, as they are measured by the Highly Sensitive Person scale. We conclude by discussing the implications of variance in NVPS for understanding individual variance in behavior and the neural substrates of consciousness.

Flash Electroretinography Parameters and Parkinson's Disease.

BACKGROUND: Parkinson's disease (PD) is known to affect retinal structure and activity. As such, retinal evaluations may be used to develop objective and possibly early PD diagnostic tools. OBJECTIVE: The aim of this study was to investigate the effects of Parkinson's disease (PD) manifestation and treatment on retinal activity. METHODS: Data were collected on 21 participants diagnosed with PD, including the number of medications taken, clinical scales and flash electroretinography (fERG) measurements, under light-adapted and dark-adapted conditions. The fERG parameters measured included a-wave and b-wave amplitude and implicit time (i.e., latency). First, we investigated correlations between symptom measure scores and the fERG parameters. Next, we divided participants into two groups based on their antiparkinsonian medication load and analyzed differences between these groups' fERG parameters. RESULTS: fERG parameters were strongly correlated with a number of clinical variables, including motor and non-motor symptoms and age at PD onset. Photoreceptor cell implicit time was longer among participants taking one or less antiparkinsonian medication as compared to those taking two or more. However, overall there was not strong evidence of a relationship between the number of antiparkinsonian medications taken and the fERG parameters. CONCLUSION: Findings suggest that fERG may be a useful, non-intrusive measure of retinal, and, perhaps overall CNS function, in PD. However, additional studies in larger samples are needed to clarify this association.

Predictions from masked motion with and without obstacles.

Predicting the future is essential for organisms like Homo sapiens, who live in a dynamic and ever-changing world. Previous research has established that conscious stimuli can lead to non-conscious predictions. Here we examine whether masked stimuli can also induce such predictions. We use masked movement-with and without obstacles-to examine predictions from masked stimuli. In six experiments a moving object was masked using continuous flash suppression (CFS). A few hundred milliseconds after the object had disappeared, a conscious probe appeared in a location that was either consistent with the masked stimulus or not. In Experiments 1-3 the movement was linear, and reaction times (RTs) indicated predictions that were based on direction and speed of movement. In Experiment 4, the masked moving object collided with an obstacle and then disappeared. Predictions in this case should reflect deflection, and indeed reaction times revealed predictions on the deflection route. In Experiments 5 and 6 we introduce an innovative way of using eye-tracking during continuous flash suppression (CFS) and report physiological evidence-in the forms of eye-movements-for masked stimuli induced predictions. We thus conclude that humans can use dynamic masked stimuli to generate active predictions about the future, and use these predictions to guide behavior. We also discuss the possible interpretations of these findings in light of the current scientific discussion regarding the relation between masked presentation, subliminal perception and awareness measurement methods.

Keep it real: rethinking the primacy of experimental control in cognitive neuroscience.

Naturalistic experimental paradigms in neuroimaging arose from a pressure to test the validity of models we derive from highly-controlled experiments in real-world contexts. In many cases, however, such efforts led to the realization that models developed under particular experimental manipulations failed to capture much variance outside the context of that manipulation. The critique of non-naturalistic experiments is not a recent development; it echoes a persistent and subversive thread in the history of modern psychology. The brain has evolved to guide behavior in a multidimensional world with many interacting variables. The assumption that artificially decoupling and manipulating these variables will lead to a satisfactory understanding of the brain may be untenable. We develop an argument for the primacy of naturalistic paradigms, and point to recent developments in machine learning as an example of the transformative power of relinquishing control. Naturalistic paradigms should not be deployed as an afterthought if we hope to build models of brain and behavior that extend beyond the laboratory into the real world.

Direct Fit to Nature: An Evolutionary Perspective on Biological and Artificial Neural Networks.

Evolution is a blind fitting process by which organisms become adapted to their environment. Does the brain use similar brute-force fitting processes to learn how to perceive and act upon the world? Recent advances in artificial neural networks have exposed the power of optimizing millions of synaptic weights over millions of observations to operate robustly in real-world contexts. These models do not learn simple, human-interpretable rules or representations of the world; rather, they use local computations to interpolate over task-relevant manifolds in a high-dimensional parameter space. Counterintuitively, similar to evolutionary processes, over-parameterized models can be simple and parsimonious, as they provide a versatile, robust solution for learning a diverse set of functions. This new family of direct-fit models present a radical challenge to many of the theoretical assumptions in psychology and neuroscience. At the same time, this shift in perspective establishes unexpected links with developmental and ecological psychology.

Face proprioception does not modulate access to visual awareness of emotional faces in a continuous flash suppression paradigm.

An important question in neuroscience is which multisensory information, presented outside of awareness, can influence the nature and speed of conscious access to our percepts. Recently, proprioceptive feedback of the hand was reported to lead to faster awareness of congruent hand images in a breaking continuous flash suppression (b-CFS) paradigm. Moreover, a vast literature suggests that spontaneous facial mimicry can improve emotion recognition, even without awareness of the stimulus face. However, integration of visual and proprioceptive information about the face to date has not been tested with CFS. The modulation of visual awareness of emotional faces by facial proprioception was investigated across three separate experiments. Face proprioception was induced with voluntary facial expressions or with spontaneous facial mimicry. Frequentist statistical analyses were complemented with Bayesian statistics. No evidence of multisensory integration was found, suggesting that proprioception does not modulate access to visual awareness of emotional faces in a CFS paradigm.

La Prensa Brasileña y sus "Cruzadas Morales": Un Análisis de los Casos del Segundo Gobierno de Getúlio Vargas y el Primer Gobierno de Lula da Silva / The Brazilian Press and its "Moral Crusades": An Analysis of Several Cases under President Getúlio Vargas' Second Term and President Lula da Silva's First Term / La Presse Brésilienne et ses "Croisades Morales": Une Analyse des Cas du Second Mandat de Getúlio Vargas et du Premier de Lula da Silva / A Imprensa Brasileira e suas "Cruzadas Morais": Análise dos Casos do Segundo Governo de Getúlio Vargas e do Primeiro Governo de Lula da Silva

RESUMEN A partir del análisis de los encuadres y discursos en los editoriales y otras secciones de los periódicos O Estado de S. Paulo y O Globo durante el segundo gobierno de Getúlio Vargas y el primer gobierno de Lula da Silva, este artículo explora las características de lo que denomina como la "moralización de la política" adoptada por la prensa en ambos períodos. En este sentido, se sostiene que la estrategia de colonizar los asuntos públicos con la distinción entre "honestos" y "corruptos" tenía por fin vaciar el debate público, centrando las acusaciones de corrupción sobre los gobiernos y ocultado la apropiación histórica de las clases dominantes. De este modo, la prensa colaboraría para la naturalización del orden social en el país. A su vez, con estos encuadres, ambos periódicos favorecerían en estos períodos la construcción de escenarios de crisis políticas de intensidad que atravesarían estos gobiernos.

ABSTRACT By means of an analysis of the contexts and debates in editorials and other sections of O Estado de S. Paulo and O Globo newspapers during Getúlio Vargas' second presidential term and Lula da Silva's first presidential term, the following article explores the characteristics of what it terms the "moralization of politics" adopted by the press during both periods. In this sense, it is argued that the strategy of colonizing public matters with the distinction between "honest" and "corrupt" was aimed at silencing public debate, focusing the accusations of corruption on governments and concealing the historical appropriation of the ruling classes. The press was therefore complicit in naturalizing the social order in Brazil, with the two newspapers also favoring the shaping of political crisis scenarios so significant as to have rocked these governments.

RÉSUMÉ À partir de l'analyse de la ligne et des discours des éditoriaux et autres sections des quotidiens O Estado de S. Paulo et O Globo durant le second mandat de Getúlio Vargas et le premier de Lula da Silva, cet article explore les caractéristiques de la supposée "moralisation de la politique" prônée par la presse lors de ces deux périodes. En ce sens, on soutiendra que la stratégie de colonisation des affaires publiques par la distinction entre "honnêtes" et "corrompus" a eu pour objectif de vider le débat public de sa substance en centrant les accusations de corruption sur les gouvernements et en occultant l'appropriation historique des classes dominantes. La presse collaborerait ainsi à la naturalisation de l'ordre social en vigueur dans le pays. En même temps, cette ligne éditoriale favoriserait dans les deux quotidiens la construction du climat de crise politique qui a traversé les deux mandats en question.

RESUMO A partir da análise de matérias, editoriais e outras seções dos jornais O Estado de São Paulo e O Globo, durante o segundo governo de Getúlio Vargas e o primeiro governo Lula, este artigo explora as características da chamada "moralização da política", adotada pela imprensa em ambos os períodos. Neste sentido, defende-se que a estratégia de submeter os assuntos públicos à distinção entre "honestos" e "corruptos" tinha o intuito de esvaziar o debate público, com ênfase em acusações de corrupção voltadas para os governos e na ocultação da apropriação histórica das classes dominantes. Assim sendo, a imprensa contribuiria para a naturalização da ordem social no país. Por sua vez, com essa postura editorial, ambos os jornais viriam a favorecer nestes períodos a construção de cenários de intensas crises políticas que permearam esses governos.

Enhanced discriminability for nonbiological motion violating the two-thirds power law.

The two-thirds power law describes the relationship between velocity and curvature in human motor movements. Interestingly, this motor law also affects visual motion perception, in which stimuli moving according to the two-thirds power law are perceived to have a constant velocity compared to stimuli actually moving at constant velocity. Thus, visual motion adhering to biological motion principles causes a kinematic illusion of smooth and velocity-invariant motion. However, it is yet unclear how this motion law affects the discrimination of visual stimuli and if its encoding requires attention. Here we tested the perceptual discrimination of stimuli following biological (two-thirds power law) or nonbiological movement under conditions in which the stimuli were degraded or masked through continuous flash suppression. Additionally, we tested subjective perception of naturalness and velocity consistency. Our results show that the discriminability of a visual target is inversely related to the perceived "naturalness" of its movement. Discrimination of stimuli following the two-thirds power law required more time than the same stimuli moving at constant velocity or nonecological variants of the two-thirds power law and was present for both masked and degraded stimuli.

Are valence and arousal separable in emotional experience?

The bipolar valence-arousal model of conscious experience of emotions is prominent in emotion research. In this work, we examine the validity of this model in the context of feelings elicited by visual stimuli. In particular, we examine whether arousal has a unique contribution over bivariate valence (separate measures for pleasure and displeasure) in explaining physiological arousal (electrodermal activity, EDA) and self-reported feelings at the level of item-specific responses across and within individuals. Our results suggest that self-reports of arousal have neither an advantage in predicting EDA nor make a unique contribution when valence is present in the model. Acceptance of the null hypothesis was confirmed with the use of the Bayesian information criterion. Arousal also showed no advantage over valence in predicting global feelings, but demonstrated a small unique component (1.5% to 4% of variance explained). These results have practical implications for both experimental design in the study of emotions and the underlying bases of their conscious experience.