RESUMO
COVID-19 forced students to rely on online learning using multimedia tools, and multimedia learning continues to impact education beyond the pandemic. In this study, we combined behavioral, eye-tracking, and neuroimaging paradigms to identify multimedia learning processes and outcomes. College students viewed four video lectures including slides with either an onscreen human instructor, an animated instructor, or no onscreen instructor. Brain activity was recorded via fMRI, visual attention was recorded via eye-tracking, and learning outcome was assessed via post-tests. Onscreen presence of instructor, compared with no instructor presence, resulted in superior post-test performance, less visual attention on the slide, more synchronized eye movements during learning, and higher neural synchronization in cortical networks associated with socio-emotional processing and working memory. Individual variation in cognitive and socio-emotional abilities and intersubject neural synchronization revealed different levels of cognitive and socio-emotional processing in different learning conditions. The instructor-present condition evoked increased synchronization, likely reflecting extra processing demands in attentional control, working memory engagement, and socio-emotional processing. Although human instructors and animated instructors led to comparable learning outcomes, the effects were due to the dynamic interplay of information processing vs. attentional distraction. These findings reflect a benefit-cost trade-off where multimedia learning outcome is enhanced only when the cognitive benefits motivated by the social presence of onscreen instructor outweigh the cognitive costs brought about by concurrent attentional distraction unrelated to learning.
Assuntos
Aprendizagem , Multimídia , Humanos , Cognição/fisiologia , Memória de Curto Prazo/fisiologia , EstudantesRESUMO
Deep convolutional neural networks (DCNNs) trained for face identification can rival and even exceed human-level performance. The ways in which the internal face representations in DCNNs relate to human cognitive representations and brain activity are not well understood. Nearly all previous studies focused on static face image processing with rapid display times and ignored the processing of naturalistic, dynamic information. To address this gap, we developed the largest naturalistic dynamic face stimulus set in human neuroimaging research (700+ naturalistic video clips of unfamiliar faces). We used this naturalistic dataset to compare representational geometries estimated from DCNNs, behavioral responses, and brain responses. We found that DCNN representational geometries were consistent across architectures, cognitive representational geometries were consistent across raters in a behavioral arrangement task, and neural representational geometries in face areas were consistent across brains. Representational geometries in late, fully connected DCNN layers, which are optimized for individuation, were much more weakly correlated with cognitive and neural geometries than were geometries in late-intermediate layers. The late-intermediate face-DCNN layers successfully matched cognitive representational geometries, as measured with a behavioral arrangement task that primarily reflected categorical attributes, and correlated with neural representational geometries in known face-selective topographies. Our study suggests that current DCNNs successfully capture neural cognitive processes for categorical attributes of faces but less accurately capture individuation and dynamic features.
Assuntos
Reconhecimento Facial , Humanos , Reconhecimento Facial/fisiologia , Redes Neurais de Computação , Processamento de Imagem Assistida por Computador/métodos , Encéfalo/diagnóstico por imagem , NeuroimagemRESUMO
When listening to spoken narratives, we must integrate information over multiple, concurrent timescales, building up from words to sentences to paragraphs to a coherent narrative. Recent evidence suggests that the brain relies on a chain of hierarchically organized areas with increasing temporal receptive windows to process naturalistic narratives. We hypothesized that the structure of this cortical processing hierarchy should result in an observable sequence of response lags between networks comprising the hierarchy during narrative comprehension. This study uses functional MRI to estimate the response lags between functional networks during narrative comprehension. We use intersubject cross-correlation analysis to capture network connectivity driven by the shared stimulus. We found a fixed temporal sequence of response lags-on the scale of several seconds-starting in early auditory areas, followed by language areas, the attention network, and lastly the default mode network. This gradient is consistent across eight distinct stories but absent in data acquired during rest or using a scrambled story stimulus, supporting our hypothesis that narrative construction gives rise to internetwork lags. Finally, we build a simple computational model for the neural dynamics underlying the construction of nested narrative features. Our simulations illustrate how the gradual accumulation of information within the boundaries of nested linguistic events, accompanied by increased activity at each level of the processing hierarchy, can give rise to the observed lag gradient.
Assuntos
Mapeamento Encefálico , Percepção da Fala , Percepção da Fala/fisiologia , Compreensão/fisiologia , Encéfalo/diagnóstico por imagem , Encéfalo/fisiologia , Imageamento por Ressonância MagnéticaRESUMO
The attention schema theory posits a specific relationship between subjective awareness and attention, in which awareness is the control model that the brain uses to aid in the endogenous control of attention. In previous experiments, we developed a behavioral paradigm in human subjects to manipulate awareness and attention. The paradigm involved a visual cue that could be used to guide attention to a target stimulus. In task 1, subjects were aware of the cue, but not aware that it provided information about the target. The cue measurably drew exogenous attention to itself. In addition, implicitly, the subjects' endogenous attention mechanism used the cue to help shift attention to the target. In task 2, subjects were no longer aware of the cue. The cue still measurably drew exogenous attention to itself, yet without awareness of the cue, the subjects' endogenous control mechanism was no longer able to use the cue to control attention. Thus, the control of attention depended on awareness. Here, we tested the two tasks while scanning brain activity in human volunteers. We predicted that the right temporoparietal junction (TPJ) would be active in relation to the process in which awareness helps control attention. This prediction was confirmed. The right TPJ was active in relation to the effect of the cue on attention in task 1; it was not measurably active in task 2. The difference was significant. In our interpretation, the right TPJ is involved in an interaction in which awareness permits the control of attention.
Assuntos
Atenção/fisiologia , Conscientização/fisiologia , Lobo Parietal/fisiologia , Lobo Temporal/fisiologia , Adolescente , Adulto , Comportamento , Mapeamento Encefálico , Feminino , Humanos , Imageamento por Ressonância Magnética , Masculino , Pessoa de Meia-Idade , Rede Nervosa/fisiologia , Análise e Desempenho de Tarefas , Adulto JovemRESUMO
Effective influence management during advice-giving requires individuals to express confidence in the advice properly and switch timely between the 'competitive' strategy and the 'defensive' strategy. However, how advisers switch between these two strategies, and whether and why there exist individual differences during this process remain elusive. We used an advice-giving game that manipulated incentive contexts (Incentivized/Non-Incentivized) to induce the adviser's confidence expression strategy switching and measured the brain activities of adviser and advisee concurrently using functional near-infrared spectroscopy (fNIRS). Behaviorally, we observed individual differences in strategy switching. Some advisers applied the 'defensive' strategy when incentivized and the 'competitive' strategy when not incentivized, while others applied the 'competitive' strategy when incentivized and the 'defensive' strategy when not incentivized. This effect was mediated by the adviser's perceived stress in each condition and was reflected by the frequencies of advice-taking in the advisees. Neurally, brain activation in the dorsolateral prefrontal cortex (DLPFC) supported strategy switching, as well as interpersonal neural synchronization (INS) in the temporoparietal junction (TPJ) that supported influence management. This two-in-one process, i.e., confidence expression strategy switching and the corresponding influence management, was linked and modulated by the strength of DLPFC-TPJ functional connectivity in the adviser. We further developed a descriptive model that contributed to understanding the adviser's strategy switching during influence management.
Assuntos
Encéfalo , Motivação , Humanos , Processos Mentais , Mapeamento Encefálico/métodos , Córtex Pré-Frontal/fisiologiaRESUMO
Recent fMRI studies of event segmentation have found that default mode regions represent high-level event structure during movie watching. In these regions, neural patterns are relatively stable during events and shift at event boundaries. Music, like narratives, contains hierarchical event structure (e.g., sections are composed of phrases). Here, we tested the hypothesis that brain activity patterns in default mode regions reflect the high-level event structure of music. We used fMRI to record brain activity from 25 participants (male and female) as they listened to a continuous playlist of 16 musical excerpts and additionally collected annotations for these excerpts by asking a separate group of participants to mark when meaningful changes occurred in each one. We then identified temporal boundaries between stable patterns of brain activity using a hidden Markov model and compared the location of the model boundaries to the location of the human annotations. We identified multiple brain regions with significant matches to the observer-identified boundaries, including auditory cortex, medial prefrontal cortex, parietal cortex, and angular gyrus. From these results, we conclude that both higher-order and sensory areas contain information relating to the high-level event structure of music. Moreover, the higher-order areas in this study overlap with areas found in previous studies of event perception in movies and audio narratives, including regions in the default mode network.
Assuntos
Córtex Auditivo , Música , Córtex Auditivo/diagnóstico por imagem , Percepção Auditiva , Mapeamento Encefálico/métodos , Feminino , Humanos , Imageamento por Ressonância Magnética/métodos , MasculinoRESUMO
A key problem in functional magnetic resonance imaging (fMRI) is to estimate spatial activity patterns from noisy high-dimensional signals. Spatial smoothing provides one approach to regularizing such estimates. However, standard smoothing methods ignore the fact that correlations in neural activity may fall off at different rates in different brain areas, or exhibit discontinuities across anatomical or functional boundaries. Moreover, such methods do not exploit the fact that widely separated brain regions may exhibit strong correlations due to bilateral symmetry or the network organization of brain regions. To capture this non-stationary spatial correlation structure, we introduce the brain kernel, a continuous covariance function for whole-brain activity patterns. We define the brain kernel in terms of a continuous nonlinear mapping from 3D brain coordinates to a latent embedding space, parametrized with a Gaussian process (GP). The brain kernel specifies the prior covariance between voxels as a function of the distance between their locations in embedding space. The GP mapping warps the brain nonlinearly so that highly correlated voxels are close together in latent space, and uncorrelated voxels are far apart. We estimate the brain kernel using resting-state fMRI data, and we develop an exact, scalable inference method based on block coordinate descent to overcome the challenges of high dimensionality (10-100K voxels). Finally, we illustrate the brain kernel's usefulness with applications to brain decoding and factor analysis with multiple task-based fMRI datasets.
Assuntos
Mapeamento Encefálico/métodos , Processamento de Imagem Assistida por Computador/métodos , Imageamento por Ressonância Magnética/métodos , Neuroimagem/métodos , Humanos , Imageamento TridimensionalRESUMO
Shared information content is represented across brains in idiosyncratic functional topographies. Hyperalignment addresses these idiosyncrasies by using neural responses to project individuals' brain data into a common model space while maintaining the geometric relationships between distinct patterns of activity or connectivity. The dimensions of this common model capture functional profiles that are shared across individuals such as cortical response profiles collected during a common time-locked stimulus presentation (e.g. movie viewing) or functional connectivity profiles. Hyperalignment can use either response-based or connectivity-based input data to derive transformations that project individuals' neural data from anatomical space into the common model space. Previously, only response or connectivity profiles were used in the derivation of these transformations. In this study, we developed a new hyperalignment algorithm, hybrid hyperalignment, that derives transformations based on both response-based and connectivity-based information. We used three different movie-viewing fMRI datasets to test the performance of our new algorithm. Hybrid hyperalignment derives a single common model space that aligns response-based information as well as or better than response hyperalignment while simultaneously aligning connectivity-based information better than connectivity hyperalignment. These results suggest that a single common information space can encode both shared cortical response and functional connectivity profiles across individuals.
Assuntos
Mapeamento Encefálico/métodos , Córtex Cerebral/diagnóstico por imagem , Processamento de Imagem Assistida por Computador/métodos , Imageamento por Ressonância Magnética/métodos , Filmes Cinematográficos , Rede Nervosa/diagnóstico por imagem , Adulto , Córtex Cerebral/fisiologia , Feminino , Humanos , Masculino , Rede Nervosa/fisiologia , Estimulação Luminosa/métodosRESUMO
Naturalistic experimental paradigms in neuroimaging arose from a pressure to test the validity of models we derive from highly-controlled experiments in real-world contexts. In many cases, however, such efforts led to the realization that models developed under particular experimental manipulations failed to capture much variance outside the context of that manipulation. The critique of non-naturalistic experiments is not a recent development; it echoes a persistent and subversive thread in the history of modern psychology. The brain has evolved to guide behavior in a multidimensional world with many interacting variables. The assumption that artificially decoupling and manipulating these variables will lead to a satisfactory understanding of the brain may be untenable. We develop an argument for the primacy of naturalistic paradigms, and point to recent developments in machine learning as an example of the transformative power of relinquishing control. Naturalistic paradigms should not be deployed as an afterthought if we hope to build models of brain and behavior that extend beyond the laboratory into the real world.
Assuntos
Encéfalo/fisiologia , Cognição/fisiologia , Neurociência Cognitiva , Aprendizado de Máquina , Humanos , Neuroimagem/métodosRESUMO
Naturalistic, dynamic movies evoke strong, consistent, and information-rich patterns of activity over a broad expanse of cortex and engage multiple perceptual and cognitive systems in parallel. The use of naturalistic stimuli enables functional brain imaging research to explore cognitive domains that are poorly sampled in highly-controlled experiments. These domains include perception and understanding of agentic action, which plays a larger role in visual representation than was appreciated from experiments using static, controlled stimuli.
Assuntos
Pesquisa Biomédica , Mapeamento Encefálico , Córtex Cerebral/fisiologia , Neurociência Cognitiva , Imageamento por Ressonância Magnética , Filmes Cinematográficos , Percepção Visual/fisiologia , Animais , Pesquisa Biomédica/métodos , Pesquisa Biomédica/normas , Pesquisa Biomédica/tendências , Neurociência Cognitiva/métodos , Neurociência Cognitiva/normas , Neurociência Cognitiva/tendências , HumanosRESUMO
Connectivity hyperalignment can be used to estimate a single shared response space across disjoint datasets. We develop a connectivity-based shared response model that factorizes aggregated fMRI datasets into a single reduced-dimension shared connectivity space and subject-specific topographic transformations. These transformations resolve idiosyncratic functional topographies and can be used to project response time series into shared space. We evaluate this algorithm on a large collection of heterogeneous, naturalistic fMRI datasets acquired while subjects listened to spoken stories. Projecting subject data into shared space dramatically improves between-subject story time-segment classification and increases the dimensionality of shared information across subjects. This improvement generalizes to subjects and stories excluded when estimating the shared space. We demonstrate that estimating a simple semantic encoding model in shared space improves between-subject forward encoding and inverted encoding model performance. The shared space estimated across all datasets is distinct from the shared space derived from any particular constituent dataset; the algorithm leverages shared connectivity to yield a consensus shared space conjoining diverse story stimuli.
Assuntos
Rede Nervosa/diagnóstico por imagem , Rede Nervosa/fisiologia , Estimulação Acústica , Adolescente , Adulto , Algoritmos , Córtex Auditivo/diagnóstico por imagem , Córtex Auditivo/fisiologia , Percepção Auditiva , Mapeamento Encefálico , Bases de Dados Factuais , Feminino , Humanos , Individualidade , Imageamento por Ressonância Magnética , Masculino , Pessoa de Meia-Idade , Modelos Neurológicos , Modelos Psicológicos , Semântica , Adulto JovemRESUMO
Current neurobiological models assign a central role to predictive processes calibrated to environmental statistics. Neuroimaging studies examining the encoding of stimulus uncertainty have relied almost exclusively on manipulations in which stimuli were presented in a single sensory modality, and further assumed that neural responses vary monotonically with uncertainty. This has left a gap in theoretical development with respect to two core issues: (i) are there cross-modal brain systems that encode input uncertainty in way that generalizes across sensory modalities, and (ii) are there brain systems that track input uncertainty in a non-monotonic fashion? We used multivariate pattern analysis to address these two issues using auditory, visual and audiovisual inputs. We found signatures of cross-modal encoding in frontoparietal, orbitofrontal, and association cortices using a searchlight cross-classification analysis where classifiers trained to discriminate levels of uncertainty in one modality were tested in another modality. Additionally, we found widespread systems encoding uncertainty non-monotonically using classifiers trained to discriminate intermediate levels of uncertainty from both the highest and lowest uncertainty levels. These findings comprise the first comprehensive report of cross-modal and non-monotonic neural sensitivity to statistical regularities in the environment, and suggest that conventional paradigms testing for monotonic responses to uncertainty in a single sensory modality may have limited generalizability.
Assuntos
Percepção Auditiva/fisiologia , Mapeamento Encefálico/métodos , Encéfalo/fisiologia , Percepção Visual/fisiologia , Estimulação Acústica/métodos , Adulto , Feminino , Humanos , Processamento de Imagem Assistida por Computador/métodos , Imageamento por Ressonância Magnética/métodos , Masculino , Estimulação Luminosa/métodos , Adulto JovemRESUMO
Fine-grained functional organization of cortex is not well-conserved across individuals. As a result, individual differences in cortical functional architecture are confounded by topographic idiosyncrasies-i.e., differences in functional-anatomical correspondence. In this study, we used hyperalignment to align information encoded in topographically variable patterns to study individual differences in fine-grained cortical functional architecture in a common representational space. We characterized the structure of individual differences using three common functional indices, and assessed the reliability of this structure across independent samples of data in a natural vision paradigm. Hyperalignment markedly improved the reliability of individual differences across all three indices by resolving topographic idiosyncrasies and accommodating information encoded in spatially fine-grained response patterns. Our results demonstrate that substantial individual differences in cortical functional architecture exist at fine spatial scales, but are inaccessible with anatomical normalization alone.
Assuntos
Mapeamento Encefálico/métodos , Córtex Cerebral/fisiologia , Processamento de Imagem Assistida por Computador/métodos , Individualidade , Imageamento por Ressonância Magnética/métodos , Adulto , Mapeamento Encefálico/normas , Córtex Cerebral/diagnóstico por imagem , Feminino , Humanos , Processamento de Imagem Assistida por Computador/normas , Imageamento por Ressonância Magnética/normas , Masculino , Reprodutibilidade dos Testes , Adulto JovemRESUMO
Humans prioritize different semantic qualities of a complex stimulus depending on their behavioral goals. These semantic features are encoded in distributed neural populations, yet it is unclear how attention might operate across these distributed representations. To address this, we presented participants with naturalistic video clips of animals behaving in their natural environments while the participants attended to either behavior or taxonomy. We used models of representational geometry to investigate how attentional allocation affects the distributed neural representation of animal behavior and taxonomy. Attending to animal behavior transiently increased the discriminability of distributed population codes for observed actions in anterior intraparietal, pericentral, and ventral temporal cortices. Attending to animal taxonomy while viewing the same stimuli increased the discriminability of distributed animal category representations in ventral temporal cortex. For both tasks, attention selectively enhanced the discriminability of response patterns along behaviorally relevant dimensions. These findings suggest that behavioral goals alter how the brain extracts semantic features from the visual world. Attention effectively disentangles population responses for downstream read-out by sculpting representational geometry in late-stage perceptual areas.
Assuntos
Atenção/fisiologia , Encéfalo/fisiologia , Percepção de Movimento/fisiologia , Semântica , Adulto , Encéfalo/diagnóstico por imagem , Mapeamento Encefálico/métodos , Feminino , Humanos , Imageamento por Ressonância Magnética , Masculino , Modelos Estatísticos , Vias Neurais/diagnóstico por imagem , Vias Neurais/fisiologia , Testes Neuropsicológicos , Reconhecimento Visual de Modelos/fisiologiaRESUMO
UNLABELLED: Common or folk knowledge about animals is dominated by three dimensions: (1) level of cognitive complexity or "animacy;" (2) dangerousness or "predacity;" and (3) size. We investigated the neural basis of the perceived dangerousness or aggressiveness of animals, which we refer to more generally as "perception of threat." Using functional magnetic resonance imaging (fMRI), we analyzed neural activity evoked by viewing images of animal categories that spanned the dissociable semantic dimensions of threat and taxonomic class. The results reveal a distributed network for perception of threat extending along the right superior temporal sulcus. We compared neural representational spaces with target representational spaces based on behavioral judgments and a computational model of early vision and found a processing pathway in which perceived threat emerges as a dominant dimension: whereas visual features predominate in early visual cortex and taxonomy in lateral occipital and ventral temporal cortices, these dimensions fall away progressively from posterior to anterior temporal cortices, leaving threat as the dominant explanatory variable. Our results suggest that the perception of threat in the human brain is associated with neural structures that underlie perception and cognition of social actions and intentions, suggesting a broader role for these regions than has been thought previously, one that includes the perception of potential threat from agents independent of their biological class. SIGNIFICANCE STATEMENT: For centuries, philosophers have wondered how the human mind organizes the world into meaningful categories and concepts. Today this question is at the core of cognitive science, but our focus has shifted to understanding how knowledge manifests in dynamic activity of neural systems in the human brain. This study advances the young field of empirical neuroepistemology by characterizing the neural systems engaged by an important dimension in our cognitive representation of the animal kingdom ontological subdomain: how the brain represents the perceived threat, dangerousness, or "predacity" of animals. Our findings reveal how activity for domain-specific knowledge of animals overlaps the social perception networks of the brain, suggesting domain-general mechanisms underlying the representation of conspecifics and other animals.
Assuntos
Encéfalo/fisiologia , Conectoma , Comportamento Predatório/classificação , Percepção Visual , Adulto , Anfíbios/fisiologia , Animais , Artrópodes/fisiologia , Encéfalo/citologia , Cognição , Feminino , Humanos , Imageamento por Ressonância Magnética , Masculino , Neurônios/fisiologia , Répteis/fisiologiaRESUMO
Complex systems are described according to two central dimensions: (a) the randomness of their output, quantified via entropy; and (b) their complexity, which reflects the organization of a system's generators. Whereas some approaches hold that complexity can be reduced to uncertainty or entropy, an axiom of complexity science is that signals with very high or very low entropy are generated by relatively non-complex systems, while complex systems typically generate outputs with entropy peaking between these two extremes. In understanding their environment, individuals would benefit from coding for both input entropy and complexity; entropy indexes uncertainty and can inform probabilistic coding strategies, whereas complexity reflects a concise and abstract representation of the underlying environmental configuration, which can serve independent purposes, e.g., as a template for generalization and rapid comparisons between environments. Using functional neuroimaging, we demonstrate that, in response to passively processed auditory inputs, functional integration patterns in the human brain track both the entropy and complexity of the auditory signal. Connectivity between several brain regions scaled monotonically with input entropy, suggesting sensitivity to uncertainty, whereas connectivity between other regions tracked entropy in a convex manner consistent with sensitivity to input complexity. These findings suggest that the human brain simultaneously tracks the uncertainty of sensory data and effectively models their environmental generators.
Assuntos
Percepção Auditiva/fisiologia , Mapeamento Encefálico/métodos , Encéfalo/fisiologia , Entropia , Vias Neurais/fisiologia , Adulto , Feminino , Humanos , Processamento de Imagem Assistida por Computador/métodos , Imageamento por Ressonância Magnética , Masculino , Modelos NeurológicosRESUMO
Storytelling-an ancient way for humans to share individual experiences with others-has been found to induce neural synchronization among listeners. In our exploration of the dynamic fluctuations in listener-listener (LL) coupling throughout stories, we uncover a significant correlation between LL and lag-speaker-listener (lag-SL) couplings over time. Using the analogy of neural pattern (dis)similarity as distances between participants, we term this phenomenon the "herding effect": like a shepherd guiding a group of sheep, the more closely listeners follow the speaker's prior brain activity patterns (higher lag-SL similarity), the more tightly they cluster together (higher LL similarity). This herding effect is particularly pronounced in brain regions where neural synchronization among listeners tracks with behavioral ratings of narrative engagement, highlighting the mediating role of narrative content in the observed multi-brain neural coupling dynamics. By integrating LL and SL neural couplings, this study illustrates how unfolding stories shape a dynamic multi-brain functional network and how the configuration of this network may be associated with moment-by-moment efficacy of communication.
RESUMO
When processing language, the brain is thought to deploy specialized computations to construct meaning from complex linguistic structures. Recently, artificial neural networks based on the Transformer architecture have revolutionized the field of natural language processing. Transformers integrate contextual information across words via structured circuit computations. Prior work has focused on the internal representations ("embeddings") generated by these circuits. In this paper, we instead analyze the circuit computations directly: we deconstruct these computations into the functionally-specialized "transformations" that integrate contextual information across words. Using functional MRI data acquired while participants listened to naturalistic stories, we first verify that the transformations account for considerable variance in brain activity across the cortical language network. We then demonstrate that the emergent computations performed by individual, functionally-specialized "attention heads" differentially predict brain activity in specific cortical regions. These heads fall along gradients corresponding to different layers and context lengths in a low-dimensional cortical space.
Assuntos
Mapeamento Encefálico , Encéfalo , Idioma , Imageamento por Ressonância Magnética , Redes Neurais de Computação , Humanos , Encéfalo/fisiologia , Encéfalo/diagnóstico por imagem , Masculino , Feminino , Adulto , Adulto Jovem , Modelos Neurológicos , Processamento de Linguagem NaturalRESUMO
Recent research has used large language models (LLMs) to study the neural basis of naturalistic language processing in the human brain. LLMs have rapidly grown in complexity, leading to improved language processing capabilities. However, neuroscience researchers haven't kept up with the quick progress in LLM development. Here, we utilized several families of transformer-based LLMs to investigate the relationship between model size and their ability to capture linguistic information in the human brain. Crucially, a subset of LLMs were trained on a fixed training set, enabling us to dissociate model size from architecture and training set size. We used electrocorticography (ECoG) to measure neural activity in epilepsy patients while they listened to a 30-minute naturalistic audio story. We fit electrode-wise encoding models using contextual embeddings extracted from each hidden layer of the LLMs to predict word-level neural signals. In line with prior work, we found that larger LLMs better capture the structure of natural language and better predict neural activity. We also found a log-linear relationship where the encoding performance peaks in relatively earlier layers as model size increases. We also observed variations in the best-performing layer across different brain regions, corresponding to an organized language processing hierarchy.
RESUMO
Effective communication hinges on a mutual understanding of word meaning in different contexts. We recorded brain activity using electrocorticography during spontaneous, face-to-face conversations in five pairs of epilepsy patients. We developed a model-based coupling framework that aligns brain activity in both speaker and listener to a shared embedding space from a large language model (LLM). The context-sensitive LLM embeddings allow us to track the exchange of linguistic information, word by word, from one brain to another in natural conversations. Linguistic content emerges in the speaker's brain before word articulation and rapidly re-emerges in the listener's brain after word articulation. The contextual embeddings better capture word-by-word neural alignment between speaker and listener than syntactic and articulatory models. Our findings indicate that the contextual embeddings learned by LLMs can serve as an explicit numerical model of the shared, context-rich meaning space humans use to communicate their thoughts to one another.