Brain-machine convergent evolution: Why finding parallels between brain and artificial systems is informative.

Central nervous system neurons manifest a rich diversity of selectivity profiles-whose precise role is still poorly understood. Following the striking success of artificial networks, a major debate has emerged concerning their usefulness in explaining neuronal properties. Here we propose that finding parallels between artificial and neuronal networks is informative precisely because these systems are so different from each other. Our argument is based on an extension of the concept of convergent evolution-well established in biology-to the domain of artificial systems. Applying this concept to different areas and levels of the cortical hierarchy can be a powerful tool for elucidating the functional role of well-known cortical selectivities. Importantly, we further demonstrate that such parallels can uncover novel functionalities by showing that grid cells in the entorhinal cortex can be modeled to function as a set of basis functions in a lossy representation such as the well-known JPEG compression. Thus, contrary to common intuition, here we illustrate that finding parallels with artificial systems provides novel and informative insights, particularly in those cases that are far removed from realistic brain biology.

A shared model-based linguistic space for transmitting our thoughts from brain to brain in natural conversations.

Effective communication hinges on a mutual understanding of word meaning in different contexts. We recorded brain activity using electrocorticography during spontaneous, face-to-face conversations in five pairs of epilepsy patients. We developed a model-based coupling framework that aligns brain activity in both speaker and listener to a shared embedding space from a large language model (LLM). The context-sensitive LLM embeddings allow us to track the exchange of linguistic information, word by word, from one brain to another in natural conversations. Linguistic content emerges in the speaker's brain before word articulation and rapidly re-emerges in the listener's brain after word articulation. The contextual embeddings better capture word-by-word neural alignment between speaker and listener than syntactic and articulatory models. Our findings indicate that the contextual embeddings learned by LLMs can serve as an explicit numerical model of the shared, context-rich meaning space humans use to communicate their thoughts to one another.

A shared linguistic space for transmitting our thoughts from brain to brain in natural conversations.

Effective communication hinges on a mutual understanding of word meaning in different contexts. The embedding space learned by large language models can serve as an explicit model of the shared, context-rich meaning space humans use to communicate their thoughts. We recorded brain activity using electrocorticography during spontaneous, face-to-face conversations in five pairs of epilepsy patients. We demonstrate that the linguistic embedding space can capture the linguistic content of word-by-word neural alignment between speaker and listener. Linguistic content emerged in the speaker's brain before word articulation, and the same linguistic content rapidly reemerged in the listener's brain after word articulation. These findings establish a computational framework to study how human brains transmit their thoughts to one another in real-world contexts.

Retrospective behavioral sampling (RBS): A method to effectively track the cognitive fluctuations driven by naturalistic stimulation.

Everyday experiences are dynamic, driving fluctuations across simultaneous cognitive processes. A key challenge in the study of naturalistic cognition is to disentangle the complexity of these dynamic processes, without altering the natural experience itself. Retrospective behavioral sampling (RBS) is a novel approach to model the cognitive fluctuations corresponding to the time-course of naturalistic stimulation, across a variety of cognitive dimensions. We tested the effectiveness and reliability of RBS in a web-based experiment, in which 53 participants viewed short movies and listened to a story, followed by retrospective reporting. Participants recalled their experience of 55 discrete events from the stimuli, rating their quality of memory, magnitude of surprise, intensity of negative and positive emotions, perceived importance, reflectivity state, and mental time travel. In addition, a subset of the original cohort re-rated their memory of events in a follow-up questionnaire. Results show highly replicable fluctuation patterns across distinct cognitive dimensions, thereby revealing a stimulus-driven experience that is substantially shared among individuals. Remarkably, memory ratings more than a week after stimulation resulted in an almost identical time-course of memorability as measured immediately following stimulation. In addition, idiosyncratic response patterns were preserved across different stimuli, indicating that RBS characterizes individual differences that are stimulus invariant. The current findings highlight the potential of RBS as a powerful tool for measuring dynamic processes of naturalistic cognition. We discuss the promising approach of matching RBS fluctuations with dynamic processes measured via other testing modalities, such as neuroimaging, to study the neural manifestations of naturalistic cognitive processing.

The "Narratives" fMRI dataset for evaluating models of naturalistic language comprehension.

The "Narratives" collection aggregates a variety of functional MRI datasets collected while human subjects listened to naturalistic spoken stories. The current release includes 345 subjects, 891 functional scans, and 27 diverse stories of varying duration totaling ~4.6 hours of unique stimuli (~43,000 words). This data collection is well-suited for naturalistic neuroimaging analysis, and is intended to serve as a benchmark for models of language and narrative comprehension. We provide standardized MRI data accompanied by rich metadata, preprocessed versions of the data ready for immediate use, and the spoken story stimuli with time-stamped phoneme- and word-level transcripts. All code and data are publicly available with full provenance in keeping with current best practices in transparent and reproducible neuroimaging.

The surprising role of the default mode network in naturalistic perception.

The default mode network (DMN) is a group of high-order brain regions recently implicated in processing external naturalistic events, yet it remains unclear what cognitive function it serves. Here we identified the cognitive states predictive of DMN fMRI coactivation. Particularly, we developed a state-fluctuation pattern analysis, matching network coactivations across a short movie with retrospective behavioral sampling of movie events. Network coactivation was selectively correlated with the state of surprise across movie events, compared to all other cognitive states (e.g. emotion, vividness). The effect was exhibited in the DMN, but not dorsal attention or visual networks. Furthermore, surprise was found to mediate DMN coactivations with hippocampus and nucleus accumbens. These unexpected findings point to the DMN as a major hub in high-level prediction-error representations.

Analysis of stimulus-induced brain dynamics during naturalistic paradigms.

Naturalistic stimuli offer promising avenues for investigating brain function across the rich, realistic spectrum of human experiences. Functional magnetic resonance imaging (fMRI) studies of brain activity during naturalistic paradigms have provided new information about dynamic neural processing in ecologically valid contexts. Yet, the complex, uncontrolled nature of such stimuli -- and the resulting mixture of neuronal and physiological responses embedded within the fMRI signals -- present challenges with respect to data analysis and interpretation. In this brief commentary, we discuss methods and open challenges in naturalistic fMRI investigations, with a focus on extracting and interpreting stimulus-induced fMRI signals.

Propagation of Information Along the Cortical Hierarchy as a Function of Attention While Reading and Listening to Stories.

How does attention route information from sensory to high-order areas as a function of task, within the relatively fixed topology of the brain? In this study, participants were simultaneously presented with 2 unrelated stories-one spoken and one written-and asked to attend one while ignoring the other. We used fMRI and a novel intersubject correlation analysis to track the spread of information along the processing hierarchy as a function of task. Processing the unattended spoken (written) information was confined to auditory (visual) cortices. In contrast, attending to the spoken (written) story enhanced the stimulus-selective responses in sensory regions and allowed it to spread into higher-order areas. Surprisingly, we found that the story-specific spoken (written) responses for the attended story also reached secondary visual (auditory) regions of the unattended sensory modality. These results demonstrate how attention enhances the processing of attended input and allows it to propagate across brain areas.

Altered topology of neural circuits in congenital prosopagnosia.

Using a novel, fMRI-based inter-subject functional correlation (ISFC) approach, which isolates stimulus-locked inter-regional correlation patterns, we compared the cortical topology of the neural circuit for face processing in participants with an impairment in face recognition, congenital prosopagnosia (CP), and matched controls. Whereas the anterior temporal lobe served as the major network hub for face processing in controls, this was not the case for the CPs. Instead, this group evinced hyper-connectivity in posterior regions of the visual cortex, mostly associated with the lateral occipital and the inferior temporal cortices. Moreover, the extent of this hyper-connectivity was correlated with the face recognition deficit. These results offer new insights into the perturbed cortical topology in CP, which may serve as the underlying neural basis of the behavioral deficits typical of this disorder. The approach adopted here has the potential to uncover altered topologies in other neurodevelopmental disorders, as well.

Measuring speaker-listener neural coupling with functional near infrared spectroscopy.

The present study investigates brain-to-brain coupling, defined as inter-subject correlations in the hemodynamic response, during natural verbal communication. We used functional near-infrared spectroscopy (fNIRS) to record brain activity of 3 speakers telling stories and 15 listeners comprehending audio recordings of these stories. Listeners' brain activity was significantly correlated with speakers' with a delay. This between-brain correlation disappeared when verbal communication failed. We further compared the fNIRS and functional Magnetic Resonance Imaging (fMRI) recordings of listeners comprehending the same story and found a significant relationship between the fNIRS oxygenated-hemoglobin concentration changes and the fMRI BOLD in brain areas associated with speech comprehension. This correlation between fNIRS and fMRI was only present when data from the same story were compared between the two modalities and vanished when data from different stories were compared; this cross-modality consistency further highlights the reliability of the spatiotemporal brain activation pattern as a measure of story comprehension. Our findings suggest that fNIRS can be used for investigating brain-to-brain coupling during verbal communication in natural settings.

Same Story, Different Story.

Differences in people's beliefs can substantially impact their interpretation of a series of events. In this functional MRI study, we manipulated subjects' beliefs, leading two groups of subjects to interpret the same narrative in different ways. We found that responses in higher-order brain areas-including the default-mode network, language areas, and subsets of the mirror neuron system-tended to be similar among people who shared the same interpretation, but different from those of people with an opposing interpretation. Furthermore, the difference in neural responses between the two groups at each moment was correlated with the magnitude of the difference in the interpretation of the narrative. This study demonstrates that brain responses to the same event tend to cluster together among people who share the same views.

Dynamic reconfiguration of the default mode network during narrative comprehension.

Does the default mode network (DMN) reconfigure to encode information about the changing environment? This question has proven difficult, because patterns of functional connectivity reflect a mixture of stimulus-induced neural processes, intrinsic neural processes and non-neuronal noise. Here we introduce inter-subject functional correlation (ISFC), which isolates stimulus-dependent inter-regional correlations between brains exposed to the same stimulus. During fMRI, we had subjects listen to a real-life auditory narrative and to temporally scrambled versions of the narrative. We used ISFC to isolate correlation patterns within the DMN that were locked to the processing of each narrative segment and specific to its meaning within the narrative context. The momentary configurations of DMN ISFC were highly replicable across groups. Moreover, DMN coupling strength predicted memory of narrative segments. Thus, ISFC opens new avenues for linking brain network dynamics to stimulus features and behaviour.

Coupled neural systems underlie the production and comprehension of naturalistic narrative speech.

Neuroimaging studies of language have typically focused on either production or comprehension of single speech utterances such as syllables, words, or sentences. In this study we used a new approach to functional MRI acquisition and analysis to characterize the neural responses during production and comprehension of complex real-life speech. First, using a time-warp based intrasubject correlation method, we identified all areas that are reliably activated in the brains of speakers telling a 15-min-long narrative. Next, we identified areas that are reliably activated in the brains of listeners as they comprehended that same narrative. This allowed us to identify networks of brain regions specific to production and comprehension, as well as those that are shared between the two processes. The results indicate that production of a real-life narrative is not localized to the left hemisphere but recruits an extensive bilateral network, which overlaps extensively with the comprehension system. Moreover, by directly comparing the neural activity time courses during production and comprehension of the same narrative we were able to identify not only the spatial overlap of activity but also areas in which the neural activity is coupled across the speaker's and listener's brains during production and comprehension of the same narrative. We demonstrate widespread bilateral coupling between production- and comprehension-related processing within both linguistic and nonlinguistic areas, exposing the surprising extent of shared processes across the two systems.

Selective and invariant neural responses to spoken and written narratives.

Linguistic content can be conveyed both in speech and in writing. But how similar is the neural processing when the same real-life information is presented in spoken and written form? Using functional magnetic resonance imaging, we recorded neural responses from human subjects who either listened to a 7 min spoken narrative or read a time-locked presentation of its transcript. Next, within each brain area, we directly compared the response time courses elicited by the written and spoken narrative. Early visual areas responded selectively to the written version, and early auditory areas to the spoken version of the narrative. In addition, many higher-order parietal and frontal areas demonstrated strong selectivity, responding far more reliably to either the spoken or written form of the narrative. By contrast, the response time courses along the superior temporal gyrus and inferior frontal gyrus were remarkably similar for spoken and written narratives, indicating strong modality-invariance of linguistic processing in these circuits. These results suggest that our ability to extract the same information from spoken and written forms arises from a mixture of selective neural processes in early (perceptual) and high-order (control) areas, and modality-invariant responses in linguistic and extra-linguistic areas.

Pre-neuronal morphological processing of object location by individual whiskers.

In the vibrissal system, touch information is conveyed by a receptorless whisker hair to follicle mechanoreceptors, which then provide input to the brain. We examined whether any processing, that is, meaningful transformation, occurs in the whisker itself. Using high-speed videography and tracking the movements of whiskers in anesthetized and behaving rats, we found that whisker-related morphological phase planes, based on angular and curvature variables, can represent the coordinates of object position after contact in a reliable manner, consistent with theoretical predictions. By tracking exposed follicles, we found that the follicle-whisker junction is rigid, which enables direct readout of whisker morphological coding by mechanoreceptors. Finally, we found that our behaving rats pushed their whiskers against objects during localization in a way that induced meaningful morphological coding and, in parallel, improved their localization performance, which suggests a role for pre-neuronal morphological computation in active vibrissal touch.

Collagenous skeleton of the rat mystacial pad.

Anatomical and functional integrity of the rat mystacial pad (MP) is dependent on the intrinsic organization of its extracellular matrix. By using collagen autofluorescence, in the rat MP, we revealed a collagenous skeleton that interconnects whisker follicles, corium, and deep collagen layers. We suggest that this skeleton supports MP tissues, mediates force transmission from muscles to whiskers, facilitates whisker retraction after protraction, and limits MP extensibility.

Temporal and spatial characteristics of vibrissa responses to motor commands.

A mechanistic description of the generation of whisker movements is essential for understanding the control of whisking and vibrissal active touch. We explore how facial-motoneuron spikes are translated, via an intrinsic muscle, to whisker movements. This is achieved by constructing, simulating, and analyzing a computational, biomechanical model of the motor plant, and by measuring spiking to movement transformations at small and large angles using high-precision whisker tracking in vivo. Our measurements revealed a supralinear summation of whisker protraction angles in response to consecutive motoneuron spikes with moderate interspike intervals (5 ms < Deltat < 30 ms). This behavior is explained by a nonlinear transformation from intracellular changes in Ca(2+) concentration to muscle force. Our model predicts the following spatial constraints: (1) Contraction of a single intrinsic muscle results in movement of its two attached whiskers with different amplitudes; the relative amplitudes depend on the resting angles and on the attachment location of the intrinsic muscle on the anterior whisker. Counterintuitively, for a certain range of resting angles, activation of a single intrinsic muscle can lead to a retraction of one of its two attached whiskers. (2) When a whisker is pulled by its two adjacent muscles with similar forces, the protraction amplitude depends only weakly on the resting angle. (3) Contractions of two adjacent muscles sums up linearly for small amplitudes and supralinearly for larger amplitudes. The model provides a direct translation from motoneuron spikes to whisker movements and can serve as a building block in closed-loop motor-sensory models of active touch.

Muscle architecture in the mystacial pad of the rat.

The vibrissal system of the rat is an example of active tactile sensing, and has recently been used as a prototype in construction of touch-oriented robots. Active vibrissal exploration and touch are enabled and controlled by musculature of the mystacial pad. So far, knowledge about motor control of the rat vibrissal system has been extracted from what is known about the vibrissal systems of other species, mainly mice and hamsters, since a detailed description of the musculature of the rat mystacial pad was lacking. In the present work, the musculature of the rat mystacial pad was revealed by slicing the mystacial pad in four different planes, staining of mystacial pad slices for cytochrome oxidase, and tracking spatial organization of mystacial pad muscles in consecutive slices. We found that the rat mystacial pad contains four superficial extrinsic muscles and five parts of the M. nasolabialis profundus. The connection scheme of the three parts of the M. nasolabialis profundus is described here for the first time. These muscles are inserted into the plate of the mystacial pad, and thus, their contraction causes whisker retraction. All the muscles of the rat mystacial pad contained three types of skeletal striated fibers (red, white, and intermediate). Although the entire rat mystacial pad usually functions as unity, our data revealed its structural segmentation into nasal and maxillary subdivisions. The mechanisms of whisking in the rat, and hypotheses concerning biomechanical interactions during whisking, are discussed with respect to the muscle architecture of the rat mystacial pad.

Generalized Stokes parameters shift keying approach to multichip differential phase encoded optical modulation formats.

Conventional and advanced modulation formats that simultaneously modulate two or more of the optical attributes of phase, amplitude, and polarization and/or utilize observation intervals longer than two chips (time slots) are designed by using a unified interpretation as signaling by means of generalized Stokes parameters. In particular, the new paradigm is applied to the recently introduced multichip extension of optical differential phase shift keying.