Gromov-Wasserstein unsupervised alignment reveals structural correspondences between the color similarity structures of humans and large language models.

Large Language Models (LLMs), such as the General Pre-trained Transformer (GPT), have shown remarkable performance in various cognitive tasks. However, it remains unclear whether these models have the ability to accurately infer human perceptual representations. Previous research has addressed this question by quantifying correlations between similarity response patterns of humans and LLMs. Correlation provides a measure of similarity, but it relies pre-defined item labels and does not distinguish category- and item- level similarity, falling short of characterizing detailed structural correspondence between humans and LLMs. To assess their structural equivalence in more detail, we propose the use of an unsupervised alignment method based on Gromov-Wasserstein optimal transport (GWOT). GWOT allows for the comparison of similarity structures without relying on pre-defined label correspondences and can reveal fine-grained structural similarities and differences that may not be detected by simple correlation analysis. Using a large dataset of similarity judgments of 93 colors, we compared the color similarity structures of humans (color-neurotypical and color-atypical participants) and two GPT models (GPT-3.5 and GPT-4). Our results show that the similarity structure of color-neurotypical participants can be remarkably well aligned with that of GPT-4 and, to a lesser extent, to that of GPT-3.5. These results contribute to the methodological advancements of comparing LLMs with human perception, and highlight the potential of unsupervised alignment methods to reveal detailed structural correspondences.

Association of bidirectional network cores in the brain with conscious perception and cognition.

The brain comprises a complex network of interacting regions. To understand the roles and mechanisms of this complex network, its structural features related to specific cognitive functions need to be elucidated. Among such relationships, recent developments in neuroscience highlight the link between network bidirectionality and conscious perception. Given the essential roles of both feedforward and feedback signals in conscious perception, it is surmised that subnetworks with bidirectional interactions are critical. However, the link between such subnetworks and conscious perception remains unclear due to the network's complexity. In this study, we propose a framework for extracting subnetworks with strong bidirectional interactions-termed the "cores" of a network-from brain activity. We applied this framework to resting-state and task-based fMRI data to identify regions forming strongly bidirectional cores. We then explored the association of these cores with conscious perception and cognitive functions. The central cores predominantly included cerebral cortical regions, which are crucial for conscious perception, rather than subcortical regions. Furthermore, the cores were composed of previously reported regions in which electrical stimulation altered conscious perception. These results suggest a link between the bidirectional cores and conscious perception. A meta-analysis and comparison of the core structure with a cortical functional connectivity gradient suggested that the central cores were related to lower-order sensorimotor functions. An ablation study emphasized the importance of incorporating bidirectionality, not merely interaction strength for these outcomes. The proposed framework provides novel insight into the roles of network cores with strong bidirectional interactions in conscious perception and lower-order sensorimotor functions.

Optimal Control Costs of Brain State Transitions in Linear Stochastic Systems.

The brain is a system that performs numerous functions by controlling its states. Quantifying the cost of this control is essential as it reveals how the brain can be controlled based on the minimization of the control cost, and which brain regions are most important to the optimal control of transitions. Despite its great potential, the current control paradigm in neuroscience uses a deterministic framework and is therefore unable to consider stochasticity, severely limiting its application to neural data. Here, to resolve this limitation, we propose a novel framework for the evaluation of control costs based on a linear stochastic model. Following our previous work, we quantified the optimal control cost as the minimal Kullback-Leibler divergence between the uncontrolled and controlled processes. In the linear model, we established an analytical expression for minimal cost and showed that we can decompose it into the cost for controlling the mean and covariance of brain activity. To evaluate the utility of our novel framework, we examined the significant brain regions in the optimal control of transitions from the resting state to seven cognitive task states in human whole-brain imaging data of either sex. We found that, in realizing the different transitions, the lower visual areas commonly played a significant role in controlling the means, while the posterior cingulate cortex commonly played a significant role in controlling the covariances.SIGNIFICANCE STATEMENT The brain performs many cognitive functions by controlling its states. Quantifying the cost of this control is essential as it reveals how the brain can be optimally controlled in terms of the cost, and which brain regions are most important to the optimal control of transitions. Here, we built a novel framework to quantify control cost that takes account of stochasticity of neural activity, which is ignored in previous studies. We established the analytical expression of the stochastic control cost, which enables us to compute the cost in high-dimensional neural data. We identified the significant brain regions for the optimal control in cognitive tasks in human whole-brain imaging data.

Bidirectionally connected cores in a mouse connectome: towards extracting the brain subnetworks essential for consciousness.

Where in the brain consciousness resides remains unclear. It has been suggested that the subnetworks supporting consciousness should be bidirectionally (recurrently) connected because both feed-forward and feedback processing are necessary for conscious experience. Accordingly, evaluating which subnetworks are bidirectionally connected and the strength of these connections would likely aid the identification of regions essential to consciousness. Here, we propose a method for hierarchically decomposing a network into cores with different strengths of bidirectional connection, as a means of revealing the structure of the complex brain network. We applied the method to a whole-brain mouse connectome. We found that cores with strong bidirectional connections consisted of regions presumably essential to consciousness (e.g. the isocortical and thalamic regions, and claustrum) and did not include regions presumably irrelevant to consciousness (e.g. cerebellum). Contrarily, we could not find such correspondence between cores and consciousness when we applied other simple methods that ignored bidirectionality. These findings suggest that our method provides a novel insight into the relation between bidirectional brain network structures and consciousness.

Quantifying brain state transition cost via Schrödinger Bridge.

Quantifying brain state transition cost is a fundamental problem in systems neuroscience. Previous studies utilized network control theory to measure the cost by considering a neural system as a deterministic dynamical system. However, this approach does not capture the stochasticity of neural systems, which is important for accurately quantifying brain state transition cost. Here, we propose a novel framework based on optimal control in stochastic systems. In our framework, we quantify the transition cost as the Kullback-Leibler divergence from an uncontrolled transition path to the optimally controlled path, which is known as Schrödinger Bridge. To test its utility, we applied this framework to functional magnetic resonance imaging data from the Human Connectome Project and computed the brain state transition cost in cognitive tasks. We demonstrate correspondence between brain state transition cost and the difficulty of tasks. The results suggest that our framework provides a general theoretical tool for investigating cognitive functions from the viewpoint of transition cost.

Information flow in the rat thalamo-cortical system: spontaneous vs. stimulus-evoked activities.

The interaction between the thalamus and sensory cortex plays critical roles in sensory processing. Previous studies have revealed pathway-specific synaptic properties of thalamo-cortical connections. However, few studies to date have investigated how each pathway routes moment-to-moment information. Here, we simultaneously recorded neural activity in the auditory thalamus (or ventral division of the medial geniculate body; MGv) and primary auditory cortex (A1) with a laminar resolution in anesthetized rats. Transfer entropy (TE) was used as an information theoretic measure to operationalize "information flow". Our analyses confirmed that communication between the thalamus and cortex was strengthened during presentation of auditory stimuli. In the resting state, thalamo-cortical communications almost disappeared, whereas intracortical communications were strengthened. The predominant source of information was the MGv at the onset of stimulus presentation and layer 5 during spontaneous activity. In turn, MGv was the major recipient of information from layer 6. TE suggested that a small but significant population of MGv-to-A1 pairs was "information-bearing," whereas A1-to-MGv pairs typically exhibiting small effects played modulatory roles. These results highlight the capability of TE analyses to unlock novel avenues for bridging the gap between well-established anatomical knowledge of canonical microcircuits and physiological correlates via the concept of dynamic information flow.

Fast, cell-resolution, contiguous-wide two-photon imaging to reveal functional network architectures across multi-modal cortical areas.

Fast and wide field-of-view imaging with single-cell resolution, high signal-to-noise ratio, and no optical aberrations have the potential to inspire new avenues of investigations in biology. However, such imaging is challenging because of the inevitable tradeoffs among these parameters. Here, we overcome these tradeoffs by combining a resonant scanning system, a large objective with low magnification and high numerical aperture, and highly sensitive large-aperture photodetectors. The result is a practically aberration-free, fast-scanning high optical invariant two-photon microscopy (FASHIO-2PM) that enables calcium imaging from a large network composed of ∼16,000 neurons at 7.5 Hz from a 9 mm2 contiguous image plane, including more than 10 sensory-motor and higher-order areas of the cerebral cortex in awake mice. Network analysis based on single-cell activities revealed that the brain exhibits small-world rather than scale-free behavior. The FASHIO-2PM is expected to enable studies on biological dynamics by simultaneously monitoring macroscopic activities and their compositional elements.

Efficient search for informational cores in complex systems: Application to brain networks.

An important step in understanding the nature of the brain is to identify "cores" in the brain network, where brain areas strongly interact with each other. Cores can be considered as essential sub-networks for brain functions. In the last few decades, an information-theoretic approach to identifying cores has been developed. In this approach, interactions between parts are measured by an information loss function, which quantifies how much information would be lost if interactions between parts were removed. Then, a core called a "complex" is defined as a subsystem wherein the amount of information loss is locally maximal. Although identifying complexes can be a novel and useful approach, its application is practically impossible because computation time grows exponentially with system size. Here we propose a fast and exact algorithm for finding complexes, called Hierarchical Partitioning for Complex search (HPC). HPC hierarchically partitions systems to narrow down candidates for complexes. The computation time of HPC is polynomial, enabling us to find complexes in large systems (up to several hundred) in a practical amount of time. We prove that HPC is exact when an information loss function satisfies a mathematical property, monotonicity. We show that mutual information is one such information loss function. We also show that a broad class of submodular functions can be considered as such information loss functions, indicating the expandability of our framework to the class. We applied HPC to electrocorticogram recordings from a monkey and demonstrated that HPC revealed temporally stable and characteristic complexes.

An Information Theoretic Approach to Reveal the Formation of Shared Representations.

Modality-invariant categorical representations, i.e., shared representation, is thought to play a key role in learning to categorize multi-modal information. We have investigated how a bimodal autoencoder can form a shared representation in an unsupervised manner with multi-modal data. We explored whether altering the depth of the network and mixing the multi-modal inputs at the input layer affect the development of the shared representations. Based on the activation of units in the hidden layers, we classified them into four different types: visual cells, auditory cells, inconsistent visual and auditory cells, and consistent visual and auditory cells. Our results show that the number and quality of the last type (i.e., shared representation) significantly differ depending on the depth of the network and are enhanced when the network receives mixed inputs as opposed to separate inputs for each modality, as occurs in typical two-stage frameworks. In the present work, we present a way to utilize information theory to understand the abstract representations formed in the hidden layers of the network. We believe that such an information theoretic approach could potentially provide insights into the development of more efficient and cost-effective ways to train neural networks using qualitative measures of the representations that cannot be captured by analyzing only the final outputs of the networks.

A general spectral decomposition of causal influences applied to integrated information.

BACKGROUND: Quantifying interactions among many neurons is fundamental to understanding system-level phenomena such as attention, learning and even conscious experience. Causal influences in the brain, quantified as integrated information, are thought to support subjective conscious experience. Recent empirical work has shown that the spectral decomposition of causal influences, for example using Granger causality, can reveal frequency-specific influences that are not observed in the time domain. However, a spectral decomposition of integrated information has not been put forward, limiting its adoption for analyzing neural data. NEW METHOD: We present a general and flexible framework for deriving the spectral decomposition of causal influences in autoregressive processes. RESULTS: We use the framework to derive a spectral decomposition of integrated information. We show that other well-known measures, including Granger causality, can be derived using the same framework. Using simulations, we demonstrate a complex interplay between the spectral decomposition of integrated information and other measures that is not observed in the time domain. COMPARISON WITH EXISTING METHODS: This paper provides a spectral decomposition of integrated information for the first time. Although a spectral decomposition of Granger causality has been derived, that approach is only applicable to uni-directional causal influences, not multi-directional causal influences as required for integrated information. CONCLUSIONS: Our novel framework can be used to derive the spectral decomposition of uni- and multi-directional measures of causal influences. We use this framework to derive a spectral decomposition of integrated information, paving the way for better understanding how frequency-specific causal influences in the brain relate to cognition.