Neural Foundations of Mental Simulation: Future Prediction of Latent Representations on Dynamic Scenes.

Humans and animals have a rich and flexible understanding of the physical world, which enables them to infer the underlying dynamical trajectories of objects and events, plausible future states, and use that to plan and anticipate the consequences of actions. However, the neural mechanisms underlying these computations are unclear. We combine a goal-driven modeling approach with dense neurophysiological data and high-throughput human behavioral readouts that contain thousands of comparisons to directly impinge on this question. Specifically, we construct and evaluate several classes of sensory-cognitive networks to predict the future state of rich, ethologically-relevant environments, ranging from self-supervised end-to-end models with pixel-wise or object-slot objectives, to models that future predict in the latent space of purely static image-pretrained or dynamic video-pretrained foundation models. We find that "scale is not all you need", and that many state-of-the-art machine learning models fail to perform well on our neural and behavioral benchmarks for future prediction. In fact, only one class of models matches these data well overall. We find that neural responses are currently best predicted by models trained to predict the future state of their environment in the latent space of pretrained foundation models optimized for dynamic scenes in a self-supervised manner. These models also approach the neurons' ability to predict the environmental state variables that are visually hidden from view, despite not being explicitly trained to do so. Finally, we find that not all foundation model latents are equal. Notably, models that future predict in the latent space of video foundation models that are optimized to support a diverse range of egocentric sensorimotor tasks, reasonably match both human behavioral error patterns and neural dynamics across all environmental scenarios that we were able to test. Overall, these findings suggest that the neural mechanisms and behaviors of primate mental simulation have strong inductive biases associated with them, and are thus far most consistent with being optimized to future predict on reusable visual representations that are useful for Embodied AI more generally.

Recurrent networks endowed with structural priors explain suboptimal animal behavior.

The strategies found by animals facing a new task are determined both by individual experience and by structural priors evolved to leverage the statistics of natural environments. Rats quickly learn to capitalize on the trial sequence correlations of two-alternative forced choice (2AFC) tasks after correct trials but consistently deviate from optimal behavior after error trials. To understand this outcome-dependent gating, we first show that recurrent neural networks (RNNs) trained in the same 2AFC task outperform rats as they can readily learn to use across-trial information both after correct and error trials. We hypothesize that, although RNNs can optimize their behavior in the 2AFC task without any a priori restrictions, rats' strategy is constrained by a structural prior adapted to a natural environment in which rewarded and non-rewarded actions provide largely asymmetric information. When pre-training RNNs in a more ecological task with more than two possible choices, networks develop a strategy by which they gate off the across-trial evidence after errors, mimicking rats' behavior. Population analyses show that the pre-trained networks form an accurate representation of the sequence statistics independently of the outcome in the previous trial. After error trials, gating is implemented by a change in the network dynamics that temporarily decouple the categorization of the stimulus from the across-trial accumulated evidence. Our results suggest that the rats' suboptimal behavior reflects the influence of a structural prior that reacts to errors by isolating the network decision dynamics from the context, ultimately constraining the performance in a 2AFC laboratory task.

Constructing neural network models from brain data reveals representational transformations linked to adaptive behavior.

The human ability to adaptively implement a wide variety of tasks is thought to emerge from the dynamic transformation of cognitive information. We hypothesized that these transformations are implemented via conjunctive activations in "conjunction hubs"-brain regions that selectively integrate sensory, cognitive, and motor activations. We used recent advances in using functional connectivity to map the flow of activity between brain regions to construct a task-performing neural network model from fMRI data during a cognitive control task. We verified the importance of conjunction hubs in cognitive computations by simulating neural activity flow over this empirically-estimated functional connectivity model. These empirically-specified simulations produced above-chance task performance (motor responses) by integrating sensory and task rule activations in conjunction hubs. These findings reveal the role of conjunction hubs in supporting flexible cognitive computations, while demonstrating the feasibility of using empirically-estimated neural network models to gain insight into cognitive computations in the human brain.

Meta-learning synaptic plasticity and memory addressing for continual familiarity detection.

Over the course of a lifetime, we process a continual stream of information. Extracted from this stream, memories must be efficiently encoded and stored in an addressable manner for retrieval. To explore potential mechanisms, we consider a familiarity detection task in which a subject reports whether an image has been previously encountered. We design a feedforward network endowed with synaptic plasticity and an addressing matrix, meta-learned to optimize familiarity detection over long intervals. We find that anti-Hebbian plasticity leads to better performance than Hebbian plasticity and replicates experimental results such as repetition suppression. A combinatorial addressing function emerges, selecting a unique neuron as an index into the synaptic memory matrix for storage or retrieval. Unlike previous models, this network operates continuously and generalizes to intervals it has not been trained on. Our work suggests a biologically plausible mechanism for continual learning and demonstrates an effective application of machine learning for neuroscience discovery.

Towards the next generation of recurrent network models for cognitive neuroscience.

Recurrent neural networks (RNNs) trained with machine learning techniques on cognitive tasks have become a widely accepted tool for neuroscientists. In this short opinion piece, we discuss fundamental challenges faced by the early work of this approach and recent steps to overcome such challenges and build next-generation RNN models for cognition. We propose several essential questions that practitioners of this approach should address to continue to build future generations of RNN models.

Evolving the olfactory system with machine learning.

The convergent evolution of the fly and mouse olfactory system led us to ask whether the anatomic connectivity and functional logic of olfactory circuits would evolve in artificial neural networks trained to perform olfactory tasks. Artificial networks trained to classify odor identity recapitulate the connectivity inherent in the olfactory system. Input units are driven by a single receptor type, and units driven by the same receptor converge to form a glomerulus. Glomeruli exhibit sparse, unstructured connectivity onto a larger expansion layer of Kenyon cells. When trained to both classify odor identity and to impart innate valence onto odors, the network develops independent pathways for identity and valence classification. Thus, the defining features of fly and mouse olfactory systems also evolved in artificial neural networks trained to perform olfactory tasks. This implies that convergent evolution reflects an underlying logic rather than shared developmental principles.

Artificial Neural Networks for Neuroscientists: A Primer.

Artificial neural networks (ANNs) are essential tools in machine learning that have drawn increasing attention in neuroscience. Besides offering powerful techniques for data analysis, ANNs provide a new approach for neuroscientists to build models for complex behaviors, heterogeneous neural activity, and circuit connectivity, as well as to explore optimization in neural systems, in ways that traditional models are not designed for. In this pedagogical Primer, we introduce ANNs and demonstrate how they have been fruitfully deployed to study neuroscientific questions. We first discuss basic concepts and methods of ANNs. Then, with a focus on bringing this mathematical framework closer to neurobiology, we detail how to customize the analysis, structure, and learning of ANNs to better address a wide range of challenges in brain research. To help readers garner hands-on experience, this Primer is accompanied with tutorial-style code in PyTorch and Jupyter Notebook, covering major topics.

Task representations in neural networks trained to perform many cognitive tasks.

The brain has the ability to flexibly perform many tasks, but the underlying mechanism cannot be elucidated in traditional experimental and modeling studies designed for one task at a time. Here, we trained single network models to perform 20 cognitive tasks that depend on working memory, decision making, categorization, and inhibitory control. We found that after training, recurrent units can develop into clusters that are functionally specialized for different cognitive processes, and we introduce a simple yet effective measure to quantify relationships between single-unit neural representations of tasks. Learning often gives rise to compositionality of task representations, a critical feature for cognitive flexibility, whereby one task can be performed by recombining instructions for other tasks. Finally, networks developed mixed task selectivity similar to recorded prefrontal neurons after learning multiple tasks sequentially with a continual-learning technique. This work provides a computational platform to investigate neural representations of many cognitive tasks.

How to study the neural mechanisms of multiple tasks.

Most biological and artificial neural systems are capable of completing multiple tasks. However, the neural mechanism by which multiple tasks are accomplished within the same system is largely unclear. We start by discussing how different tasks can be related, and methods to generate large sets of inter-related tasks to study how neural networks and animals perform multiple tasks. We then argue that there are mechanisms that emphasize either specialization or flexibility. We will review two such neural mechanisms underlying multiple tasks at the neuronal level (modularity and mixed selectivity), and discuss how different mechanisms can emerge depending on training methods in neural networks.

Inter-areal Balanced Amplification Enhances Signal Propagation in a Large-Scale Circuit Model of the Primate Cortex.

Understanding reliable signal transmission represents a notable challenge for cortical systems, which display a wide range of weights of feedforward and feedback connections among heterogeneous areas. We re-examine the question of signal transmission across the cortex in a network model based on mesoscopic directed and weighted inter-areal connectivity data of the macaque cortex. Our findings reveal that, in contrast to purely feedforward propagation models, the presence of long-range excitatory feedback projections could compromise stable signal propagation. Using population rate models as well as a spiking network model, we find that effective signal propagation can be accomplished by balanced amplification across cortical areas while ensuring dynamical stability. Moreover, the activation of prefrontal cortex in our model requires the input strength to exceed a threshold, which is consistent with the ignition model of conscious processing. These findings demonstrate our model as an anatomically realistic platform for investigations of global primate cortex dynamics.

A disinhibitory circuit motif and flexible information routing in the brain.

In the mammalian neocortex, an area typically receives inputs from, and projects to, dozens of other areas. Mechanisms are needed to flexibly route information to the right place at the right time, which we term 'pathway gating'. For instance, a region in your brain that receives signals from both visual and auditory pathways may want to 'gate in' the visual pathway while 'gating out' the auditory pathway when you try to read a book surrounded by people in a noisy café. In this review, we marshall experimental and computational evidence in support of a circuit mechanism for flexible pathway gating realized by a disinhibitory motif. Moreover, recent work shows an increasing preponderance of this disinhibitory motif from sensory areas to association areas of the mammalian cortex. Pathway input gating is briefly compared with alternative or complementary gating mechanisms. Predictions and open questions for future research on this puzzle about the complex brain system will be discussed.

Paradoxical response reversal of top-down modulation in cortical circuits with three interneuron types.

Pyramidal cells and interneurons expressing parvalbumin (PV), somatostatin (SST), and vasoactive intestinal peptide (VIP) show cell-type-specific connectivity patterns leading to a canonical microcircuit across cortex. Experiments recording from this circuit often report counterintuitive and seemingly contradictory findings. For example, the response of SST cells in mouse V1 to top-down behavioral modulation can change its sign when the visual input changes, a phenomenon that we call response reversal. We developed a theoretical framework to explain these seemingly contradictory effects as emerging phenomena in circuits with two key features: interactions between multiple neural populations and a nonlinear neuronal input-output relationship. Furthermore, we built a cortical circuit model which reproduces counterintuitive dynamics observed in mouse V1. Our analytical calculations pinpoint connection properties critical to response reversal, and predict additional novel types of complex dynamics that could be tested in future experiments.

Brain-wide Maps Reveal Stereotyped Cell-Type-Based Cortical Architecture and Subcortical Sexual Dimorphism.

The stereotyped features of neuronal circuits are those most likely to explain the remarkable capacity of the brain to process information and govern behaviors, yet it has not been possible to comprehensively quantify neuronal distributions across animals or genders due to the size and complexity of the mammalian brain. Here we apply our quantitative brain-wide (qBrain) mapping platform to document the stereotyped distributions of mainly inhibitory cell types. We discover an unexpected cortical organizing principle: sensory-motor areas are dominated by output-modulating parvalbumin-positive interneurons, whereas association, including frontal, areas are dominated by input-modulating somatostatin-positive interneurons. Furthermore, we identify local cell type distributions with more cells in the female brain in 10 out of 11 sexually dimorphic subcortical areas, in contrast to the overall larger brains in males. The qBrain resource can be further mined to link stereotyped aspects of neuronal distributions to known and unknown functions of diverse brain regions.

A dendritic disinhibitory circuit mechanism for pathway-specific gating.

While reading a book in a noisy café, how does your brain 'gate in' visual information while filtering out auditory stimuli? Here we propose a mechanism for such flexible routing of information flow in a complex brain network (pathway-specific gating), tested using a network model of pyramidal neurons and three classes of interneurons with connection probabilities constrained by data. We find that if inputs from different pathways cluster on a pyramidal neuron dendrite, a pathway can be gated-on by a disinhibitory circuit motif. The branch-specific disinhibition can be achieved despite dense interneuronal connectivity, even with random connections. Moreover, clustering of input pathways on dendrites can naturally emerge through synaptic plasticity regulated by dendritic inhibition. This gating mechanism in a neural circuit is further demonstrated by performing a context-dependent decision-making task. The model suggests that cognitive flexibility engages top-down signalling of behavioural rule or context that targets specific classes of inhibitory neurons.