A Unified, Scalable Framework for Neural Population Decoding.

Our ability to use deep learning approaches to decipher neural activity would likely benefit from greater scale, in terms of both model size and datasets. However, the integration of many neural recordings into one unified model is challenging, as each recording contains the activity of different neurons from different individual animals. In this paper, we introduce a training framework and architecture designed to model the population dynamics of neural activity across diverse, large-scale neural recordings. Our method first tokenizes individual spikes within the dataset to build an efficient representation of neural events that captures the fine temporal structure of neural activity. We then employ cross-attention and a PerceiverIO backbone to further construct a latent tokenization of neural population activities. Utilizing this architecture and training framework, we construct a large-scale multi-session model trained on large datasets from seven nonhuman primates, spanning over 158 different sessions of recording from over 27,373 neural units and over 100 hours of recordings. In a number of different tasks, we demonstrate that our pretrained model can be rapidly adapted to new, unseen sessions with unspecified neuron correspondence, enabling few-shot performance with minimal labels. This work presents a powerful new approach for building deep learning tools to analyze neural data and stakes out a clear path to training at scale.

Label-free imaging of nuclear membrane for analysis of nuclear import of viral complexes.

HIV-1 enters the nucleus of non-dividing cells through the nuclear pore complex where it integrates into the host genome. The mechanism of HIV-1 nuclear import remains poorly understood. A powerful means to investigate the docking of HIV-1 at the nuclear pore and nuclear import of viral complexes is through single virus tracking in live cells. This approach necessitates fluorescence labeling of HIV-1 particles and the nuclear envelope, which may be challenging, especially in the context of primary cells. Here, we leveraged a deep neural network model for label-free visualization of the nuclear envelope using transmitted light microscopy. A training image set of cells with fluorescently labeled nuclear Lamin B1 (ground truth), along with the corresponding transmitted light images, was acquired and used to train our model to predict the morphology of the nuclear envelope in fixed cells. This protocol yielded accurate predictions of the nuclear membrane and was used in conjunction with virus infection to examine the nuclear entry of fluorescently labeled HIV-1 complexes. Analyses of HIV-1 nuclear import as a function of virus input yielded identical numbers of fluorescent viral complexes per nucleus using the ground truth and predicted nuclear membrane images. We also demonstrate the utility of predicting the nuclear envelope based on transmitted light images for multicolor fluorescence microscopy of infected cells. Importantly, we show that our model can be adapted to predict the nuclear membrane of live cells imaged at 37 °C, making this approach compatible with single virus tracking. Collectively, these findings demonstrate the utility of deep learning approaches for label-free imaging of cellular structures during early stages of virus infection.

Detecting change points in neural population activity with contrastive metric learning.

Finding points in time where the distribution of neural responses changes (change points) is an important step in many neural data analysis pipelines. However, in complex and free behaviors, where we see different types of shifts occurring at different rates, it can be difficult to use existing methods for change point (CP) detection because they can't necessarily handle different types of changes that may occur in the underlying neural distribution. Additionally, response changes are often sparse in high dimensional neural recordings, which can make existing methods detect spurious changes. In this work, we introduce a new approach for finding changes in neural population states across diverse activities and arousal states occurring in free behavior. Our model follows a contrastive learning approach: we learn a metric for CP detection based on maximizing the Sinkhorn divergences of neuron firing rates across two sides of a labeled CP. We apply this method to a 12-hour neural recording of a freely behaving mouse to detect changes in sleep stages and behavior. We show that when we learn a metric, we can better detect change points and also yield insights into which neurons and sub-groups are important for detecting certain types of switches that occur in the brain.

Learning signatures of decision making from many individuals playing the same game.

Human behavior is incredibly complex and the factors that drive decision making-from instinct, to strategy, to biases between individuals-often vary over multiple timescales. In this paper, we design a predictive framework that learns representations to encode an individual's 'behavioral style', i.e. long-term behavioral trends, while simultaneously predicting future actions and choices. The model explicitly separates representations into three latent spaces: the recent past space, the short-term space, and the long-term space where we hope to capture individual differences. To simultaneously extract both global and local variables from complex human behavior, our method combines a multi-scale temporal convolutional network with latent prediction tasks, where we encourage embeddings across the entire sequence, as well as subsets of the sequence, to be mapped to similar points in the latent space. We develop and apply our method to a large-scale behavioral dataset from 1,000 humans playing a 3-armed bandit task, and analyze what our model's resulting embeddings reveal about the human decision making process. In addition to predicting future choices, we show that our model can learn rich representations of human behavior over multiple timescales and provide signatures of differences in individuals.

Half-Hop: A graph upsampling approach for slowing down message passing.

Message passing neural networks have shown a lot of success on graph-structured data. However, there are many instances where message passing can lead to over-smoothing or fail when neighboring nodes belong to different classes. In this work, we introduce a simple yet general framework for improving learning in message passing neural networks. Our approach essentially upsamples edges in the original graph by adding "slow nodes" at each edge that can mediate communication between a source and a target node. Our method only modifies the input graph, making it plug-and-play and easy to use with existing models. To understand the benefits of slowing down message passing, we provide theoretical and empirical analyses. We report results on several supervised and self-supervised benchmarks, and show improvements across the board, notably in heterophilic conditions where adjacent nodes are more likely to have different labels. Finally, we show how our approach can be used to generate augmentations for self-supervised learning, where slow nodes are randomly introduced into different edges in the graph to generate multi-scale views with variable path lengths.

Tauopathy severely disrupts homeostatic set-points in emergent neural dynamics but not in the activity of individual neurons.

The homeostatic regulation of neuronal activity is essential for robust computation; key set-points, such as firing rate, are actively stabilized to compensate for perturbations. From this perspective, the disruption of brain function central to neurodegenerative disease should reflect impairments of computationally essential set-points. Despite connecting neurodegeneration to functional outcomes, the impact of disease on set-points in neuronal activity is unknown. Here we present a comprehensive, theory-driven investigation of the effects of tau-mediated neurodegeneration on homeostatic set-points in neuronal activity. In a mouse model of tauopathy, we examine 27,000 hours of hippocampal recordings during free behavior throughout disease progression. Contrary to our initial hypothesis that tauopathy would impact set-points in spike rate and variance, we found that cell-level set-points are resilient to even the latest stages of disease. Instead, we find that tauopathy disrupts neuronal activity at the network-level, which we quantify using both pairwise measures of neuron interactions as well as measurement of the network's nearness to criticality, an ideal computational regime that is known to be a homeostatic set-point. We find that shifts in network criticality 1) track with symptoms, 2) predict underlying anatomical and molecular pathology, 3) occur in a sleep/wake dependent manner, and 4) can be used to reliably classify an animal's genotype. Our data suggest that the critical set-point is intact, but that homeostatic machinery is progressively incapable of stabilizing hippocampal networks, particularly during waking. This work illustrates how neurodegenerative processes can impact the computational capacity of neurobiological systems, and suggest an important connection between molecular pathology, circuit function, and animal behavior.

A non-oscillatory, millisecond-scale embedding of brain state provides insight into behavior.

Sleep and wake are understood to be slow, long-lasting processes that span the entire brain. Brain states correlate with many neurophysiological changes, yet the most robust and reliable signature of state is enriched in rhythms between 0.1 and 20 Hz. The possibility that the fundamental unit of brain state could be a reliable structure at the scale of milliseconds and microns has not been addressed due to the physical limits associated with oscillation-based definitions. Here, by analyzing high resolution neural activity recorded in 10 anatomically and functionally diverse regions of the murine brain over 24 h, we reveal a mechanistically distinct embedding of state in the brain. Sleep and wake states can be accurately classified from on the order of 100 to 101 ms of neuronal activity sampled from 100 µm of brain tissue. In contrast to canonical rhythms, this embedding persists above 1,000 Hz. This high frequency embedding is robust to substates and rapid events such as sharp wave ripples and cortical ON/OFF states. To ascertain whether such fast and local structure is meaningful, we leveraged our observation that individual circuits intermittently switch states independently of the rest of the brain. Brief state discontinuities in subsets of circuits correspond with brief behavioral discontinuities during both sleep and wake. Our results suggest that the fundamental unit of state in the brain is consistent with the spatial and temporal scale of neuronal computation, and that this resolution can contribute to an understanding of cognition and behavior.

De novo evolution of macroscopic multicellularity.

While early multicellular lineages necessarily started out as relatively simple groups of cells, little is known about how they became Darwinian entities capable of sustained multicellular evolution1-3. Here we investigate this with a multicellularity long-term evolution experiment, selecting for larger group size in the snowflake yeast (Saccharomyces cerevisiae) model system. Given the historical importance of oxygen limitation4, our ongoing experiment consists of three metabolic treatments5-anaerobic, obligately aerobic and mixotrophic yeast. After 600 rounds of selection, snowflake yeast in the anaerobic treatment group evolved to be macroscopic, becoming around 2 × 104 times larger (approximately mm scale) and about 104-fold more biophysically tough, while retaining a clonal multicellular life cycle. This occurred through biophysical adaptation-evolution of increasingly elongate cells that initially reduced the strain of cellular packing and then facilitated branch entanglements that enabled groups of cells to stay together even after many cellular bonds fracture. By contrast, snowflake yeast competing for low oxygen5 remained microscopic, evolving to be only around sixfold larger, underscoring the critical role of oxygen levels in the evolution of multicellular size. Together, this research provides unique insights into an ongoing evolutionary transition in individuality, showing how simple groups of cells overcome fundamental biophysical limitations through gradual, yet sustained, multicellular evolution.

Learning signatures of decision making from many individuals playing the same game.

Human behavior is incredibly complex and the factors that drive decision making--from instinct, to strategy, to biases between individuals--often vary over multiple timescales. In this paper, we design a predictive framework that learns representations to encode an individual's 'behavioral style', i.e. long-term behavioral trends, while simultaneously predicting future actions and choices. The model explicitly separates representations into three latent spaces: the recent past space, the short-term space, and the long-term space where we hope to capture individual differences. To simultaneously extract both global and local variables from complex human behavior, our method combines a multi-scale temporal convolutional network with latent prediction tasks, where we encourage embeddings across the entire sequence, as well as subsets of the sequence, to be mapped to similar points in the latent space. We develop and apply our method to a large-scale behavioral dataset from 1,000 humans playing a 3-armed bandit task, and analyze what our model's resulting embeddings reveal about the human decision making process. In addition to predicting future choices, we show that our model can learn rich representations of human behavior over multiple timescales and provide signatures of differences in individuals.

Transcriptomic cell type structures in vivo neuronal activity across multiple timescales.

Cell type is hypothesized to be a key determinant of a neuron's role within a circuit. Here, we examine whether a neuron's transcriptomic type influences the timing of its activity. We develop a deep-learning architecture that learns features of interevent intervals across timescales (ms to >30 min). We show that transcriptomic cell-class information is embedded in the timing of single neuron activity in the intact brain of behaving animals (calcium imaging and extracellular electrophysiology) as well as in a bio-realistic model of the visual cortex. Further, a subset of excitatory cell types are distinguishable but can be classified with higher accuracy when considering cortical layer and projection class. Finally, we show that computational fingerprints of cell types may be universalizable across structured stimuli and naturalistic movies. Our results indicate that transcriptomic class and type may be imprinted in the timing of single neuron activity across diverse stimuli.

Seeing the forest and the tree: Building representations of both individual and collective dynamics with transformers.

Complex time-varying systems are often studied by abstracting away from the dynamics of individual components to build a model of the population-level dynamics from the start. However, when building a population-level description, it can be easy to lose sight of each individual and how they contribute to the larger picture. In this paper, we present a novel transformer architecture for learning from time-varying data that builds descriptions of both the individual as well as the collective population dynamics. Rather than combining all of our data into our model at the onset, we develop a separable architecture that operates on individual time-series first before passing them forward; this induces a permutation-invariance property and can be used to transfer across systems of different size and order. After demonstrating that our model can be applied to successfully recover complex interactions and dynamics in many-body systems, we apply our approach to populations of neurons in the nervous system. On neural activity datasets, we show that our model not only yields robust decoding performance, but also provides impressive performance in transfer across recordings of different animals without any neuron-level correspondence. By enabling flexible pre-training that can be transferred to neural recordings of different size and order, our work provides a first step towards creating a foundation model for neural decoding.

MTNeuro: A Benchmark for Evaluating Representations of Brain Structure Across Multiple Levels of Abstraction.

There are multiple scales of abstraction from which we can describe the same image, depending on whether we are focusing on fine-grained details or a more global attribute of the image. In brain mapping, learning to automatically parse images to build representations of both small-scale features (e.g., the presence of cells or blood vessels) and global properties of an image (e.g., which brain region the image comes from) is a crucial and open challenge. However, most existing datasets and benchmarks for neuroanatomy consider only a single downstream task at a time. To bridge this gap, we introduce a new dataset, annotations, and multiple downstream tasks that provide diverse ways to readout information about brain structure and architecture from the same image. Our multi-task neuroimaging benchmark (MTNeuro) is built on volumetric, micrometer-resolution X-ray microtomography images spanning a large thalamocortical section of mouse brain, encompassing multiple cortical and subcortical regions. We generated a number of different prediction challenges and evaluated several supervised and self-supervised models for brain-region prediction and pixel-level semantic segmentation of microstructures. Our experiments not only highlight the rich heterogeneity of this dataset, but also provide insights into how self-supervised approaches can be used to learn representations that capture multiple attributes of a single image and perform well on a variety of downstream tasks. Datasets, code, and pre-trained baseline models are provided at: https://mtneuro.github.io/.

Making transport more robust and interpretable by moving data through a small number of anchor points.

Optimal transport (OT) is a widely used technique for distribution alignment, with applications throughout the machine learning, graphics, and vision communities. Without any additional structural assumptions on transport, however, OT can be fragile to outliers or noise, especially in high dimensions. Here, we introduce Latent Optimal Transport (LOT), a new approach for OT that simultaneously learns low-dimensional structure in data while leveraging this structure to solve the alignment task. The idea behind our approach is to learn two sets of "anchors" that constrain the flow of transport between a source and target distribution. In both theoretical and empirical studies, we show that LOT regularizes the rank of transport and makes it more robust to outliers and the sampling density. We show that by allowing the source and target to have different anchors, and using LOT to align the latent spaces between anchors, the resulting transport plan has better structural interpretability and highlights connections between both the individual data points and the local geometry of the datasets.

Drop, Swap, and Generate: A Self-Supervised Approach for Generating Neural Activity.

Meaningful and simplified representations of neural activity can yield insights into how and what information is being processed within a neural circuit. However, without labels, finding representations that reveal the link between the brain and behavior can be challenging. Here, we introduce a novel unsupervised approach for learning disentangled representations of neural activity called Swap-VAE. Our approach combines a generative modeling framework with an instance-specific alignment loss that tries to maximize the representational similarity between transformed views of the input (brain state). These transformed (or augmented) views are created by dropping out neurons and jittering samples in time, which intuitively should lead the network to a representation that maintains both temporal consistency and invariance to the specific neurons used to represent the neural state. Through evaluations on both synthetic data and neural recordings from hundreds of neurons in different primate brains, we show that it is possible to build representations that disentangle neural datasets along relevant latent dimensions linked to behavior.

A three-dimensional thalamocortical dataset for characterizing brain heterogeneity.

Neural microarchitecture is heterogeneous, varying both across and within brain regions. The consistent identification of regions of interest is one of the most critical aspects in examining neurocircuitry, as these structures serve as the vital landmarks with which to map brain pathways. Access to continuous, three-dimensional volumes that span multiple brain areas not only provides richer context for identifying such landmarks, but also enables a deeper probing of the microstructures within. Here, we describe a three-dimensional X-ray microtomography imaging dataset of a well-known and validated thalamocortical sample, encompassing a range of cortical and subcortical structures from the mouse brain . In doing so, we provide the field with access to a micron-scale anatomical imaging dataset ideal for studying heterogeneity of neural structure.

Generative models and abstractions for large-scale neuroanatomy datasets.

Neural datasets are increasing rapidly in both resolution and volume. In neuroanatomy, this trend has been accelerated by innovations in imaging technology. As full datasets are impractical and unnecessary for many applications, it is important to identify abstractions that distill useful features of neural structure, organization, and anatomy. In this review article, we discuss several such abstractions and highlight recent algorithmic advances in working with these models. In particular, we discuss the use of generative models in neuroanatomy; such models may be considered 'meta-abstractions' that capture distributions over other abstractions.

Approximating Cellular Densities from High-Resolution Neuroanatomical Imaging Data.

Characterizing the cellular architecture (cytoar-chitecture) of tissues in the nervous system is critical for modeling disease progression, defining boundaries between brain regions, and informing models of neural information processing. Extracting this information from anatomical data requires the expertise of trained neuroanatomists, and is a challenging task for inexperienced analysts. To address this need, we present an unbiased, automated method to estimate cellular density of retinal and neocortical datasets. Our approach leverages the fact that within retinal and neurocortical datasets, cells are organized into "layers" of constant density to approximate cytoarchitecture with a small number of known basis elements. We introduce methods for patch extraction, cell detection, and sparse approximation of inhomogeneous Poisson processes to differentiate changes in cellular densities and detect layers. Our results demonstrate the feasibility of using automation to reveal the cytoarchitecture of large-scale biological samples.

Latent Factors and Dynamics in Motor Cortex and Their Application to Brain-Machine Interfaces.

In the 1960s, Evarts first recorded the activity of single neurons in motor cortex of behaving monkeys (Evarts, 1968). In the 50 years since, great effort has been devoted to understanding how single neuron activity relates to movement. Yet these single neurons exist within a vast network, the nature of which has been largely inaccessible. With advances in recording technologies, algorithms, and computational power, the ability to study these networks is increasing exponentially. Recent experimental results suggest that the dynamical properties of these networks are critical to movement planning and execution. Here we discuss this dynamical systems perspective and how it is reshaping our understanding of the motor cortices. Following an overview of key studies in motor cortex, we discuss techniques to uncover the "latent factors" underlying observed neural population activity. Finally, we discuss efforts to use these factors to improve the performance of brain-machine interfaces, promising to make these findings broadly relevant to neuroengineering as well as systems neuroscience.