SLEAP: A deep learning system for multi-animal pose tracking.

The desire to understand how the brain generates and patterns behavior has driven rapid methodological innovation in tools to quantify natural animal behavior. While advances in deep learning and computer vision have enabled markerless pose estimation in individual animals, extending these to multiple animals presents unique challenges for studies of social behaviors or animals in their natural environments. Here we present Social LEAP Estimates Animal Poses (SLEAP), a machine learning system for multi-animal pose tracking. This system enables versatile workflows for data labeling, model training and inference on previously unseen data. SLEAP features an accessible graphical user interface, a standardized data model, a reproducible configuration system, over 30 model architectures, two approaches to part grouping and two approaches to identity tracking. We applied SLEAP to seven datasets across flies, bees, mice and gerbils to systematically evaluate each approach and architecture, and we compare it with other existing approaches. SLEAP achieves greater accuracy and speeds of more than 800 frames per second, with latencies of less than 3.5 ms at full 1,024 × 1,024 image resolution. This makes SLEAP usable for real-time applications, which we demonstrate by controlling the behavior of one animal on the basis of the tracking and detection of social interactions with another animal.

Spontaneous activity in cortical neurons is stereotyped and non-Poisson.

Neurons fire even in the absence of sensory stimulation or task demands. Numerous theoretical studies have modeled this spontaneous activity as a Poisson process with uncorrelated intervals between successive spikes and a variance in firing rate equal to the mean. Experimental tests of this hypothesis have yielded variable results, though most have concluded that firing is not Poisson. However, these tests say little about the ways firing might deviate from randomness. Nor are they definitive because many different distributions can have equal means and variances. Here, we characterized spontaneous spiking patterns in extracellular recordings from monkey, cat, and mouse cerebral cortex neurons using rate-normalized spike train autocorrelation functions (ACFs) and a logarithmic timescale. If activity was Poisson, this function should be flat. This was almost never the case. Instead, ACFs had diverse shapes, often with characteristic peaks in the 1-700 ms range. Shapes were stable over time, up to the longest recording periods used (51 min). They did not fall into obvious clusters. ACFs were often unaffected by visual stimulation, though some abruptly changed during brain state shifts. These behaviors may have their origin in the intrinsic biophysics and dendritic anatomy of the cells or in the inputs they receive.

Fully integrated silicon probes for high-density recording of neural activity.

Sensory, motor and cognitive operations involve the coordinated action of large neuronal populations across multiple brain regions in both superficial and deep structures. Existing extracellular probes record neural activity with excellent spatial and temporal (sub-millisecond) resolution, but from only a few dozen neurons per shank. Optical Ca2+ imaging offers more coverage but lacks the temporal resolution needed to distinguish individual spikes reliably and does not measure local field potentials. Until now, no technology compatible with use in unrestrained animals has combined high spatiotemporal resolution with large volume coverage. Here we design, fabricate and test a new silicon probe known as Neuropixels to meet this need. Each probe has 384 recording channels that can programmably address 960 complementary metal-oxide-semiconductor (CMOS) processing-compatible low-impedance TiN sites that tile a single 10-mm long, 70 × 20-µm cross-section shank. The 6 × 9-mm probe base is fabricated with the shank on a single chip. Voltage signals are filtered, amplified, multiplexed and digitized on the base, allowing the direct transmission of noise-free digital data from the probe. The combination of dense recording sites and high channel count yielded well-isolated spiking activity from hundreds of neurons per probe implanted in mice and rats. Using two probes, more than 700 well-isolated single neurons were recorded simultaneously from five brain structures in an awake mouse. The fully integrated functionality and small size of Neuropixels probes allowed large populations of neurons from several brain structures to be recorded in freely moving animals. This combination of high-performance electrode technology and scalable chip fabrication methods opens a path towards recording of brain-wide neural activity during behaviour.

Voltage distributions in extracellular brain recordings.

Extracellular recordings of brain voltage signals have many uses, including the identification of spikes and the characterization of brain states via analysis of local field potential (LFP) or EEG recordings. Though the factors underlying the generation of these signals are time varying and complex, their analysis may be facilitated by an understanding of their statistical properties. To this end, we analyzed the voltage distributions of high-pass extracellular recordings from a variety of structures, including cortex, thalamus, and hippocampus, in monkeys, cats, and rodents. We additionally investigated LFP signals in these recordings as well as human EEG signals obtained during different sleep stages. In all cases, the distributions were accurately described by a Gaussian within ±1.5 standard deviations from zero. Outside these limits, voltages tended to be distributed exponentially, that is, they fell off linearly on log-linear frequency plots, with variable heights and slopes. A possible explanation for this is that sporadically and independently occurring events with individual Gaussian size distributions can sum to produce approximately exponential distributions. For the high-pass recordings, a second explanation results from a model of the noisy behavior of ion channels that produce action potentials via Hodgkin-Huxley kinetics. The distributions produced by this model, relative to the averaged potential, were also Gaussian with approximately exponential flanks. The model also predicted time-varying noise distributions during action potentials, which were observed in the extracellular spike signals. These findings suggest a principled method for detecting spikes in high-pass recordings and transient events in LFP and EEG signals.NEW & NOTEWORTHY We show that the voltage distributions in brain recordings, including high-pass extracellular recordings, the LFP, and human EEG, are accurately described by a Gaussian within ±1.5 standard deviations from zero, with heavy, exponential tails outside these limits. This offers a principled way of setting event detection thresholds in high-pass recordings. It also offers a means for identifying event-like, transient signals in LFP and EEG recordings which may correlate with other neural phenomena.

Nonlinear Decoding of Natural Images From Large-Scale Primate Retinal Ganglion Recordings.

Decoding sensory stimuli from neural activity can provide insight into how the nervous system might interpret the physical environment, and facilitates the development of brain-machine interfaces. Nevertheless, the neural decoding problem remains a significant open challenge. Here, we present an efficient nonlinear decoding approach for inferring natural scene stimuli from the spiking activities of retinal ganglion cells (RGCs). Our approach uses neural networks to improve on existing decoders in both accuracy and scalability. Trained and validated on real retinal spike data from more than 1000 simultaneously recorded macaque RGC units, the decoder demonstrates the necessity of nonlinear computations for accurate decoding of the fine structures of visual stimuli. Specifically, high-pass spatial features of natural images can only be decoded using nonlinear techniques, while low-pass features can be extracted equally well by linear and nonlinear methods. Together, these results advance the state of the art in decoding natural stimuli from large populations of neurons.

LFP clustering in cortex reveals a taxonomy of Up states and near-millisecond, ordered phase-locking in cortical neurons.

During slow-wave sleep and anesthesia, mammalian cortex exhibits a synchronized state during which neurons shift from a largely nonfiring to a firing state, known as an Up-state transition. Up-state transitions may constitute the default activity pattern of the entire cortex (Neske GT. Front Neural Circuits 9: 88, 2016) and could be critical to understanding cortical function, yet the genesis of such transitions and their interaction with single neurons is not well understood. It was recently shown that neurons firing at rates >2 Hz fire spikes in a stereotyped order during Up-state transitions (Luczak A, McNaughton BL, Harris KD. Nat Rev Neurosci 16: 745-755, 2015), yet it is still unknown if Up states are homogeneous and whether spiking order is present in neurons with rates <2 Hz (the majority). Using extracellular recordings from anesthetized cats and mice and from naturally sleeping rats, we show for the first time that Up-state transitions can be classified into several types based on the shape of the local field potential (LFP) during each transition. Individual LFP events could be localized in time to within 1-4 ms, more than an order of magnitude less than in previous studies. The majority of recorded neurons synchronized their firing to within ±5-15 ms relative to each Up-state transition. Simultaneous electrophysiology and wide-field imaging in mouse confirmed that LFP event clusters are cortex-wide phenomena. Our findings show that Up states are of different types and point to the potential importance of temporal order and millisecond-scale signaling by cortical neurons.NEW & NOTEWORTHY During cortical Up-state transitions in sleep and anesthesia, neurons undergo brief periods of increased firing in an order similar to that occurring in awake states. We show that these transitions can be classified into distinct types based on the shape of the local field potential. Transition times can be defined to <5 ms. Most neurons synchronize their firing to within ±5-15 ms of the transitions and fire in a consistent order.

Inferring cortical function in the mouse visual system through large-scale systems neuroscience.

The scientific mission of the Project MindScope is to understand neocortex, the part of the mammalian brain that gives rise to perception, memory, intelligence, and consciousness. We seek to quantitatively evaluate the hypothesis that neocortex is a relatively homogeneous tissue, with smaller functional modules that perform a common computational function replicated across regions. We here focus on the mouse as a mammalian model organism with genetics, physiology, and behavior that can be readily studied and manipulated in the laboratory. We seek to describe the operation of cortical circuitry at the computational level by comprehensively cataloging and characterizing its cellular building blocks along with their dynamics and their cell type-specific connectivities. The project is also building large-scale experimental platforms (i.e., brain observatories) to record the activity of large populations of cortical neurons in behaving mice subject to visual stimuli. A primary goal is to understand the series of operations from visual input in the retina to behavior by observing and modeling the physical transformations of signals in the corticothalamic system. We here focus on the contribution that computer modeling and theory make to this long-term effort.

Unsupervised discovery of family specific vocal usage in the Mongolian gerbil.

In nature, animal vocalizations can provide crucial information about identity, including kinship and hierarchy. However, lab-based vocal behavior is typically studied during brief interactions between animals with no prior social relationship, and under environmental conditions with limited ethological relevance. Here, we address this gap by establishing long-term acoustic recordings from Mongolian gerbil families, a core social group that uses an array of sonic and ultrasonic vocalizations. Three separate gerbil families were transferred to an enlarged environment and continuous 20-day audio recordings were obtained. Using a variational autoencoder (VAE) to quantify 583,237 vocalizations, we show that gerbils exhibit a more elaborate vocal repertoire than has been previously reported and that vocal repertoire usage differs significantly by family. By performing gaussian mixture model clustering on the VAE latent space, we show that families preferentially use characteristic sets of vocal clusters and that these usage preferences remain stable over weeks. Furthermore, gerbils displayed family-specific transitions between vocal clusters. Since gerbils live naturally as extended families in complex underground burrows that are adjacent to other families, these results suggest the presence of a vocal dialect which could be exploited by animals to represent kinship. These findings position the Mongolian gerbil as a compelling animal model to study the neural basis of vocal communication and demonstrates the potential for using unsupervised machine learning with uninterrupted acoustic recordings to gain insights into naturalistic animal behavior.

Behavioral Classification of Sequential Neural Activity Using Time Varying Recurrent Neural Networks.

Shifts in data distribution across time can strongly affect early classification of time-series data. When decoding behavior from neural activity, early detection of behavior may help in devising corrective neural stimulation before the onset of behavior. Recurrent Neural Networks (RNNs) are common models for sequence data. However, standard RNNs are not able to handle data with temporal distributional shifts to guarantee robust classification across time. To enable the network to utilize all temporal features of the neural input data, and to enhance the memory of an RNN, we propose a novel approach: RNNs with time-varying weights, here termed Time-Varying RNNs (TV-RNNs). These models are able to not only predict the class of the time-sequence correctly but also lead to accurate classification earlier in the sequence than standard RNNs. In this work, we focus on early sequential classification of brain-wide neural activity across time using TV-RNNs applied to a variety of neural data from mice and humans, as subjects perform motor tasks. Finally, we explore the contribution of different brain regions on behavior classification using SHapley Additive exPlanation (SHAP) value, and find that the somatosensory and premotor regions play a large role in behavioral classification.

Mesoscale cortex-wide neural dynamics predict self-initiated actions in mice several seconds prior to movement.

Volition - the sense of control or agency over one's voluntary actions - is widely recognized as the basis of both human subjective experience and natural behavior in nonhuman animals. Several human studies have found peaks in neural activity preceding voluntary actions, for example the readiness potential (RP), and some have shown upcoming actions could be decoded even before awareness. Others propose that random processes underlie and explain pre-movement neural activity. Here, we seek to address these issues by evaluating whether pre-movement neural activity in mice contains structure beyond that present in random neural activity. Implementing a self-initiated water-rewarded lever-pull paradigm in mice while recording widefield [Ca++] neural activity we find that cortical activity changes in variance seconds prior to movement and that upcoming lever pulls could be predicted between 3 and 5 s (or more in some cases) prior to movement. We found inhibition of motor cortex starting at approximately 5 s prior to lever pulls and activation of motor cortex starting at approximately 2 s prior to a random unrewarded left limb movement. We show that mice, like humans, are biased toward commencing self-initiated actions during specific phases of neural activity but that the pre-movement neural code changes over time in some mice and is widely distributed as behavior prediction improved when using all vs. single cortical areas. These findings support the presence of structured multi-second neural dynamics preceding self-initiated action beyond that expected from random processes. Our results also suggest that neural mechanisms underlying self-initiated action could be preserved between mice and humans.

Uncovering the effect of different brain regions on behavioral classification using recurrent neural networks.

As our ability to record neural activity from a larger number of brain areas increases, we need to develop tools to understand how this activity is related to ongoing behavior. Recurrent neural networks (RNNs) have been shown to perform successful classification for sequence data. However, they are black box models: once trained, it is difficult to uncover the mechanisms that they are using to classify. In this study, we analyze the effect of RNNs on classifying behavior using a simulated dataset and a widefield neural activity dataset as mice perform a self-initiated behavior. We show that RNNs are comparable to, or outperform, traditional classification methods such as Support Vector Machine (SVM), and can also lead to accurate prediction of behavior. Using dimensionality reduction, we visualize the activity of the RNNs to better understand the classification mechanisms of the RNNs. Finally, we are able to accurately pinpoint the effect of different regions on behavioral classification. This study highlights the utility and interpretability of RNNs while classifying behavior using neural activity from different regions.

BioNet: A Python interface to NEURON for modeling large-scale networks.

There is a significant interest in the neuroscience community in the development of large-scale network models that would integrate diverse sets of experimental data to help elucidate mechanisms underlying neuronal activity and computations. Although powerful numerical simulators (e.g., NEURON, NEST) exist, data-driven large-scale modeling remains challenging due to difficulties involved in setting up and running network simulations. We developed a high-level application programming interface (API) in Python that facilitates building large-scale biophysically detailed networks and simulating them with NEURON on parallel computer architecture. This tool, termed "BioNet", is designed to support a modular workflow whereby the description of a constructed model is saved as files that could be subsequently loaded for further refinement and/or simulation. The API supports both NEURON's built-in as well as user-defined models of cells and synapses. It is capable of simulating a variety of observables directly supported by NEURON (e.g., spikes, membrane voltage, intracellular [Ca++]), as well as plugging in modules for computing additional observables (e.g. extracellular potential). The high-level API platform obviates the time-consuming development of custom code for implementing individual models, and enables easy model sharing via standardized files. This tool will help refocus neuroscientists on addressing outstanding scientific questions rather than developing narrow-purpose modeling code.

A Visual Guide to Sorting Electrophysiological Recordings Using 'SpikeSorter'.

Few stand-alone software applications are available for sorting spikes from recordings made with multi-electrode arrays. Ideally, an application should be user friendly with a graphical user interface, able to read data files in a variety of formats, and provide users with a flexible set of tools giving them the ability to detect and sort extracellular voltage waveforms from different units with some degree of reliability. Previously published spike sorting methods are now available in a software program, SpikeSorter, intended to provide electrophysiologists with a complete set of tools for sorting, starting from raw recorded data file and ending with the export of sorted spikes times. Procedures are automated to the extent this is currently possible. The article explains and illustrates the use of the program. A representative data file is opened, extracellular traces are filtered, events are detected and then clustered. A number of problems that commonly occur during sorting are illustrated, including the artefactual over-splitting of units due to the tendency of some units to fire spikes in pairs where the second spike is significantly smaller than the first, and over-splitting caused by slow variation in spike height over time encountered in some units. The accuracy of SpikeSorter's performance has been tested with surrogate ground truth data and found to be comparable to that of other algorithms in current development.

Mesoscale brain explorer, a flexible python-based image analysis and visualization tool.

Imaging of mesoscale brain activity is used to map interactions between brain regions. This work has benefited from the pioneering studies of Grinvald et al., who employed optical methods to image brain function by exploiting the properties of intrinsic optical signals and small molecule voltage-sensitive dyes. Mesoscale interareal brain imaging techniques have been advanced by cell targeted and selective recombinant indicators of neuronal activity. Spontaneous resting state activity is often collected during mesoscale imaging to provide the basis for mapping of connectivity relationships using correlation. However, the information content of mesoscale datasets is vast and is only superficially presented in manuscripts given the need to constrain measurements to a fixed set of frequencies, regions of interest, and other parameters. We describe a new open source tool written in python, termed mesoscale brain explorer (MBE), which provides an interface to process and explore these large datasets. The platform supports automated image processing pipelines with the ability to assess multiple trials and combine data from different animals. The tool provides functions for temporal filtering, averaging, and visualization of functional connectivity relations using time-dependent correlation. Here, we describe the tool and show applications, where previously published datasets were reanalyzed using MBE.

Mapping cortical mesoscopic networks of single spiking cortical or sub-cortical neurons.

Understanding the basis of brain function requires knowledge of cortical operations over wide-spatial scales, but also within the context of single neurons. In vivo, wide-field GCaMP imaging and sub-cortical/cortical cellular electrophysiology were used in mice to investigate relationships between spontaneous single neuron spiking and mesoscopic cortical activity. We make use of a rich set of cortical activity motifs that are present in spontaneous activity in anesthetized and awake animals. A mesoscale spike-triggered averaging procedure allowed the identification of motifs that are preferentially linked to individual spiking neurons by employing genetically targeted indicators of neuronal activity. Thalamic neurons predicted and reported specific cycles of wide-scale cortical inhibition/excitation. In contrast, spike-triggered maps derived from single cortical neurons yielded spatio-temporal maps expected for regional cortical consensus function. This approach can define network relationships between any point source of neuronal spiking and mesoscale cortical maps.