SLEAP: A deep learning system for multi-animal pose tracking.

The desire to understand how the brain generates and patterns behavior has driven rapid methodological innovation in tools to quantify natural animal behavior. While advances in deep learning and computer vision have enabled markerless pose estimation in individual animals, extending these to multiple animals presents unique challenges for studies of social behaviors or animals in their natural environments. Here we present Social LEAP Estimates Animal Poses (SLEAP), a machine learning system for multi-animal pose tracking. This system enables versatile workflows for data labeling, model training and inference on previously unseen data. SLEAP features an accessible graphical user interface, a standardized data model, a reproducible configuration system, over 30 model architectures, two approaches to part grouping and two approaches to identity tracking. We applied SLEAP to seven datasets across flies, bees, mice and gerbils to systematically evaluate each approach and architecture, and we compare it with other existing approaches. SLEAP achieves greater accuracy and speeds of more than 800 frames per second, with latencies of less than 3.5 ms at full 1,024 × 1,024 image resolution. This makes SLEAP usable for real-time applications, which we demonstrate by controlling the behavior of one animal on the basis of the tracking and detection of social interactions with another animal.

LFP clustering in cortex reveals a taxonomy of Up states and near-millisecond, ordered phase-locking in cortical neurons.

During slow-wave sleep and anesthesia, mammalian cortex exhibits a synchronized state during which neurons shift from a largely nonfiring to a firing state, known as an Up-state transition. Up-state transitions may constitute the default activity pattern of the entire cortex (Neske GT. Front Neural Circuits 9: 88, 2016) and could be critical to understanding cortical function, yet the genesis of such transitions and their interaction with single neurons is not well understood. It was recently shown that neurons firing at rates >2 Hz fire spikes in a stereotyped order during Up-state transitions (Luczak A, McNaughton BL, Harris KD. Nat Rev Neurosci 16: 745-755, 2015), yet it is still unknown if Up states are homogeneous and whether spiking order is present in neurons with rates <2 Hz (the majority). Using extracellular recordings from anesthetized cats and mice and from naturally sleeping rats, we show for the first time that Up-state transitions can be classified into several types based on the shape of the local field potential (LFP) during each transition. Individual LFP events could be localized in time to within 1-4 ms, more than an order of magnitude less than in previous studies. The majority of recorded neurons synchronized their firing to within ±5-15 ms relative to each Up-state transition. Simultaneous electrophysiology and wide-field imaging in mouse confirmed that LFP event clusters are cortex-wide phenomena. Our findings show that Up states are of different types and point to the potential importance of temporal order and millisecond-scale signaling by cortical neurons.NEW & NOTEWORTHY During cortical Up-state transitions in sleep and anesthesia, neurons undergo brief periods of increased firing in an order similar to that occurring in awake states. We show that these transitions can be classified into distinct types based on the shape of the local field potential. Transition times can be defined to <5 ms. Most neurons synchronize their firing to within ±5-15 ms of the transitions and fire in a consistent order.

Mapping cortical mesoscopic networks of single spiking cortical or sub-cortical neurons.

Understanding the basis of brain function requires knowledge of cortical operations over wide-spatial scales, but also within the context of single neurons. In vivo, wide-field GCaMP imaging and sub-cortical/cortical cellular electrophysiology were used in mice to investigate relationships between spontaneous single neuron spiking and mesoscopic cortical activity. We make use of a rich set of cortical activity motifs that are present in spontaneous activity in anesthetized and awake animals. A mesoscale spike-triggered averaging procedure allowed the identification of motifs that are preferentially linked to individual spiking neurons by employing genetically targeted indicators of neuronal activity. Thalamic neurons predicted and reported specific cycles of wide-scale cortical inhibition/excitation. In contrast, spike-triggered maps derived from single cortical neurons yielded spatio-temporal maps expected for regional cortical consensus function. This approach can define network relationships between any point source of neuronal spiking and mesoscale cortical maps.