Reconstruction of neocortex: Organelles, compartments, cells, circuits, and activity.

We assembled a semi-automated reconstruction of L2/3 mouse primary visual cortex from ∼250 × 140 × 90 µm3 of electron microscopic images, including pyramidal and non-pyramidal neurons, astrocytes, microglia, oligodendrocytes and precursors, pericytes, vasculature, nuclei, mitochondria, and synapses. Visual responses of a subset of pyramidal cells are included. The data are publicly available, along with tools for programmatic and three-dimensional interactive access. Brief vignettes illustrate the breadth of potential applications relating structure to function in cortical circuits and neuronal cell biology. Mitochondria and synapse organization are characterized as a function of path length from the soma. Pyramidal connectivity motif frequencies are predicted accurately using a configuration model of random graphs. Pyramidal cells receiving more connections from nearby cells exhibit stronger and more reliable visual responses. Sample code shows data access and analysis.

NeuroPAL: A Multicolor Atlas for Whole-Brain Neuronal Identification in C. elegans.

Comprehensively resolving neuronal identities in whole-brain images is a major challenge. We achieve this in C. elegans by engineering a multicolor transgene called NeuroPAL (a neuronal polychromatic atlas of landmarks). NeuroPAL worms share a stereotypical multicolor fluorescence map for the entire hermaphrodite nervous system that resolves all neuronal identities. Neurons labeled with NeuroPAL do not exhibit fluorescence in the green, cyan, or yellow emission channels, allowing the transgene to be used with numerous reporters of gene expression or neuronal dynamics. We showcase three applications that leverage NeuroPAL for nervous-system-wide neuronal identification. First, we determine the brainwide expression patterns of all metabotropic receptors for acetylcholine, GABA, and glutamate, completing a map of this communication network. Second, we uncover changes in cell fate caused by transcription factor mutations. Third, we record brainwide activity in response to attractive and repulsive chemosensory cues, characterizing multimodal coding for these stimuli.

Bayesian Sparse Regression Analysis Documents the Diversity of Spinal Inhibitory Interneurons.

Documenting the extent of cellular diversity is a critical step in defining the functional organization of tissues and organs. To infer cell-type diversity from partial or incomplete transcription factor expression data, we devised a sparse Bayesian framework that is able to handle estimation uncertainty and can incorporate diverse cellular characteristics to optimize experimental design. Focusing on spinal V1 inhibitory interneurons, for which the spatial expression of 19 transcription factors has been mapped, we infer the existence of ~50 candidate V1 neuronal types, many of which localize in compact spatial domains in the ventral spinal cord. We have validated the existence of inferred cell types by direct experimental measurement, establishing this Bayesian framework as an effective platform for cell-type characterization in the nervous system and elsewhere.

Primacy of Flexor Locomotor Pattern Revealed by Ancestral Reversion of Motor Neuron Identity.

Spinal circuits can generate locomotor output in the absence of sensory or descending input, but the principles of locomotor circuit organization remain unclear. We sought insight into these principles by considering the elaboration of locomotor circuits across evolution. The identity of limb-innervating motor neurons was reverted to a state resembling that of motor neurons that direct undulatory swimming in primitive aquatic vertebrates, permitting assessment of the role of motor neuron identity in determining locomotor pattern. Two-photon imaging was coupled with spike inference to measure locomotor firing in hundreds of motor neurons in isolated mouse spinal cords. In wild-type preparations, we observed sequential recruitment of motor neurons innervating flexor muscles controlling progressively more distal joints. Strikingly, after reversion of motor neuron identity, virtually all firing patterns became distinctly flexor like. Our findings show that motor neuron identity directs locomotor circuit wiring and indicate the evolutionary primacy of flexor pattern generation.

Lightning Pose: improved animal pose estimation via semi-supervised learning, Bayesian ensembling and cloud-native open-source tools.

Contemporary pose estimation methods enable precise measurements of behavior via supervised deep learning with hand-labeled video frames. Although effective in many cases, the supervised approach requires extensive labeling and often produces outputs that are unreliable for downstream analyses. Here, we introduce 'Lightning Pose', an efficient pose estimation package with three algorithmic contributions. First, in addition to training on a few labeled video frames, we use many unlabeled videos and penalize the network whenever its predictions violate motion continuity, multiple-view geometry and posture plausibility (semi-supervised learning). Second, we introduce a network architecture that resolves occlusions by predicting pose on any given frame using surrounding unlabeled frames. Third, we refine the pose predictions post hoc by combining ensembling and Kalman smoothing. Together, these components render pose trajectories more accurate and scientifically usable. We released a cloud application that allows users to label data, train networks and process new videos directly from the browser.

Removing direct photocurrent artifacts in optogenetic connectivity mapping data via constrained matrix factorization.

Monosynaptic connectivity mapping is crucial for building circuit-level models of neural computation. Two-photon optogenetic stimulation, when combined with whole-cell recording, enables large-scale mapping of physiological circuit parameters. In this experimental setup, recorded postsynaptic currents are used to infer the presence and strength of connections. For many cell types, nearby connections are those we expect to be strongest. However, when the postsynaptic cell expresses opsin, optical excitation of nearby cells can induce direct photocurrents in the postsynaptic cell. These photocurrent artifacts contaminate synaptic currents, making it difficult or impossible to probe connectivity for nearby cells. To overcome this problem, we developed a computational tool, Photocurrent Removal with Constraints (PhoRC). Our method is based on a constrained matrix factorization model which leverages the fact that photocurrent kinetics are less variable than those of synaptic currents. We demonstrate on real and simulated data that PhoRC consistently removes photocurrents while preserving synaptic currents, despite variations in photocurrent kinetics across datasets. Our method allows the discovery of synaptic connections which would have been otherwise obscured by photocurrent artifacts, and may thus reveal a more complete picture of synaptic connectivity. PhoRC runs faster than real time and is available as open source software.

Versatile multiple object tracking in sparse 2D/3D videos via deformable image registration.

Tracking body parts in behaving animals, extracting fluorescence signals from cells embedded in deforming tissue, and analyzing cell migration patterns during development all require tracking objects with partially correlated motion. As dataset sizes increase, manual tracking of objects becomes prohibitively inefficient and slow, necessitating automated and semi-automated computational tools. Unfortunately, existing methods for multiple object tracking (MOT) are either developed for specific datasets and hence do not generalize well to other datasets, or require large amounts of training data that are not readily available. This is further exacerbated when tracking fluorescent sources in moving and deforming tissues, where the lack of unique features and sparsely populated images create a challenging environment, especially for modern deep learning techniques. By leveraging technology recently developed for spatial transformer networks, we propose ZephIR, an image registration framework for semi-supervised MOT in 2D and 3D videos. ZephIR can generalize to a wide range of biological systems by incorporating adjustable parameters that encode spatial (sparsity, texture, rigidity) and temporal priors of a given data class. We demonstrate the accuracy and versatility of our approach in a variety of applications, including tracking the body parts of a behaving mouse and neurons in the brain of a freely moving C. elegans. We provide an open-source package along with a web-based graphical user interface that allows users to provide small numbers of annotations to interactively improve tracking results.

Voltage imaging and optogenetics reveal behaviour-dependent changes in hippocampal dynamics.

A technology that simultaneously records membrane potential from multiple neurons in behaving animals will have a transformative effect on neuroscience research1,2. Genetically encoded voltage indicators are a promising tool for these purposes; however, these have so far been limited to single-cell recordings with a marginal signal-to-noise ratio in vivo3-5. Here we developed improved near-infrared voltage indicators, high-speed microscopes and targeted gene expression schemes that enabled simultaneous in vivo recordings of supra- and subthreshold voltage dynamics in multiple neurons in the hippocampus of behaving mice. The reporters revealed subcellular details of back-propagating action potentials and correlations in subthreshold voltage between multiple cells. In combination with stimulation using optogenetics, the reporters revealed changes in neuronal excitability that were dependent on the behavioural state, reflecting the interplay of excitatory and inhibitory synaptic inputs. These tools open the possibility for detailed explorations of network dynamics in the context of behaviour. Fig. 1 PHOTOACTIVATED QUASAR3 (PAQUASAR3) REPORTS NEURONAL ACTIVITY IN VIVO.: a, Schematic of the paQuasAr3 construct. b, Photoactivation by blue light enhanced voltage signals excited by red light in cultured neurons that expressed paQuasAr3 (representative example of n = 4 cells). c, Model of the photocycle of paQuasAr3. d, Confocal images of sparsely expressed paQuasAr3 in brain slices. Scale bars, 50 µm. Representative images, experiments were repeated in n = 3 mice. e, Simultaneous fluorescence and patch-clamp recordings from a neuron expressing paQuasAr3 in acute brain slice. Top, magnification of boxed regions. Schematic shows brain slice, patch pipette and microscope objective. f, Simultaneous fluorescence and patch-clamp recordings of inhibitory post synaptic potentials in an L2-3 neuron induced by electrical stimulation of L5-6 in acute slice. g, Normalized change in fluorescence (ΔF/F) and SNR of optically recorded post-synaptic potentials (PSPs) as a function of the amplitude of the post-synaptic potentials. The voltage sensitivity was ΔF/F = 40 ± 1.7% per 100 mV. The SNR was 0.93 ± 0.07 per 1 mV in a 1-kHz bandwidth (n = 42 post-synaptic potentials from 5 cells, data are mean ± s.d.). Schematic shows brain slice, patch pipette, field stimulation electrodes and microscope objective. h, Optical measurements of paQuasAr3 fluorescence in the CA1 region of the hippocampus (top) and glomerular layer of the olfactory bulb (bottom) of anaesthetized mice (representative traces from n = 7 CA1 cells and n = 13 olfactory bulb cells, n = 3 mice). Schematics show microscope objective and the imaged brain region. i, STA fluorescence from 88 spikes in a CA1 oriens neuron. j, Frames from the STA video showing the delay in the back-propagating action potential in the dendrites relative to the soma. k, Sub-Nyquist fitting of the action potential delay and width shows electrical compartmentalization in the dendrites. Experiments in k-m were repeated in n = 2 cells from n = 2 mice.

Visualizing the organization and differentiation of the male-specific nervous system of C. elegans.

Sex differences in the brain are prevalent throughout the animal kingdom and particularly well appreciated in the nematode Caenorhabditis elegans, where male animals contain a little-studied set of 93 male-specific neurons. To make these neurons amenable for future study, we describe here how a multicolor reporter transgene, NeuroPAL, is capable of visualizing the distinct identities of all male-specific neurons. We used NeuroPAL to visualize and characterize a number of features of the male-specific nervous system. We provide several proofs of concept for using NeuroPAL to identify the sites of expression of gfp-tagged reporter genes and for cellular fate analysis by analyzing the effect of removal of several developmental patterning genes on neuronal identity acquisition. We use NeuroPAL and its intrinsic cohort of more than 40 distinct differentiation markers to show that, even though male-specific neurons are generated throughout all four larval stages, they execute their terminal differentiation program in a coordinated manner in the fourth larval stage. This coordinated wave of differentiation, which we call 'just-in-time' differentiation, couples neuronal maturation programs with the appearance of sexual organs.

Rapid mesoscale volumetric imaging of neural activity with synaptic resolution.

Imaging neurons and neural circuits over large volumes at high speed and subcellular resolution is a difficult task. Incorporating a Bessel focus module into a two-photon fluorescence mesoscope, we achieved rapid volumetric imaging of neural activity over the mesoscale with synaptic resolution. We applied the technology to calcium imaging of entire dendritic spans of neurons as well as neural ensembles within multiple cortical regions over two hemispheres of the awake mouse brain.

Blind demixing methods for recovering dense neuronal morphology from barcode imaging data.

Cellular barcoding methods offer the exciting possibility of 'infinite-pseudocolor' anatomical reconstruction-i.e., assigning each neuron its own random unique barcoded 'pseudocolor,' and then using these pseudocolors to trace the microanatomy of each neuron. Here we use simulations, based on densely-reconstructed electron microscopy microanatomy, with signal structure matched to real barcoding data, to quantify the feasibility of this procedure. We develop a new blind demixing approach to recover the barcodes that label each neuron, and validate this method on real data with known barcodes. We also develop a neural network which uses the recovered barcodes to reconstruct the neuronal morphology from the observed fluorescence imaging data, 'connecting the dots' between discontiguous barcode amplicon signals. We find that accurate recovery should be feasible, provided that the barcode signal density is sufficiently high. This study suggests the possibility of mapping the morphology and projection pattern of many individual neurons simultaneously, at high resolution and at large scale, via conventional light microscopy.

BARcode DEmixing through Non-negative Spatial Regression (BarDensr).

Modern spatial transcriptomics methods can target thousands of different types of RNA transcripts in a single slice of tissue. Many biological applications demand a high spatial density of transcripts relative to the imaging resolution, leading to partial mixing of transcript rolonies in many voxels; unfortunately, current analysis methods do not perform robustly in this highly-mixed setting. Here we develop a new analysis approach, BARcode DEmixing through Non-negative Spatial Regression (BarDensr): we start with a generative model of the physical process that leads to the observed image data and then apply sparse convex optimization methods to estimate the underlying (demixed) rolony densities. We apply BarDensr to simulated and real data and find that it achieves state of the art signal recovery, particularly in densely-labeled regions or data with low spatial resolution. Finally, BarDensr is fast and parallelizable. We provide open-source code as well as an implementation for the 'NeuroCAAS' cloud platform.

Partitioning variability in animal behavioral videos using semi-supervised variational autoencoders.

Recent neuroscience studies demonstrate that a deeper understanding of brain function requires a deeper understanding of behavior. Detailed behavioral measurements are now often collected using video cameras, resulting in an increased need for computer vision algorithms that extract useful information from video data. Here we introduce a new video analysis tool that combines the output of supervised pose estimation algorithms (e.g. DeepLabCut) with unsupervised dimensionality reduction methods to produce interpretable, low-dimensional representations of behavioral videos that extract more information than pose estimates alone. We demonstrate this tool by extracting interpretable behavioral features from videos of three different head-fixed mouse preparations, as well as a freely moving mouse in an open field arena, and show how these interpretable features can facilitate downstream behavioral and neural analyses. We also show how the behavioral features produced by our model improve the precision and interpretation of these downstream analyses compared to using the outputs of either fully supervised or fully unsupervised methods alone.

Nonlinear Decoding of Natural Images From Large-Scale Primate Retinal Ganglion Recordings.

Decoding sensory stimuli from neural activity can provide insight into how the nervous system might interpret the physical environment, and facilitates the development of brain-machine interfaces. Nevertheless, the neural decoding problem remains a significant open challenge. Here, we present an efficient nonlinear decoding approach for inferring natural scene stimuli from the spiking activities of retinal ganglion cells (RGCs). Our approach uses neural networks to improve on existing decoders in both accuracy and scalability. Trained and validated on real retinal spike data from more than 1000 simultaneously recorded macaque RGC units, the decoder demonstrates the necessity of nonlinear computations for accurate decoding of the fine structures of visual stimuli. Specifically, high-pass spatial features of natural images can only be decoded using nonlinear techniques, while low-pass features can be extracted equally well by linear and nonlinear methods. Together, these results advance the state of the art in decoding natural stimuli from large populations of neurons.

Localized semi-nonnegative matrix factorization (LocaNMF) of widefield calcium imaging data.

Widefield calcium imaging enables recording of large-scale neural activity across the mouse dorsal cortex. In order to examine the relationship of these neural signals to the resulting behavior, it is critical to demix the recordings into meaningful spatial and temporal components that can be mapped onto well-defined brain regions. However, no current tools satisfactorily extract the activity of the different brain regions in individual mice in a data-driven manner, while taking into account mouse-specific and preparation-specific differences. Here, we introduce Localized semi-Nonnegative Matrix Factorization (LocaNMF), a method that efficiently decomposes widefield video data and allows us to directly compare activity across multiple mice by outputting mouse-specific localized functional regions that are significantly more interpretable than more traditional decomposition techniques. Moreover, it provides a natural subspace to directly compare correlation maps and neural dynamics across different behaviors, mice, and experimental conditions, and enables identification of task- and movement-related brain regions.

Community-based benchmarking improves spike rate inference from two-photon calcium imaging data.

In recent years, two-photon calcium imaging has become a standard tool to probe the function of neural circuits and to study computations in neuronal populations. However, the acquired signal is only an indirect measurement of neural activity due to the comparatively slow dynamics of fluorescent calcium indicators. Different algorithms for estimating spike rates from noisy calcium measurements have been proposed in the past, but it is an open question how far performance can be improved. Here, we report the results of the spikefinder challenge, launched to catalyze the development of new spike rate inference algorithms through crowd-sourcing. We present ten of the submitted algorithms which show improved performance compared to previously evaluated methods. Interestingly, the top-performing algorithms are based on a wide range of principles from deep neural networks to generative models, yet provide highly correlated estimates of the neural activity. The competition shows that benchmark challenges can drive algorithmic developments in neuroscience.

Fast online deconvolution of calcium imaging data.

Fluorescent calcium indicators are a popular means for observing the spiking activity of large neuronal populations, but extracting the activity of each neuron from raw fluorescence calcium imaging data is a nontrivial problem. We present a fast online active set method to solve this sparse non-negative deconvolution problem. Importantly, the algorithm 3progresses through each time series sequentially from beginning to end, thus enabling real-time online estimation of neural activity during the imaging session. Our algorithm is a generalization of the pool adjacent violators algorithm (PAVA) for isotonic regression and inherits its linear-time computational complexity. We gain remarkable increases in processing speed: more than one order of magnitude compared to currently employed state of the art convex solvers relying on interior point methods. Unlike these approaches, our method can exploit warm starts; therefore optimizing model hyperparameters only requires a handful of passes through the data. A minor modification can further improve the quality of activity inference by imposing a constraint on the minimum spike size. The algorithm enables real-time simultaneous deconvolution of O(105) traces of whole-brain larval zebrafish imaging data on a laptop.

Electrical stimulus artifact cancellation and neural spike detection on large multi-electrode arrays.

Simultaneous electrical stimulation and recording using multi-electrode arrays can provide a valuable technique for studying circuit connectivity and engineering neural interfaces. However, interpreting these measurements is challenging because the spike sorting process (identifying and segregating action potentials arising from different neurons) is greatly complicated by electrical stimulation artifacts across the array, which can exhibit complex and nonlinear waveforms, and overlap temporarily with evoked spikes. Here we develop a scalable algorithm based on a structured Gaussian Process model to estimate the artifact and identify evoked spikes. The effectiveness of our methods is demonstrated in both real and simulated 512-electrode recordings in the peripheral primate retina with single-electrode and several types of multi-electrode stimulation. We establish small error rates in the identification of evoked spikes, with a computational complexity that is compatible with real-time data analysis. This technology may be helpful in the design of future high-resolution sensory prostheses based on tailored stimulation (e.g., retinal prostheses), and for closed-loop neural stimulation at a much larger scale than currently possible.

Multi-scale approaches for high-speed imaging and analysis of large neural populations.

Progress in modern neuroscience critically depends on our ability to observe the activity of large neuronal populations with cellular spatial and high temporal resolution. However, two bottlenecks constrain efforts towards fast imaging of large populations. First, the resulting large video data is challenging to analyze. Second, there is an explicit tradeoff between imaging speed, signal-to-noise, and field of view: with current recording technology we cannot image very large neuronal populations with simultaneously high spatial and temporal resolution. Here we describe multi-scale approaches for alleviating both of these bottlenecks. First, we show that spatial and temporal decimation techniques based on simple local averaging provide order-of-magnitude speedups in spatiotemporally demixing calcium video data into estimates of single-cell neural activity. Second, once the shapes of individual neurons have been identified at fine scale (e.g., after an initial phase of conventional imaging with standard temporal and spatial resolution), we find that the spatial/temporal resolution tradeoff shifts dramatically: after demixing we can accurately recover denoised fluorescence traces and deconvolved neural activity of each individual neuron from coarse scale data that has been spatially decimated by an order of magnitude. This offers a cheap method for compressing this large video data, and also implies that it is possible to either speed up imaging significantly, or to "zoom out" by a corresponding factor to image order-of-magnitude larger neuronal populations with minimal loss in accuracy or temporal resolution.

Neuroprosthetic Decoder Training as Imitation Learning.

Neuroprosthetic brain-computer interfaces function via an algorithm which decodes neural activity of the user into movements of an end effector, such as a cursor or robotic arm. In practice, the decoder is often learned by updating its parameters while the user performs a task. When the user's intention is not directly observable, recent methods have demonstrated value in training the decoder against a surrogate for the user's intended movement. Here we show that training a decoder in this way is a novel variant of an imitation learning problem, where an oracle or expert is employed for supervised training in lieu of direct observations, which are not available. Specifically, we describe how a generic imitation learning meta-algorithm, dataset aggregation (DAgger), can be adapted to train a generic brain-computer interface. By deriving existing learning algorithms for brain-computer interfaces in this framework, we provide a novel analysis of regret (an important metric of learning efficacy) for brain-computer interfaces. This analysis allows us to characterize the space of algorithmic variants and bounds on their regret rates. Existing approaches for decoder learning have been performed in the cursor control setting, but the available design principles for these decoders are such that it has been impossible to scale them to naturalistic settings. Leveraging our findings, we then offer an algorithm that combines imitation learning with optimal control, which should allow for training of arbitrary effectors for which optimal control can generate goal-oriented control. We demonstrate this novel and general BCI algorithm with simulated neuroprosthetic control of a 26 degree-of-freedom model of an arm, a sophisticated and realistic end effector.