Continual Learning in a Multi-Layer Network of an Electric Fish.

Distributing learning across multiple layers has proven extremely powerful in artificial neural networks. However, little is known about how multi-layer learning is implemented in the brain. Here, we provide an account of learning across multiple processing layers in the electrosensory lobe (ELL) of mormyrid fish and report how it solves problems well known from machine learning. Because the ELL operates and learns continuously, it must reconcile learning and signaling functions without switching its mode of operation. We show that this is accomplished through a functional compartmentalization within intermediate layer neurons in which inputs driving learning differentially affect dendritic and axonal spikes. We also find that connectivity based on learning rather than sensory response selectivity assures that plasticity at synapses onto intermediate-layer neurons is matched to the requirements of output neurons. The mechanisms we uncover have relevance to learning in the cerebellum, hippocampus, and cerebral cortex, as well as in artificial systems.

Collective sensing in electric fish.

A number of organisms, including dolphins, bats and electric fish, possess sophisticated active sensory systems that use self-generated signals (for example, acoustic or electrical emissions) to probe the environment1,2. Studies of active sensing in social groups have typically focused on strategies for minimizing interference from conspecific emissions2-4. However, it is well known from engineering that multiple spatially distributed emitters and receivers can greatly enhance environmental sensing (for example, multistatic radar and sonar)5-8. Here we provide evidence from modelling, neural recordings and behavioural experiments that the African weakly electric fish Gnathonemus petersii utilizes the electrical pulses of conspecifics to extend its electrolocation range, discriminate objects and increase information transmission. These results provide evidence for a new, collective mode of active sensing in which individual perception is enhanced by the energy emissions of nearby group members.

Lightning Pose: improved animal pose estimation via semi-supervised learning, Bayesian ensembling and cloud-native open-source tools.

Contemporary pose estimation methods enable precise measurements of behavior via supervised deep learning with hand-labeled video frames. Although effective in many cases, the supervised approach requires extensive labeling and often produces outputs that are unreliable for downstream analyses. Here, we introduce 'Lightning Pose', an efficient pose estimation package with three algorithmic contributions. First, in addition to training on a few labeled video frames, we use many unlabeled videos and penalize the network whenever its predictions violate motion continuity, multiple-view geometry and posture plausibility (semi-supervised learning). Second, we introduce a network architecture that resolves occlusions by predicting pose on any given frame using surrounding unlabeled frames. Third, we refine the pose predictions post hoc by combining ensembling and Kalman smoothing. Together, these components render pose trajectories more accurate and scientifically usable. We released a cloud application that allows users to label data, train networks and process new videos directly from the browser.

Neural Mechanisms for Predicting the Sensory Consequences of Behavior: Insights from Electrosensory Systems.

Perception of the environment requires differentiating between external sensory inputs and those that are self-generated. Some of the clearest insights into the neural mechanisms underlying this process have come from studies of the electrosensory systems of fish. Neurons at the first stage of electrosensory processing generate negative images of the electrosensory consequences of the animal's own behavior. By canceling out the effects of predictable, self-generated inputs, negative images allow for the selective encoding of unpredictable, externally generated stimuli. Combined experimental and theoretical studies of electrosensory systems have led to detailed accounts of how negative images are formed at the level of synaptic plasticity rules, cells, and circuits. Here, I review these accounts and discuss their implications for understanding how predictions of the sensory consequences of behavior may be generated in other sensory structures and the cerebellum.

Responses of cerebellar Purkinje cells during fictive optomotor behavior in larval zebrafish.

Although most studies of the cerebellum have been conducted in mammals, cerebellar circuitry is highly conserved across vertebrates, suggesting that studies of simpler systems may be useful for understanding cerebellar function. The larval zebrafish is particularly promising in this regard because of its accessibility to optical monitoring and manipulations of neural activity. Although several studies suggest that the cerebellum plays a role in behavior at larval stages, little is known about the signals conveyed by particular classes of cerebellar neurons. Here we use electrophysiological recordings to characterize subthreshold, simple spike, and climbing fiber responses in larval zebrafish Purkinje cells in the context of the fictive optomotor response (OMR)-a paradigm in which fish adjust motor output to stabilize their virtual position relative to a visual stimulus. Although visual responses were prominent in Purkinje cells, they lacked the direction or velocity sensitivity that would be expected for controlling the OMR. On the other hand, Purkinje cells exhibited strong responses during fictive swim bouts. Temporal characteristics of these responses are suggestive of a general role for the larval zebrafish cerebellum in controlling swimming. Climbing fibers encoded both visual and motor signals but did not appear to encode signals that could be used to adjust OMR gain, such as retinal slip. Finally, the observation of diverse relationships between simple spikes and climbing fiber responses in individual Purkinje cells highlights the importance of distinguishing between these two types of activity in calcium imaging experiments.

A role for mixed corollary discharge and proprioceptive signals in predicting the sensory consequences of movements.

Animals must distinguish behaviorally relevant patterns of sensory stimulation from those that are attributable to their own movements. In principle, this distinction could be made based on internal signals related to motor commands, known as corollary discharge (CD), sensory feedback, or some combination of both. Here we use an advantageous model system--the electrosensory lobe (ELL) of weakly electric mormyrid fish--to directly examine how CD and proprioceptive feedback signals are transformed into negative images of the predictable electrosensory consequences of the fish's motor commands and/or movements. In vivo recordings from ELL neurons and theoretical modeling suggest that negative images are formed via anti-Hebbian plasticity acting on random, nonlinear mixtures of CD and proprioception. In support of this, we find that CD and proprioception are randomly mixed in spinal mossy fibers and that properties of granule cells are consistent with a nonlinear recoding of these signals. The mechanistic account provided here may be relevant to understanding how internal models of movement consequences are implemented in other systems in which similar components (e.g., mixed sensory and motor signals and synaptic plasticity) are found.

Sensory processing and corollary discharge effects in posterior caudal lobe Purkinje cells in a weakly electric mormyrid fish.

Although it has been suggested that the cerebellum functions to predict the sensory consequences of motor commands, how such predictions are implemented in cerebellar circuitry remains largely unknown. A detailed and relatively complete account of predictive mechanisms has emerged from studies of cerebellum-like sensory structures in fish, suggesting that comparisons of the cerebellum and cerebellum-like structures may be useful. Here we characterize electrophysiological response properties of Purkinje cells in a region of the cerebellum proper of weakly electric mormyrid fish, the posterior caudal lobe (LCp), which receives the same mossy fiber inputs and projects to the same target structures as the electrosensory lobe (ELL), a well-studied cerebellum-like structure. We describe patterns of simple spike and climbing fiber activation in LCp Purkinje cells in response to motor corollary discharge, electrosensory, and proprioceptive inputs and provide evidence for two functionally distinct Purkinje cell subtypes within LCp. Protocols that induce rapid associative plasticity in ELL fail to induce plasticity in LCp, suggesting differences in the adaptive functions of the two structures. Similarities and differences between LCp and ELL are discussed in light of these results.

Lightning Pose: improved animal pose estimation via semi-supervised learning, Bayesian ensembling, and cloud-native open-source tools.

Contemporary pose estimation methods enable precise measurements of behavior via supervised deep learning with hand-labeled video frames. Although effective in many cases, the supervised approach requires extensive labeling and often produces outputs that are unreliable for downstream analyses. Here, we introduce "Lightning Pose," an efficient pose estimation package with three algorithmic contributions. First, in addition to training on a few labeled video frames, we use many unlabeled videos and penalize the network whenever its predictions violate motion continuity, multiple-view geometry, and posture plausibility (semi-supervised learning). Second, we introduce a network architecture that resolves occlusions by predicting pose on any given frame using surrounding unlabeled frames. Third, we refine the pose predictions post-hoc by combining ensembling and Kalman smoothing. Together, these components render pose trajectories more accurate and scientifically usable. We release a cloud application that allows users to label data, train networks, and predict new videos directly from the browser.

Ultra-high density electrodes improve detection, yield, and cell type identification in neuronal recordings.

To understand the neural basis of behavior, it is essential to sensitively and accurately measure neural activity at single neuron and single spike resolution. Extracellular electrophysiology delivers this, but it has biases in the neurons it detects and it imperfectly resolves their action potentials. To minimize these limitations, we developed a silicon probe with much smaller and denser recording sites than previous designs, called Neuropixels Ultra (NP Ultra). This device samples neuronal activity at ultra-high spatial density (~10 times higher than previous probes) with low noise levels, while trading off recording span. NP Ultra is effectively an implantable voltage-sensing camera that captures a planar image of a neuron's electrical field. We use a spike sorting algorithm optimized for these probes to demonstrate that the yield of visually-responsive neurons in recordings from mouse visual cortex improves up to ~3-fold. We show that NP Ultra can record from small neuronal structures including axons and dendrites. Recordings across multiple brain regions and four species revealed a subset of extracellular action potentials with unexpectedly small spatial spread and axon-like features. We share a large-scale dataset of these brain-wide recordings in mice as a resource for studies of neuronal biophysics. Finally, using ground-truth identification of three major inhibitory cortical cell types, we found that these cell types were discriminable with approximately 75% success, a significant improvement over lower-resolution recordings. NP Ultra improves spike sorting performance, detection of subcellular compartments, and cell type classification to enable more powerful dissection of neural circuit activity during behavior.

An internal model for canceling self-generated sensory input in freely behaving electric fish.

Internal models that predict the sensory consequences of motor actions are vital for sensory, motor, and cognitive functions. However, the relationship between motor action and sensory input is complex, often varying from one moment to another depending on the state of the animal and the environment. The neural mechanisms for generating predictions under such challenging, real-world conditions remain largely unknown. Using novel methods for underwater neural recording, a quantitative analysis of unconstrained behavior, and computational modeling, we provide evidence for an unexpectedly sophisticated internal model at the first stage of active electrosensory processing in mormyrid fish. Closed-loop manipulations reveal that electrosensory lobe neurons are capable of simultaneously learning and storing multiple predictions of the sensory consequences of motor commands specific to different sensory states. These results provide mechanistic insights into how internal motor signals and information about the sensory environment are combined within a cerebellum-like circuitry to predict the sensory consequences of natural behavior.

Collective Sensing in Electric Fish.

A number of organisms, including dolphins, bats, and electric fish, possess sophisticated active sensory systems that use self-generated signals (e.g. acoustic or electrical emissions) to probe the environment1,2. Studies of active sensing in social groups have typically focused on strategies for minimizing interference from conspecific emissions2-4. However, it is well-known from engineering that multiple spatially distributed emitters and receivers can greatly enhance environmental sensing (e.g. multistatic radar and sonar)5-8. Here we provide evidence from modeling, neural recordings, and behavioral experiments that the African weakly electric fish Gnathonemus petersii utilizes the electrical pulses of conspecifics to extend electrolocation range, discriminate objects, and increase information transmission. These results suggest a novel, collective mode of active sensing in which individual perception is enhanced by the energy emissions of nearby group members.

A mechanism for differential control of axonal and dendritic spiking underlying learning in a cerebellum-like circuit.

In addition to the action potentials used for axonal signaling, many neurons generate dendritic "spikes" associated with synaptic plasticity. However, in order to control both plasticity and signaling, synaptic inputs must be able to differentially modulate the firing of these two spike types. Here, we investigate this issue in the electrosensory lobe (ELL) of weakly electric mormyrid fish, where separate control over axonal and dendritic spikes is essential for the transmission of learned predictive signals from inhibitory interneurons to the output stage of the circuit. Through a combination of experimental and modeling studies, we uncover a novel mechanism by which sensory input selectively modulates the rate of dendritic spiking by adjusting the amplitude of backpropagating axonal action potentials. Interestingly, this mechanism does not require spatially segregated synaptic inputs or dendritic compartmentalization but relies instead on an electrotonically distant spike initiation site in the axon-a common biophysical feature of neurons.

A Dense Conformal Electrode Array for High Spatial Resolution Stimulation of Electrosensory Systems.

Studies of electrosensory systems have led to insights into to a number of general issues in biology. However, investigations of these systems have been limited by the inability to precisely control spatial patterns of electrosensory input. In this paper, an electrode array and a system to selectively stimulate spatially restricted regions of an electroreceptor array is presented. The array has 96 channels consisting of chrome/gold electrodes patterned on a flexible parylene-C substrate and encapsulated with another parylene-C layer. The conformability of the electrode array allows for optimal current driving and surface interface conditions. Recordings of neural activity at the first central processing stage in weakly electric mormyrid fish support the potential of this system for high spatial resolution stimulation and mapping of electrosensory systems.

Neural readout of a latency code in the active electrosensory system.

The latency of spikes relative to a stimulus conveys sensory information across modalities. However, in most cases, it remains unclear whether and how such latency codes are utilized by postsynaptic neurons. In the active electrosensory system of mormyrid fish, a latency code for stimulus amplitude in electroreceptor afferent nerve fibers (EAs) is hypothesized to be read out by a central reference provided by motor corollary discharge (CD). Here, we demonstrate that CD enhances sensory responses in postsynaptic granular cells of the electrosensory lobe but is not required for reading out EA input. Instead, diverse latency and spike count tuning across the EA population give rise to graded information about stimulus amplitude that can be read out by standard integration of converging excitatory synaptic inputs. Inhibitory control over the temporal window of integration renders two granular cell subclasses differentially sensitive to information derived from relative spike latency versus spike count.

Neurobiology: The power of pauses in electrocommunication.

A new study of social communication behavior in weakly electric fish identifies neural mechanisms that may account for the significance of silent pauses in communication.

A rapid whisker-based decision underlying skilled locomotion in mice.

Skilled motor behavior requires rapidly integrating external sensory input with information about internal state to decide which movements to make next. Using machine learning approaches for high-resolution kinematic analysis, we uncover the logic of a rapid decision underlying sensory-guided locomotion in mice. After detecting obstacles with their whiskers mice select distinct kinematic strategies depending on a whisker-derived estimate of obstacle location together with the position and velocity of their body. Although mice rely on whiskers for obstacle avoidance, lesions of primary whisker sensory cortex had minimal impact. While motor cortex manipulations affected the execution of the chosen strategy, the decision-making process remained largely intact. These results highlight the potential of machine learning for reductionist analysis of naturalistic behaviors and provide a case in which subcortical brain structures appear sufficient for mediating a relatively sophisticated sensorimotor decision.

Instructive effect of visual experience in mouse visual cortex.

We describe a form of experience-dependent response enhancement in the visual cortex of awake mice. Repeated presentations of grating stimuli of a single orientation result in a persistent enhancement of responses evoked by the test stimulus. Response potentiation is specific to the orientation of the test stimulus, develops gradually over the course of several training sessions, and occurs in both juvenile and adult mice. The stimulus-selective response potentiation (SRP) can mask deprivation-induced response depression in adult mice. SRP requires NMDA receptor activation and is prevented by viral delivery of a peptide that interferes with AMPA receptor trafficking. SRP may reveal the mechanisms involved in certain forms of perceptual learning.

Transformations of electrosensory encoding associated with an adaptive filter.

Sensory information is often acquired through active exploration. However, an animal's own movements may result in changes in patterns of sensory input that could interfere with the detection and processing of behaviorally relevant sensory signals. Neural mechanisms for predicting the sensory consequences of movements are thus likely to be of general importance for sensory systems. Such mechanisms have been identified in cerebellum-like structures associated with electrosensory processing in fish. These structures are hypothesized to act as adaptive filters, removing correlations between incoming sensory input and central predictive signals through associative plasticity at parallel fiber synapses. The present study tests the adaptive filter hypothesis in the electrosensory lobe (ELL) of weakly electric mormyrid fish. We compared the ability of electroreceptors and ELL efferent neurons to encode the position of moving objects in the presence and absence of self-generated electrosensory signals caused by tail movements. Tail movements had strong effects on the responses of electroreceptors, substantially reducing the amount of information they conveyed about object position. In contrast, responses of efferent neurons were relatively unaffected by tail movements, and the information they conveyed about object position was preserved. We provide evidence that the electrosensory consequences of tail bending are opposed by proprioceptive inputs conveyed by parallel fibers and that the effects of proprioceptive inputs to efferent cells are plastic. These results support the idea that cerebellum-like structures learn and remove the predictable sensory consequences of behavior and link mechanisms of adaptive filtering to selective encoding of behaviorally relevant sensory information.

Generalization of learned responses in the mormyrid electrosensory lobe.

Appropriate generalization of learned responses to new situations is vital for adaptive behavior. We provide a circuit-level account of generalization in the electrosensory lobe (ELL) of weakly electric mormyrid fish. Much is already known in this system about a form of learning in which motor corollary discharge signals cancel responses to the uninformative input evoked by the fish's own electric pulses. However, for this cancellation to be useful under natural circumstances, it must generalize accurately across behavioral regimes, specifically different electric pulse rates. We show that such generalization indeed occurs in ELL neurons, and develop a circuit-level model explaining how this may be achieved. The mechanism involves regularized synaptic plasticity and an approximate matching of the temporal dynamics of motor corollary discharge and electrosensory inputs. Recordings of motor corollary discharge signals in mossy fibers and granule cells provide direct evidence for such matching.

NMDA receptor-dependent ocular dominance plasticity in adult visual cortex.

The binocular region of mouse visual cortex is strongly dominated by inputs from the contralateral eye. Here we show in adult mice that depriving the dominant contralateral eye of vision leads to a persistent, NMDA receptor-dependent enhancement of the weak ipsilateral-eye inputs. These data provide in vivo evidence for metaplasticity as a mechanism for binocular competition and demonstrate that an ocular dominance shift can occur solely by the mechanisms of response enhancement. They also show that adult mouse visual cortex has a far greater potential for experience-dependent plasticity than previously appreciated. These insights may force a revision in how data on ocular dominance plasticity in mutant mice have been interpreted.