Sensorimotor delays constrain robust locomotion in a 3D kinematic model of fly walking.

Walking animals must maintain stability in the presence of external perturbations, despite significant temporal delays in neural signaling and muscle actuation. Here, we develop a 3D kinematic model with a layered control architecture to investigate how sensorimotor delays constrain robustness of walking behavior in the fruit fly, Drosophila. Motivated by the anatomical architecture of insect locomotor control circuits, our model consists of three component layers: a neural network that generates realistic 3D joint kinematics for each leg, an optimal controller that executes the joint kinematics while accounting for delays, and an inter-leg coordinator. The model generates realistic simulated walking that matches real fly walking kinematics and sustains walking even when subjected to unexpected perturbations, generalizing beyond its training data. However, we found that the model's robustness to perturbations deteriorates when sensorimotor delay parameters exceed the physiological range. These results suggest that fly sensorimotor control circuits operate close to the temporal limit at which they can detect and respond to external perturbations. More broadly, we show how a modular, layered model architecture can be used to investigate physiological constraints on animal behavior.

Presynaptic inhibition selectively suppresses leg proprioception in behaving Drosophila.

The sense of proprioception is mediated by internal mechanosensory neurons that detect joint position and movement. To support a diverse range of functions, from stabilizing posture to coordinating movements, proprioceptive feedback to limb motor control circuits must be tuned in a context-dependent manner. How proprioceptive feedback signals are tuned to match behavioral demands remains poorly understood. Using calcium imaging in behaving Drosophila, we find that the axons of position-encoding leg proprioceptors are active across behaviors, whereas the axons of movement-encoding leg proprioceptors are suppressed during walking and grooming. Using connectomics, we identify a specific class of interneurons that provide GABAergic presynaptic inhibition to the axons of movement-encoding proprioceptors. These interneurons are active during self-generated but not passive leg movements and receive input from descending neurons, suggesting they are driven by predictions of leg movement originating in the brain. Predictively suppressing expected proprioceptive feedback provides a mechanism to attenuate reflexes that would otherwise interfere with voluntary movement.

Emergent behaviour and neural dynamics in artificial agents tracking odour plumes.

Tracking an odour plume to locate its source under variable wind and plume statistics is a complex task. Flying insects routinely accomplish such tracking, often over long distances, in pursuit of food or mates. Several aspects of this remarkable behaviour and its underlying neural circuitry have been studied experimentally. Here we take a complementary in silico approach to develop an integrated understanding of their behaviour and neural computations. Specifically, we train artificial recurrent neural network agents using deep reinforcement learning to locate the source of simulated odour plumes that mimic features of plumes in a turbulent flow. Interestingly, the agents' emergent behaviours resemble those of flying insects, and the recurrent neural networks learn to compute task-relevant variables with distinct dynamic structures in population activity. Our analyses put forward a testable behavioural hypothesis for tracking plumes in changing wind direction, and we provide key intuitions for memory requirements and neural dynamics in odour plume tracking.

Intraspecific variation in the placement of campaniform sensilla on the wings of the hawkmoth Manduca sexta.

Flight control requires active sensory feedback, and insects have many sensors that help them estimate their current locomotor state, including campaniform sensilla, which are mechanoreceptors that sense strain resulting from deformation of the cuticle. Campaniform sensilla on the wing detect bending and torsional forces encountered during flight, providing input to the flight feedback control system. During flight, wings experience complex spatio-temporal strain patterns. Because campaniform sensilla detect only local strain, their placement on the wing is presumably critical for determining the overall representation of wing deformation; however, how these sensilla are distributed across wings is largely unknown. Here, we test the hypothesis that campaniform sensilla are found in stereotyped locations across individuals of Manduca sexta, a hawkmoth. We found that although campaniform sensilla are consistently found on the same veins or in the same regions of the wings, their total number and distribution can vary extensively. This suggests that there is some robustness to variation in sensory feedback in the insect flight control system. The regions where campaniform sensilla are consistently found provide clues to their functional roles, although some patterns might be reflective of developmental processes. Collectively, our results on intraspecific variation in campaniform sensilla placement on insect wings will help reshape our thinking on the utility of mechanosensory feedback for insect flight control and guide further experimental and comparative studies.

Mechanistic Hypotheses for Proprioceptive Sensing Within the Avian Lumbosacral Spinal Cord.

Animals need to accurately sense changes in their body position to perform complex movements. It is increasingly clear that the vertebrate central nervous system contains a variety of cells capable of detecting body motion, in addition to the comparatively well-understood mechanosensory cells of the vestibular system and the peripheral proprioceptors. One such intriguing system is the lower spinal cord and column in birds, also known as the avian lumbosacral organ (LSO), which is thought to act as a set of balance sensors that allow birds to detect body movements separately from head movements detected by the vestibular system. Here, we take what is known about proprioceptive, mechanosensory spinal neurons in other vertebrates to explore hypotheses for how the LSO might sense mechanical information related to movement. Although the LSO is found only in birds, recent immunohistochemical studies of the avian LSO have hinted at similarities between cells in the LSO and the known spinal proprioceptors in other vertebrates. In addition to describing possible connections between avian spinal anatomy and recent findings on spinal proprioception as well as sensory and sensorimotor spinal networks, we also present some new data that suggest a role for sensory afferent peptides in LSO function. Thus, this perspective articulates a set of testable ideas on mechanisms of LSO function grounded in the emerging spinal proprioception scientific literature.

Nonuniform structural properties of wings confer sensing advantages.

Sensory feedback is essential to both animals and robotic systems for achieving coordinated, precise movements. Mechanosensory feedback, which provides information about body deformation, depends not only on the properties of sensors but also on the structure in which they are embedded. In insects, wing structure plays a particularly important role in flapping flight: in addition to generating aerodynamic forces, wings provide mechanosensory feedback necessary for guiding flight while undergoing dramatic deformations during each wingbeat. However, the role that wing structure plays in determining mechanosensory information is relatively unexplored. Insect wings exhibit characteristic stiffness gradients and are subject to both aerodynamic and structural damping. Here we examine how both of these properties impact sensory performance, using finite element analysis combined with sensor placement optimization approaches. We show that wings with nonuniform stiffness exhibit several advantages over uniform stiffness wings, resulting in higher accuracy of rotation detection and lower sensitivity to the placement of sensors on the wing. Moreover, we show that higher damping generally improves the accuracy with which body rotations can be detected. These results contribute to our understanding of the evolution of the nonuniform stiffness patterns in insect wings, as well as suggest design principles for robotic systems.

Arousal as a universal embedding for spatiotemporal brain dynamics.

Neural activity in awake organisms shows widespread and spatiotemporally diverse correlations with behavioral and physiological measurements. We propose that this covariation reflects in part the dynamics of a unified, arousal-related process that regulates brain-wide physiology on the timescale of seconds. Taken together with theoretical foundations in dynamical systems, this interpretation leads us to a surprising prediction: that a single, scalar measurement of arousal (e.g., pupil diameter) should suffice to reconstruct the continuous evolution of multimodal, spatiotemporal measurements of large-scale brain physiology. To test this hypothesis, we perform multimodal, cortex-wide optical imaging and behavioral monitoring in awake mice. We demonstrate that spatiotemporal measurements of neuronal calcium, metabolism, and blood-oxygen can be accurately and parsimoniously modeled from a low-dimensional state-space reconstructed from the time history of pupil diameter. Extending this framework to behavioral and electrophysiological measurements from the Allen Brain Observatory, we demonstrate the ability to integrate diverse experimental data into a unified generative model via mappings from an intrinsic arousal manifold. Our results support the hypothesis that spontaneous, spatially structured fluctuations in brain-wide physiology-widely interpreted to reflect regionally-specific neural communication-are in large part reflections of an arousal-related process. This enriched view of arousal dynamics has broad implications for interpreting observations of brain, body, and behavior as measured across modalities, contexts, and scales.

Structured random receptive fields enable informative sensory encodings.

Brains must represent the outside world so that animals survive and thrive. In early sensory systems, neural populations have diverse receptive fields structured to detect important features in inputs, yet significant variability has been ignored in classical models of sensory neurons. We model neuronal receptive fields as random, variable samples from parameterized distributions and demonstrate this model in two sensory modalities using data from insect mechanosensors and mammalian primary visual cortex. Our approach leads to a significant theoretical connection between the foundational concepts of receptive fields and random features, a leading theory for understanding artificial neural networks. The modeled neurons perform a randomized wavelet transform on inputs, which removes high frequency noise and boosts the signal. Further, these random feature neurons enable learning from fewer training samples and with smaller networks in artificial tasks. This structured random model of receptive fields provides a unifying, mathematically tractable framework to understand sensory encodings across both spatial and temporal domains.

Large-scale waves of activity in the neonatal mouse brain in vivo occur almost exclusively during sleep cycles.

Spontaneous electrical activity plays major roles in the development of cortical circuitry. This activity can occur highly localized regions or can propagate over the entire cortex. Both types of activity coexist during early development. To investigate how different forms of spontaneous activity might be temporally segregated, we used wide-field trans-cranial calcium imaging over an entire hemisphere in P1-P8 mouse pups. We found that spontaneous waves of activity that propagate to cover the majority of the cortex (large-scale waves; LSWs) are generated at the end of the first postnatal week, along with several other forms of more localized activity. We further found that LSWs are segregated into sleep cycles. In contrast, cortical activity during wake states is more spatially restricted and the few large-scale forms of activity that occur during wake can be distinguished from LSWs in sleep based on their initiation in the motor cortex and their correlation with body movements. This change in functional cortical circuitry to a state that is permissive for large-scale activity may temporally segregate different forms of activity during critical stages when activity-dependent circuit development occurs over many spatial scales. Our data also suggest that LSWs in early development may be a functional precursor to slow sleep waves in the adult, which play critical roles in memory consolidation and synaptic rescaling.

Molecular Markers of Mechanosensation in Glycinergic Neurons in the Avian Lumbosacral Spinal Cord.

Birds are exceptionally adept at controlling their body position. For example, they can coordinate rapid movements of their body while stabilizing their head. Intriguingly, this ability may rely in part on a mechanosensory organ in the avian lower spinal cord called the lumbosacral organ (LSO). However, molecular mechanotransduction mechanisms have not been identified in the avian spinal cord. Here, we report the presence of glycinergic neurons in the LSO that exhibit immunoreactivity for myosin7a and espin, molecules essential for function and maintenance of hair cells in the inner ear. Specifically, we find glycinergic cell bodies near the central canal and processes that extend laterally to the accessory lobes and spinal ligaments. These LSO neurons are reminiscent of glycinergic neurons in a recently-described lateral spinal proprioceptive organ in zebrafish that detects spinal bending. The avian LSO, however, is located inside a series of fused vertebrae called the synsacrum, which constrains spinal bending. We suggest the LSO may be a modification and elaboration of a preexisting mechanosensory spinal network in vertebrates. A mechanistic understanding of its function may be an important clue to understanding the evolution and development of avian locomotion.

Learning neural decoders without labels using multiple data streams.

Objective.Recent advances in neural decoding have accelerated the development of brain-computer interfaces aimed at assisting users with everyday tasks such as speaking, walking, and manipulating objects. However, current approaches for training neural decoders commonly require large quantities of labeled data, which can be laborious or infeasible to obtain in real-world settings. Alternatively, self-supervised models that share self-generated pseudo-labels between two data streams have shown exceptional performance on unlabeled audio and video data, but it remains unclear how well they extend to neural decoding.Approach.We learn neural decoders without labels by leveraging multiple simultaneously recorded data streams, including neural, kinematic, and physiological signals. Specifically, we apply cross-modal, self-supervised deep clustering to train decoders that can classify movements from brain recordings. After training, we then isolate the decoders for each input data stream and compare the accuracy of decoders trained using cross-modal deep clustering against supervised and unimodal, self-supervised models.Main results.We find that sharing pseudo-labels between two data streams during training substantially increases decoding performance compared to unimodal, self-supervised models, with accuracies approaching those of supervised decoders trained on labeled data. Next, we extend cross-modal decoder training to three or more modalities, achieving state-of-the-art neural decoding accuracy that matches or slightly exceeds the performance of supervised models.Significance.We demonstrate that cross-modal, self-supervised decoding can be applied to train neural decoders when few or no labels are available and extend the cross-modal framework to share information among three or more data streams, further improving self-supervised training.

AJILE12: Long-term naturalistic human intracranial neural recordings and pose.

Understanding the neural basis of human movement in naturalistic scenarios is critical for expanding neuroscience research beyond constrained laboratory paradigms. Here, we describe our Annotated Joints in Long-term Electrocorticography for 12 human participants (AJILE12) dataset, the largest human neurobehavioral dataset that is publicly available; the dataset was recorded opportunistically during passive clinical epilepsy monitoring. AJILE12 includes synchronized intracranial neural recordings and upper body pose trajectories across 55 semi-continuous days of naturalistic movements, along with relevant metadata, including thousands of wrist movement events and annotated behavioral states. Neural recordings are available at 500 Hz from at least 64 electrodes per participant, for a total of 1280 hours. Pose trajectories at 9 upper-body keypoints were estimated from 118 million video frames. To facilitate data exploration and reuse, we have shared AJILE12 on The DANDI Archive in the Neurodata Without Borders (NWB) data standard and developed a browser-based dashboard.

Decoding Happiness from Neural and Video Recordings for Better Insight Into Emotional Processing in the Brain.

Gaining a better understanding of which brain regions are responsible for emotional processing is crucial for the development of novel treatments for neuropsychiatric disorders. Current approaches rely on sparse assessments of subjects' emotional states, rarely reaching more than a hundred per patient. Additionally, data are usually obtained in a task solving scenario, possibly influencing their emotions by study design. Here, we utilize several days worth of near-continuous neural and video recordings of subjects in a naturalistic environment to predict the emotional state of happiness from neural data. We are able to obtain high-frequency and high-volume happiness labels for this task by first predicting happiness from video data in an intermediary step, achieving good results (F1 = .75) and providing us with more than 6 million happiness assessments per patient, on average. We then utilize these labels for a classifier on neural data (F1 = .71). Our findings provide a potential pathway for future work on emotional processing that circumvents the mentioned restrictions.

A leg to stand on: computational models of proprioception.

Dexterous motor control requires feedback from proprioceptors, internal mechanosensory neurons that sense the body's position and movement. An outstanding question in neuroscience is how diverse proprioceptive feedback signals contribute to flexible motor control. Genetic tools now enable targeted recording and perturbation of proprioceptive neurons in behaving animals; however, these experiments can be challenging to interpret, due to the tight coupling of proprioception and motor control. Here, we argue that understanding the role of proprioceptive feedback in controlling behavior will be aided by the development of multiscale models of sensorimotor loops. We review current phenomenological and structural models for proprioceptor encoding and discuss how they may be integrated with existing models of posture, movement, and body state estimation.

Learning to feed in the dark: how light level influences feeding in the hawkmoth Manduca sexta.

Nocturnal insects like moths are essential for pollination, providing resilience to the diurnal pollination networks. Moths use both vision and mechanosensation to locate the nectary opening in the flowers with their proboscis. However, increased light levels due to artificial light at night (ALAN) pose a serious threat to nocturnal insects. Here, we examined how light levels influence the efficacy by which the crepuscular hawkmoth Manduca sexta locates the nectary. We used three-dimensional-printed artificial flowers fitted with motion sensors in the nectary and machine vision to track the motion of hovering moths under two light levels: 0.1 lux (moonlight) and 50 lux (dawn/dusk). We found that moths in higher light conditions took significantly longer to find the nectary, even with repeated visits to the same flower. In addition to taking longer, moths in higher light conditions hovered further from the flower during feeding. Increased light levels adversely affect learning and motor control in these animals.

Anipose: A toolkit for robust markerless 3D pose estimation.

Quantifying movement is critical for understanding animal behavior. Advances in computer vision now enable markerless tracking from 2D video, but most animals move in 3D. Here, we introduce Anipose, an open-source toolkit for robust markerless 3D pose estimation. Anipose is built on the 2D tracking method DeepLabCut, so users can expand their existing experimental setups to obtain accurate 3D tracking. It consists of four components: (1) a 3D calibration module, (2) filters to resolve 2D tracking errors, (3) a triangulation module that integrates temporal and spatial regularization, and (4) a pipeline to structure processing of large numbers of videos. We evaluate Anipose on a calibration board as well as mice, flies, and humans. By analyzing 3D leg kinematics tracked with Anipose, we identify a key role for joint rotation in motor control of fly walking. To help users get started with 3D tracking, we provide tutorials and documentation at http://anipose.org/.

Go with the FLOW: visualizing spatiotemporal dynamics in optical widefield calcium imaging.

Widefield calcium imaging has recently emerged as a powerful experimental technique to record coordinated large-scale brain activity. These measurements present a unique opportunity to characterize spatiotemporally coherent structures that underlie neural activity across many regions of the brain. In this work, we leverage analytic techniques from fluid dynamics to develop a visualization framework that highlights features of flow across the cortex, mapping wavefronts that may be correlated with behavioural events. First, we transform the time series of widefield calcium images into time-varying vector fields using optic flow. Next, we extract concise diagrams summarizing the dynamics, which we refer to as FLOW (flow lines in optical widefield imaging) portraits. These FLOW portraits provide an intuitive map of dynamic calcium activity, including regions of initiation and termination, as well as the direction and extent of activity spread. To extract these structures, we use the finite-time Lyapunov exponent technique developed to analyse time-varying manifolds in unsteady fluids. Importantly, our approach captures coherent structures that are poorly represented by traditional modal decomposition techniques. We demonstrate the application of FLOW portraits on three simple synthetic datasets and two widefield calcium imaging datasets, including cortical waves in the developing mouse and spontaneous cortical activity in an adult mouse.

Wing structure and neural encoding jointly determine sensing strategies in insect flight.

Animals rely on sensory feedback to generate accurate, reliable movements. In many flying insects, strain-sensitive neurons on the wings provide rapid feedback that is critical for stable flight control. While the impacts of wing structure on aerodynamic performance have been widely studied, the impacts of wing structure on sensing are largely unexplored. In this paper, we show how the structural properties of the wing and encoding by mechanosensory neurons interact to jointly determine optimal sensing strategies and performance. Specifically, we examine how neural sensors can be placed effectively on a flapping wing to detect body rotation about different axes, using a computational wing model with varying flexural stiffness. A small set of mechanosensors, conveying strain information at key locations with a single action potential per wingbeat, enable accurate detection of body rotation. Optimal sensor locations are concentrated at either the wing base or the wing tip, and they transition sharply as a function of both wing stiffness and neural threshold. Moreover, the sensing strategy and performance is robust to both external disturbances and sensor loss. Typically, only five sensors are needed to achieve near-peak accuracy, with a single sensor often providing accuracy well above chance. Our results show that small-amplitude, dynamic signals can be extracted efficiently with spatially and temporally sparse sensors in the context of flight. The demonstrated interaction of wing structure and neural encoding properties points to the importance of understanding each in the context of their joint evolution.