Lightning Pose: improved animal pose estimation via semi-supervised learning, Bayesian ensembling and cloud-native open-source tools.

Contemporary pose estimation methods enable precise measurements of behavior via supervised deep learning with hand-labeled video frames. Although effective in many cases, the supervised approach requires extensive labeling and often produces outputs that are unreliable for downstream analyses. Here, we introduce 'Lightning Pose', an efficient pose estimation package with three algorithmic contributions. First, in addition to training on a few labeled video frames, we use many unlabeled videos and penalize the network whenever its predictions violate motion continuity, multiple-view geometry and posture plausibility (semi-supervised learning). Second, we introduce a network architecture that resolves occlusions by predicting pose on any given frame using surrounding unlabeled frames. Third, we refine the pose predictions post hoc by combining ensembling and Kalman smoothing. Together, these components render pose trajectories more accurate and scientifically usable. We released a cloud application that allows users to label data, train networks and process new videos directly from the browser.

Collective sensing in electric fish.

A number of organisms, including dolphins, bats and electric fish, possess sophisticated active sensory systems that use self-generated signals (for example, acoustic or electrical emissions) to probe the environment1,2. Studies of active sensing in social groups have typically focused on strategies for minimizing interference from conspecific emissions2-4. However, it is well known from engineering that multiple spatially distributed emitters and receivers can greatly enhance environmental sensing (for example, multistatic radar and sonar)5-8. Here we provide evidence from modelling, neural recordings and behavioural experiments that the African weakly electric fish Gnathonemus petersii utilizes the electrical pulses of conspecifics to extend its electrolocation range, discriminate objects and increase information transmission. These results provide evidence for a new, collective mode of active sensing in which individual perception is enhanced by the energy emissions of nearby group members.

Lightning Pose: improved animal pose estimation via semi-supervised learning, Bayesian ensembling, and cloud-native open-source tools.

Contemporary pose estimation methods enable precise measurements of behavior via supervised deep learning with hand-labeled video frames. Although effective in many cases, the supervised approach requires extensive labeling and often produces outputs that are unreliable for downstream analyses. Here, we introduce "Lightning Pose," an efficient pose estimation package with three algorithmic contributions. First, in addition to training on a few labeled video frames, we use many unlabeled videos and penalize the network whenever its predictions violate motion continuity, multiple-view geometry, and posture plausibility (semi-supervised learning). Second, we introduce a network architecture that resolves occlusions by predicting pose on any given frame using surrounding unlabeled frames. Third, we refine the pose predictions post-hoc by combining ensembling and Kalman smoothing. Together, these components render pose trajectories more accurate and scientifically usable. We release a cloud application that allows users to label data, train networks, and predict new videos directly from the browser.

Collective Sensing in Electric Fish.

A number of organisms, including dolphins, bats, and electric fish, possess sophisticated active sensory systems that use self-generated signals (e.g. acoustic or electrical emissions) to probe the environment1,2. Studies of active sensing in social groups have typically focused on strategies for minimizing interference from conspecific emissions2-4. However, it is well-known from engineering that multiple spatially distributed emitters and receivers can greatly enhance environmental sensing (e.g. multistatic radar and sonar)5-8. Here we provide evidence from modeling, neural recordings, and behavioral experiments that the African weakly electric fish Gnathonemus petersii utilizes the electrical pulses of conspecifics to extend electrolocation range, discriminate objects, and increase information transmission. These results suggest a novel, collective mode of active sensing in which individual perception is enhanced by the energy emissions of nearby group members.

The Use of Supervised Learning Models in Studying Agonistic Behavior and Communication in Weakly Electric Fish.

Despite considerable advances, studying electrocommunication of weakly electric fish, particularly in pulse-type species, is challenging as very short signal epochs at variable intervals from a few hertz up to more than 100 Hz need to be assigned to individuals. In this study, we show that supervised learning approaches offer a promising tool to automate or semiautomate the workflow, and thereby allowing the analysis of much longer episodes of behavior in a reasonable amount of time. We provide a detailed workflow mainly based on open resource software. We demonstrate the usefulness by applying the approach to the analysis of dyadic interactions of Gnathonemus petersii. Coupling of the proposed methods with a boundary element modeling approach, we are thereby able to model the information gained and provided during agonistic encounters. The data indicate that the passive electrosensory input, in particular, provides sufficient information to localize a contender during the pre-contest phase, fish did not use or rely on the theoretically also available sensory information of the contest outcome-determining size difference between contenders before engaging in agonistic behavior.

Neurobiology: The power of pauses in electrocommunication.

A new study of social communication behavior in weakly electric fish identifies neural mechanisms that may account for the significance of silent pauses in communication.

Task-Related Sensorimotor Adjustments Increase the Sensory Range in Electrolocation.

Perception and motor control traditionally are studied separately. However, motor activity can serve as a scaffold to shape the sensory flow. This tight link between motor actions and sensing is particularly evident in active sensory systems. Here, we investigate how the weakly electric mormyrid fish Gnathonemus petersii of undetermined sex structure their sensing and motor behavior while learning a perceptual task. We find systematic adjustments of the motor behavior that correlate with an increased performance. Using a model to compute the electrosensory input, we show that these behavioral adjustments improve the sensory input. As we find low neuronal detection thresholds at the level of medullary electrosensory neurons, it seems that the behavior-driven improvements of the sensory input are highly suitable to overcome the sensory limitations, thereby increasing the sensory range. Our results show that motor control is an active component of sensory learning, demonstrating that a detailed understanding of contribution of motor actions to sensing is needed to understand even seemingly simple behaviors.SIGNIFICANCE STATEMENT Motor-guided sensation and perception are intertwined, with motor behavior serving as a scaffold to shape the sensory input. We characterized how the weakly electric mormyrid fish Gnathonemus petersii, as it learns a perceptual task, restructures its sensorimotor behavior. We find that systematic adjustments of the motor behavior correlate with increased performance and a shift of the sensory attention of the animal. Analyzing the afferent electrosensory input shows that a significant gain in information results from these sensorimotor adjustments. Our results show that motor control can be an active component of sensory learning. Researching the sensory corollaries of motor control thus can be crucial to understand sensory sensation and perception under naturalistic conditions.

Motion parallax in electric sensing.

A crucial step in forming spatial representations of the environment involves the estimation of relative distance. Active sampling through specific movements is considered essential for optimizing the sensory flow that enables the extraction of distance cues. However, in electric sensing, direct evidence for the generation and exploitation of sensory flow is lacking. Weakly electric fish rely on a self-generated electric field to navigate and capture prey in the dark. This electric sense provides a blurred representation of the environment, making the exquisite sensory abilities of electric fish enigmatic. Stereotyped back-and-forth swimming patterns reminiscent of visual peering movements are suggestive of the active generation of sensory flow, but how motion contributes to the disambiguation of the electrosensory world remains unclear. Here, we show that a dipole-like electric field geometry coupled to motion provides the physical basis for a nonvisual parallax. We then show in a behavioral assay that this cue is used for electrosensory distance perception across phylogenetically distant taxa of weakly electric fish. Notably, these species electrically sample the environment in temporally distinct ways (using discrete pulses or quasisinusoidal waves), suggesting a ubiquitous role for parallax in electric sensing. Our results demonstrate that electrosensory information is extracted from sensory flow and used in a behaviorally relevant context. A better understanding of motion-based electric sensing will provide insight into the sensorimotor coordination required for active sensing in general and may lead to improved electric field-based imaging applications in a variety of contexts.

Passive and active electroreception during agonistic encounters in the weakly electric fish Gymnotus omarorum.

Agonistic behaviour related to territorial defence is likely to be costly in terms of energy loss and risk of injury. Hence information about the fighting ability of a potential opponent could influence the outcome of the contest. We here study electric images of the territorial and aggressive weakly electric fish Gymnotus omarorum in the context of agonistic behaviour. We show that passive and active electric images may drive the approach towards an opponent. The likelihood of first attacks can be predicted in these fish based on electric image information, suggesting that aggressive interactions may in fact be triggered through the passive electrosensory information.

Computational modeling of electric imaging in weakly electric fish: insights for physiology, behavior and evolution.

Weakly electric fish can sense electric signals produced by other animals whether they are conspecifics, preys or predators. These signals, sensed by passive electroreception, sustain electrocommunication, mating and agonistic behavior. Weakly electric fish can also generate a weak electrical discharge with which they can actively sense the animate and inanimate objects in their surroundings. Understanding both sensory modalities depends on our knowledge of how pre-receptorial electric images are formed and how movements modify them during behavior. The inability of effectively measuring pre-receptorial fields at the level of the skin contrasts with the amount of knowledge on electric fields and the availability of computational methods for estimating them. In this work we review past work on modeling of electric organ discharge and electric images, showing the usefulness of these methods to calculate the field and providing a brief explanation of their principles. In addition, we focus on recent work demonstrating the potential of electric image modeling and what the method has to offer for experimentalists studying sensory physiology, behavior and evolution.

Electric imaging through evolution, a modeling study of commonalities and differences.

Modeling the electric field and images in electric fish contributes to a better understanding of the pre-receptor conditioning of electric images. Although the boundary element method has been very successful for calculating images and fields, complex electric organ discharges pose a challenge for active electroreception modeling. We have previously developed a direct method for calculating electric images which takes into account the structure and physiology of the electric organ as well as the geometry and resistivity of fish tissues. The present article reports a general application of our simulator for studying electric images in electric fish with heterogeneous, extended electric organs. We studied three species of Gymnotiformes, including both wave-type (Apteronotus albifrons) and pulse-type (Gymnotus obscurus and Gymnotus coropinae) fish, with electric organs of different complexity. The results are compared with the African (Gnathonemus petersii) and American (Gymnotus omarorum) electric fish studied previously. We address the following issues: 1) how to calculate equivalent source distributions based on experimental measurements, 2) how the complexity of the electric organ discharge determines the features of the electric field and 3) how the basal field determines the characteristics of electric images. Our findings allow us to generalize the hypothesis (previously posed for G. omarorum) in which the perioral region and the rest of the body play different sensory roles. While the "electrosensory fovea" appears suitable for exploring objects in detail, the rest of the body is likened to a "peripheral retina" for detecting the presence and movement of surrounding objects. We discuss the commonalities and differences between species. Compared to African species, American electric fish show a weaker field. This feature, derived from the complexity of distributed electric organs, may endow Gymnotiformes with the ability to emit site-specific signals to be detected in the short range by a conspecific and the possibility to evolve predator avoidance strategies.