Lightning Pose: improved animal pose estimation via semi-supervised learning, Bayesian ensembling, and cloud-native open-source tools.

Contemporary pose estimation methods enable precise measurements of behavior via supervised deep learning with hand-labeled video frames. Although effective in many cases, the supervised approach requires extensive labeling and often produces outputs that are unreliable for downstream analyses. Here, we introduce "Lightning Pose," an efficient pose estimation package with three algorithmic contributions. First, in addition to training on a few labeled video frames, we use many unlabeled videos and penalize the network whenever its predictions violate motion continuity, multiple-view geometry, and posture plausibility (semi-supervised learning). Second, we introduce a network architecture that resolves occlusions by predicting pose on any given frame using surrounding unlabeled frames. Third, we refine the pose predictions post-hoc by combining ensembling and Kalman smoothing. Together, these components render pose trajectories more accurate and scientifically usable. We release a cloud application that allows users to label data, train networks, and predict new videos directly from the browser.

Neural Correlates of Multisensory Detection Behavior: Comparison of Primary and Higher-Order Visual Cortex.

We act upon stimuli in our surrounding environment by gathering the multisensory information they convey and by integrating this information to decide on a behavioral action. We hypothesized that the anterolateral secondary visual cortex (area AL) of the mouse brain may serve as a hub for sensorimotor transformation of audiovisual information. We imaged neuronal activity in primary visual cortex (V1) and AL of the mouse during a detection task using visual, auditory, and audiovisual stimuli. We found that AL neurons were more sensitive to weak uni- and multisensory stimuli compared to V1. Depending on contrast, different subsets of AL and V1 neurons showed cross-modal modulation of visual responses. During audiovisual stimulation, AL neurons showed stronger differentiation of behaviorally reported versus unreported stimuli compared to V1, whereas V1 showed this distinction during unisensory visual stimulation. Thus, neural population activity in area AL correlates more closely with multisensory detection behavior than V1.

The circuit architecture of cortical multisensory processing: Distinct functions jointly operating within a common anatomical network.

Our perceptual systems continuously process sensory inputs from different modalities and organize these streams of information such that our subjective representation of the outside world is a unified experience. By doing so, they also enable further cognitive processing and behavioral action. While cortical multisensory processing has been extensively investigated in terms of psychophysics and mesoscale neural correlates, an in depth understanding of the underlying circuit-level mechanisms is lacking. Previous studies on circuit-level mechanisms of multisensory processing have predominantly focused on cue integration, i.e. the mechanism by which sensory features from different modalities are combined to yield more reliable stimulus estimates than those obtained by using single sensory modalities. In this review, we expand the framework on the circuit-level mechanisms of cortical multisensory processing by highlighting that multisensory processing is a family of functions - rather than a single operation - which involves not only the integration but also the segregation of modalities. In addition, multisensory processing not only depends on stimulus features, but also on cognitive resources, such as attention and memory, as well as behavioral context, to determine the behavioral outcome. We focus on rodent models as a powerful instrument to study the circuit-level bases of multisensory processes, because they enable combining cell-type-specific recording and interventional techniques with complex behavioral paradigms. We conclude that distinct multisensory processes share overlapping anatomical substrates, are implemented by diverse neuronal micro-circuitries that operate in parallel, and are flexibly recruited based on factors such as stimulus features and behavioral constraints.

Audiovisual Integration Enhances Stimulus Detection Performance in Mice.

The detection of objects in the external world improves when humans and animals integrate object features of multiple sensory modalities. Behavioral and neuronal mechanisms underlying multisensory stimulus detection are poorly understood, mainly because they have not been investigated with suitable behavioral paradigms. Such behavioral paradigms should (i) elicit a robust multisensory gain, (ii) incorporate systematic calibration of stimulus amplitude to the sensory capacities of the individual subject, (iii) yield a high trial count, and (iv) be easily compatible with a large variety of neurophysiological recording techniques. We developed an audiovisual stimulus detection task for head-fixed mice which meets all of these critical behavioral constraints. Behavioral data obtained with this task indicated a robust increase in detection performance of multisensory stimuli compared with unisensory cues, which was maximal when both stimulus constituents were presented at threshold intensity. The multisensory behavioral effect was associated with a change in the perceptual performance which consisted of two components. First, the visual and auditory perceptual systems increased their sensitivity meaning that low intensity stimuli were more often detected. Second, enhanced acuity enabled the systems to better classify whether there was a stimulus or not. Fitting our data to signal detection models revealed that the multisensory gain was more likely to be achieved by integration of sensory signals rather than by stimulus redundancy or competition. This validated behavioral paradigm can be exploited to reliably investigate the neuronal correlates of multisensory stimulus detection at the level of single neurons, microcircuits, and larger perceptual systems.

Conditioning sharpens the spatial representation of rewarded stimuli in mouse primary visual cortex.

Reward is often employed as reinforcement in behavioral paradigms but it is unclear how the visuospatial aspect of a stimulus-reward association affects the cortical representation of visual space. Using a head-fixed paradigm, we conditioned mice to associate the same visual pattern in adjacent retinotopic regions with availability and absence of reward. Time-lapse intrinsic optical signal imaging under anesthesia showed that conditioning increased the spatial separation of mesoscale cortical representations of reward predicting- and non-reward predicting stimuli. Subsequent in vivo two-photon calcium imaging revealed that this improved separation correlated with enhanced population coding for retinotopic location, specifically for the trained orientation and spatially confined to the V1 region where rewarded and non-rewarded stimulus representations bordered. These results are corroborated by conditioning-induced differences in the correlation structure of population activity. Thus, the cortical representation of visual space is sharpened as consequence of associative stimulus-reward learning while the overall retinotopic map remains unaltered.

Audiovisual Modulation in Mouse Primary Visual Cortex Depends on Cross-Modal Stimulus Configuration and Congruency.

The sensory neocortex is a highly connected associative network that integrates information from multiple senses, even at the level of the primary sensory areas. Although a growing body of empirical evidence supports this view, the neural mechanisms of cross-modal integration in primary sensory areas, such as the primary visual cortex (V1), are still largely unknown. Using two-photon calcium imaging in awake mice, we show that the encoding of audiovisual stimuli in V1 neuronal populations is highly dependent on the features of the stimulus constituents. When the visual and auditory stimulus features were modulated at the same rate (i.e., temporally congruent), neurons responded with either an enhancement or suppression compared with unisensory visual stimuli, and their prevalence was balanced. Temporally incongruent tones or white-noise bursts included in audiovisual stimulus pairs resulted in predominant response suppression across the neuronal population. Visual contrast did not influence multisensory processing when the audiovisual stimulus pairs were congruent; however, when white-noise bursts were used, neurons generally showed response suppression when the visual stimulus contrast was high whereas this effect was absent when the visual contrast was low. Furthermore, a small fraction of V1 neurons, predominantly those located near the lateral border of V1, responded to sound alone. These results show that V1 is involved in the encoding of cross-modal interactions in a more versatile way than previously thought.SIGNIFICANCE STATEMENT The neural substrate of cross-modal integration is not limited to specialized cortical association areas but extends to primary sensory areas. Using two-photon imaging of large groups of neurons, we show that multisensory modulation of V1 populations is strongly determined by the individual and shared features of cross-modal stimulus constituents, such as contrast, frequency, congruency, and temporal structure. Congruent audiovisual stimulation resulted in a balanced pattern of response enhancement and suppression compared with unisensory visual stimuli, whereas incongruent or dissimilar stimuli at full contrast gave rise to a population dominated by response-suppressing neurons. Our results indicate that V1 dynamically integrates nonvisual sources of information while still attributing most of its resources to coding visual information.

Reward Expectancy Strengthens CA1 Theta and Beta Band Synchronization and Hippocampal-Ventral Striatal Coupling.

The use of information from the hippocampal memory system in motivated behavior depends on its communication with the ventral striatum. When an animal encounters cues that signal subsequent reward, its reward expectancy is raised. It is unknown, however, how this process affects hippocampal dynamics and their influence on target structures, such as ventral striatum. We show that, in rats, reward-predictive cues result in enhanced hippocampal theta and beta band rhythmic activity during subsequent action, compared with uncued goal-directed navigation. The beta band component, also labeled theta's harmonic, involves selective hippocampal CA1 cell groups showing frequency doubling of firing periodicity relative to theta rhythmicity and it partitions the theta cycle into segments showing clear versus poor spike timing organization. We found that theta phase precession occurred over a wider range than previously reported. This was apparent from spikes emitted near the peak of the theta cycle exhibiting large "phase precessing jumps" relative to spikes in foregoing cycles. Neither this phenomenon nor the regular manifestation of theta phase precession was affected by reward expectancy. Ventral striatal neuronal firing phase-locked not only to hippocampal theta, but also to beta band activity. Both hippocampus and ventral striatum showed increased synchronization between neuronal firing and local field potential activity during cued compared with uncued goal approaches. These results suggest that cue-triggered reward expectancy intensifies hippocampal output to target structures, such as the ventral striatum, by which the hippocampus may gain prioritized access to systems modulating motivated behaviors. SIGNIFICANCE STATEMENT: Here we show that temporally discrete cues raising reward expectancy enhance both theta and beta band activity in the hippocampus once goal-directed navigation has been initiated. These rhythmic activities are associated with increased synchronization of neuronal firing patterns in the hippocampus and the connected ventral striatum. When transmitted to downstream target structures, this expectancy-related state of intensified processing in the hippocampus may modulate goal-directed action.

Population-Level Neural Codes Are Robust to Single-Neuron Variability from a Multidimensional Coding Perspective.

Sensory neurons are often tuned to particular stimulus features, but their responses to repeated presentation of the same stimulus can vary over subsequent trials. This presents a problem for understanding the functioning of the brain, because downstream neuronal populations ought to construct accurate stimulus representations, even upon singular exposure. To study how trial-by-trial fluctuations (i.e., noise) in activity influence cortical representations of sensory input, we performed chronic calcium imaging of GCaMP6-expressing populations in mouse V1. We observed that high-dimensional response correlations, i.e., dependencies in activation strength among multiple neurons, can be used to predict single-trial, single-neuron noise. These multidimensional correlations are structured such that variability in the response of single neurons is relatively harmless to population representations of visual stimuli. We propose that multidimensional coding may represent a canonical principle of cortical circuits, explaining why the apparent noisiness of neuronal responses is compatible with accurate neural representations of stimulus features.