Learning the sound inventory of a complex vocal skill via an intrinsic reward.

Reinforcement learning (RL) is thought to underlie the acquisition of vocal skills like birdsong and speech, where sounding like one's "tutor" is rewarding. However, what RL strategy generates the rich sound inventories for song or speech? We find that the standard actor-critic model of birdsong learning fails to explain juvenile zebra finches' efficient learning of multiple syllables. However, when we replace a single actor with multiple independent actors that jointly maximize a common intrinsic reward, then birds' empirical learning trajectories are accurately reproduced. The influence of each actor (syllable) on the magnitude of global reward is competitively determined by its acoustic similarity to target syllables. This leads to each actor matching the target it is closest to and, occasionally, to the competitive exclusion of an actor from the learning process (i.e., the learned song). We propose that a competitive-cooperative multi-actor RL (MARL) algorithm is key for the efficient learning of the action inventory of a complex skill.

Interactive extraction of diverse vocal units from a planar embedding without the need for prior sound segmentation.

Annotating and proofreading data sets of complex natural behaviors such as vocalizations are tedious tasks because instances of a given behavior need to be correctly segmented from background noise and must be classified with minimal false positive error rate. Low-dimensional embeddings have proven very useful for this task because they can provide a visual overview of a data set in which distinct behaviors appear in different clusters. However, low-dimensional embeddings introduce errors because they fail to preserve distances; and embeddings represent only objects of fixed dimensionality, which conflicts with vocalizations that have variable dimensions stemming from their variable durations. To mitigate these issues, we introduce a semi-supervised, analytical method for simultaneous segmentation and clustering of vocalizations. We define a given vocalization type by specifying pairs of high-density regions in the embedding plane of sound spectrograms, one region associated with vocalization onsets and the other with offsets. We demonstrate our two-neighborhood (2N) extraction method on the task of clustering adult zebra finch vocalizations embedded with UMAP. We show that 2N extraction allows the identification of short and long vocal renditions from continuous data streams without initially committing to a particular segmentation of the data. Also, 2N extraction achieves much lower false positive error rate than comparable approaches based on a single defining region. Along with our method, we present a graphical user interface (GUI) for visualizing and annotating data.

Fast Retrograde Access to Projection Neuron Circuits Underlying Vocal Learning in Songbirds.

Understanding the structure and function of neural circuits underlying speech and language is a vital step toward better treatments for diseases of these systems. Songbirds, among the few animal orders that share with humans the ability to learn vocalizations from a conspecific, have provided many insights into the neural mechanisms of vocal development. However, research into vocal learning circuits has been hindered by a lack of tools for rapid genetic targeting of specific neuron populations to meet the quick pace of developmental learning. Here, we present a viral tool that enables fast and efficient retrograde access to projection neuron populations. In zebra finches, Bengalese finches, canaries, and mice, we demonstrate fast retrograde labeling of cortical or dopaminergic neurons. We further demonstrate the suitability of our construct for detailed morphological analysis, for in vivo imaging of calcium activity, and for multi-color brainbow labeling.

Sensory substitution reveals a manipulation bias.

Sensory substitution is a promising therapeutic approach for replacing a missing or diseased sensory organ by translating inaccessible information into another sensory modality. However, many substitution systems are not well accepted by subjects. To explore the effect of sensory substitution on voluntary action repertoires and their associated affective valence, we study deaf songbirds to which we provide visual feedback as a substitute of auditory feedback. Surprisingly, deaf birds respond appetitively to song-contingent binary visual stimuli. They skillfully adapt their songs to increase the rate of visual stimuli, showing that auditory feedback is not required for making targeted changes to vocal repertoires. We find that visually instructed song learning is basal-ganglia dependent. Because hearing birds respond aversively to the same visual stimuli, sensory substitution reveals a preference for actions that elicit sensory feedback over actions that do not, suggesting that substitution systems should be designed to exploit the drive to manipulate.

Undirected singing rate as a non-invasive tool for welfare monitoring in isolated male zebra finches.

Research on the songbird zebra finch (Taeniopygia guttata) has advanced our behavioral, hormonal, neuronal, and genetic understanding of vocal learning. However, little is known about the impact of typical experimental manipulations on the welfare of these birds. Here we explore whether the undirected singing rate can be used as an indicator of welfare. We tested this idea by performing a post hoc analysis of singing behavior in isolated male zebra finches subjected to interactive white noise, to surgery, or to tethering. We find that the latter two experimental manipulations transiently but reliably decreased singing rates. By contraposition, we infer that a high-sustained singing rate is suggestive of successful coping or improved welfare in these experiments. Our analysis across more than 300 days of song data suggests that a singing rate above a threshold of several hundred song motifs per day implies an absence of an acute stressor or a successful coping with stress. Because singing rate can be measured in a completely automatic fashion, its observation can help to reduce experimenter bias in welfare monitoring. Because singing rate measurements are non-invasive, we expect this study to contribute to the refinement of the current welfare monitoring tools in zebra finches.

Nearest neighbours reveal fast and slow components of motor learning.

Changes in behaviour resulting from environmental influences, development and learning1-5 are commonly quantified on the basis of a few hand-picked features2-4,6,7 (for example, the average pitch of acoustic vocalizations3), assuming discrete classes of behaviours (such as distinct vocal syllables)2,3,8-10. However, such methods generalize poorly across different behaviours and model systems and may miss important components of change. Here we present a more-general account of behavioural change that is based on nearest-neighbour statistics11-13, and apply it to song development in a songbird, the zebra finch3. First, we introduce the concept of 'repertoire dating', whereby each rendition of a behaviour (for example, each vocalization) is assigned a repertoire time, reflecting when similar renditions were typical in the behavioural repertoire. Repertoire time isolates the components of vocal variability that are congruent with long-term changes due to vocal learning and development, and stratifies the behavioural repertoire into 'regressions', 'anticipations' and 'typical renditions'. Second, we obtain a holistic, yet low-dimensional, description of vocal change in terms of a stratified 'behavioural trajectory', revealing numerous previously unrecognized components of behavioural change on fast and slow timescales, as well as distinct patterns of overnight consolidation1,2,4,14,15 across the behavioral repertoire. We find that diurnal changes in regressions undergo only weak consolidation, whereas anticipations and typical renditions consolidate fully. Because of its generality, our nonparametric description of how behaviour evolves relative to itself-rather than to a potentially arbitrary, experimenter-defined goal2,3,14,16-appears well suited for comparing learning and change across behaviours and species17,18, as well as biological and artificial systems5.

Learning auditory discriminations from observation is efficient but less robust than learning from experience.

Social learning enables complex societies. However, it is largely unknown how insights obtained from observation compare with insights gained from trial-and-error, in particular in terms of their robustness. Here, we use aversive reinforcement to train "experimenter" zebra finches to discriminate between auditory stimuli in the presence of an "observer" finch. We show that experimenters are slow to successfully discriminate the stimuli, but immediately generalize their ability to a new set of similar stimuli. By contrast, observers subjected to the same task are able to discriminate the initial stimulus set, but require more time for successful generalization. Drawing on concepts from machine learning, we suggest that observer learning has evolved to rapidly absorb sensory statistics without pressure to minimize neural resources, whereas learning from experience is endowed with a form of regularization that enables robust inference.

Author Correction: Deep sleep maintains learning efficiency of the human brain.

This corrects the article DOI: 10.1038/ncomms15405.

Songbirds work around computational complexity by learning song vocabulary independently of sequence.

While acquiring motor skills, animals transform their plastic motor sequences to match desired targets. However, because both the structure and temporal position of individual gestures are adjustable, the number of possible motor transformations increases exponentially with sequence length. Identifying the optimal transformation towards a given target is therefore a computationally intractable problem. Here we show an evolutionary workaround for reducing the computational complexity of song learning in zebra finches. We prompt juveniles to modify syllable phonology and sequence in a learned song to match a newly introduced target song. Surprisingly, juveniles match each syllable to the most spectrally similar sound in the target, regardless of its temporal position, resulting in unnecessary sequence errors, that they later try to correct. Thus, zebra finches prioritize efficient learning of syllable vocabulary, at the cost of inefficient syntax learning. This strategy provides a non-optimal but computationally manageable solution to the task of vocal sequence learning.

Deep sleep maintains learning efficiency of the human brain.

It is hypothesized that deep sleep is essential for restoring the brain's capacity to learn efficiently, especially in regions heavily activated during the day. However, causal evidence in humans has been lacking due to the inability to sleep deprive one target area while keeping the natural sleep pattern intact. Here we introduce a novel approach to focally perturb deep sleep in motor cortex, and investigate the consequences on behavioural and neurophysiological markers of neuroplasticity arising from dedicated motor practice. We show that the capacity to undergo neuroplastic changes is reduced by wakefulness but restored during unperturbed sleep. This restorative process is markedly attenuated when slow waves are selectively perturbed in motor cortex, demonstrating that deep sleep is a requirement for maintaining sustainable learning efficiency.

A Bayesian Account of Vocal Adaptation to Pitch-Shifted Auditory Feedback.

Motor systems are highly adaptive. Both birds and humans compensate for synthetically induced shifts in the pitch (fundamental frequency) of auditory feedback stemming from their vocalizations. Pitch-shift compensation is partial in the sense that large shifts lead to smaller relative compensatory adjustments of vocal pitch than small shifts. Also, compensation is larger in subjects with high motor variability. To formulate a mechanistic description of these findings, we adapt a Bayesian model of error relevance. We assume that vocal-auditory feedback loops in the brain cope optimally with known sensory and motor variability. Based on measurements of motor variability, optimal compensatory responses in our model provide accurate fits to published experimental data. Optimal compensation correctly predicts sensory acuity, which has been estimated in psychophysical experiments as just-noticeable pitch differences. Our model extends the utility of Bayesian approaches to adaptive vocal behaviors.

A Neural Code That Is Isometric to Vocal Output and Correlates with Its Sensory Consequences.

What cortical inputs are provided to motor control areas while they drive complex learned behaviors? We study this question in the nucleus interface of the nidopallium (NIf), which is required for normal birdsong production and provides the main source of auditory input to HVC, the driver of adult song. In juvenile and adult zebra finches, we find that spikes in NIf projection neurons precede vocalizations by several tens of milliseconds and are insensitive to distortions of auditory feedback. We identify a local isometry between NIf output and vocalizations: quasi-identical notes produced in different syllables are preceded by highly similar NIf spike patterns. NIf multiunit firing during song precedes responses in auditory cortical neurons by about 50 ms, revealing delayed congruence between NIf spiking and a neural representation of auditory feedback. Our findings suggest that NIf codes for imminent acoustic events within vocal performance.

Rhythmic Continuous-Time Coding in the Songbird Analog of Vocal Motor Cortex.

Songbirds learn and produce complex sequences of vocal gestures. Adult birdsong requires premotor nucleus HVC, in which projection neurons (PNs) burst sparsely at stereotyped times in the song. It has been hypothesized that PN bursts, as a population, form a continuous sequence, while a different model of HVC function proposes that both HVC PN and interneuron activity is tightly organized around motor gestures. Using a large dataset of PNs and interneurons recorded in singing birds, we test several predictions of these models. We find that PN bursts in adult birds are continuously and nearly uniformly distributed throughout song. However, we also find that PN and interneuron firing rates exhibit significant 10-Hz rhythmicity locked to song syllables, peaking prior to syllable onsets and suppressed prior to offsets-a pattern that predominates PN and interneuron activity in HVC during early stages of vocal learning.

Linear Methods for Efficient and Fast Separation of Two Sources Recorded with a Single Microphone.

This letter addresses the problem of separating two speakers from a single microphone recording. Three linear methods are tested for source separation, all of which operate directly on sound spectrograms: (1) eigenmode analysis of covariance difference to identify spectro-temporal features associated with large variance for one source and small variance for the other source; (2) maximum likelihood demixing in which the mixture is modeled as the sum of two gaussian signals and maximum likelihood is used to identify the most likely sources; and (3) suppression-regression, in which autoregressive models are trained to reproduce one source and suppress the other. These linear approaches are tested on the problem of separating a known male from a known female speaker. The performance of these algorithms is assessed in terms of the residual error of estimated source spectrograms, waveform signal-to-noise ratio, and perceptual evaluation of speech quality scores. This work shows that the algorithms compare favorably to nonlinear approaches such as nonnegative sparse coding in terms of simplicity, performance, and suitability for real-time implementations, and they provide benchmark solutions for monaural source separation tasks.

Reconstruction of vocal interactions in a group of small songbirds.

The main obstacle for investigating vocal interactions in vertebrates is the difficulty of discriminating individual vocalizations of rapidly moving, sometimes simultaneously vocalizing individuals. We developed a method of recording and analyzing individual vocalizations in free-ranging animals using ultraminiature back-attached sound and acceleration recorders. Our method allows the separation of zebra finch vocalizations irrespective of background noise and the number of vocalizing animals nearby.

A higher sensory brain region is involved in reversing reinforcement-induced vocal changes in a songbird.

Many animals exhibit flexible behaviors that they can adjust to increase reward or avoid harm (learning by positive or aversive reinforcement). But what neural mechanisms allow them to restore their original behavior (motor program) after reinforcement is withdrawn? One possibility is that motor restoration relies on brain areas that have a role in memorization but no role in either motor production or in sensory processing relevant for expressing the behavior and its refinement. We investigated the role of a higher auditory brain area in the songbird for modifying and restoring the stereotyped adult song. We exposed zebra finches to aversively reinforcing white noise stimuli contingent on the pitch of one of their stereotyped song syllables. In response, birds significantly changed the pitch of that syllable to avoid the aversive reinforcer. After we withdrew reinforcement, birds recovered their original song within a few days. However, we found that large bilateral lesions in the caudal medial nidopallium (NCM, a high auditory area) impaired recovery of the original pitch even several weeks after withdrawal of the reinforcing stimuli. Because NCM lesions spared both successful noise-avoidance behavior and birds' auditory discrimination ability, our results show that NCM is not needed for directed motor changes or for auditory discriminative processing, but is implied in memorizing or recalling the memory of the recent song target.

Evidence for a causal inverse model in an avian cortico-basal ganglia circuit.

Learning by imitation is fundamental to both communication and social behavior and requires the conversion of complex, nonlinear sensory codes for perception into similarly complex motor codes for generating action. To understand the neural substrates underlying this conversion, we study sensorimotor transformations in songbird cortical output neurons of a basal-ganglia pathway involved in song learning. Despite the complexity of sensory and motor codes, we find a simple, temporally specific, causal correspondence between them. Sensory neural responses to song playback mirror motor-related activity recorded during singing, with a temporal offset of roughly 40 ms, in agreement with short feedback loop delays estimated using electrical and auditory stimulation. Such matching of mirroring offsets and loop delays is consistent with a recent Hebbian theory of motor learning and suggests that cortico-basal ganglia pathways could support motor control via causal inverse models that can invert the rich correspondence between motor exploration and sensory feedback.

Dynamic alignment models for neural coding.

Recently, there have been remarkable advances in modeling the relationships between the sensory environment, neuronal responses, and behavior. However, most models cannot encompass variable stimulus-response relationships such as varying response latencies and state or context dependence of the neural code. Here, we consider response modeling as a dynamic alignment problem and model stimulus and response jointly by a mixed pair hidden Markov model (MPH). In MPHs, multiple stimulus-response relationships (e.g., receptive fields) are represented by different states or groups of states in a Markov chain. Each stimulus-response relationship features temporal flexibility, allowing modeling of variable response latencies, including noisy ones. We derive algorithms for learning of MPH parameters and for inference of spike response probabilities. We show that some linear-nonlinear Poisson cascade (LNP) models are a special case of MPHs. We demonstrate the efficiency and usefulness of MPHs in simulations of both jittered and switching spike responses to white noise and natural stimuli. Furthermore, we apply MPHs to extracellular single and multi-unit data recorded in cortical brain areas of singing birds to showcase a novel method for estimating response lag distributions. MPHs allow simultaneous estimation of receptive fields, latency statistics, and hidden state dynamics and so can help to uncover complex stimulus response relationships that are subject to variable timing and involve diverse neural codes.

Activity in a premotor cortical nucleus of zebra finches is locally organized and exhibits auditory selectivity in neurons but not in glia.

Motor functions are often guided by sensory experience, most convincingly illustrated by complex learned behaviors. Key to sensory guidance in motor areas may be the structural and functional organization of sensory inputs and their evoked responses. We study sensory responses in large populations of neurons and neuron-assistive cells in the songbird motor area HVC, an auditory-vocal brain area involved in sensory learning and in adult song production. HVC spike responses to auditory stimulation display remarkable preference for the bird's own song (BOS) compared to other stimuli. Using two-photon calcium imaging in anesthetized zebra finches we measure the spatio-temporal structure of baseline activity and of auditory evoked responses in identified populations of HVC cells. We find strong correlations between calcium signal fluctuations in nearby cells of a given type, both in identified neurons and in astroglia. In identified HVC neurons only, auditory stimulation decorrelates ongoing calcium signals, less for BOS than for other sound stimuli. Overall, calcium transients show strong preference for BOS in identified HVC neurons but not in astroglia, showing diversity in local functional organization among identified neuron and astroglia populations.

At the interface of the auditory and vocal motor systems: NIf and its role in vocal processing, production and learning.

Communication between auditory and vocal motor nuclei is essential for vocal learning. In songbirds, the nucleus interfacialis of the nidopallium (NIf) is part of a sensorimotor loop, along with auditory nucleus avalanche (Av) and song system nucleus HVC, that links the auditory and song systems. Most of the auditory information comes through this sensorimotor loop, with the projection from NIf to HVC representing the largest single source of auditory information to the song system. In addition to providing the majority of HVC's auditory input, NIf is also the primary driver of spontaneous activity and premotor-like bursting during sleep in HVC. Like HVC and RA, two nuclei critical for song learning and production, NIf exhibits behavioral-state dependent auditory responses and strong motor bursts that precede song output. NIf also exhibits extended periods of fast gamma oscillations following vocal production. Based on the converging evidence from studies of physiology and functional connectivity it would be reasonable to expect NIf to play an important role in the learning, maintenance, and production of song. Surprisingly, however, lesions of NIf in adult zebra finches have no effect on song production or maintenance. Only the plastic song produced by juvenile zebra finches during the sensorimotor phase of song learning is affected by NIf lesions. In this review, we carefully examine what is known about NIf at the anatomical, physiological, and behavioral levels. We reexamine conclusions drawn from previous studies in the light of our current understanding of the song system, and establish what can be said with certainty about NIf's involvement in song learning, maintenance, and production. Finally, we review recent theories of song learning integrating possible roles for NIf within these frameworks and suggest possible parallels between NIf and sensorimotor areas that form part of the neural circuitry for speech processing in humans.