Mesostriatal dopamine is sensitive to changes in specific cue-reward contingencies.

Learning causal relationships relies on understanding how often one event precedes another. To investigate how dopamine neuron activity and neurotransmitter release change when a retrospective relationship is degraded for a specific pair of events, we used outcome-selective Pavlovian contingency degradation in rats. Conditioned responding was attenuated for the cue-reward contingency that was degraded, as was dopamine neuron activity in the midbrain and dopamine release in the ventral striatum in response to the cue and subsequent reward. Contingency degradation also abolished the trial-by-trial history dependence of the dopamine responses at the time of trial outcome. This profile of changes in cue- and reward-evoked responding is not easily explained by a standard reinforcement learning model. An alternative model based on learning causal relationships was better able to capture dopamine responses during contingency degradation, as well as conditioned behavior following optogenetic manipulations of dopamine during noncontingent rewards. Our results suggest that mesostriatal dopamine encodes the contingencies between meaningful events during learning.

Mesolimbic dopamine ramps reflect environmental timescales.

Mesolimbic dopamine activity occasionally exhibits ramping dynamics, reigniting debate on theories of dopamine signaling. This debate is ongoing partly because the experimental conditions under which dopamine ramps emerge remain poorly understood. Here, we show that during Pavlovian and instrumental conditioning, mesolimbic dopamine ramps are only observed when the inter-trial interval is short relative to the trial period. These results constrain theories of dopamine signaling and identify a critical variable determining the emergence of dopamine ramps.

Sensory cortical ensembles exhibit differential coupling to ripples in distinct hippocampal subregions.

Cortical neurons activated during recent experiences often reactivate with dorsal hippocampal CA1 ripples during subsequent rest. Less is known about cortical interactions with intermediate hippocampal CA1, whose connectivity, functions, and ripple events differ from dorsal CA1. We identified three clusters of putative excitatory neurons in mouse visual cortex that are preferentially excited together with either dorsal or intermediate CA1 ripples or suppressed before both ripples. Neurons in each cluster were evenly distributed across primary and higher visual cortices and co-active even in the absence of ripples. These ensembles exhibited similar visual responses but different coupling to thalamus and pupil-indexed arousal. We observed a consistent activity sequence preceding and predicting ripples: (1) suppression of ripple-suppressed cortical neurons, (2) thalamic silence, and (3) activation of intermediate CA1-ripple-activated cortical neurons. We propose that coordinated dynamics of these ensembles relay visual experiences to distinct hippocampal subregions for incorporation into different cognitive maps.

An open-source behavior controller for associative learning and memory (B-CALM).

Associative learning and memory, i.e., learning and remembering the associations between environmental stimuli, self-generated actions, and outcomes such as rewards or punishments, are critical for the well-being of animals. Hence, the neural mechanisms underlying these processes are extensively studied using behavioral tasks in laboratory animals. Traditionally, these tasks have been controlled using commercial hardware and software, which limits scalability and accessibility due to their cost. More recently, due to the revolution in microcontrollers or microcomputers, several general-purpose and open-source solutions have been advanced for controlling neuroscientific behavioral tasks. While these solutions have great strength due to their flexibility and general-purpose nature, for the same reasons, they suffer from some disadvantages including the need for considerable programming expertise, limited online visualization, or slower than optimal response latencies for any specific task. Here, to mitigate these concerns, we present an open-source behavior controller for associative learning and memory (B-CALM). B-CALM provides an integrated suite that can control a host of associative learning and memory behaviors. As proof of principle for its applicability, we show data from head-fixed mice learning Pavlovian conditioning, operant conditioning, discrimination learning, as well as a timing task and a choice task. These can be run directly from a user-friendly graphical user interface (GUI) written in MATLAB that controls many independently running Arduino Mega microcontrollers in parallel (one per behavior box). In sum, B-CALM will enable researchers to execute a wide variety of associative learning and memory tasks in a scalable, accurate, and user-friendly manner.

Few-shot learning: temporal scaling in behavioral and dopaminergic learning.

How do we learn associations in the world (e.g., between cues and rewards)? Cue-reward associative learning is controlled in the brain by mesolimbic dopamine1-4. It is widely believed that dopamine drives such learning by conveying a reward prediction error (RPE) in accordance with temporal difference reinforcement learning (TDRL) algorithms5. TDRL implementations are "trial-based": learning progresses sequentially across individual cue-outcome experiences. Accordingly, a foundational assumption-often considered a mere truism-is that the more cue-reward pairings one experiences, the more one learns this association. Here, we disprove this assumption, thereby falsifying a foundational principle of trial-based learning algorithms. Specifically, when a group of head-fixed mice received ten times fewer experiences over the same total time as another, a single experience produced as much learning as ten experiences in the other group. This quantitative scaling also holds for mesolimbic dopaminergic learning, with the increase in learning rate being so high that the group with fewer experiences exhibits dopaminergic learning in as few as four cue-reward experiences and behavioral learning in nine. An algorithm implementing reward-triggered retrospective learning explains these findings. The temporal scaling and few-shot learning observed here fundamentally changes our understanding of the neural algorithms of associative learning.

Sensory cortical ensembles exhibit differential coupling to ripples in distinct hippocampal subregions.

Cortical neurons activated during recent experiences often reactivate with dorsal hippocampal CA1 sharp-wave ripples (SWRs) during subsequent rest. Less is known about cortical interactions with intermediate hippocampal CA1, whose connectivity, functions, and SWRs differ from those of dorsal CA1. We identified three clusters of visual cortical excitatory neurons that are excited together with either dorsal or intermediate CA1 SWRs, or suppressed before both SWRs. Neurons in each cluster were distributed across primary and higher visual cortices and co-active even in the absence of SWRs. These ensembles exhibited similar visual responses but different coupling to thalamus and pupil-indexed arousal. We observed a consistent activity sequence: (i) suppression of SWR-suppressed cortical neurons, (ii) thalamic silence, and (iii) activation of the cortical ensemble preceding and predicting intermediate CA1 SWRs. We propose that the coordinated dynamics of these ensembles relay visual experiences to distinct hippocampal subregions for incorporation into different cognitive maps.

Mesolimbic dopamine release conveys causal associations.

Learning to predict rewards based on environmental cues is essential for survival. It is believed that animals learn to predict rewards by updating predictions whenever the outcome deviates from expectations, and that such reward prediction errors (RPEs) are signaled by the mesolimbic dopamine system-a key controller of learning. However, instead of learning prospective predictions from RPEs, animals can infer predictions by learning the retrospective cause of rewards. Hence, whether mesolimbic dopamine instead conveys a causal associative signal that sometimes resembles RPE remains unknown. We developed an algorithm for retrospective causal learning and found that mesolimbic dopamine release conveys causal associations but not RPE, thereby challenging the dominant theory of reward learning. Our results reshape the conceptual and biological framework for associative learning.

How do real animals account for the passage of time during associative learning?

Animals routinely learn to associate environmental stimuli and self-generated actions with their outcomes such as rewards. One of the most popular theoretical models of such learning is the reinforcement learning (RL) framework. The simplest form of RL, model-free RL, is widely applied to explain animal behavior in numerous neuroscientific studies. More complex RL versions assume that animals build and store an explicit model of the world in memory. To apply these approaches to explain animal behavior, typical neuroscientific RL models make implicit assumptions about how real animals represent the passage of time. In this perspective, I explicitly list these assumptions and show that they have several problematic implications. I hope that the explicit discussion of these problems encourages the field to seriously examine the assumptions underlying timing and reinforcement learning. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Interoceptive Inception in Insula.

The neural mechanisms underlying interoception-the sensation of internal physiological states-remain largely unresolved. In this issue of Neuron, Livneh et al. (2020) demonstrate that the insula bridges different timescales of interoception by tracking and predicting thirst and hunger states.

Social Stimuli Induce Activation of Oxytocin Neurons Within the Paraventricular Nucleus of the Hypothalamus to Promote Social Behavior in Male Mice.

Oxytocin (OT) is critical for the expression of social behavior across a wide array of species; however, the role of this system in the encoding of socially relevant information is not well understood. In the present study, we show that chemogenetic activation of OT neurons within the paraventricular nucleus of the hypothalamus (PVH) of male mice (OT-Ires-Cre) enhanced social investigation during a social choice test, while chemogenetic inhibition of these neurons abolished typical social preferences. These data suggest that activation of the OT system is necessary to direct behavior preferentially toward social stimuli. To determine whether the presence of a social stimulus is sufficient to induce activation of PVH-OT neurons, we performed the first definitive recording of OT neurons in awake mice using two-photon calcium imaging. These recordings demonstrate that social stimuli activate PVH-OT neurons and that these neurons differentially encode social and nonsocial stimuli, suggesting that PVH-OT neurons may act to convey social salience of environmental stimuli. Finally, an attenuation of social salience is associated with social disorders, such as autism. We therefore also examined possible OT system dysfunction in a mouse model of autism, Shank3b knock-out (KO) mice. Male Shank3b KO mice showed a marked reduction in PVH-OT neuron number and administration of an OT receptor agonist improved social deficits. Overall, these data suggest that the presence of a social stimulus induces activation of the PVH-OT neurons to promote adaptive social behavior responses.SIGNIFICANCE STATEMENT Although the oxytocin (OT) system is well known to regulate a diverse array of social behaviors, the mechanism in which OT acts to promote the appropriate social response is poorly understood. One hypothesis is that the presence of social conspecifics activates the OT system to generate an adaptive social response. Here, we selectively recorded from OT neurons in the paraventricular hypothalamic nucleus (PVH) to show that social stimulus exposure indeed induces activation of the OT system. We also show that activation of the OT system is necessary to promote social behavior and that mice with abnormal social behavior have reduced numbers of PVH-OT neurons. Finally, aberrant social behavior in these mice was rescued by administration of an OT receptor agonist.

Persistent activation of central amygdala CRF neurons helps drive the immediate fear extinction deficit.

Fear extinction is an active learning process whereby previously established conditioned responses to a conditioned stimulus are suppressed. Paradoxically, when extinction training is performed immediately following fear acquisition, the extinction memory is weakened. Here, we demonstrate that corticotrophin-releasing factor (CRF)-expressing neurons in the central amygdala (CeA) antagonize the extinction memory following immediate extinction training. CeA-CRF neurons transition from responding to the unconditioned stimulus to the conditioned stimulus during the acquisition of a fear memory that persists during immediate extinction training, but diminishes during delayed extinction training. Inhibition of CeA-CRF neurons during immediate extinction training is sufficient to promote enhanced extinction memories, and activation of these neurons following delay extinction training is sufficient to reinstate a previously extinguished fear memory. These results demonstrate CeA-CRF neurons are an important substrate for the persistence of fear and have broad implications for the neural basis of persistent negative affective behavioral states.

Single-cell activity tracking reveals that orbitofrontal neurons acquire and maintain a long-term memory to guide behavioral adaptation.

Learning to predict rewards based on environmental cues is essential for survival. The orbitofrontal cortex (OFC) contributes to such learning by conveying reward-related information to brain areas such as the ventral tegmental area (VTA). Despite this, how cue-reward memory representations form in individual OFC neurons and are modified based on new information is unknown. To address this, using in vivo two-photon calcium imaging in mice, we tracked the response evolution of thousands of OFC output neurons, including those projecting to VTA, through multiple days and stages of cue-reward learning. Collectively, we show that OFC contains several functional clusters of neurons distinctly encoding cue-reward memory representations, with only select responses routed downstream to VTA. Unexpectedly, these representations were stably maintained by the same neurons even after extinction of the cue-reward pairing, and supported behavioral learning and memory. Thus, OFC neuronal activity represents a long-term cue-reward associative memory to support behavioral adaptation.

The Timing of Reward-Seeking Action Tracks Visually Cued Theta Oscillations in Primary Visual Cortex.

An emerging body of work challenges the view that primary visual cortex (V1) represents the visual world faithfully. Theta oscillations in the local field potential (LFP) of V1 have been found to convey temporal expectations and, specifically, to express the delay between a visual stimulus and the reward that it portends. We extend this work by showing how these oscillatory states in male, wild-type rats can even relate to the timing of a visually cued reward-seeking behavior. In particular, we show that, with training, high precision and accuracy in behavioral timing tracks the power of these oscillations and the time of action execution covaries with their duration. These LFP oscillations are also intimately related to spiking responses at the single-unit level, which themselves carry predictive timing information. Together, these observations extend our understanding of the role of cortical oscillations in timing generally and the role of V1 in the timing of visually cued behaviors specifically.SIGNIFICANCE STATEMENT Traditionally, primary visual cortex (V1) has been regarded as playing a purely perceptual role in stimulus-driven behaviors. Recent work has challenged that view by showing that theta oscillations in rodent V1 may come to convey timed expectations. Here, we show that these theta oscillations carry predictive information about timed reward-seeking actions, thus elucidating a behavioral role for theta oscillations in V1 and extending our understanding of the role of V1 in decision making.

The habenula.

The habenula is a tiny brain region the size of a pea in humans. This region is highly conserved across vertebrates and has been traditionally overlooked by neuroscientists. The name habenula is derived from the Latin word habena, meaning "little rein", because of its elongated shape. Originally its function was thought to be related to the regulation of the nearby pineal gland (which Rene Descartes described as the "principal seat of the soul"). More recent evidence, however, demonstrates that the habenula acts as a critical neuroanatomical hub that connects and regulates brain regions important for divergent motivational states and cognition. In this Primer, we will discuss the recent and converging evidence that points to the habenula as a key brain region for motivation and decision-making.

Cell-Type-Specific Optogenetics in Monkeys.

The recent advent of technologies enabling cell-type-specific recording and manipulation of neuronal activity spurred tremendous progress in neuroscience. However, they have been largely limited to mice, which lack the richness in behavior of primates. Stauffer et al. now present a generalizable method for achieving cell-type specificity in monkeys.

Rationalizing spatial exploration patterns of wild animals and humans through a temporal discounting framework.

Understanding the exploration patterns of foragers in the wild provides fundamental insight into animal behavior. Recent experimental evidence has demonstrated that path lengths (distances between consecutive turns) taken by foragers are well fitted by a power law distribution. Numerous theoretical contributions have posited that "Lévy random walks"-which can produce power law path length distributions-are optimal for memoryless agents searching a sparse reward landscape. It is unclear, however, whether such a strategy is efficient for cognitively complex agents, from wild animals to humans. Here, we developed a model to explain the emergence of apparent power law path length distributions in animals that can learn about their environments. In our model, the agent's goal during search is to build an internal model of the distribution of rewards in space that takes into account the cost of time to reach distant locations (i.e., temporally discounting rewards). For an agent with such a goal, we find that an optimal model of exploration in fact produces hyperbolic path lengths, which are well approximated by power laws. We then provide support for our model by showing that humans in a laboratory spatial exploration task search space systematically and modify their search patterns under a cost of time. In addition, we find that path length distributions in a large dataset obtained from free-ranging marine vertebrates are well described by our hyperbolic model. Thus, we provide a general theoretical framework for understanding spatial exploration patterns of cognitively complex foragers.

Visualization of cortical, subcortical and deep brain neural circuit dynamics during naturalistic mammalian behavior with head-mounted microscopes and chronically implanted lenses.

Genetically encoded calcium indicators for visualizing dynamic cellular activity have greatly expanded our understanding of the brain. However, owing to the light-scattering properties of the brain, as well as the size and rigidity of traditional imaging technology, in vivo calcium imaging has been limited to superficial brain structures during head-fixed behavioral tasks. These limitations can now be circumvented by using miniature, integrated microscopes in conjunction with an implantable microendoscopic lens to guide light into and out of the brain, thus permitting optical access to deep brain (or superficial) neural ensembles during naturalistic behaviors. Here we describe steps to conduct such imaging studies using mice. However, we anticipate that the protocol can be easily adapted for use in other small vertebrates. Successful completion of this protocol will permit cellular imaging of neuronal activity and the generation of data sets with sufficient statistical power to correlate neural activity with stimulus presentation, physiological state and other aspects of complex behavioral tasks. This protocol takes 6-11 weeks to complete.

Analytical Calculation of Errors in Time and Value Perception Due to a Subjective Time Accumulator: A Mechanistic Model and the Generation of Weber's Law.

It has been previously shown (Namboodiri, Mihalas, Marton, & Hussain Shuler, 2014) that an evolutionary theory of decision making and time perception is capable of explaining numerous behavioral observations regarding how humans and animals decide between differently delayed rewards of differing magnitudes and how they perceive time. An implementation of this theory using a stochastic drift-diffusion accumulator model (Namboodiri, Mihalas, & Hussain Shuler, 2014a) showed that errors in time perception and decision making approximately obey Weber's law for a range of parameters. However, prior calculations did not have a clear mechanistic underpinning. Further, these calculations were only approximate, with the range of parameters being limited. In this letter, we provide a full analytical treatment of such an accumulator model, along with a mechanistic implementation, to calculate the expression of these errors for the entirety of the parameter space. In our mechanistic model, Weber's law results from synaptic facilitation and depression within the feedback synapses of the accumulator. Our theory also makes the prediction that the steepness of temporal discounting can be affected by requiring the precise timing of temporal intervals. Thus, by presenting exact quantitative calculations, this work provides falsifiable predictions for future experimental testing.

Visually cued action timing in the primary visual cortex.

Most behaviors are generated in three steps: sensing the external world, processing that information to instruct decision-making, and producing a motor action. Sensory areas, especially primary sensory cortices, have long been held to be involved only in the first step of this sequence. Here, we develop a visually cued interval timing task that requires rats to decide when to perform an action following a brief visual stimulus. Using single-unit recordings and optogenetics in this task, we show that activity generated by the primary visual cortex (V1) embodies the target interval and may instruct the decision to time the action on a trial-by-trial basis. A spiking neuronal model of local recurrent connections in V1 produces neural responses that predict and drive the timing of future actions, rationalizing our observations. Our data demonstrate that the primary visual cortex may contribute to the instruction of visually cued timed actions.

A temporal basis for Weber's law in value perception.

Weber's law-the observation that the ability to perceive changes in magnitudes of stimuli is proportional to the magnitude-is a widely observed psychophysical phenomenon. It is also believed to underlie the perception of reward magnitudes and the passage of time. Since many ecological theories state that animals attempt to maximize reward rates, errors in the perception of reward magnitudes and delays must affect decision-making. Using an ecological theory of decision-making (TIMERR), we analyze the effect of multiple sources of noise (sensory noise, time estimation noise, and integration noise) on reward magnitude and subjective value perception. We show that the precision of reward magnitude perception is correlated with the precision of time perception and that Weber's law in time estimation can lead to Weber's law in value perception. The strength of this correlation is predicted to depend on the reward history of the animal. Subsequently, we show that sensory integration noise (either alone or in combination with time estimation noise) also leads to Weber's law in reward magnitude perception in an accumulator model, if it has balanced Poisson feedback. We then demonstrate that the noise in subjective value of a delayed reward, due to the combined effect of noise in both the perception of reward magnitude and delay, also abides by Weber's law. Thus, in our theory we prove analytically that the perception of reward magnitude, time, and subjective value change all approximately obey Weber's law.