Prioritizing replay when future goals are unknown.

Although hippocampal place cells replay nonlocal trajectories, the computational function of these events remains controversial. One hypothesis, formalized in a prominent reinforcement learning account, holds that replay plans routes to current goals. However, recent puzzling data appear to contradict this perspective by showing that replayed destinations lag current goals. These results may support an alternative hypothesis that replay updates route information to build a "cognitive map." Yet no similar theory exists to formalize this view, and it is unclear how such a map is represented or what role replay plays in computing it. We address these gaps by introducing a theory of replay that learns a map of routes to candidate goals, before reward is available or when its location may change. Our work extends the planning account to capture a general map-building function for replay, reconciling it with data, and revealing an unexpected relationship between the seemingly distinct hypotheses.

Dopamine-independent effect of rewards on choices through hidden-state inference.

Dopamine is implicated in adaptive behavior through reward prediction error (RPE) signals that update value estimates. There is also accumulating evidence that animals in structured environments can use inference processes to facilitate behavioral flexibility. However, it is unclear how these two accounts of reward-guided decision-making should be integrated. Using a two-step task for mice, we show that dopamine reports RPEs using value information inferred from task structure knowledge, alongside information about reward rate and movement. Nonetheless, although rewards strongly influenced choices and dopamine activity, neither activating nor inhibiting dopamine neurons at trial outcome affected future choice. These data were recapitulated by a neural network model where cortex learned to track hidden task states by predicting observations, while basal ganglia learned values and actions via RPEs. This shows that the influence of rewards on choices can stem from dopamine-independent information they convey about the world's state, not the dopaminergic RPEs they produce.

Lights, fiber, action! A primer on in vivo fiber photometry.

Fiber photometry is a key technique for characterizing brain-behavior relationships in vivo. Initially, it was primarily used to report calcium dynamics as a proxy for neural activity via genetically encoded indicators. This generated new insights into brain functions including movement, memory, and motivation at the level of defined circuits and cell types. Recently, the opportunity for discovery with fiber photometry has exploded with the development of an extensive range of fluorescent sensors for biomolecules including neuromodulators and peptides that were previously inaccessible in vivo. This critical advance, combined with the new availability of affordable "plug-and-play" recording systems, has made monitoring molecules with high spatiotemporal precision during behavior highly accessible. However, while opening exciting new avenues for research, the rapid expansion in fiber photometry applications has occurred without coordination or consensus on best practices. Here, we provide a comprehensive guide to help end-users execute, analyze, and suitably interpret fiber photometry studies.

Propagation of activity through the cortical hierarchy and perception are determined by neural variability.

Brains are composed of anatomically and functionally distinct regions performing specialized tasks, but regions do not operate in isolation. Orchestration of complex behaviors requires communication between brain regions, but how neural dynamics are organized to facilitate reliable transmission is not well understood. Here we studied this process directly by generating neural activity that propagates between brain regions and drives behavior, assessing how neural populations in sensory cortex cooperate to transmit information. We achieved this by imaging two densely interconnected regions-the primary and secondary somatosensory cortex (S1 and S2)-in mice while performing two-photon photostimulation of S1 neurons and assigning behavioral salience to the photostimulation. We found that the probability of perception is determined not only by the strength of the photostimulation but also by the variability of S1 neural activity. Therefore, maximizing the signal-to-noise ratio of the stimulus representation in cortex relative to the noise or variability is critical to facilitate activity propagation and perception.

Control of sustained attention and impulsivity by Gq-protein signalling in parvalbumin interneurons of the anterior cingulate cortex.

The anterior cingulate cortex (ACC) has been implicated in attention deficit hyperactivity disorder (ADHD). More specifically, an appropriate balance of excitatory and inhibitory activity in the ACC may be critical for the control of impulsivity, hyperactivity, and sustained attention which are centrally affected in ADHD. Hence, pharmacological augmentation of parvalbumin- (PV) or somatostatin-positive (Sst) inhibitory ACC interneurons could be a potential treatment strategy. We, therefore, tested whether stimulation of Gq-protein-coupled receptors (GqPCRs) in these interneurons could improve attention or impulsivity assessed with the 5-choice-serial reaction-time task in male mice. When challenging impulse control behaviourally or pharmacologically, activation of the chemogenetic GqPCR hM3Dq in ACC PV-cells caused a selective decrease of active erroneous-i.e. incorrect and premature-responses, indicating improved attentional and impulse control. When challenging attention, in contrast, omissions were increased, albeit without extension of reward latencies or decreases of attentional accuracy. These effects largely resembled those of the ADHD medication atomoxetine. Additionally, they were mostly independent of each other within individual animals. GqPCR activation in ACC PV-cells also reduced hyperactivity. In contrast, if hM3Dq was activated in Sst-interneurons, no improvement of impulse control was observed, and a reduction of incorrect responses was only induced at high agonist levels and accompanied by reduced motivational drive. These results suggest that the activation of GqPCRs expressed specifically in PV-cells of the ACC may be a viable strategy to improve certain aspects of sustained attention, impulsivity and hyperactivity in ADHD.

Complementary task representations in hippocampus and prefrontal cortex for generalizing the structure of problems.

Humans and other animals effortlessly generalize prior knowledge to solve novel problems, by abstracting common structure and mapping it onto new sensorimotor specifics. To investigate how the brain achieves this, in this study, we trained mice on a series of reversal learning problems that shared the same structure but had different physical implementations. Performance improved across problems, indicating transfer of knowledge. Neurons in medial prefrontal cortex (mPFC) maintained similar representations across problems despite their different sensorimotor correlates, whereas hippocampal (dCA1) representations were more strongly influenced by the specifics of each problem. This was true for both representations of the events that comprised each trial and those that integrated choices and outcomes over multiple trials to guide an animal's decisions. These data suggest that prefrontal cortex and hippocampus play complementary roles in generalization of knowledge: PFC abstracts the common structure among related problems, and hippocampus maps this structure onto the specifics of the current situation.

Explicit knowledge of task structure is a primary determinant of human model-based action.

Explicit information obtained through instruction profoundly shapes human choice behaviour. However, this has been studied in computationally simple tasks, and it is unknown how model-based and model-free systems, respectively generating goal-directed and habitual actions, are affected by the absence or presence of instructions. We assessed behaviour in a variant of a computationally more complex decision-making task, before and after providing information about task structure, both in healthy volunteers and in individuals suffering from obsessive-compulsive or other disorders. Initial behaviour was model-free, with rewards directly reinforcing preceding actions. Model-based control, employing predictions of states resulting from each action, emerged with experience in a minority of participants, and less in those with obsessive-compulsive disorder. Providing task structure information strongly increased model-based control, similarly across all groups. Thus, in humans, explicit task structural knowledge is a primary determinant of model-based reinforcement learning and is most readily acquired from instruction rather than experience.

Open-source, Python-based, hardware and software for controlling behavioural neuroscience experiments.

Laboratory behavioural tasks are an essential research tool. As questions asked of behaviour and brain activity become more sophisticated, the ability to specify and run richly structured tasks becomes more important. An increasing focus on reproducibility also necessitates accurate communication of task logic to other researchers. To these ends, we developed pyControl, a system of open-source hardware and software for controlling behavioural experiments comprising a simple yet flexible Python-based syntax for specifying tasks as extended state machines, hardware modules for building behavioural setups, and a graphical user interface designed for efficiently running high-throughput experiments on many setups in parallel, all with extensive online documentation. These tools make it quicker, easier, and cheaper to implement rich behavioural tasks at scale. As important, pyControl facilitates communication and reproducibility of behavioural experiments through a highly readable task definition syntax and self-documenting features. Here, we outline the system's design and rationale, present validation experiments characterising system performance, and demonstrate example applications in freely moving and head-fixed mouse behaviour.

A low-cost open-source 5-choice operant box system optimized for electrophysiology and optophysiology in mice.

Operant boxes enable the application of complex behavioural paradigms to support circuit neuroscience and drug discovery research. However, commercial operant box systems are expensive and often not optimised for combining behaviour with neurophysiology. Here we introduce a fully open-source Python-based operant-box system in a 5-choice design (pyOS-5) that enables assessment of multiple cognitive and affective functions. It is optimized for fast turn-over between animals, and for testing of tethered mice for simultaneous physiological recordings or optogenetic manipulation. For reward delivery, we developed peristaltic and syringe pumps based on a stepper motor and 3D-printed parts. Tasks are specified using a Python-based syntax implemented on custom-designed printed circuit boards that are commercially available at low cost. We developed an open-source graphical user interface (GUI) and task definition scripts to conduct assays assessing operant learning, attention, impulsivity, working memory, or cognitive flexibility, alleviating the need for programming skills of the end user. All behavioural events are recorded with millisecond resolution, and TTL-outputs and -inputs allow straightforward integration with physiological recordings and closed-loop manipulations. This combination of features realizes a cost-effective, nose-poke-based operant box system that allows reliable circuit-neuroscience experiments investigating correlates of cognition and emotion in large cohorts of subjects.

Distinct roles for dopamine clearance mechanisms in regulating behavioral flexibility.

Dopamine plays a crucial role in adaptive behavior, and dysfunctional dopamine is implicated in multiple psychiatric conditions characterized by inflexible or inconsistent choices. However, the precise relationship between dopamine and flexible decision making remains unclear. One reason is that, while many studies have focused on the activity of dopamine neurons, efficient dopamine signaling also relies on clearance mechanisms, notably the dopamine transporter (DAT), which predominates in striatum, and catechol-O-methyltransferase (COMT), which predominates in cortex. The exact locus, extent, and timescale of the effects of DAT and COMT are uncertain. Moreover, there is limited data on how acute disruption of either mechanism affects flexible decision making strategies mediated by cortico-striatal networks. To address these issues, we combined pharmacological modulation of DAT and COMT with electrochemistry and behavior in mice. DAT blockade, but not COMT inhibition, regulated sub-second dopamine release in the nucleus accumbens core, but surprisingly neither clearance mechanism affected evoked release in prelimbic cortex. This was not due to a lack of sensitivity, as both amphetamine and atomoxetine changed the kinetics of sub-second release. In a multi-step decision making task where mice had to respond to reversals in either reward probabilities or the choice sequence to reach the goal, DAT blockade selectively impaired, and COMT inhibition improved, performance after reward reversals, but neither manipulation affected the adaptation of choices after action-state transition reversals. Together, our data suggest that DAT and COMT shape specific aspects of behavioral flexibility by regulating different aspects of the kinetics of striatal and cortical dopamine, respectively.

Control of impulsivity by Gi-protein signalling in layer-5 pyramidal neurons of the anterior cingulate cortex.

Pathological impulsivity is a debilitating symptom of multiple psychiatric diseases with few effective treatment options. To identify druggable receptors with anti-impulsive action we developed a systematic target discovery approach combining behavioural chemogenetics and gene expression analysis. Spatially restricted inhibition of three subdivisions of the prefrontal cortex of mice revealed that the anterior cingulate cortex (ACC) regulates premature responding, a form of motor impulsivity. Probing three G-protein cascades with designer receptors, we found that the activation of Gi-signalling in layer-5 pyramidal cells (L5-PCs) of the ACC strongly, reproducibly, and selectively decreased challenge-induced impulsivity. Differential gene expression analysis across murine ACC cell-types and 402 GPCRs revealed that - among Gi-coupled receptor-encoding genes - Grm2 is the most selectively expressed in L5-PCs while alternative targets were scarce. Validating our approach, we confirmed that mGluR2 activation reduced premature responding. These results suggest Gi-coupled receptors in ACC L5-PCs as therapeutic targets for impulse control disorders.

What is dopamine doing in model-based reinforcement learning?

Experiments have implicated dopamine in model-based reinforcement learning (RL). These findings are unexpected as dopamine is thought to encode a reward prediction error (RPE), which is the key teaching signal in model-free RL. Here we examine two possible accounts for dopamine's involvement in model-based RL: the first that dopamine neurons carry a prediction error used to update a type of predictive state representation called a successor representation, the second that two well established aspects of dopaminergic activity, RPEs and surprise signals, can together explain dopamine's involvement in model-based RL.

The Anterior Cingulate Cortex Predicts Future States to Mediate Model-Based Action Selection.

Behavioral control is not unitary. It comprises parallel systems, model based and model free, that respectively generate flexible and habitual behaviors. Model-based decisions use predictions of the specific consequences of actions, but how these are implemented in the brain is poorly understood. We used calcium imaging and optogenetics in a sequential decision task for mice to show that the anterior cingulate cortex (ACC) predicts the state that actions will lead to, not simply whether they are good or bad, and monitors whether outcomes match these predictions. ACC represents the complete state space of the task, with reward signals that depend strongly on the state where reward is obtained but minimally on the preceding choice. Accordingly, ACC is necessary only for updating model-based strategies, not for basic reward-driven action reinforcement. These results reveal that ACC is a critical node in model-based control, with a specific role in predicting future states given chosen actions.

pyPhotometry: Open source Python based hardware and software for fiber photometry data acquisition.

Fiber photometry is the process of recording bulk neural activity by measuring fluorescence changes in activity sensitive indicators such as GCaMP through an optical fiber. We present a system of open source hardware and software for fiber photometry data acquisition consisting of a compact, low cost, data acquisition board built around the Micropython microcontroller, and a cross platform graphical user interface (GUI) for controlling acquisition and visualising signals. The system can acquire two analog and two digital signals, and control two external LEDs via built in LED drivers. Time-division multiplexed illumination allows independent readout of fluorescence evoked by different excitation wavelengths from a single photoreceiver signal. Validation experiments indicate this approach offers better signal to noise for a given average excitation light intensity than sinusoidally-modulated illumination. pyPhotometry is substantially cheaper than commercial hardware filling the same role, and we anticipate, as an open source and comparatively simple tool, it will be easily adaptable and therefore of broad interest to a wide range of users.

Analogue closed-loop optogenetic modulation of hippocampal pyramidal cells dissociates gamma frequency and amplitude.

Gamma-band oscillations are implicated in modulation of attention, integration of sensory information and flexible communication among anatomically connected brain areas. How networks become entrained is incompletely understood. Specifically, it is unclear how the spectral and temporal characteristics of network oscillations can be altered on rapid timescales needed for efficient communication. We use closed-loop optogenetic modulation of principal cell excitability in mouse hippocampal slices to interrogate the dynamical properties of hippocampal oscillations. Gamma frequency and amplitude can be modulated bi-directionally, and dissociated, by phase-advancing or delaying optogenetic feedback to pyramidal cells. Closed-loop modulation alters the synchrony rather than average frequency of action potentials, in principle avoiding disruption of population rate-coding of information. Modulation of phasic excitatory currents in principal neurons is sufficient to manipulate oscillations, suggesting that feed-forward excitation of pyramidal cells has an important role in determining oscillatory dynamics and the ability of networks to couple with one another.

(Reinforcement?) Learning to forage optimally.

Foraging effectively is critical to the survival of all animals and this imperative is thought to have profoundly shaped brain evolution. Decisions made by foraging animals often approximate optimal strategies, but the learning and decision mechanisms generating these choices remain poorly understood. Recent work with laboratory foraging tasks in humans suggest their behaviour is poorly explained by model-free reinforcement learning, with simple heuristic strategies better describing behaviour in some tasks, and in others evidence of prospective prediction of the future state of the environment. We suggest that model-based average reward reinforcement learning may provide a common framework for understanding these apparently divergent foraging strategies.

Simple Plans or Sophisticated Habits? State, Transition and Learning Interactions in the Two-Step Task.

The recently developed 'two-step' behavioural task promises to differentiate model-based from model-free reinforcement learning, while generating neurophysiologically-friendly decision datasets with parametric variation of decision variables. These desirable features have prompted its widespread adoption. Here, we analyse the interactions between a range of different strategies and the structure of transitions and outcomes in order to examine constraints on what can be learned from behavioural performance. The task involves a trade-off between the need for stochasticity, to allow strategies to be discriminated, and a need for determinism, so that it is worth subjects' investment of effort to exploit the contingencies optimally. We show through simulation that under certain conditions model-free strategies can masquerade as being model-based. We first show that seemingly innocuous modifications to the task structure can induce correlations between action values at the start of the trial and the subsequent trial events in such a way that analysis based on comparing successive trials can lead to erroneous conclusions. We confirm the power of a suggested correction to the analysis that can alleviate this problem. We then consider model-free reinforcement learning strategies that exploit correlations between where rewards are obtained and which actions have high expected value. These generate behaviour that appears model-based under these, and also more sophisticated, analyses. Exploiting the full potential of the two-step task as a tool for behavioural neuroscience requires an understanding of these issues.

When brains flip coins.

In a recent study in the journal Cell, Tervo et al. (2014) show that animals can implement stochastic choice policies in environments unfavorable to predictive strategies. The shift toward stochastic behavior was driven by noradrenergic signaling in the anterior cingulate cortex.

Oscillatory multiplexing of population codes for selective communication in the mammalian brain.

Mammalian brains exhibit population oscillations, the structures of which vary in time and space according to behavioural state. A proposed function of these oscillations is to control the flow of signals among anatomically connected networks. However, the nature of neural coding that may support selective communication that depends on oscillations has received relatively little attention. Here, we consider the role of multiplexing, whereby multiple information streams share a common neural substrate. We suggest that multiplexing implemented through periodic modulation of firing-rate population codes enables flexible reconfiguration of effective connectivity among brain areas.