Mesolimbic dopamine adapts the rate of learning from action.

Recent success in training artificial agents and robots derives from a combination of direct learning of behavioural policies and indirect learning through value functions1-3. Policy learning and value learning use distinct algorithms that optimize behavioural performance and reward prediction, respectively. In animals, behavioural learning and the role of mesolimbic dopamine signalling have been extensively evaluated with respect to reward prediction4; however, so far there has been little consideration of how direct policy learning might inform our understanding5. Here we used a comprehensive dataset of orofacial and body movements to understand how behavioural policies evolved as naive, head-restrained mice learned a trace conditioning paradigm. Individual differences in initial dopaminergic reward responses correlated with the emergence of learned behavioural policy, but not the emergence of putative value encoding for a predictive cue. Likewise, physiologically calibrated manipulations of mesolimbic dopamine produced several effects inconsistent with value learning but predicted by a neural-network-based model that used dopamine signals to set an adaptive rate, not an error signal, for behavioural policy learning. This work provides strong evidence that phasic dopamine activity can regulate direct learning of behavioural policies, expanding the explanatory power of reinforcement learning models for animal learning6.

Basal Ganglia Circuits for Action Specification.

Behavior is readily classified into patterns of movements with inferred common goals-actions. Goals may be discrete; movements are continuous. Through the careful study of isolated movements in laboratory settings, or via introspection, it has become clear that animals can exhibit exquisite graded specification to their movements. Moreover, graded control can be as fundamental to success as the selection of which action to perform under many naturalistic scenarios: a predator adjusting its speed to intercept moving prey, or a tool-user exerting the perfect amount of force to complete a delicate task. The basal ganglia are a collection of nuclei in vertebrates that extend from the forebrain (telencephalon) to the midbrain (mesencephalon), constituting a major descending extrapyramidal pathway for control over midbrain and brainstem premotor structures. Here we discuss how this pathway contributes to the continuous specification of movements that endows our voluntary actions with vigor and grace.

The contribution of extrasynaptic signaling to cerebellar information processing.

The diversity of synapses within the simple modular structure of the cerebellum has been crucial for study of the phasic extrasynaptic signaling by fast neurotransmitters collectively referred to as "spillover." Additionally, the accessibility of cerebellar components for in vivo recordings and their recruitment by simple behaviors or sensory stimuli has allowed for both direct and indirect demonstrations of the effects of transmitter spillover in the intact brain. The continued study of spillover in the cerebellum not only promotes our understanding of information transfer through cerebellar structures but also how extrasynaptic signaling may be regulated and interpreted throughout the CNS.

Afferent convergence to a shared population of interneuron AMPA receptors.

Precise alignment of pre- and postsynaptic elements optimizes the activation of glutamate receptors at excitatory synapses. Nonetheless, glutamate that diffuses out of the synaptic cleft can have actions at distant receptors, a mode of transmission called spillover. To uncover the extrasynaptic actions of glutamate, we localized AMPA receptors (AMPARs) mediating spillover transmission between climbing fibers and molecular layer interneurons in the cerebellar cortex. We found that climbing fiber spillover generates calcium transients mediated by Ca2+-permeable AMPARs at parallel fiber synapses. Spillover occludes parallel fiber synaptic currents, indicating that separate, independently regulated afferent pathways converge onto a common pool of AMPARs. Together these findings demonstrate a circuit motif wherein glutamate 'spill-in' from an unconnected afferent pathway co-opts synaptic receptors, allowing activation of postsynaptic AMPARs even when canonical glutamate release is suppressed.

In Vivo Optogenetics with Stimulus Calibration.

Optogenetic reagents allow for depolarization and hyperpolarization of cells with light. This provides unprecedented spatial and temporal resolution to the control of neuronal activity both in vitro and in vivo. In the intact animal this requires strategies to deliver light deep into the highly scattering tissue of the brain. A general approach that we describe here is to implant optical fibers just above brain regions targeted for light delivery. In part due to the fact that expression of optogenetic proteins is accomplished by techniques with inherent variability (e.g., viral expression levels), it also requires strategies to measure and calibrate the effect of stimulation. Here we describe general procedures that allow one to simultaneously stimulate neurons and use photometry with genetically encoded activity indicators to precisely calibrate stimulation.

Dissociable contributions of phasic dopamine activity to reward and prediction.

Sensory cues that precede reward acquire predictive (expected value) and incentive (drive reward-seeking action) properties. Mesolimbic dopamine neurons' responses to sensory cues correlate with both expected value and reward-seeking action. This has led to the proposal that phasic dopamine responses may be sufficient to inform value-based decisions, elicit actions, and/or induce motivational states; however, causal tests are incomplete. Here, we show that direct dopamine neuron stimulation, both calibrated to physiological and greater intensities, at the time of reward can be sufficient to induce and maintain reward seeking (reinforcing) although replacement of a cue with stimulation is insufficient to induce reward seeking or act as an informative cue. Stimulation of descending cortical inputs, one synapse upstream, are sufficient for reinforcement and cues to future reward. Thus, physiological activation of mesolimbic dopamine neurons can be sufficient for reinforcing properties of reward without being sufficient for the predictive and incentive properties of cues.

Learning from Action: Reconsidering Movement Signaling in Midbrain Dopamine Neuron Activity.

Animals infer when and where a reward is available from experience with informative sensory stimuli and their own actions. In vertebrates, this is thought to depend upon the release of dopamine from midbrain dopaminergic neurons. Studies of the role of dopamine have focused almost exclusively on their encoding of informative sensory stimuli; however, many dopaminergic neurons are active just prior to movement initiation, even in the absence of sensory stimuli. How should current frameworks for understanding the role of dopamine incorporate these observations? To address this question, we review recent anatomical and functional evidence for action-related dopamine signaling. We conclude by proposing a framework in which dopaminergic neurons encode subjective signals of action initiation to solve an internal credit assignment problem.

The timing of action determines reward prediction signals in identified midbrain dopamine neurons.

Animals adapt their behavior in response to informative sensory cues using multiple brain circuits. The activity of midbrain dopaminergic neurons is thought to convey a critical teaching signal: reward-prediction error. Although reward-prediction error signals are thought to be essential to learning, little is known about the dynamic changes in the activity of midbrain dopaminergic neurons as animals learn about novel sensory cues and appetitive rewards. Here we describe a large dataset of cell-attached recordings of identified dopaminergic neurons as naive mice learned a novel cue-reward association. During learning midbrain dopaminergic neuron activity results from the summation of sensory cue-related and movement initiation-related response components. These components are both a function of reward expectation yet they are dissociable. Learning produces an increasingly precise coordination of action initiation following sensory cues that results in apparent reward-prediction error correlates. Our data thus provide new insights into the circuit mechanisms that underlie a critical computation in a highly conserved learning circuit.

Non-synaptic signaling from cerebellar climbing fibers modulates Golgi cell activity.

Golgi cells are the principal inhibitory neurons at the input stage of the cerebellum, providing feedforward and feedback inhibition through mossy fiber and parallel fiber synapses. In vivo studies have shown that Golgi cell activity is regulated by climbing fiber stimulation, yet there is little functional or anatomical evidence for synapses between climbing fibers and Golgi cells. Here, we show that glutamate released from climbing fibers activates ionotropic and metabotropic receptors on Golgi cells through spillover-mediated transmission. The interplay of excitatory and inhibitory conductances provides flexible control over Golgi cell spiking, allowing either excitation or a biphasic sequence of excitation and inhibition following single climbing fiber stimulation. Together with prior studies of spillover transmission to molecular layer interneurons, these results reveal that climbing fibers exert control over inhibition at both the input and output layers of the cerebellar cortex.

Spillover-mediated feedforward inhibition functionally segregates interneuron activity.

Neurotransmitter spillover represents a form of neural transmission not restricted to morphologically defined synaptic connections. Communication between climbing fibers (CFs) and molecular layer interneurons (MLIs) in the cerebellum is mediated exclusively by glutamate spillover. Here, we show how CF stimulation functionally segregates MLIs based on their location relative to glutamate release. Excitation of MLIs that reside within the domain of spillover diffusion coordinates inhibition of MLIs outside the diffusion limit. CF excitation of MLIs is dependent on extrasynaptic NMDA receptors that enhance the spatial and temporal spread of CF signaling. Activity mediated by functionally segregated MLIs converges onto neighboring Purkinje cells (PCs) to generate a long-lasting biphasic change in inhibition. These data demonstrate how glutamate release from single CFs modulates excitability of neighboring PCs, thus expanding the influence of CFs on cerebellar cortical activity in a manner not predicted by anatomical connectivity.

Increased expression of glutamate transporter GLT-1 in peritumoral tissue associated with prolonged survival and decreases in tumor growth in a rat model of experimental malignant glioma.

OBJECT: Gliomas are known to release excessive amounts of glutamate, inducing glutamate excitotoxic cell death in the peritumoral region and allowing the tumor to grow and to expand. Glutamate transporter upregulation has been shown to be neuroprotective by removing extracellular glutamate in a number of preclinical animal models of neurodegenerative diseases, including amyotrophic lateral sclerosis and Parkinson disease as well as psychiatric disorders such as depression. The authors therefore hypothesized that the protective mechanism of glutamate transporter upregulation would be useful for the treatment of gliomas as well. METHODS: In this study 9L gliosarcoma cells were treated with a glutamate transporter upregulating agent, thiamphenicol, an antibiotic approved in Europe, which has been shown previously to increase glutamate transporter expression and has recently been validated in a human Phase I biomarker trial for glutamate transporter upregulation. Cells were monitored in vitro for glutamate transporter levels and cell proliferation. In vivo, rats were injected intracranially with 9L cells and were treated with increasing doses of thiamphenicol. Animals were monitored for survival. In addition, postmortem brain tissue was analyzed for tumor size, glutamate transporter levels, and neuron count. RESULTS: Thiamphenicol showed little effects on proliferation of 9L gliosarcoma cells in vitro and did not change glutamate transporter levels in these cells. However, when delivered locally in an experimental glioma model in rats, thiamphenicol dose dependently (10-5000 µM) significantly increased survival up to 7 days and concomitantly decreased tumor size from 46.2 mm(2) to 10.2 mm(2) when compared with lesions in nontreated controls. Furthermore, immunohistochemical and biochemical analysis of peritumoral tissue confirmed an 84% increase in levels of glutamate transporter protein and a 72% increase in the number of neuronal cells in the tissue adjacent to the tumor. CONCLUSIONS: These results show that increasing glutamate transporter expression in peritumoral tissue is neuroprotective. It suggests that glutamate transporter upregulation for the treatment of gliomas should be further investigated and potentially be part of a combination therapy with standard chemotherapeutic agents.