Wave-like dopamine dynamics as a mechanism for spatiotemporal credit assignment.

Significant evidence supports the view that dopamine shapes learning by encoding reward prediction errors. However, it is unknown whether striatal targets receive tailored dopamine dynamics based on regional functional specialization. Here, we report wave-like spatiotemporal activity patterns in dopamine axons and release across the dorsal striatum. These waves switch between activational motifs and organize dopamine transients into localized clusters within functionally related striatal subregions. Notably, wave trajectories were tailored to task demands, propagating from dorsomedial to dorsolateral striatum when rewards are contingent on animal behavior and in the opponent direction when rewards are independent of behavioral responses. We propose a computational architecture in which striatal dopamine waves are sculpted by inference about agency and provide a mechanism to direct credit assignment to specialized striatal subregions. Supporting model predictions, dorsomedial dopamine activity during reward-pursuit signaled the extent of instrumental control and interacted with reward waves to predict future behavioral adjustments.

Functional diversity for body actions in the mesencephalic locomotor region.

The mesencephalic locomotor region (MLR) is a key midbrain center with roles in locomotion. Despite extensive studies and clinical trials aimed at therapy-resistant Parkinson's disease (PD), debate on its function remains. Here, we reveal the existence of functionally diverse neuronal populations with distinct roles in control of body movements. We identify two spatially intermingled glutamatergic populations separable by axonal projections, mouse genetics, neuronal activity profiles, and motor functions. Most spinally projecting MLR neurons encoded the full-body behavior rearing. Loss- and gain-of-function optogenetic perturbation experiments establish a function for these neurons in controlling body extension. In contrast, Rbp4-transgene-positive MLR neurons project in an ascending direction to basal ganglia, preferentially encode the forelimb behaviors handling and grooming, and exhibit a role in modulating movement. Thus, the MLR contains glutamatergic neuronal subpopulations stratified by projection target exhibiting roles in action control not restricted to locomotion.

Molecular Diversity and Specializations among the Cells of the Adult Mouse Brain.

The mammalian brain is composed of diverse, specialized cell populations. To systematically ascertain and learn from these cellular specializations, we used Drop-seq to profile RNA expression in 690,000 individual cells sampled from 9 regions of the adult mouse brain. We identified 565 transcriptionally distinct groups of cells using computational approaches developed to distinguish biological from technical signals. Cross-region analysis of these 565 cell populations revealed features of brain organization, including a gene-expression module for synthesizing axonal and presynaptic components, patterns in the co-deployment of voltage-gated ion channels, functional distinctions among the cells of the vasculature and specialization of glutamatergic neurons across cortical regions. Systematic neuronal classifications for two complex basal ganglia nuclei and the striatum revealed a rare population of spiny projection neurons. This adult mouse brain cell atlas, accessible through interactive online software (DropViz), serves as a reference for development, disease, and evolution.

Optogenetic Editing Reveals the Hierarchical Organization of Learned Action Sequences.

The organization of action into sequences underlies complex behaviors that are essential for organismal survival and reproduction. Despite extensive studies of innate sequences in relation to central pattern generators, how learned action sequences are controlled and whether they are organized as a chain or a hierarchy remain largely unknown. By training mice to perform heterogeneous action sequences, we demonstrate that striatal direct and indirect pathways preferentially encode different behavioral levels of sequence structure. State-dependent closed-loop optogenetic stimulation of the striatal direct pathway can selectively insert a single action element into the sequence without disrupting the overall sequence length. Optogenetic manipulation of the striatal indirect pathway completely removes the ongoing subsequence while leaving the following subsequence to be executed with the appropriate timing and length. These results suggest that learned action sequences are not organized in a serial but rather a hierarchical structure that is distinctly controlled by basal ganglia pathways.

The Striatum Organizes 3D Behavior via Moment-to-Moment Action Selection.

Many naturalistic behaviors are built from modular components that are expressed sequentially. Although striatal circuits have been implicated in action selection and implementation, the neural mechanisms that compose behavior in unrestrained animals are not well understood. Here, we record bulk and cellular neural activity in the direct and indirect pathways of dorsolateral striatum (DLS) as mice spontaneously express action sequences. These experiments reveal that DLS neurons systematically encode information about the identity and ordering of sub-second 3D behavioral motifs; this encoding is facilitated by fast-timescale decorrelations between the direct and indirect pathways. Furthermore, lesioning the DLS prevents appropriate sequence assembly during exploratory or odor-evoked behaviors. By characterizing naturalistic behavior at neural timescales, these experiments identify a code for elemental 3D pose dynamics built from complementary pathway dynamics, support a role for DLS in constructing meaningful behavioral sequences, and suggest models for how actions are sculpted over time.

Circuit-Specific Deep Brain Stimulation Provides Insights into Movement Control.

Deep brain stimulation (DBS), a method in which electrical stimulation is delivered to specific areas of the brain, is an effective treatment for managing symptoms of a number of neurological and neuropsychiatric disorders. Clinical access to neural circuits during DBS provides an opportunity to study the functional link between neural circuits and behavior. This review discusses how the use of DBS in Parkinson's disease and dystonia has provided insights into the brain networks and physiological mechanisms that underlie motor control. In parallel, insights from basic science about how patterns of electrical stimulation impact plasticity and communication within neural circuits are transforming DBS from a therapy for treating symptoms to a therapy for treating circuits, with the goal of training the brain out of its diseased state.

Chronic Stress Alters Striosome-Circuit Dynamics, Leading to Aberrant Decision-Making.

Effective evaluation of costs and benefits is a core survival capacity that in humans is considered as optimal, "rational" decision-making. This capacity is vulnerable in neuropsychiatric disorders and in the aftermath of chronic stress, in which aberrant choices and high-risk behaviors occur. We report that chronic stress exposure in rodents produces abnormal evaluation of costs and benefits resembling non-optimal decision-making in which choices of high-cost/high-reward options are sharply increased. Concomitantly, alterations in the task-related spike activity of medial prefrontal neurons correspond with increased activity of their striosome-predominant striatal projection neuron targets and with decreased and delayed striatal fast-firing interneuron activity. These effects of chronic stress on prefronto-striatal circuit dynamics could be blocked or be mimicked by selective optogenetic manipulation of these circuits. We suggest that altered excitation-inhibition dynamics of striosome-based circuit function could be an underlying mechanism by which chronic stress contributes to disorders characterized by aberrant decision-making under conflict. VIDEO ABSTRACT.

Striosomes and Matrisomes: Scaffolds for Dynamic Coupling of Volition and Action.

Striosomes form neurochemically specialized compartments of the striatum embedded in a large matrix made up of modules called matrisomes. Striosome-matrix architecture is multiplexed with the canonical direct-indirect organization of the striatum. Striosomal functions remain to be fully clarified, but key information is emerging. First, striosomes powerfully innervate nigral dopamine-containing neurons and can completely shut down their activity, with a following rebound excitation. Second, striosomes receive limbic and cognition-related corticostriatal afferents and are dynamically modulated in relation to value-based actions. Third, striosomes are spatially interspersed among matrisomes and interneurons and are influenced by local and global neuromodulatory and oscillatory activities. Fourth, striosomes tune engagement and the motivation to perform reinforcement learning, to manifest stereotypical behaviors, and to navigate valence conflicts and valence discriminations. We suggest that, at an algorithmic level, striosomes could serve as distributed scaffolds to provide formats of the striatal computations generated through development and refined through learning. We propose that striosomes affect subjective states. By transforming corticothalamic and other inputs to the functional formats of the striatum, they could implement state transitions in nigro-striato-nigral circuits to affect bodily and cognitive actions according to internal motives whose functions are compromised in neuropsychiatric conditions.

Brainstem Circuits for Locomotion.

Locomotion is a universal motor behavior that is expressed as the output of many integrated brain functions. Locomotion is organized at several levels of the nervous system, with brainstem circuits acting as the gate between brain areas regulating innate, emotional, or motivational locomotion and executive spinal circuits. Here we review recent advances on brainstem circuits involved in controlling locomotion. We describe how delineated command circuits govern the start, speed, stop, and steering of locomotion. We also discuss how these pathways interface between executive circuits in the spinal cord and diverse brain areas important for context-specific selection of locomotion. A recurrent theme is the need to establish a functional connectome to and from brainstem command circuits. Finally, we point to unresolved issues concerning the integrated function of locomotor control.

Basal Ganglia Circuits for Action Specification.

Behavior is readily classified into patterns of movements with inferred common goals-actions. Goals may be discrete; movements are continuous. Through the careful study of isolated movements in laboratory settings, or via introspection, it has become clear that animals can exhibit exquisite graded specification to their movements. Moreover, graded control can be as fundamental to success as the selection of which action to perform under many naturalistic scenarios: a predator adjusting its speed to intercept moving prey, or a tool-user exerting the perfect amount of force to complete a delicate task. The basal ganglia are a collection of nuclei in vertebrates that extend from the forebrain (telencephalon) to the midbrain (mesencephalon), constituting a major descending extrapyramidal pathway for control over midbrain and brainstem premotor structures. Here we discuss how this pathway contributes to the continuous specification of movements that endows our voluntary actions with vigor and grace.

Current Principles of Motor Control, with Special Reference to Vertebrate Locomotion.

The vertebrate control of locomotion involves all levels of the nervous system from cortex to the spinal cord. Here, we aim to cover all main aspects of this complex behavior, from the operation of the microcircuits in the spinal cord to the systems and behavioral levels and extend from mammalian locomotion to the basic undulatory movements of lamprey and fish. The cellular basis of propulsion represents the core of the control system, and it involves the spinal central pattern generator networks (CPGs) controlling the timing of different muscles, the sensory compensation for perturbations, and the brain stem command systems controlling the level of activity of the CPGs and the speed of locomotion. The forebrain and in particular the basal ganglia are involved in determining which motor programs should be recruited at a given point of time and can both initiate and stop locomotor activity. The propulsive control system needs to be integrated with the postural control system to maintain body orientation. Moreover, the locomotor movements need to be steered so that the subject approaches the goal of the locomotor episode, or avoids colliding with elements in the environment or simply escapes at high speed. These different aspects will all be covered in the review.

Feasibility of dopamine as a vector-valued feedback signal in the basal ganglia.

It is well established that midbrain dopaminergic neurons support reinforcement learning (RL) in the basal ganglia by transmitting a reward prediction error (RPE) to the striatum. In particular, different computational models and experiments have shown that a striatum-wide RPE signal can support RL over a small discrete set of actions (e.g., no/no-go, choose left/right). However, there is accumulating evidence that the basal ganglia functions not as a selector between predefined actions but rather as a dynamical system with graded, continuous outputs. To reconcile this view with RL, there is a need to explain how dopamine could support learning of continuous outputs, rather than discrete action values. Inspired by the recent observations that besides RPE, the firing rates of midbrain dopaminergic neurons correlate with motor and cognitive variables, we propose a model in which dopamine signal in the striatum carries a vector-valued error feedback signal (a loss gradient) instead of a homogeneous scalar error (a loss). We implement a local, "three-factor" corticostriatal plasticity rule involving the presynaptic firing rate, a postsynaptic factor, and the unique dopamine concentration perceived by each striatal neuron. With this learning rule, we show that such a vector-valued feedback signal results in an increased capacity to learn a multidimensional series of real-valued outputs. Crucially, we demonstrate that this plasticity rule does not require precise nigrostriatal synapses but remains compatible with experimental observations of random placement of varicosities and diffuse volume transmission of dopamine.

Neuronal implementation of the temporal difference learning algorithm in the midbrain dopaminergic system.

The temporal difference learning (TDL) algorithm has been essential to conceptualizing the role of dopamine in reinforcement learning (RL). Despite its theoretical importance, it remains unknown whether a neuronal implementation of this algorithm exists in the brain. Here, we provide an interpretation of the recently described signaling properties of ventral tegmental area (VTA) GABAergic neurons and show that a circuitry of these neurons implements the TDL algorithm. Specifically, we identified the neuronal mechanism of three key components of the TDL model: a sustained state value signal encoded by an afferent input to the VTA, a temporal differentiation circuit formed by two types of VTA GABAergic neurons the combined output of which computes momentary reward prediction (RP) as the derivative of the state value, and the computation of reward prediction errors (RPEs) in dopamine neurons utilizing the output of the differentiation circuit. Using computational methods, we also show that this mechanism is optimally adapted to the biophysics of RPE signaling in dopamine neurons, mechanistically links the emergence of conditioned reinforcement to RP, and can naturally account for the temporal discounting of reinforcement. Elucidating the implementation of the TDL algorithm may further the investigation of RL in biological and artificial systems.

Neural Representations of Post-Decision Accuracy and Reward Expectation in the Caudate Nucleus and Frontal Eye Field.

Performance monitoring that supports ongoing behavioral adjustments is often examined in the context of either choice confidence for perceptual decisions (i.e., "did I get it right?") or reward expectation for reward-based decisions (i.e., "what reward will I receive?"). However, our understanding of how the brain encodes these distinct evaluative signals remains limited because they are easily conflated, particularly in commonly used two-alternative tasks with symmetric rewards for correct choices. Previously we used a motion-discrimination task with asymmetric rewards to identify neural substrates of forming reward-biased perceptual decisions in the caudate nucleus (part of the striatum in the basal ganglia) and the frontal eye field (FEF, in prefrontal cortex). Here we leveraged this task design to partially decouple estimates of accuracy and reward expectation and examine their impacts on subsequent decisions and their representations in those two brain areas. We identified distinguishable representations of these two evaluative signals in individual caudate and FEF neurons, with regional differences in their distribution patterns and time courses. We observed that well-trained monkeys (both sexes) used both evaluative signals, infrequently but consistently, to adjust their subsequent decisions. We found further that these behavioral adjustments had reliable relationships with the neural representations of both evaluative signals in caudate, but not FEF. These results suggest that the cortico-striatal decision network may use diverse evaluative signals to monitor and adjust decision-making behaviors, adding to our understanding of the different roles that the FEF and caudate nucleus play in a diversity of decision-related computations.

Synaptic Changes in Pallidostriatal Circuits Observed in the Parkinsonian Model Triggers Abnormal Beta Synchrony with Accurate Spatio-temporal Properties across the Basal Ganglia.

Excessive oscillatory activity across basal ganglia (BG) nuclei in the ß frequencies (12-30 Hz) is a hallmark of Parkinson's disease (PD). While the link between oscillations and symptoms remains debated, exaggerated ß oscillations constitute an important biomarker for therapeutic effectiveness in PD. The neuronal mechanisms of ß-oscillation generation however remain unknown. Many existing models rely on a central role of the subthalamic nucleus (STN) or cortical inputs to BG. Contrarily, neural recordings and optogenetic manipulations in normal and parkinsonian rats recently highlighted the central role of the external pallidum (GPe) in abnormal ß oscillations, while showing that the integrity of STN or motor cortex is not required. Here, we evaluate the mechanisms for the generation of abnormal ß oscillations in a BG network model where neuronal and synaptic time constants, connectivity, and firing rate distributions are strongly constrained by experimental data. Guided by a mean-field approach, we show in a spiking neural network that several BG sub-circuits can drive oscillations. Strong recurrent STN-GPe connections or collateral intra-GPe connections drive γ oscillations (>40 Hz), whereas strong pallidostriatal loops drive low-ß (10-15 Hz) oscillations. We show that pathophysiological strengthening of striatal and pallidal synapses following dopamine depletion leads to the emergence of synchronized oscillatory activity in the mid-ß range with spike-phase relationships between BG neuronal populations in-line with experiments. Furthermore, inhibition of GPe, contrary to STN, abolishes oscillations. Our modeling study uncovers the neural mechanisms underlying PD ß oscillations and may thereby guide the future development of therapeutic strategies.

Subthalamic Activity for Motor Execution and Cancelation in Monkeys.

The subthalamic nucleus (STN) receives cortical inputs via the hyperdirect and indirect pathways, projects to the output nuclei of the basal ganglia, and plays a critical role in the control of voluntary movements and movement disorders. STN neurons change their activity during execution of movements, while recent studies emphasize STN activity specific to cancelation of movements. To address the relationship between execution and cancelation functions, we examined STN activity in two Japanese monkeys (Macaca fuscata, both sexes) who performed a goal-directed reaching task with a delay that included Go, Cancel, and NoGo trials. We first examined responses to the stimulation of the forelimb regions in the primary motor cortex and/or supplementary motor area. STN neurons with motor cortical inputs were found in the dorsal somatomotor region of the STN. All these STN neurons showed activity changes in Go trials, suggesting their involvement in execution of movements. Part of them exhibited activity changes in Cancel trials and sustained activity during delay periods, suggesting their involvement in cancelation of planed movements and preparation of movements, respectively. The STN neurons rarely showed activity changes in NoGo trials. Go- and Cancel-related activity was selective to the direction of movements, and the selectivity was higher in Cancel trials than in Go trials. Changes in Go- and Cancel-related activity occurred early enough to initiate and cancel movements, respectively. These results suggest that the dorsal somatomotor region of the STN, which receives motor cortical inputs, is involved in preparation and execution of movements and cancelation of planned movements.

Value-Based Search Efficiency Is Encoded in the Substantia Nigra Reticulata Firing Rate, Spiking Irregularity and Local Field Potential.

Recent results show that valuable objects can pop out in visual search, yet its neural mechanisms remain unexplored. Given the role of substantia nigra reticulata (SNr) in object value memory and control of gaze, we recorded its single-unit activity while male macaque monkeys engaged in efficient or inefficient search for a valuable target object among low-value objects. The results showed that efficient search was concurrent with stronger inhibition and higher spiking irregularity in the target-present (TP) compared with the target-absent (TA) trials in SNr. Importantly, the firing rate differentiation of TP and TA trials happened within ∼100 ms of display onset, and its magnitude was significantly correlated with the search times and slopes (search efficiency). Time-frequency analyses of local field potential (LFP) after display onset revealed significant modulations of the gamma band power with search efficiency. The greater reduction of SNr firing in TP trials in efficient search can create a stronger disinhibition of downstream superior colliculus, which in turn can facilitate saccade to obtain valuable targets in competitive environments.

What Have We Learned About Movement Disorders from Functional Neurosurgery?

Modern functional neurosurgery for movement disorders such as Parkinson's disease, tremor, and dystonia involves the placement of focal lesions or the application of deep brain stimulation (DBS) within circuits that modulate motor function. Precise targeting of these motor structures can be further refined by the use of electrophysiological approaches. In particular, microelectrode recordings enable the delineation of neuroanatomic structures. In the course of these operations, there is an opportunity not only to map basal ganglia structures but also to gain insights into how disturbances in neural activity produce movement disorders. In this review, we aim to highlight what the field has uncovered thus far about movement disorders through DBS. The work to date lays the foundation for future studies that will shed further light on dysfunctional circuits mediating diseases of the nervous system and how we might modulate these circuits therapeutically.

Putaminal dopamine modulates movement motivation in Parkinson's disease.

The relative inability to produce effortful movements is the most specific motor sign of Parkinson's disease, which is primarily characterized by loss of dopaminergic terminals in the putamen. The motor motivation hypothesis suggests that this motor deficit may not reflect a deficiency in motor control per se, but a deficiency in cost-benefit considerations for motor effort. For the first time, we investigated the quantitative effect of dopamine depletion on the motivation of motor effort in Parkinson's disease. A total of 21 early-stage, unmedicated patients with Parkinson's disease and 26 healthy controls were included. An incentivized force task was used to capture the amount of effort participants were willing to invest for different monetary incentive levels and dopamine transporter depletion in the bilateral putamen was assessed. Our results demonstrate that patients with Parkinson's disease applied significantly less grip force than healthy controls, especially for low incentive levels. Congruously, decrease of motor effort with greater loss of putaminal dopaminergic terminals was most pronounced for low incentive levels. This signifies that putaminal dopamine is most critical to motor effort when the trade-off with the benefit is poor. Taken together, we provide direct evidence that the reduction of effortful movements in Parkinson's disease depends on motivation and that this effect is associated with putaminal dopaminergic degeneration.

Subthalamic control of impulsive actions: insights from deep brain stimulation in Parkinson's disease.

Control of actions allows adaptive, goal-directed behaviour. The basal ganglia, including the subthalamic nucleus, are thought to play a central role in dynamically controlling actions through recurrent negative feedback loops with the cerebral cortex. Here, we summarize recent translational studies that used deep brain stimulation to record neural activity from and apply electrical stimulation to the subthalamic nucleus in people with Parkinson's disease. These studies have elucidated spatial, spectral and temporal features of the neural mechanisms underlying the controlled delay of actions in cortico-subthalamic networks and demonstrated their causal effects on behaviour in distinct processing windows. While these mechanisms have been conceptualized as control signals for suppressing impulsive response tendencies in conflict tasks and as decision threshold adjustments in value-based and perceptual decisions, we propose a common framework linking decision-making, cognition and movement. Within this framework, subthalamic deep brain stimulation can lead to suboptimal choices by reducing the time that patients take for deliberation before committing to an action. However, clinical studies have consistently shown that the occurrence of impulse control disorders is reduced, not increased, after subthalamic deep brain stimulation surgery. This apparent contradiction can be reconciled when recognizing the multifaceted nature of impulsivity, its underlying mechanisms and modulation by treatment. While subthalamic deep brain stimulation renders patients susceptible to making decisions without proper forethought, this can be disentangled from effects related to dopamine comprising sensitivity to benefits versus costs, reward delay aversion and learning from outcomes. Alterations in these dopamine-mediated mechanisms are thought to underlie the development of impulse control disorders and can be relatively spared with reduced dopaminergic medication after subthalamic deep brain stimulation. Together, results from studies using deep brain stimulation as an experimental tool have improved our understanding of action control in the human brain and have important implications for treatment of patients with neurological disorders.