An opponent striatal circuit for distributional reinforcement learning.

Machine learning research has achieved large performance gains on a wide range of tasks by expanding the learning target from mean rewards to entire probability distributions of rewards - an approach known as distributional reinforcement learning (RL)1. The mesolimbic dopamine system is thought to underlie RL in the mammalian brain by updating a representation of mean value in the striatum2,3, but little is known about whether, where, and how neurons in this circuit encode information about higher-order moments of reward distributions4. To fill this gap, we used high-density probes (Neuropixels) to acutely record striatal activity from well-trained, water-restricted mice performing a classical conditioning task in which reward mean, reward variance, and stimulus identity were independently manipulated. In contrast to traditional RL accounts, we found robust evidence for abstract encoding of variance in the striatum. Remarkably, chronic ablation of dopamine inputs disorganized these distributional representations in the striatum without interfering with mean value coding. Two-photon calcium imaging and optogenetics revealed that the two major classes of striatal medium spiny neurons - D1 and D2 MSNs - contributed to this code by preferentially encoding the right and left tails of the reward distribution, respectively. We synthesize these findings into a new model of the striatum and mesolimbic dopamine that harnesses the opponency between D1 and D2 MSNs5-15 to reap the computational benefits of distributional RL.

M1 receptors interacting with NMDAR enhance delay-related neuronal firing and improve working memory performance.

The recurrent excitatory circuits in dlPFC underlying working memory are known to require activation of glutamatergic NMDA receptors (NMDAR). The neurons in these circuits also rely on acetylcholine to maintain persistent activity, with evidence for actions at both nicotinic α7 receptors and muscarinic M1 receptors (M1R). It is known that nicotinic α7 receptors interact with NMDAR in these circuits, but the interactions between M1R and NMDAR on dlPFC neuronal activity are unknown. Here, we investigated whether M1Rs contribute to the permissive effects of ACh in dlPFC circuitry underlying working memory via interactions with NMDA receptors. We tested interactions between M1Rs and NMDARs in vivo on single neuron activity in rhesus macaques performing a working memory task, as well as on working memory behavior in rodents following infusion of M1R and NMDAR compounds into mPFC. We report that M1R antagonists block the enhancing effects of NMDA application, consistent with M1R permissive actions. Conversely, M1R positive allosteric modulators prevented the detrimental effects of NMDAR blockade in single neurons in dlPFC and on working memory performance in rodents. These data support an interaction between M1R and NMDARs in working memory circuitry in both primates and rats, and suggest M1Rs contribute to the permissive actions of ACh in primate dlPFC. These results are consistent with recent data suggesting that M1R agonists may be helpful in the treatment of schizophrenia, a cognitive disorder associated with NMDAR dysfunction.

Distributional Reinforcement Learning in the Brain.

Learning about rewards and punishments is critical for survival. Classical studies have demonstrated an impressive correspondence between the firing of dopamine neurons in the mammalian midbrain and the reward prediction errors of reinforcement learning algorithms, which express the difference between actual reward and predicted mean reward. However, it may be advantageous to learn not only the mean but also the complete distribution of potential rewards. Recent advances in machine learning have revealed a biologically plausible set of algorithms for reconstructing this reward distribution from experience. Here, we review the mathematical foundations of these algorithms as well as initial evidence for their neurobiological implementation. We conclude by highlighting outstanding questions regarding the circuit computation and behavioral readout of these distributional codes.

Muscarinic M1 Receptors Modulate Working Memory Performance and Activity via KCNQ Potassium Channels in the Primate Prefrontal Cortex.

Working memory relies on the dorsolateral prefrontal cortex (dlPFC), where microcircuits of pyramidal neurons enable persistent firing in the absence of sensory input, maintaining information through recurrent excitation. This activity relies on acetylcholine, although the molecular mechanisms for this dependence are not thoroughly understood. This study investigated the role of muscarinic M1 receptors (M1Rs) in the dlPFC using iontophoresis coupled with single-unit recordings from aging monkeys with naturally occurring cholinergic depletion. We found that M1R stimulation produced an inverted-U dose response on cell firing and behavioral performance when given systemically to aged monkeys. Immunoelectron microscopy localized KCNQ isoforms (Kv7.2, Kv7.3, and Kv7.5) on layer III dendrites and spines, similar to M1Rs. Iontophoretic manipulation of KCNQ channels altered cell firing and reversed the effects of M1R compounds, suggesting that KCNQ channels are one mechanism for M1R actions in the dlPFC. These results indicate that M1Rs may be an appropriate target to treat cognitive disorders with cholinergic alterations.

A Structural Model of the Inactivation Gate of Voltage-Activated Potassium Channels.

After opening, the Shaker voltage-gated potassium (KV) channel rapidly inactivates when one of its four N-termini enters and occludes the channel pore. Although it is known that the tip of the N-terminus reaches deep into the central cavity, the conformation adopted by this domain during inactivation and the nature of its interactions with the rest of the channel remain unclear. Here, we use molecular dynamics simulations coupled with electrophysiology experiments to reveal the atomic-scale mechanisms of inactivation. We find that the first six amino acids of the N-terminus spontaneously enter the central cavity in an extended conformation, establishing hydrophobic contacts with residues lining the pore. A second portion of the N-terminus, consisting of a long 24 amino acid α-helix, forms numerous polar contacts with residues in the intracellular entryway of the T1 domain. Double mutant cycle analysis revealed a strong relationship between predicted interatomic distances and empirically observed thermodynamic coupling, establishing a plausible model of the transition of KV channels to the inactivated state.

Seeing structure: Shape skeletons modulate perceived similarity.

An intrinsic part of seeing objects is seeing how similar or different they are relative to one another. This experience requires that objects be mentally represented in a common format over which such comparisons can be carried out. What is that representational format? Objects could be compared in terms of their superficial features (e.g., degree of pixel-by-pixel overlap), but a more intriguing possibility is that they are compared on the basis of a deeper structure. One especially promising candidate that has enjoyed success in the computer vision literature is the shape skeleton-a geometric transformation that represents objects according to their inferred underlying organization. Despite several hints that shape skeletons are computed in human vision, it remains unclear how much they actually matter for subsequent performance. Here, we explore the possibility that shape skeletons help mediate the ability to extract visual similarity. Observers completed a same/different task in which two shapes could vary either in their skeletal structure (without changing superficial features such as size, orientation, and internal angular separation) or in large surface-level ways (without changing overall skeletal organization). Discrimination was better for skeletally dissimilar shapes: observers had difficulty appreciating even surprisingly large differences when those differences did not reorganize the underlying skeletons. This pattern also generalized beyond line drawings to 3-D volumes whose skeletons were less readily inferable from the shapes' visible contours. These results show how shape skeletons may influence the perception of similarity-and more generally, how they have important consequences for downstream visual processing.