Distributional reinforcement learning in prefrontal cortex.

The prefrontal cortex is crucial for learning and decision-making. Classic reinforcement learning (RL) theories center on learning the expectation of potential rewarding outcomes and explain a wealth of neural data in the prefrontal cortex. Distributional RL, on the other hand, learns the full distribution of rewarding outcomes and better explains dopamine responses. In the present study, we show that distributional RL also better explains macaque anterior cingulate cortex neuronal responses, suggesting that it is a common mechanism for reward-guided learning.

Common Mechanisms of Learning in Motor and Cognitive Systems.

Rapid progress in our understanding of the brain's learning mechanisms has been accomplished over the past decade, particularly with conceptual advances, including representing behavior as a dynamical system, large-scale neural population recordings, and new methods of analysis of neuronal populations. However, motor and cognitive systems have been traditionally studied with different methods and paradigms. Recently, some common principles, evident in both behavior and neural activity, that underlie these different types of learning have become to emerge. Here we review results from motor and cognitive learning, relying on different techniques and studying different systems to understand the mechanisms of learning. Movement is intertwined with cognitive operations, and its dynamics reflect cognitive variables. Training, in either motor or cognitive tasks, involves recruitment of previously unresponsive neurons and reorganization of neural activity in a low dimensional manifold. Mapping of new variables in neural activity can be very rapid, instantiating flexible learning of new tasks. Communication between areas is just as critical a part of learning as are patterns of activity within an area emerging with learning. Common principles across systems provide a map for future research.

Distinct neural representations during a brain-machine interface and manual reaching task in motor cortex, prefrontal cortex, and striatum.

Although brain-machine interfaces (BMIs) are directly controlled by the modulation of a select local population of neurons, distributed networks consisting of cortical and subcortical areas have been implicated in learning and maintaining control. Previous work in rodents has demonstrated the involvement of the striatum in BMI learning. However, the prefrontal cortex has been largely ignored when studying motor BMI control despite its role in action planning, action selection, and learning abstract tasks. Here, we compare local field potentials simultaneously recorded from primary motor cortex (M1), dorsolateral prefrontal cortex (DLPFC), and the caudate nucleus of the striatum (Cd) while nonhuman primates perform a two-dimensional, self-initiated, center-out task under BMI control and manual control. Our results demonstrate the presence of distinct neural representations for BMI and manual control in M1, DLPFC, and Cd. We find that neural activity from DLPFC and M1 best distinguishes control types at the go cue and target acquisition, respectively, while M1 best predicts target-direction at both task events. We also find effective connectivity from DLPFC → M1 throughout both control types and Cd → M1 during BMI control. These results suggest distributed network activity between M1, DLPFC, and Cd during BMI control that is similar yet distinct from manual control.

Value dynamics affect choice preparation during decision-making.

During decision-making, neurons in the orbitofrontal cortex (OFC) sequentially represent the value of each option in turn, but it is unclear how these dynamics are translated into a choice response. One brain region that may be implicated in this process is the anterior cingulate cortex (ACC), which strongly connects with OFC and contains many neurons that encode the choice response. We investigated how OFC value signals interacted with ACC neurons encoding the choice response by performing simultaneous high-channel count recordings from the two areas in nonhuman primates. ACC neurons encoding the choice response steadily increased their firing rate throughout the decision-making process, peaking shortly before the time of the choice response. Furthermore, the value dynamics in OFC affected ACC ramping-when OFC represented the more valuable option, ACC ramping accelerated. Because OFC tended to represent the more valuable option more frequently and for a longer duration, this interaction could explain how ACC selects the more valuable response.

Distinct neural representations during a brain-machine interface and manual reaching task in motor cortex, prefrontal cortex, and striatum.

Although brain-machine interfaces (BMIs) are directly controlled by the modulation of a select local population of neurons, distributed networks consisting of cortical and subcortical areas have been implicated in learning and maintaining control. Previous work in rodent BMI has demonstrated the involvement of the striatum in BMI learning. However, the prefrontal cortex has been largely ignored when studying motor BMI control despite its role in action planning, action selection, and learning abstract tasks. Here, we compare local field potentials simultaneously recorded from the primary motor cortex (M1), dorsolateral prefrontal cortex (DLPFC), and the caudate nucleus of the striatum (Cd) while nonhuman primates perform a two-dimensional, self-initiated, center-out task under BMI control and manual control. Our results demonstrate the presence of distinct neural representations for BMI and manual control in M1, DLPFC, and Cd. We find that neural activity from DLPFC and M1 best distinguish between control types at the go cue and target acquisition, respectively. We also found effective connectivity from DLPFC→M1 throughout trials across both control types and Cd→M1 during BMI control. These results suggest distributed network activity between M1, DLPFC, and Cd during BMI control that is similar yet distinct from manual control.

Abstraction of Reward Context Facilitates Relative Reward Coding in Neural Populations of the Macaque Anterior Cingulate Cortex.

The anterior cingulate cortex (ACC) is believed to be involved in many cognitive processes, including linking goals to actions and tracking decision-relevant contextual information. ACC neurons robustly encode expected outcomes, but how this relates to putative functions of ACC remains unknown. Here, we approach this question from the perspective of population codes by analyzing neural spiking data in the ventral and dorsal banks of the ACC in two male monkeys trained to perform a stimulus-motor mapping task to earn rewards or avoid losses. We found that neural populations favor a low dimensional representational geometry that emphasizes the valence of potential outcomes while also facilitating the independent, abstract representation of multiple task-relevant variables. Valence encoding persisted throughout the trial, and realized outcomes were primarily encoded in a relative sense, such that cue valence acted as a context for outcome encoding. This suggests that the population coding we observe could be a mechanism that allows feedback to be interpreted in a context-dependent manner. Together, our results point to a prominent role for ACC in context setting and relative interpretation of outcomes, facilitated by abstract, or untangled, representations of task variables.SIGNIFICANCE STATEMENT The ability to interpret events in light of the current context is a critical facet of higher-order cognition. The ACC is suggested to be important for tracking contextual information, whereas alternate views hold that its function is more related to the motor system and linking goals to appropriate actions. We evaluated these possibilities by analyzing geometric properties of neural population activity in monkey ACC when contexts were determined by the valence of potential outcomes and found that this information was represented as a dominant, abstract concept. Ensuing outcomes were then coded relative to these contexts, suggesting an important role for these representations in context-dependent evaluation. Such mechanisms may be critical for the abstract reasoning and generalization characteristic of biological intelligence.

Decoding cognition in real-time.

How can we study unobservable cognitive processes that cannot be measured directly? This has been an enduring challenge for cognitive scientists. In this essay we discuss advances in neurotechnology that could allow cognitive processes to be decoded in real-time and the implications that this may have for cognitive science and the treatment of neuropsychiatric disease.

Fast and slow contributions to decision-making in corticostriatal circuits.

We make complex decisions using both fast judgments and slower, more deliberative reasoning. For example, during value-based decision-making, animals make rapid value-guided orienting eye movements after stimulus presentation that bias the upcoming decision. The neural mechanisms underlying these processes remain unclear. To address this, we recorded from the caudate nucleus and orbitofrontal cortex while animals made value-guided decisions. Using population-level decoding, we found a rapid, phasic signal in caudate that predicted the choice response and closely aligned with animals' initial orienting eye movements. In contrast, the dynamics in orbitofrontal cortex were more consistent with a deliberative system serially representing the value of each available option. The phasic caudate value signal and the deliberative orbitofrontal value signal were largely independent from each other, consistent with value-guided orienting and value-guided decision-making being independent processes.

Taking stock of value in the orbitofrontal cortex.

People with damage to the orbitofrontal cortex (OFC) have specific problems making decisions, whereas their other cognitive functions are spared. Neurophysiological studies have shown that OFC neurons fire in proportion to the value of anticipated outcomes. Thus, a central role of the OFC is to guide optimal decision-making by signalling values associated with different choices. Until recently, this view of OFC function dominated the field. New data, however, suggest that the OFC may have a much broader role in cognition by representing cognitive maps that can be used to guide behaviour and that value is just one of many variables that are important for behavioural control. In this Review, we critically evaluate these two alternative accounts of OFC function and examine how they might be reconciled.

Cognitive strategies shift information from single neurons to populations in prefrontal cortex.

Neurons in primate lateral prefrontal cortex (LPFC) play a critical role in working memory (WM) and cognitive strategies. Consistent with adaptive coding models, responses of these neurons are not fixed but flexibly adjust on the basis of cognitive demands. However, little is known about how these adjustments affect population codes. Here, we investigated ensemble coding in LPFC while monkeys implemented different strategies in a WM task. Although single neurons were less tuned when monkeys used more stereotyped strategies, task information could still be accurately decoded from neural populations. This was due to changes in population codes that distributed information among a greater number of neurons, each contributing less to the overall population. Moreover, this shift occurred for task-relevant, but not irrelevant, information. These results demonstrate that cognitive strategies that impose structure on information held in mind rearrange population codes in LPFC, such that information becomes more distributed among neurons in an ensemble.

Hybrid dedicated and distributed coding in PMd/M1 provides separation and interaction of bilateral arm signals.

Pronounced activity is observed in both hemispheres of the motor cortex during preparation and execution of unimanual movements. The organizational principles of bi-hemispheric signals and the functions they serve throughout motor planning remain unclear. Using an instructed-delay reaching task in monkeys, we identified two components in population responses spanning PMd and M1. A "dedicated" component, which segregated activity at the level of individual units, emerged in PMd during preparation. It was most prominent following movement when M1 became strongly engaged, and principally involved the contralateral hemisphere. In contrast to recent reports, these dedicated signals solely accounted for divergence of arm-specific neural subspaces. The other "distributed" component mixed signals for each arm within units, and the subspace containing it did not discriminate between arms at any stage. The statistics of the population response suggest two functional aspects of the cortical network: one that spans both hemispheres for supporting preparatory and ongoing processes, and another that is predominantly housed in the contralateral hemisphere and specifies unilateral output.

Hippocampal neurons construct a map of an abstract value space.

The hippocampus is thought to encode a "cognitive map," a structural organization of knowledge about relationships in the world. Place cells, spatially selective hippocampal neurons that have been extensively studied in rodents, are one component of this map, describing the relative position of environmental features. However, whether this map extends to abstract, cognitive information remains unknown. Using the relative reward value of cues to define continuous "paths" through an abstract value space, we show that single neurons in primate hippocampus encode this space through value place fields, much like a rodent's place neurons encode paths through physical space. Value place fields remapped when cues changed but also became increasingly correlated across contexts, allowing maps to become generalized. Our findings help explain the critical contribution of the hippocampus to value-based decision-making, providing a mechanism by which knowledge of relationships in the world can be incorporated into reward predictions for guiding decisions.

Gender bias in academia: A lifetime problem that needs solutions.

Despite increased awareness of the lack of gender equity in academia and a growing number of initiatives to address issues of diversity, change is slow, and inequalities remain. A major source of inequity is gender bias, which has a substantial negative impact on the careers, work-life balance, and mental health of underrepresented groups in science. Here, we argue that gender bias is not a single problem but manifests as a collection of distinct issues that impact researchers' lives. We disentangle these facets and propose concrete solutions that can be adopted by individuals, academic institutions, and society.

Parameterizing neural power spectra into periodic and aperiodic components.

Electrophysiological signals exhibit both periodic and aperiodic properties. Periodic oscillations have been linked to numerous physiological, cognitive, behavioral and disease states. Emerging evidence demonstrates that the aperiodic component has putative physiological interpretations and that it dynamically changes with age, task demands and cognitive states. Electrophysiological neural activity is typically analyzed using canonically defined frequency bands, without consideration of the aperiodic (1/f-like) component. We show that standard analytic approaches can conflate periodic parameters (center frequency, power, bandwidth) with aperiodic ones (offset, exponent), compromising physiological interpretations. To overcome these limitations, we introduce an algorithm to parameterize neural power spectra as a combination of an aperiodic component and putative periodic oscillatory peaks. This algorithm requires no a priori specification of frequency bands. We validate this algorithm on simulated data, and demonstrate how it can be used in applications ranging from analyzing age-related changes in working memory to large-scale data exploration and analysis.

Stable and dynamic representations of value in the prefrontal cortex.

Optimal decision-making requires that stimulus-value associations are kept up to date by constantly comparing the expected value of a stimulus with its experienced outcome. To do this, value information must be held in mind when a stimulus and outcome are separated in time. However, little is known about the neural mechanisms of working memory (WM) for value. Contradicting theories have suggested WM requires either persistent or transient neuronal activity, with stable or dynamic representations, respectively. To test these hypotheses, we recorded neuronal activity in the orbitofrontal and anterior cingulate cortex of two monkeys performing a valuation task. We found that features of all hypotheses were simultaneously present in prefrontal activity, and no single hypothesis was exclusively supported. Instead, mixed dynamics supported robust, time invariant value representations while also encoding the information in a temporally specific manner. We suggest that this hybrid coding is a critical mechanism supporting flexible cognitive abilities.

Closed-Loop Theta Stimulation in the Orbitofrontal Cortex Prevents Reward-Based Learning.

Neuronal oscillations in the frontal cortex have been hypothesized to play a role in the organization of high-level cognition. Within the orbitofrontal cortex (OFC), there is a prominent oscillation in the theta frequency (4-8 Hz) during reward-guided behavior, but it is unclear whether this oscillation has causal significance. One methodological challenge is that it is difficult to manipulate theta without affecting other neural signals, such as single-neuron firing rates. A potential solution is to use closed-loop control to record theta in real time and use this signal to control the application of electrical microstimulation to the OFC. Using this method, we show that theta oscillations in the OFC are critically important for reward-guided learning and that they are driven by theta oscillations in the hippocampus (HPC). The ability to disrupt OFC computations via spatially localized and temporally precise stimulation could lead to novel treatment strategies for neuropsychiatric disorders involving OFC dysfunction.

Reward.

Neurons throughout frontal cortex show robust responses to rewards, but a challenge is determining the specific function served by these different reward signals. Most neuropsychiatric disorders involve dysfunction of circuits between frontal cortex and subcortical structures, such as the striatum. There are multiple frontostriatal loops, and different neuropsychiatric disorders involve different loops to greater or lesser extents. Understanding the role of reward in each of these different circuits is a necessary step in developing novel treatments for these disorders. This chapter summarizes the recent literature that has identified the role of reward in different subregions of the frontal cortex. Orbitofrontal cortex integrates information about multiple aspects of expected rewards in order to derive their value, which can then be used to decide between alternative potential rewards. Neurons in anterior cingulate cortex encode the difference between the expected reward and the actual outcome. This information is useful for learning, since it can ensure that behavior changes when the outcome was not anticipated. Reward also affects signals in lateral prefrontal cortex related to attention and response selection, ensuring that behaviors are optimally prioritized. Finally, the chapter discusses how reward signals contribute to social processing and autonomic control.

A model-based approach for targeted neurophysiology in the behaving non-human primate.

Acute neurophysiology in the behaving primate typically relies on traditional manufacturing approaches for the instrumentation necessary for recording. For example, our previous approach consisted of distributing single microelectrodes in a fixed plane situated over a circular patch of frontal cortex using conventionally-milled recording grids. With the advent of robust, multisite linear probes, and the introduction of commercially-available, high-resolution rapid prototyping systems, we have been able to improve upon traditional approaches. Here, we report our methodology for producing flexible, MR-informed recording platforms that allow us to precisely target brain structures of interest, including those that would be unreachable using previous methods. We have increased our single-session recording yields by an order of magnitude and recorded neural activity from widely-distributed regions using only a single recording chamber. This approach both speeds data collection, reduces the damage done to neural tissue over the course of a single experiment, and reduces the number of surgical procedures experienced by the animal.

Decoding Cognitive Processes from Neural Ensembles.

An intrinsic difficulty in studying cognitive processes is that they are unobservable states that exist in between observable responses to the sensory environment. Cognitive states must be inferred from indirect behavioral measures. Neuroscience potentially provides the tools necessary to measure cognitive processes directly, but it is challenged on two fronts. First, neuroscientific measures often lack the spatiotemporal resolution to identify the neural computations that underlie a cognitive process. Second, the activity of a single neuron, which is the fundamental building block of neural computation, is too noisy to provide accurate measurements of a cognitive process. In this paper, I examine recent developments in neurophysiological recording and analysis methods that provide a potential solution to these problems.

Reconciling persistent and dynamic hypotheses of working memory coding in prefrontal cortex.

Competing accounts propose that working memory (WM) is subserved either by persistent activity in single neurons or by dynamic (time-varying) activity across a neural population. Here, we compare these hypotheses across four regions of prefrontal cortex (PFC) in an oculomotor-delayed-response task, where an intervening cue indicated the reward available for a correct saccade. WM representations were strongest in ventrolateral PFC neurons with higher intrinsic temporal stability (time-constant). At the population-level, although a stable mnemonic state was reached during the delay, this tuning geometry was reversed relative to cue-period selectivity, and was disrupted by the reward cue. Single-neuron analysis revealed many neurons switched to coding reward, rather than maintaining task-relevant spatial selectivity until saccade. These results imply WM is fulfilled by dynamic, population-level activity within high time-constant neurons. Rather than persistent activity supporting stable mnemonic representations that bridge subsequent salient stimuli, PFC neurons may stabilise a dynamic population-level process supporting WM.