Dorsal raphe neurons integrate the values of reward amount, delay, and uncertainty in multi-attribute decision-making.

The dorsal raphe nucleus (DRN) is implicated in psychiatric disorders that feature impaired sensitivity to reward amount, impulsivity when facing reward delays, and risk-seeking when confronting reward uncertainty. However, it has been unclear whether and how DRN neurons signal reward amount, reward delay, and reward uncertainty during multi-attribute value-based decision-making, where subjects consider these attributes to make a choice. We recorded DRN neurons as monkeys chose between offers whose attributes, namely expected reward amount, reward delay, and reward uncertainty, varied independently. Many DRN neurons signaled offer attributes, and this population tended to integrate the attributes in a manner that reflected monkeys' preferences for amount, delay, and uncertainty. After decision-making, in response to post-decision feedback, these same neurons signaled signed reward prediction errors, suggesting a broader role in tracking value across task epochs and behavioral contexts. Our data illustrate how the DRN participates in value computations, guiding theories about the role of the DRN in decision-making and psychiatric disease.

A neural mechanism for conserved value computations integrating information and rewards.

Behavioral and economic theory dictate that we decide between options based on their values. However, humans and animals eagerly seek information about uncertain future rewards, even when this does not provide any objective value. This implies that decisions are made by endowing information with subjective value and integrating it with the value of extrinsic rewards, but the mechanism is unknown. Here, we show that human and monkey value judgements obey strikingly conserved computational principles during multi-attribute decisions trading off information and extrinsic reward. We then identify a neural substrate in a highly conserved ancient structure, the lateral habenula (LHb). LHb neurons signal subjective value, integrating information's value with extrinsic rewards, and the LHb predicts and causally influences ongoing decisions. Neurons in key input areas to the LHb largely signal components of these computations, not integrated value signals. Thus, our data uncover neural mechanisms of conserved computations underlying decisions to seek information about the future.

Dorsal raphe neurons signal integrated value during multi-attribute decision-making.

The dorsal raphe nucleus (DRN) is implicated in psychiatric disorders that feature impaired sensitivity to reward amount, impulsivity when facing reward delays, and risk-seeking when grappling with reward uncertainty. However, whether and how DRN neurons signal reward amount, reward delay, and reward uncertainty during multi-attribute value-based decision-making, where subjects consider all these attributes to make a choice, is unclear. We recorded DRN neurons as monkeys chose between offers whose attributes, namely expected reward amount, reward delay, and reward uncertainty, varied independently. Many DRN neurons signaled offer attributes. Remarkably, these neurons commonly integrated offer attributes in a manner that reflected monkeys' overall preferences for amount, delay, and uncertainty. After decision-making, in response to post-decision feedback, these same neurons signaled signed reward prediction errors, suggesting a broader role in tracking value across task epochs and behavioral contexts. Our data illustrate how DRN participates in integrated value computations, guiding theories of DRN in decision-making and psychiatric disease.

Laser stimulation of the skin for quantitative study of decision-making and motivation.

Neuroeconomics studies how decision-making is guided by the value of rewards and punishments. But to date, little is known about how noxious experiences impact decisions. A challenge is the lack of an aversive stimulus that is dynamically adjustable in intensity and location, readily usable over many trials in a single experimental session, and compatible with multiple ways to measure neuronal activity. We show that skin laser stimulation used in human studies of aversion can be used for this purpose in several key animal models. We then use laser stimulation to study how neurons in the orbitofrontal cortex (OFC), an area whose many roles include guiding decisions among different rewards, encode the value of rewards and punishments. We show that some OFC neurons integrated the positive value of rewards with the negative value of aversive laser stimulation, suggesting that the OFC can play a role in more complex choices than previously appreciated.

Surprise and recency in novelty detection in the primate brain.

Primates and other animals must detect novel objects. However, the neuronal mechanisms of novelty detection remain unclear. Prominent theories propose that visual object novelty is either derived from the computation of recency (how long ago a stimulus was experienced) or is a form of sensory surprise (stimulus unpredictability). Here, we use high-channel electrophysiology in primates to show that in many primate prefrontal, temporal, and subcortical brain areas, object novelty detection is intertwined with the computations of recency and sensory surprise. Also, distinct circuits could be engaged by expected versus unexpected sensory surprise. Finally, we studied neuronal novelty-to-familiarity transformations during learning across many days. We found a diversity of timescales in neurons' learning rates and between-session forgetting rates, both within and across brain areas, that are well suited to support flexible behavior and learning in response to novelty. Our findings show that novelty sensitivity arises on multiple timescales across single neurons due to diverse but related computations of sensory surprise and recency and shed light on the computational underpinnings of novelty detection in the primate brain.

How the value of the environment controls persistence in visual search.

Classic foraging theory predicts that humans and animals aim to gain maximum reward per unit time. However, in standard instrumental conditioning tasks individuals adopt an apparently suboptimal strategy: they respond slowly when the expected value is low. This reward-related bias is often explained as reduced motivation in response to low rewards. Here we present evidence this behavior is associated with a complementary increased motivation to search the environment for alternatives. We trained monkeys to search for reward-related visual targets in environments with different values. We found that the reward-related bias scaled with environment value, was consistent with persistent searching after the target was already found, and was associated with increased exploratory gaze to objects in the environment. A novel computational model of foraging suggests that this search strategy could be adaptive in naturalistic settings where both environments and the objects within them provide partial information about hidden, uncertain rewards.

A prefrontal network integrates preferences for advance information about uncertain rewards and punishments.

Humans and animals can be strongly motivated to seek information to resolve uncertainty about rewards and punishments. In particular, despite its clinical and societal relevance, very little is known about information seeking about punishments. We show that attitudes toward information about punishments and rewards are distinct and separable at both behavioral and neuronal levels. We demonstrate the existence of prefrontal neuronal populations that anticipate opportunities to gain information in a relatively valence-specific manner, separately anticipating information about either punishments or rewards. These neurons are located in anatomically interconnected subregions of anterior cingulate cortex (ACC) and ventrolateral prefrontal cortex (vlPFC) in area 12o/47. Unlike ACC, vlPFC also contains a population of neurons that integrate attitudes toward both reward and punishment information, to encode the overall preference for information in a bivalent manner. This cortical network is well suited to mediate information seeking by integrating the desire to resolve uncertainty about multiple, distinct motivational outcomes.

The Value of Beliefs.

We construct our beliefs to meet two sometimes conflicting goals: forming accurate beliefs to inform our decisions and forming desirable beliefs that we value for their own sake. In this NeuroView, we consider emerging neuroscience evidence on how the brain motivates itself to form particular beliefs and why it does so.

Neural circuitry of information seeking.

Humans and animals navigate uncertain environments by seeking information about the future. Remarkably, we often seek information even when it has no instrumental value for aiding our decisions - as if the information is a source of value in its own right. In recent years, there has been a flourishing of research into these non-instrumental information preferences and their implementation in the brain. Individuals value information about uncertain future rewards, and do so for multiple reasons, including valuing resolution of uncertainty and overweighting desirable information. The brain motivates this information seeking by tapping into some of the same circuitry as primary rewards like food and water. However, it also employs cortex and basal ganglia circuitry that predicts and values information as distinct from primary reward. Uncovering how these circuits cooperate will be fundamental to understanding information seeking and motivated behavior as a whole, in our increasingly complex and information-rich world.

A neural network for information seeking.

Humans and other animals often show a strong desire to know the uncertain rewards their future has in store, even when they cannot use this information to influence the outcome. However, it is unknown how the brain predicts opportunities to gain information and motivates this information-seeking behavior. Here we show that neurons in a network of interconnected subregions of primate anterior cingulate cortex and basal ganglia predict the moment of gaining information about uncertain rewards. Spontaneous increases in their information prediction signals are followed by gaze shifts toward objects associated with resolving uncertainty, and pharmacologically disrupting this network reduces the motivation to seek information. These findings demonstrate a cortico-basal ganglia mechanism responsible for motivating actions to resolve uncertainty by seeking knowledge about the future.

Valuation of knowledge and ignorance in mesolimbic reward circuitry.

The pursuit of knowledge is a basic feature of human nature. However, in domains ranging from health to finance people sometimes choose to remain ignorant. Here, we show that valence is central to the process by which the human brain evaluates the opportunity to gain information, explaining why knowledge may not always be preferred. We reveal that the mesolimbic reward circuitry selectively treats the opportunity to gain knowledge about future favorable outcomes, but not unfavorable outcomes, as if it has positive utility. This neural coding predicts participants' tendency to choose knowledge about future desirable outcomes more often than undesirable ones, and to choose ignorance about future undesirable outcomes more often than desirable ones. Strikingly, participants are willing to pay both for knowledge and ignorance as a function of the expected valence of knowledge. The orbitofrontal cortex (OFC), however, responds to the opportunity to receive knowledge over ignorance regardless of the valence of the information. Connectivity between the OFC and mesolimbic circuitry could contribute to a general preference for knowledge that is also modulated by valence. Our findings characterize the importance of valence in information seeking and its underlying neural computation. This mechanism could lead to suboptimal behavior, such as when people reject medical screenings or monitor investments more during bull than bear markets.

Orbitofrontal cortex uses distinct codes for different choice attributes in decisions motivated by curiosity.

Decision makers are curious and consequently value advance information about future events. We made use of this fact to test competing theories of value representation in area 13 of orbitofrontal cortex (OFC). In a new task, we found that monkeys reliably sacrificed primary reward (water) to view advance information about gamble outcomes. While monkeys integrated information value with primary reward value to make their decisions, OFC neurons had no systematic tendency to integrate these variables, instead encoding them in orthogonal manners. These results suggest that the predominant role of the OFC is to encode variables relevant for learning, attention, and decision making, rather than integrating them into a single scale of value. They also suggest that OFC may be placed at a relatively early stage in the hierarchy of information-seeking decisions, before evaluation is complete. Thus, our results delineate a circuit for information-seeking decisions and suggest a neural basis for curiosity.

Lateral habenula neurons signal errors in the prediction of reward information.

Humans and animals have the ability to predict future events, which they cultivate by continuously searching their environment for sources of predictive information. However, little is known about the neural systems that motivate this behavior. We hypothesized that information-seeking is assigned value by the same circuits that support reward-seeking, such that neural signals encoding reward prediction errors (RPEs) include analogous information prediction errors (IPEs). To test this, we recorded from neurons in the lateral habenula, a nucleus that encodes RPEs, while monkeys chose between cues that provided different chances to view information about upcoming rewards. We found that a subpopulation of lateral habenula neurons transmitted signals resembling IPEs, responding when reward information was unexpectedly cued, delivered or denied. These signals evaluated information sources reliably, even when the monkey's decisions did not. These neurons could provide a common instructive signal for reward-seeking and information-seeking behavior.

Dopamine in motivational control: rewarding, aversive, and alerting.

Midbrain dopamine neurons are well known for their strong responses to rewards and their critical role in positive motivation. It has become increasingly clear, however, that dopamine neurons also transmit signals related to salient but nonrewarding experiences such as aversive and alerting events. Here we review recent advances in understanding the reward and nonreward functions of dopamine. Based on this data, we propose that dopamine neurons come in multiple types that are connected with distinct brain networks and have distinct roles in motivational control. Some dopamine neurons encode motivational value, supporting brain networks for seeking, evaluation, and value learning. Others encode motivational salience, supporting brain networks for orienting, cognition, and general motivation. Both types of dopamine neurons are augmented by an alerting signal involved in rapid detection of potentially important sensory cues. We hypothesize that these dopaminergic pathways for value, salience, and alerting cooperate to support adaptive behavior.

Multiple timescales of memory in lateral habenula and dopamine neurons.

Midbrain dopamine neurons are thought to signal predictions about future rewards based on the memory of past rewarding experience. Little is known about the source of their reward memory and the factors that control its timescale. Here we recorded from dopamine neurons, as well as one of their sources of input, the lateral habenula, while animals predicted upcoming rewards based on the past reward history. We found that lateral habenula and dopamine neurons accessed two distinct reward memories: a short-timescale memory expressed at the start of the task and a near-optimal long-timescale memory expressed when a future reward outcome was revealed. The short- and long-timescale memories were expressed in different forms of reward-oriented eye movements. Our data show that the habenula-dopamine pathway contains multiple timescales of memory and provide evidence for their role in motivated behavior.

Distinct tonic and phasic anticipatory activity in lateral habenula and dopamine neurons.

Dopamine has a crucial role in anticipation of motivational events. To investigate the underlying mechanisms of this process, we analyzed the activity of dopamine neurons and one of their major sources of input, neurons in the lateral habenula, while animals anticipated upcoming behavioral tasks. We found that lateral habenula and dopamine neurons anticipated tasks in two distinct manners. First, neurons encoded the timing distribution of upcoming tasks through gradual changes in their tonic activity. This tonic signal encoded rewarding tasks in preference to punishing tasks and was correlated with classic phasic coding of motivational value. Second, neurons transmitted a phasic signal marking the time when a task began. This phasic signal encoded rewarding and punishing tasks in similar manners, as though reflecting motivational salience. Our data suggest that the habenula-dopamine pathway motivates anticipation through a combination of tonic reward-related and phasic salience-related signals.

A pallidus-habenula-dopamine pathway signals inferred stimulus values.

The reward value of a stimulus can be learned through two distinct mechanisms: reinforcement learning through repeated stimulus-reward pairings and abstract inference based on knowledge of the task at hand. The reinforcement mechanism is often identified with midbrain dopamine neurons. Here we show that a neural pathway controlling the dopamine system does not rely exclusively on either stimulus-reward pairings or abstract inference but instead uses a combination of the two. We trained monkeys to perform a reward-biased saccade task in which the reward values of two saccade targets were related in a systematic manner. Animals used each trial's reward outcome to learn the values of both targets: the target that had been presented and whose reward outcome had been experienced (experienced value) and the target that had not been presented but whose value could be inferred from the reward statistics of the task (inferred value). We then recorded from three populations of reward-coding neurons: substantia nigra dopamine neurons; a major input to dopamine neurons, the lateral habenula; and neurons that project to the lateral habenula, located in the globus pallidus. All three populations encoded both experienced values and inferred values. In some animals, neurons encoded experienced values more strongly than inferred values, and the animals showed behavioral evidence of learning faster from experience than from inference. Our data indicate that the pallidus-habenula-dopamine pathway signals reward values estimated through both experience and inference.

Coding of task reward value in the dorsal raphe nucleus.

The dorsal raphe nucleus and its serotonin-releasing neurons are thought to regulate motivation and reward-seeking. These neurons are known to be active during motivated behavior, but the underlying principles that govern their activity are unknown. Here we show that a group of dorsal raphe neurons encode behavioral tasks in a systematic manner, tracking progress toward upcoming rewards. We analyzed dorsal raphe neuron activity recorded while animals performed two reward-oriented saccade tasks. There was a strong correlation between the tonic activity level of a neuron during behavioral tasks and its encoding of reward-related cues and outcomes. Neurons that were tonically excited during the task predominantly carried positive reward signals. Neurons that were tonically inhibited during the task predominantly carried negative reward signals. Neurons that did not change their tonic activity levels during the task had weak reward signals with no tendency for a positive or negative direction. This form of correlated task and reward coding accounted for the majority of systematic variation in dorsal raphe response patterns in our tasks. A smaller component of neural activity reflected detection of reward delivery. Our data suggest that the dorsal raphe nucleus encodes participation in a behavioral task in terms of its future motivational outcomes.

Midbrain dopamine neurons signal preference for advance information about upcoming rewards.

The desire to know what the future holds is a powerful motivator in everyday life, but it is unknown how this desire is created by neurons in the brain. Here we show that when macaque monkeys are offered a water reward of variable magnitude, they seek advance information about its size. Furthermore, the same midbrain dopamine neurons that signal the expected amount of water also signal the expectation of information, in a manner that is correlated with the strength of the animal's preference. Our data show that single dopamine neurons process both primitive and cognitive rewards, and suggest that current theories of reward-seeking must be revised to include information-seeking.