Distinct replay signatures for prospective decision-making and memory preservation.

Theories of neural replay propose that it supports a range of functions, most prominently planning and memory consolidation. Here, we test the hypothesis that distinct signatures of replay in the same task are related to model-based decision-making ("planning") and memory preservation. We designed a reward learning task wherein participants utilized structure knowledge for model-based evaluation, while at the same time had to maintain knowledge of two independent and randomly alternating task environments. Using magnetoencephalography and multivariate analysis, we first identified temporally compressed sequential reactivation, or replay, both prior to choice and following reward feedback. Before choice, prospective replay strength was enhanced for the current task-relevant environment when a model-based planning strategy was beneficial. Following reward receipt, and consistent with a memory preservation role, replay for the alternative distal task environment was enhanced as a function of decreasing recency of experience with that environment. Critically, these planning and memory preservation relationships were selective to pre-choice and post-feedback periods, respectively. Our results provide support for key theoretical proposals regarding the functional role of replay and demonstrate that the relative strength of planning and memory-related signals are modulated by ongoing computational and task demands.

Reactivation of Single-Episode Pain Patterns in the Hippocampus and Decision Making.

Aversive and rewarding experiences can exert a strong influence on subsequent behavior. While decisions are often supported by the value of single past episodes, most research has focused on the role of well-learned value associations. Recent studies have begun to investigate the influence of reward-associated episodes, but it is unclear whether these results generalize to negative experiences, such as pain. To investigate whether and how the value of previous aversive experiences modulates behavior and brain activity, in our experiments female and male human participants experienced episodes of high or low pain in conjunction with incidental, trial-unique neutral pictures. In an incentive-compatible surprise test phase, we found that participants avoided pain-paired objects. In a separate fMRI experiment, at test, participants exhibited significant pain value memory. Neurally, when participants were re-exposed to pain-paired objects, we found no evidence for reactivation of pain-related patterns in pain-responsive regions, such as the anterior insula. Critically, however, we found significant reactivation of pain-related patterns of activity in the hippocampus, such that activity significantly discriminated high versus low pain episodes. Further, stronger reactivation in the anterior hippocampus was related to improved pain value memory performance. Our results demonstrate that single incidental aversive experiences can build memories that affect decision-making and that this influence may be supported by the hippocampus.SIGNIFICANCE STATEMENT Aversive and rewarding experiences can exert a strong influence on our subsequent behavior. While decisions are often supported by single past negative or positive episodes, most research has focused on the role of well-learned value associations. In experiments using aversive heat pain in conjunction with incidental objects, we found that participants' choices were biased by the level of pain associated with the objects. Further, when participants saw the objects again, pain-related neural patterns in the hippocampus were re-expressed, and this was related to pain value memory performance. These results suggest a mechanism by which even single negative experiences can guide our later decisions.

Reward learning and working memory: Effects of massed versus spaced training and post-learning delay period.

Neuroscience research has illuminated the mechanisms supporting learning from reward feedback, demonstrating a critical role for the striatum and midbrain dopamine system. However, in humans, short-term working memory that is dependent on frontal and parietal cortices can also play an important role, particularly in commonly used paradigms in which learning is relatively condensed in time. Given the growing use of reward-based learning tasks in translational studies in computational psychiatry, it is important to understand the extent of the influence of working memory and also how core gradual learning mechanisms can be better isolated. In our experiments, we manipulated the spacing between repetitions along with a post-learning delay preceding a test phase. We found that learning was slower for stimuli repeated after a long delay (spaced-trained) compared to those repeated immediately (massed-trained), likely reflecting the remaining contribution of feedback learning mechanisms when working memory is not available. For massed learning, brief interruptions led to drops in subsequent performance, and individual differences in working memory capacity positively correlated with overall performance. Interestingly, when tested after a delay period but not immediately, relative preferences decayed in the massed condition and increased in the spaced condition. Our results provide additional support for a large role of working memory in reward-based learning in temporally condensed designs. We suggest that spacing training within or between sessions is a promising approach to better isolate and understand mechanisms supporting gradual reward-based learning, with particular importance for understanding potential learning dysfunctions in addiction and psychiatric disorders.

Reward Learning over Weeks Versus Minutes Increases the Neural Representation of Value in the Human Brain.

Over the past few decades, neuroscience research has illuminated the neural mechanisms supporting learning from reward feedback. Learning paradigms are increasingly being extended to study mood and psychiatric disorders as well as addiction. However, one potentially critical characteristic that this research ignores is the effect of time on learning: human feedback learning paradigms are usually conducted in a single rapidly paced session, whereas learning experiences in ecologically relevant circumstances and in animal research are almost always separated by longer periods of time. In our experiments, we examined reward learning in short condensed sessions distributed across weeks versus learning completed in a single "massed" session in male and female participants. As expected, we found that after equal amounts of training, accuracy was matched between the spaced and massed conditions. However, in a 3-week follow-up, we found that participants exhibited significantly greater memory for the value of spaced-trained stimuli. Supporting a role for short-term memory in massed learning, we found a significant positive correlation between initial learning and working memory capacity. Neurally, we found that patterns of activity in the medial temporal lobe and prefrontal cortex showed stronger discrimination of spaced- versus massed-trained reward values. Further, patterns in the striatum discriminated between spaced- and massed-trained stimuli overall. Our results indicate that single-session learning tasks engage partially distinct learning mechanisms from distributed training. Our studies begin to address a large gap in our knowledge of human learning from reinforcement, with potential implications for our understanding of mood disorders and addiction.SIGNIFICANCE STATEMENT Humans and animals learn to associate predictive value with stimuli and actions, and these values then guide future behavior. Such reinforcement-based learning often happens over long time periods, in contrast to most studies of reward-based learning in humans. In experiments that tested the effect of spacing on learning, we found that associations learned in a single massed session were correlated with short-term memory and significantly decayed over time, whereas associations learned in short massed sessions over weeks were well maintained. Additionally, patterns of activity in the medial temporal lobe and prefrontal cortex discriminated the values of stimuli learned over weeks but not minutes. These results highlight the importance of studying learning over time, with potential applications to drug addiction and psychiatry.

Reactivation of Reward-Related Patterns from Single Past Episodes Supports Memory-Based Decision Making.

Rewarding experiences exert a strong influence on later decision making. While decades of neuroscience research have shown how reinforcement gradually shapes preferences, decisions are often influenced by single past experiences. Surprisingly, relatively little is known about the influence of single learning episodes. Although recent work has proposed a role for episodes in decision making, it is largely unknown whether and how episodic experiences contribute to value-based decision making and how the values of single episodes are represented in the brain. In multiple behavioral experiments and an fMRI experiment, we tested whether and how rewarding episodes could support later decision making. Participants experienced episodes of high reward or low reward in conjunction with incidental, trial-unique neutral pictures. In a surprise test phase, we found that participants could indeed remember the associated level of reward, as evidenced by accurate source memory for value and preferences to re-engage with rewarded objects. Further, in a separate experiment, we found that high-reward objects shown as primes before a gambling task increased financial risk taking. Neurally, re-exposure to objects in the test phase led to significant reactivation of reward-related patterns. Importantly, individual variability in the strength of reactivation predicted value memory performance. Our results provide a novel demonstration that affect-related neural patterns are reactivated during later experience. Reactivation of value information represents a mechanism by which memory can guide decision making.

Social Cognition as Reinforcement Learning: Feedback Modulates Emotion Inference.

Neuroscientific studies of social cognition typically employ paradigms in which perceivers draw single-shot inferences about the internal states of strangers. Real-world social inference features much different parameters: People often encounter and learn about particular social targets (e.g., friends) over time and receive feedback about whether their inferences are correct or incorrect. Here, we examined this process and, more broadly, the intersection between social cognition and reinforcement learning. Perceivers were scanned using fMRI while repeatedly encountering three social targets who produced conflicting visual and verbal emotional cues. Perceivers guessed how targets felt and received feedback about whether they had guessed correctly. Visual cues reliably predicted one target's emotion, verbal cues predicted a second target's emotion, and neither reliably predicted the third target's emotion. Perceivers successfully used this information to update their judgments over time. Furthermore, trial-by-trial learning signals-estimated using two reinforcement learning models-tracked activity in ventral striatum and ventromedial pFC, structures associated with reinforcement learning, and regions associated with updating social impressions, including TPJ. These data suggest that learning about others' emotions, like other forms of feedback learning, relies on domain-general reinforcement mechanisms as well as domain-specific social information processing.

Episodic memory encoding interferes with reward learning and decreases striatal prediction errors.

Learning is essential for adaptive decision making. The striatum and its dopaminergic inputs are known to support incremental reward-based learning, while the hippocampus is known to support encoding of single events (episodic memory). Although traditionally studied separately, in even simple experiences, these two types of learning are likely to co-occur and may interact. Here we sought to understand the nature of this interaction by examining how incremental reward learning is related to concurrent episodic memory encoding. During the experiment, human participants made choices between two options (colored squares), each associated with a drifting probability of reward, with the goal of earning as much money as possible. Incidental, trial-unique object pictures, unrelated to the choice, were overlaid on each option. The next day, participants were given a surprise memory test for these pictures. We found that better episodic memory was related to a decreased influence of recent reward experience on choice, both within and across participants. fMRI analyses further revealed that during learning the canonical striatal reward prediction error signal was significantly weaker when episodic memory was stronger. This decrease in reward prediction error signals in the striatum was associated with enhanced functional connectivity between the hippocampus and striatum at the time of choice. Our results suggest a mechanism by which memory encoding may compete for striatal processing and provide insight into how interactions between different forms of learning guide reward-based decision making.

Transfer of learning relates to intrinsic connectivity between hippocampus, ventromedial prefrontal cortex, and large-scale networks.

An important aspect of adaptive learning is the ability to flexibly use past experiences to guide new decisions. When facing a new decision, some people automatically leverage previously learned associations, while others do not. This variability in transfer of learning across individuals has been demonstrated repeatedly and has important implications for understanding adaptive behavior, yet the source of these individual differences remains poorly understood. In particular, it is unknown why such variability in transfer emerges even among homogeneous groups of young healthy participants who do not vary on other learning-related measures. Here we hypothesized that individual differences in the transfer of learning could be related to relatively stable differences in intrinsic brain connectivity, which could constrain how individuals learn. To test this, we obtained a behavioral measure of memory-based transfer outside of the scanner and on a separate day acquired resting-state functional MRI images in 42 participants. We then analyzed connectivity across independent component analysis-derived brain networks during rest, and tested whether intrinsic connectivity in learning-related networks was associated with transfer. We found that individual differences in transfer were related to intrinsic connectivity between the hippocampus and the ventromedial prefrontal cortex, and between these regions and large-scale functional brain networks. Together, the findings demonstrate a novel role for intrinsic brain dynamics in flexible learning-guided behavior, both within a set of functionally specific regions known to be important for learning, as well as between these regions and the default and frontoparietal networks, which are thought to serve more general cognitive functions.

Generalization of value in reinforcement learning by humans.

Research in decision-making has focused on the role of dopamine and its striatal targets in guiding choices via learned stimulus-reward or stimulus-response associations, behavior that is well described by reinforcement learning theories. However, basic reinforcement learning is relatively limited in scope and does not explain how learning about stimulus regularities or relations may guide decision-making. A candidate mechanism for this type of learning comes from the domain of memory, which has highlighted a role for the hippocampus in learning of stimulus-stimulus relations, typically dissociated from the role of the striatum in stimulus-response learning. Here, we used functional magnetic resonance imaging and computational model-based analyses to examine the joint contributions of these mechanisms to reinforcement learning. Humans performed a reinforcement learning task with added relational structure, modeled after tasks used to isolate hippocampal contributions to memory. On each trial participants chose one of four options, but the reward probabilities for pairs of options were correlated across trials. This (uninstructed) relationship between pairs of options potentially enabled an observer to learn about option values based on experience with the other options and to generalize across them. We observed blood oxygen level-dependent (BOLD) activity related to learning in the striatum and also in the hippocampus. By comparing a basic reinforcement learning model to one augmented to allow feedback to generalize between correlated options, we tested whether choice behavior and BOLD activity were influenced by the opportunity to generalize across correlated options. Although such generalization goes beyond standard computational accounts of reinforcement learning and striatal BOLD, both choices and striatal BOLD activity were better explained by the augmented model. Consistent with the hypothesized role for the hippocampus in this generalization, functional connectivity between the ventral striatum and hippocampus was modulated, across participants, by the ability of the augmented model to capture participants' choice. Our results thus point toward an interactive model in which striatal reinforcement learning systems may employ relational representations typically associated with the hippocampus.

Neural predictors of purchases.

Microeconomic theory maintains that purchases are driven by a combination of consumer preference and price. Using event-related fMRI, we investigated how people weigh these factors to make purchasing decisions. Consistent with neuroimaging evidence suggesting that distinct circuits anticipate gain and loss, product preference activated the nucleus accumbens (NAcc), while excessive prices activated the insula and deactivated the mesial prefrontal cortex (MPFC) prior to the purchase decision. Activity from each of these regions independently predicted immediately subsequent purchases above and beyond self-report variables. These findings suggest that activation of distinct neural circuits related to anticipatory affect precedes and supports consumers' purchasing decisions.

Episodic memory retrieval success is associated with rapid replay of episode content.

Retrieval of everyday experiences is fundamental for informing our future decisions. The fine-grained neurophysiological mechanisms that support such memory retrieval are largely unknown. We studied participants who first experienced, without repetition, unique multicomponent 40-80-s episodes. One day later, they engaged in cued retrieval of these episodes while undergoing magnetoencephalography. By decoding individual episode elements, we found that trial-by-trial successful retrieval was supported by the sequential replay of episode elements, with a temporal compression factor of >60. The direction of replay supporting retrieval, either backward or forward, depended on whether the task goal was to retrieve elements of an episode that followed or preceded, respectively, a retrieval cue. This sequential replay was weaker in very-high-performing participants, in whom instead we found evidence for simultaneous clustered reactivation. Our results demonstrate that memory-mediated decisions are supported by a rapid replay mechanism that can flexibly shift in direction in response to task goals.

Nucleus accumbens activation mediates the influence of reward cues on financial risk taking.

In functional magnetic resonance imaging research, nucleus accumbens (NAcc) activation spontaneously increases before financial risk taking. As anticipation of diverse rewards can increase NAcc activation, even incidental reward cues may influence financial risk taking. Using event-related functional magnetic resonance imaging, we predicted and found that anticipation of viewing rewarding stimuli (erotic pictures for 15 heterosexual men) increased financial risk taking, and that this effect was partially mediated by increases in NAcc activation. These results are consistent with the notion that incidental reward cues influence financial risk taking by altering anticipatory affect, and so identify a neuropsychological mechanism that may underlie effective emotional appeals in financial, marketing, and political domains.

Retroactive and graded prioritization of memory by reward.

Many decisions are based on an internal model of the world. Yet, how such a model is constructed from experience and represented in memory remains unknown. We test the hypothesis that reward shapes memory for sequences of events by retroactively prioritizing memory for objects as a function of their distance from reward. Human participants encountered neutral objects while exploring a series of mazes for reward. Across six data sets, we find that reward systematically modulates memory for neutral objects, retroactively prioritizing memory for objects closest to the reward. This effect of reward on memory emerges only after a 24-hour delay and is stronger for mazes followed by a longer rest interval, suggesting a role for post-reward replay and overnight consolidation, as predicted by neurobiological data in animals. These findings demonstrate that reward retroactively prioritizes memory along a sequential gradient, consistent with the role of memory in supporting adaptive decision-making.

Splitting the difference: how does the brain code reward episodes?

Animal research and human brain imaging findings suggest that reward processing involves distinct anticipation and outcome phases. Error terms in popular models of reward learning (such as the temporal difference [TD] model) do not distinguish between the updating of expectations in response to reward cues and outcomes. Thus, correlating a single error term with neural activation assumes recruitment of similar neural substrates at each update. Here, we split the error term to separately model reward prediction and prediction errors, and compare the fit of single versus split error terms to functional magnetic resonance imaging (FMRI) data acquired during a monetary incentive delay task. We speculate and find that while the nucleus accumbens computes gain prediction in response to cues, the mesial prefrontal cortex (MPFC) computes gain prediction errors in response to outcomes. In addition to offering a more comprehensive and anatomically situated view of reward processing, split error terms generate novel predictions about psychiatric symptoms and lesion-induced deficits.

Representation of aversive prediction errors in the human periaqueductal gray.

Pain is a primary driver of learning and motivated action. It is also a target of learning, as nociceptive brain responses are shaped by learning processes. We combined an instrumental pain avoidance task with an axiomatic approach to assessing fMRI signals related to prediction errors (PEs), which drive reinforcement-based learning. We found that pain PEs were encoded in the periaqueductal gray (PAG), a structure important for pain control and learning in animal models. Axiomatic tests combined with dynamic causal modeling suggested that ventromedial prefrontal cortex, supported by putamen, provides an expected value-related input to the PAG, which then conveys PE signals to prefrontal regions important for behavioral regulation, including orbitofrontal, anterior mid-cingulate and dorsomedial prefrontal cortices. Thus, pain-related learning involves distinct neural circuitry, with implications for behavior and pain dynamics.

Preference by association: how memory mechanisms in the hippocampus bias decisions.

Every day people make new choices between alternatives that they have never directly experienced. Yet, such decisions are often made rapidly and confidently. Here, we show that the hippocampus, traditionally known for its role in building long-term declarative memories, enables the spread of value across memories, thereby guiding decisions between new choice options. Using functional brain imaging in humans, we discovered that giving people monetary rewards led to activation of a preestablished network of memories, spreading the positive value of reward to nonrewarded items stored in memory. Later, people were biased to choose these nonrewarded items. This decision bias was predicted by activity in the hippocampus, reactivation of associated memories, and connectivity between memory and reward regions in the brain. These findings explain how choices among new alternatives emerge automatically from the associative mechanisms by which the brain builds memories. Further, our findings demonstrate a previously unknown role for the hippocampus in value-based decisions.

Saving for the future self: neural measures of future self-continuity predict temporal discounting.

Despite increases in the human life span, people have not increased their rate of saving. In a phenomenon known as 'temporal discounting', people value immediate gains over future gains. According to a future self-continuity hypothesis, individuals perceive and treat the future self differently from the present self, and so might fail to save for their future. Neuroimaging offers a novel means of testing this hypothesis, since previous research indicates that self- vs other-judgments elicit activation in the rostral anterior cingulate (rACC). Using event-related functional magnetic resonance imaging, we predicted and found not only individual differences in rACC activation while rating the current vs future self, but also that individual differences in current vs future self activation predicted temporal discounting assessed behaviorally a week after scanning. In addition to supporting the future self-continuity hypothesis, these findings hold implications for significant financial decisions, such as choosing whether to save for the future or spend in the present.

Available alternative incentives modulate anticipatory nucleus accumbens activation.

A reward or punishment can seem better or worse depending on what else might have happened. Little is known, however, about how neural representations of an anticipated incentive might be influenced by the available alternatives. We used event-related FMRI to investigate the activation in the nucleus accumbens (NAcc), while we varied the available alternative incentives in a monetary incentive delay task. Some task blocks included only uncertain gains and losses; others included the same uncertain gains and losses intermixed with certain gains and losses. The availability of certain gains and losses increased NAcc activation for uncertain losses and decreased the difference between uncertain gains and losses. We suggest that this pattern of activation can result from reference point changes across blocks, and that the worst available loss may serve as an important anchor for NAcc activation. These findings imply that NAcc activation represents anticipated incentive value relative to the current context of available alternative gains and losses.

Neural antecedents of the endowment effect.

The "endowment effect" refers to the tendency to place greater value on items that one owns-an anomaly that violates the reference-independence assumption of rational choice theories. We investigated neural antecedents of the endowment effect in an event-related functional magnetic resonance imaging (fMRI) study. During scanning, 24 subjects considered six products paired with 18 different prices under buying, choosing, or selling conditions. Subjects showed greater nucleus accumbens (NAcc) activation for preferred products across buy and sell conditions combined, but greater mesial prefrontal cortex (MPFC) activation in response to low prices when buying versus selling. During selling, right insular activation for preferred products predicted individual differences in susceptibility to the endowment effect. These findings are consistent with a reference-dependent account in which ownership increases value by enhancing the salience of the possible loss of preferred products.