Overharvesting in human patch foraging reflects rational structure learning and adaptive planning.

Patch foraging presents a sequential decision-making problem widely studied across organisms-stay with a current option or leave it in search of a better alternative? Behavioral ecology has identified an optimal strategy for these decisions, but, across species, foragers systematically deviate from it, staying too long with an option or "overharvesting" relative to this optimum. Despite the ubiquity of this behavior, the mechanism underlying it remains unclear and an object of extensive investigation. Here, we address this gap by approaching foraging as both a decision-making and learning problem. Specifically, we propose a model in which foragers 1) rationally infer the structure of their environment and 2) use their uncertainty over the inferred structure representation to adaptively discount future rewards. We find that overharvesting can emerge from this rational statistical inference and uncertainty adaptation process. In a patch-leaving task, we show that human participants adapt their foraging to the richness and dynamics of the environment in ways consistent with our model. These findings suggest that definitions of optimal foraging could be extended by considering how foragers reduce and adapt to uncertainty over representations of their environment.

Expanding horizons in reinforcement learning for curious exploration and creative planning.

Curiosity and creativity are expressions of the trade-off between leveraging that with which we are familiar or seeking out novelty. Through the computational lens of reinforcement learning, we describe how formulating the value of information seeking and generation via their complementary effects on planning horizons formally captures a range of solutions to striking this balance.

Associative memory retrieval modulates upcoming perceptual decisions.

Expectations can inform fast, accurate decisions. But what informs expectations? Here we test the hypothesis that expectations are set by dynamic inference from memory. Participants performed a cue-guided perceptual decision task with independently-varying memory and sensory evidence. Cues established expectations by reminding participants of past stimulus-stimulus pairings, which predicted the likely target in a subsequent noisy image stream. Participant's responses used both memory and sensory information, in accordance to their relative reliability. Formal model comparison showed that the sensory inference was best explained when its parameters were set dynamically at each trial by evidence sampled from memory. Supporting this model, neural pattern analysis revealed that responses to the probe were modulated by the specific content and fidelity of memory reinstatement that occurred before the probe appeared. Together, these results suggest that perceptual decisions arise from the continuous sampling of memory and sensory evidence.

Refresh my memory: Episodic memory reinstatements intrude on working memory maintenance.

A fundamental question in memory research is how different forms of memory interact. Previous research has shown that people rely on working memory (WM) in short-term recognition tasks; a common view is that episodic memory (EM) only influences performance on these tasks when WM maintenance is disrupted. However, retrieval of memories from EM has been widely observed during brief periods of quiescence, raising the possibility that EM retrievals during maintenance-critically, before a response can be prepared-might affect short-term recognition memory performance even in the absence of distraction. We hypothesized that this influence would be mediated by the lingering presence of reactivated EM content in WM. We obtained support for this hypothesis in three experiments, showing that delay-period EM reactivation introduces incidentally associated information (context) into WM, and that these retrieved associations negatively impact subsequent recognition, leading to substitution errors (Experiment 1) and slowing of accurate responses (Experiment 2). FMRI pattern analysis showed that slowing is mediated by the content of EM reinstatement (Experiment 3). These results expose a previously hidden influence of EM on WM, raising new questions about the adaptive nature of their interaction.

Cortical and hippocampal correlates of deliberation during model-based decisions for rewards in humans.

How do we use our memories of the past to guide decisions we've never had to make before? Although extensive work describes how the brain learns to repeat rewarded actions, decisions can also be influenced by associations between stimuli or events not directly involving reward - such as when planning routes using a cognitive map or chess moves using predicted countermoves - and these sorts of associations are critical when deciding among novel options. This process is known as model-based decision making. While the learning of environmental relations that might support model-based decisions is well studied, and separately this sort of information has been inferred to impact decisions, there is little evidence concerning the full cycle by which such associations are acquired and drive choices. Of particular interest is whether decisions are directly supported by the same mnemonic systems characterized for relational learning more generally, or instead rely on other, specialized representations. Here, building on our previous work, which isolated dual representations underlying sequential predictive learning, we directly demonstrate that one such representation, encoded by the hippocampal memory system and adjacent cortical structures, supports goal-directed decisions. Using interleaved learning and decision tasks, we monitor predictive learning directly and also trace its influence on decisions for reward. We quantitatively compare the learning processes underlying multiple behavioral and fMRI observables using computational model fits. Across both tasks, a quantitatively consistent learning process explains reaction times, choices, and both expectation- and surprise-related neural activity. The same hippocampal and ventral stream regions engaged in anticipating stimuli during learning are also engaged in proportion to the difficulty of decisions. These results support a role for predictive associations learned by the hippocampal memory system to be recalled during choice formation.

Interval Timing as a Computational Pathway From Early Life Adversity to Affective Disorders.

Adverse early life experiences can have remarkably enduring negative consequences on mental health, with numerous, varied psychiatric conditions sharing this developmental origin. Yet, the mechanisms linking adverse experiences to these conditions remain poorly understood. Here, we draw on a principled model of interval timing to propose that statistically optimal adaptation of temporal representations to an unpredictable early life environment can produce key characteristics of anhedonia, a transdiagnostic symptom associated with affective disorders like depression and anxiety. The core observation is that early temporal unpredictability produces broader, more imprecise temporal expectations. As a result, reward anticipation is diminished, and associative learning is slowed. When agents with such representations are later introduced to more stable environments, they demonstrate a negativity bias, responding more to the omission of reward than its receipt. Increased encoding of negative events has been proposed to contribute to disorders with anhedonia as a symptom. We then examined how unpredictability interacts with another form of adversity, low reward availability. We found that unpredictability's effect was most strongly felt in richer environments, potentially leading to categorically different phenotypic expressions. In sum, our formalization suggests a single mechanism can help to link early life adversity to a range of behaviors associated with anhedonia, and offers novel insights into the interactive impacts of multiple adversities.

The causal structure and computational value of narratives.

Many human behavioral and brain imaging studies have used narratively structured stimuli (e.g., written, audio, or audiovisual stories) to better emulate real-world experience in the laboratory. However, narratives are a special class of real-world experience, largely defined by their causal connections across time. Much contemporary neuroscience research does not consider this key property. We review behavioral and neuroscientific work that speaks to how causal structure shapes comprehension of and memory for narratives. We further draw connections between this work and reinforcement learning, highlighting how narratives help link causes to outcomes in complex environments. By incorporating the plausibility of causal connections between classes of actions and outcomes, reinforcement learning models may become more ecologically valid, while simultaneously elucidating the value of narratives.

Memory precision and age differentially predict the use of decision-making strategies across the lifespan.

Memory function declines in normal aging, in a relatively continuous fashion following middle-age. The effect of aging on decision-making is less well-understood, with seemingly conflicting results on both the nature and direction of these age effects. One route for clarifying these mixed findings is to understand how age-related differences in memory affect decisions. Recent work has proposed memory sampling as a specific computational role for memory in decision-making, alongside well-studied mechanisms of reinforcement learning (RL). Here, we tested the hypothesis that age-related declines in episodic memory alter memory sampling. Participants (total N = 361; ages 18-77) performed one of two variants of a standard reward-guided decision experiment with additional trial-unique mnemonic content and a separately-administered task for assessing memory precision. When we fit participants' choices with a hybrid computational model implementing both memory-based and RL-driven valuation side-by-side, we found that memory precision tracked the contribution of memory sampling to choice. At the same time, age corresponded to decreasing influence of RL and increasing perseveration. A second experiment confirmed these results and further revealed that memory precision tracked the specificity of memories selected for sampling. Together, these findings suggest that differences in decision-making across the lifespan may be related to memory function, and that interventions which aim to improve the former may benefit from targeting the latter.

Dissociating hippocampal and striatal contributions to sequential prediction learning.

Behavior may be generated on the basis of many different kinds of learned contingencies. For instance, responses could be guided by the direct association between a stimulus and response, or by sequential stimulus-stimulus relationships (as in model-based reinforcement learning or goal-directed actions). However, the neural architecture underlying sequential predictive learning is not well understood, in part because it is difficult to isolate its effect on choice behavior. To track such learning more directly, we examined reaction times (RTs) in a probabilistic sequential picture identification task in healthy individuals. We used computational learning models to isolate trial-by-trial effects of two distinct learning processes in behavior, and used these as signatures to analyse the separate neural substrates of each process. RTs were best explained via the combination of two delta rule learning processes with different learning rates. To examine neural manifestations of these learning processes, we used functional magnetic resonance imaging to seek correlates of time-series related to expectancy or surprise. We observed such correlates in two regions, hippocampus and striatum. By estimating the learning rates best explaining each signal, we verified that they were uniquely associated with one of the two distinct processes identified behaviorally. These differential correlates suggest that complementary anticipatory functions drive each region's effect on behavior. Our results provide novel insights as to the quantitative computational distinctions between medial temporal and basal ganglia learning networks and enable experiments that exploit trial-by-trial measurement of the unique contributions of both hippocampus and striatum to response behavior.

Mixing memory and desire: How memory reactivation supports deliberative decision-making.

Memories affect nearly every aspect of our mental life. They allow us to both resolve uncertainty in the present and to construct plans for the future. Recently, renewed interest in the role memory plays in adaptive behavior has led to new theoretical advances and empirical observations. We review key findings, with particular emphasis on how the retrieval of many kinds of memories affects deliberative action selection. These results are interpreted in a sequential inference framework, in which reinstatements from memory serve as "samples" of potential action outcomes. The resulting model suggests a central role for the dynamics of memory reactivation in determining the influence of different kinds of memory in decisions. We propose that representation-specific dynamics can implement a bottom-up "product of experts" rule that integrates multiple sets of action-outcome predictions weighted based on their uncertainty. We close by reviewing related findings and identifying areas for further research. This article is categorized under: Psychology > Reasoning and Decision Making Neuroscience > Cognition Neuroscience > Computation.

Humans can navigate complex graph structures acquired during latent learning.

Humans appear to represent many forms of knowledge in associative networks whose nodes are multiply connected, including sensory, spatial, and semantic. Recent work has shown that explicitly augmenting artificial agents with such graph-structured representations endows them with more human-like capabilities of compositionality and transfer learning. An open question is how humans acquire these representations. Previously, it has been shown that humans can learn to navigate graph-structured conceptual spaces on the basis of direct experience with trajectories that intentionally draw the network contours (Schapiro, Kustner, & Turk-Browne, 2012; Schapiro, Turk-Browne, Botvinick, & Norman, 2016), or through direct experience with rewards that covary with the underlying associative distance (Wu, Schulz, Speekenbrink, Nelson, & Meder, 2018). Here, we provide initial evidence that this capability is more general, extending to learning to reason about shortest-path distances across a graph structure acquired across disjoint experiences with randomized edges of the graph - a form of latent learning. In other words, we show that humans can infer graph structures, assembling them from disordered experiences. We further show that the degree to which individuals learn to reason correctly and with reference to the structure of the graph corresponds to their propensity, in a separate task, to use model-based reinforcement learning to achieve rewards. This connection suggests that the correct acquisition of graph-structured relationships is a central ability underlying forward planning and reasoning, and may be a core computation across the many domains in which graph-based reasoning is advantageous.

Context-dependent choice and evaluation in real-world consumer behavior.

A body of work spanning neuroscience, economics, and psychology indicates that decision-making is context-dependent, which means that the value of an option depends not only on the option in question, but also on the other options in the choice set-or the 'context'. While context effects have been observed primarily in small-scale laboratory studies with tightly constrained, artificially constructed choice sets, it remains to be determined whether these context effects take hold in real-world choice problems, where choice sets are large and decisions driven by rich histories of direct experience. Here, we investigate whether valuations are context-dependent in real-world choice by analyzing a massive restaurant rating dataset as well as two independent replication datasets which provide complementary operationalizations of restaurant choice. We find that users make fewer ratings-maximizing choices in choice sets with higher-rated options-a hallmark of context-dependent choice-and that post-choice restaurant ratings also varied systematically with the ratings of unchosen restaurants. Furthermore, in a follow-up laboratory experiment using hypothetical choice sets matched to the real-world data, we find further support for the idea that subjective valuations of restaurants are scaled in accordance with the choice context, providing corroborating evidence for a general mechanistic-level account of these effects. Taken together, our results provide a potent demonstration of context-dependent choice in real-world choice settings, manifesting both in decisions and subjective valuation of options.

High-resolution fMRI of content-sensitive subsequent memory responses in human medial temporal lobe.

The essential role of the medial temporal lobe (MTL) in long-term memory for individual events is well established, yet important questions remain regarding the mnemonic functions of the component structures that constitute the region. Within the hippocampus, recent functional neuroimaging findings suggest that formation of new memories depends on the dentate gyrus and the CA(3) field, whereas the contribution of the subiculum may be limited to retrieval. During encoding, it has been further hypothesized that structures within MTL cortex contribute to encoding in a content-sensitive manner, whereas hippocampal structures may contribute to encoding in a more domain-general manner. In the current experiment, high-resolution fMRI techniques were utilized to assess novelty and subsequent memory effects in MTL subregions for two classes of stimuli--faces and scenes. During scanning, participants performed an incidental encoding (target detection) task with novel and repeated faces and scenes. Subsequent recognition memory was indexed for the novel stimuli encountered during scanning. Analyses revealed voxels sensitive to both novel faces and novel scenes in all MTL regions. However, similar percentages of voxels were sensitive to novel faces and scenes in perirhinal cortex, entorhinal cortex, and a combined region comprising the dentate gyrus, CA(2), and CA(3), whereas parahippocampal cortex, CA(1), and subiculum demonstrated greater sensitivity to novel scene stimuli. Paralleling these findings, subsequent memory effects in perirhinal cortex were observed for both faces and scenes, with the magnitude of encoding activation being related to later memory strength, as indexed by a graded response tracking recognition confidence, whereas subsequent memory effects were scene-selective in parahippocampal cortex. Within the hippocampus, encoding activation in the subiculum correlated with subsequent memory for both stimulus classes, with the magnitude of encoding activation varying in a graded manner with later memory strength. Collectively, these findings suggest a gradient of content sensitivity from posterior (parahippocampal) to anterior (perirhinal) MTL cortex, with MTL cortical regions differentially contributing to successful encoding based on event content. In contrast to recent suggestions, the present data further indicate that the subiculum may contribute to successful encoding irrespective of event content.

"Chasing the first high": memory sampling in drug choice.

Although vivid memories of drug experiences are prevalent within clinical contexts and addiction folklore ("chasing the first high"), little is known about the relevance of cognitive processes governing memory retrieval to substance use disorder. Drawing on recent work that identifies episodic memory's influence on decisions for reward, we propose a framework in which drug choices are biased by selective sampling of individual memories during two phases of addiction: (i) downward spiral into persistent use and (ii) relapse. Consideration of how memory retrieval influences the addiction process suggests novel treatment strategies. Rather than try to break learned associations between drug cues and drug rewards, treatment should aim to strengthen existing and/or create new associations between drug cues and drug-inconsistent rewards.

Reward prediction errors create event boundaries in memory.

We remember when things change. Particularly salient are experiences where there is a change in rewards, eliciting reward prediction errors (RPEs). How do RPEs influence our memory of those experiences? One idea is that this signal directly enhances the encoding of memory. Another, not mutually exclusive, idea is that the RPE signals a deeper change in the environment, leading to the mnemonic separation of subsequent experiences from what came before, thereby creating a new latent context and a more separate memory trace. We tested this in four experiments where participants learned to predict rewards associated with a series of trial-unique images. High-magnitude RPEs indicated a change in the underlying distribution of rewards. To test whether these large RPEs created a new latent context, we first assessed recognition priming for sequential pairs that included a high-RPE event or not (Exp. 1: n = 27 & Exp. 2: n = 83). We found evidence of recognition priming for the high-RPE event, indicating that the high-RPE event is bound to its predecessor in memory. Given that high-RPE events are themselves preferentially remembered (Rouhani, Norman, & Niv, 2018), we next tested whether there was an event boundary across a high-RPE event (i.e., excluding the high-RPE event itself; Exp. 3: n = 85). Here, sequential pairs across a high RPE no longer showed recognition priming whereas pairs within the same latent reward state did, providing initial evidence for an RPE-modulated event boundary. We then investigated whether RPE event boundaries disrupt temporal memory by asking participants to order and estimate the distance between two events that had either included a high-RPE event between them or not (Exp. 4). We found (n = 49) and replicated (n = 77) worse sequence memory for events across a high RPE. In line with our recognition priming results, we did not find sequence memory to be impaired between the high-RPE event and its predecessor, but instead found worse sequence memory for pairs across a high-RPE event. Moreover, greater distance between events at encoding led to better sequence memory for events across a low-RPE event, but not a high-RPE event, suggesting separate mechanisms for the temporal ordering of events within versus across a latent reward context. Altogether, these findings demonstrate that high-RPE events are both more strongly encoded, show intact links with their predecessor, and act as event boundaries that interrupt the sequential integration of events. We captured these effects in a variant of the Context Maintenance and Retrieval model (CMR; Polyn, Norman, & Kahana, 2009), modified to incorporate RPEs into the encoding process.

Rats exhibit similar biases in foraging and intertemporal choice tasks.

Animals, including humans, consistently exhibit myopia in two different contexts: foraging, in which they harvest locally beyond what is predicted by optimal foraging theory, and intertemporal choice, in which they exhibit a preference for immediate vs. delayed rewards beyond what is predicted by rational (exponential) discounting. Despite the similarity in behavior between these two contexts, previous efforts to reconcile these observations in terms of a consistent pattern of time preferences have failed. Here, via extensive behavioral testing and quantitative modeling, we show that rats exhibit similar time preferences in both contexts: they prefer immediate vs. delayed rewards and they are sensitive to opportunity costs of delays to future decisions. Further, a quasi-hyperbolic discounting model, a form of hyperbolic discounting with separate components for short- and long-term rewards, explains individual rats' time preferences across both contexts, providing evidence for a common mechanism for myopic behavior in foraging and intertemporal choice.

Suicidal thoughts and behaviors are associated with an increased decision-making bias for active responses to escape aversive states.

Suicide is a leading cause of death worldwide. Despite decades of clinical and theoretical accounts that suggest that suicidal thoughts and behaviors are efforts to escape painful emotions, little prior research has examined decision making involved in escaping aversive states. We compared the performance of 85 suicidal participants to 44 nonsuicidal psychiatric patients on a novel reinforcement learning task with choices to make either active (i.e., "go") or passive responses (i.e., "no-go") to either escape or avoid an aversive stimulus. We used a computational cognitive model to isolate decision-making biases. We hypothesized that suicidal participants would exhibit a relatively elevated bias for making active responses to escape an aversive state and would show worse performance when escape required a passive response (i.e., "doing nothing" to escape). Our hypotheses were supported: The computational model revealed that suicidal participants exhibited a higher bias for an active response to escape compared with nonsuicidal psychiatric controls, suggesting that this finding was not just the result of the presence of psychopathology. The bias parameter also accounted for unique variance in predicting group status among several constructs previously related to suicidal thoughts and behaviors. This study provides a new method for testing escape decision making and does so using a computational cognitive model, allowing us to precisely index processes underlying suicidal and related behaviors. Future research examining escape decision making from a computational perspective could help link neural processes or environmental stressors to suicidal thoughts or behaviors. (PsycINFO Database Record (c) 2019 APA, all rights reserved).