Two routes to value-based decisions in Parkinson's disease: differentiating incremental reinforcement learning from episodic memory.

Patients with Parkinson's disease are impaired at incremental reward-based learning. It is typically assumed that this impairment reflects a loss of striatal dopamine. However, many open questions remain about the nature of reward-based learning deficits in Parkinson's. Recent studies have found that a combination of different cognitive and computational strategies contribute even to simple reward-based learning tasks, suggesting a possible role for episodic memory. These findings raise critical questions about how incremental learning and episodic memory interact to support learning from past experience and what their relative contributions are to impaired decision-making in Parkinson's disease. Here we addressed these questions by asking patients with Parkinson's disease (n=26) both on and off their dopamine replacement medication and age- and education-matched healthy controls (n=26) to complete a task designed to isolate the contributions of incremental learning and episodic memory to reward-based learning and decision-making. We found that Parkinson's patients performed as well as healthy controls when using episodic memory, but were impaired at incremental reward-based learning. Dopamine replacement medication remediated this deficit while enhancing subsequent episodic memory for the value of motivationally relevant stimuli. These results demonstrate that Parkinson's patients are impaired at learning about reward from trial-and-error when episodic memory is properly controlled for, and that learning based on the value of single experiences remains intact in patients with Parkinson's disease.

Emergent neural dynamics and geometry for generalization in a transitive inference task.

Relational cognition-the ability to infer relationships that generalize to novel combinations of objects-is fundamental to human and animal intelligence. Despite this importance, it remains unclear how relational cognition is implemented in the brain due in part to a lack of hypotheses and predictions at the levels of collective neural activity and behavior. Here we discovered, analyzed, and experimentally tested neural networks (NNs) that perform transitive inference (TI), a classic relational task (if A > B and B > C, then A > C). We found NNs that (i) generalized perfectly, despite lacking overt transitive structure prior to training, (ii) generalized when the task required working memory (WM), a capacity thought to be essential to inference in the brain, (iii) emergently expressed behaviors long observed in living subjects, in addition to a novel order-dependent behavior, and (iv) expressed different task solutions yielding alternative behavioral and neural predictions. Further, in a large-scale experiment, we found that human subjects performing WM-based TI showed behavior inconsistent with a class of NNs that characteristically expressed an intuitive task solution. These findings provide neural insights into a classical relational ability, with wider implications for how the brain realizes relational cognition.

The Role of the Cerebellum in Learning to Predict Reward: Evidence from Cerebellar Ataxia.

Recent findings in animals have challenged the traditional view of the cerebellum solely as the site of motor control, suggesting that the cerebellum may also be important for learning to predict reward from trial-and-error feedback. Yet, evidence for the role of the cerebellum in reward learning in humans is lacking. Moreover, open questions remain about which specific aspects of reward learning the cerebellum may contribute to. Here we address this gap through an investigation of multiple forms of reward learning in individuals with cerebellum dysfunction, represented by cerebellar ataxia cases. Nineteen participants with cerebellar ataxia and 57 age- and sex-matched healthy controls completed two separate tasks that required learning about reward contingencies from trial-and-error. To probe the selectivity of reward learning processes, the tasks differed in their underlying structure: while one task measured incremental reward learning ability alone, the other allowed participants to use an alternative learning strategy based on episodic memory alongside incremental reward learning. We found that individuals with cerebellar ataxia were profoundly impaired at reward learning from trial-and-error feedback on both tasks, but retained the ability to learn to predict reward based on episodic memory. These findings provide evidence from humans for a specific and necessary role for the cerebellum in incremental learning of reward associations based on reinforcement. More broadly, the findings suggest that alongside its role in motor learning, the cerebellum likely operates in concert with the basal ganglia to support reinforcement learning from reward.

Proactive and reactive construction of memory-based preferences.

We are often faced with decisions we have never encountered before, requiring us to infer possible outcomes before making a choice. Computational theories suggest that one way to make these types of decisions is by accessing and linking related experiences stored in memory. Past work has shown that such memory-based preference construction can occur at a number of different timepoints relative to the moment a decision is made. Some studies have found that memories are integrated at the time a decision is faced (reactively) while others found that memory integration happens earlier, when memories are encoded (proactively). Here we offer a resolution to this inconsistency. We demonstrate behavioral and neural evidence for both strategies and for how they tradeoff rationally depending on the associative structure of memory. Using fMRI to decode patterns of brain responses unique to categories of images in memory, we found that proactive memory access is more common and allows more efficient inference. However, participants also use reactive access when choice options are linked to more numerous memory associations. Together, these results indicate that the brain judiciously conducts proactive inference by accessing memories ahead of time in conditions when this strategy is most favorable.

The role of memory in counterfactual valuation.

Value-based decisions are often guided by past experience. If a choice led to a good outcome, we are more likely to repeat it. This basic idea is well-captured by reinforcement-learning models. However, open questions remain about how we assign value to options we did not choose and which we therefore never had the chance to learn about directly. One solution to this problem is proposed by policy gradient reinforcement-learning models; these do not require direct learning of value, instead optimizing choices according to a behavioral policy. For example, a logistic policy predicts that if a chosen option was rewarded, the unchosen option would be deemed less desirable. Here, we test the relevance of these models to human behavior and explore the role of memory in this phenomenon. We hypothesize that a policy may emerge from an associative memory trace formed during deliberation between choice options. In a preregistered study (n = 315) we show that people tend to invert the value of unchosen options relative to the outcome of chosen options, a phenomenon we term inverse decision bias. The inverse decision bias is correlated with memory for the association between choice options; moreover, it is reduced when memory formation is experimentally interfered with. Finally, we present a new memory-based policy gradient model that predicts both the inverse decision bias and its dependence on memory. Our findings point to a significant role of associative memory in valuation of unchosen options and introduce a new perspective on the interaction between decision-making, memory, and counterfactual reasoning. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Visuospatial information foraging describes search behavior in learning latent environmental features.

In the real world, making sequences of decisions to achieve goals often depends upon the ability to learn aspects of the environment that are not directly perceptible. Learning these so-called latent features requires seeking information about them. Prior efforts to study latent feature learning often used single decisions, used few features, and failed to distinguish between reward-seeking and information-seeking. To overcome this, we designed a task in which humans and monkeys made a series of choices to search for shapes hidden on a grid. On our task, the effects of reward and information outcomes from uncovering parts of shapes could be disentangled. Members of both species adeptly learned the shapes and preferred to select tiles expected to be informative earlier in trials than previously rewarding ones, searching a part of the grid until their outcomes dropped below the average information outcome-a pattern consistent with foraging behavior. In addition, how quickly humans learned the shapes was predicted by how well their choice sequences matched the foraging pattern, revealing an unexpected connection between foraging and learning. This adaptive search for information may underlie the ability in humans and monkeys to learn latent features to support goal-directed behavior in the long run.

Uncertainty alters the balance between incremental learning and episodic memory.

A key question in decision-making is how humans arbitrate between competing learning and memory systems to maximize reward. We address this question by probing the balance between the effects, on choice, of incremental trial-and-error learning versus episodic memories of individual events. Although a rich literature has studied incremental learning in isolation, the role of episodic memory in decision-making has only recently drawn focus, and little research disentangles their separate contributions. We hypothesized that the brain arbitrates rationally between these two systems, relying on each in circumstances to which it is most suited, as indicated by uncertainty. We tested this hypothesis by directly contrasting contributions of episodic and incremental influence to decisions, while manipulating the relative uncertainty of incremental learning using a well-established manipulation of reward volatility. Across two large, independent samples of young adults, participants traded these influences off rationally, depending more on episodic information when incremental summaries were more uncertain. These results support the proposal that the brain optimizes the balance between different forms of learning and memory according to their relative uncertainties and elucidate the circumstances under which episodic memory informs decisions.

Reinforcement learning with associative or discriminative generalization across states and actions: fMRI at 3 T and 7 T.

The model-free algorithms of "reinforcement learning" (RL) have gained clout across disciplines, but so too have model-based alternatives. The present study emphasizes other dimensions of this model space in consideration of associative or discriminative generalization across states and actions. This "generalized reinforcement learning" (GRL) model, a frugal extension of RL, parsimoniously retains the single reward-prediction error (RPE), but the scope of learning goes beyond the experienced state and action. Instead, the generalized RPE is efficiently relayed for bidirectional counterfactual updating of value estimates for other representations. Aided by structural information but as an implicit rather than explicit cognitive map, GRL provided the most precise account of human behavior and individual differences in a reversal-learning task with hierarchical structure that encouraged inverse generalization across both states and actions. Reflecting inference that could be true, false (i.e., overgeneralization), or absent (i.e., undergeneralization), state generalization distinguished those who learned well more so than action generalization. With high-resolution high-field fMRI targeting the dopaminergic midbrain, the GRL model's RPE signals (alongside value and decision signals) were localized within not only the striatum but also the substantia nigra and the ventral tegmental area, including specific effects of generalization that also extend to the hippocampus. Factoring in generalization as a multidimensional process in value-based learning, these findings shed light on complexities that, while challenging classic RL, can still be resolved within the bounds of its core computations.

An energizing role for motivation in information-seeking during the early phase of the COVID-19 pandemic.

The COVID-19 pandemic has highlighted the importance of understanding and managing information seeking behavior. Information-seeking in humans is often viewed as irrational rather than utility maximizing. Here, we hypothesized that this apparent disconnect between utility and information-seeking is due to a latent third variable, motivation. We quantified information-seeking, learning, and COVID-19-related concern (which we used as a proxy for motivation regarding COVID-19 and the changes in circumstance it caused) in a US-based sample (n = 5376) during spring 2020. We found that self-reported levels of COVID-19 concern were associated with directed seeking of COVID-19-related content and better memory for such information. Interestingly, this specific motivational state was also associated with a general enhancement of information-seeking for content unrelated to COVID-19. These effects were associated with commensurate changes to utility expectations and were dissociable from the influence of non-specific anxiety. Thus, motivation both directs and energizes epistemic behavior, linking together utility and curiosity.

Preserved motor memory in Parkinson's disease.

Patients with Parkinson's disease, who lose the dopaminergic projections to the striatum, are impaired in certain aspects of motor learning. Recent evidence suggests that, in addition to its role in motor performance, the striatum plays a key role in the memory of motor learning. Whether Parkinson's patients have impaired motor memory and whether motor memory is modulated by dopamine at the time of initial learning is unknown. To address these questions, we measured memory of a learned motor sequence in Parkinson's patients who were either On or Off their dopaminergic medications at the time of initial learning. We compared them to a group of older and younger controls. Contrary to our predictions, motor memory was not impaired in patients compared to older controls, and was not influenced by dopamine state at the time of initial learning. To probe post-learning consolidation processes, we also tested whether learning a new sequence shortly after learning the initial sequence would interfere with later memory. We found that, in contrast to younger adults, neither older adults nor patients were susceptible to this interference. These findings suggest that motor memory is preserved in Parkinson's patients and raise the possibility that motor memory in patients is supported by compensatory non-dopamine sensitive mechanisms. Furthermore, given the similar performance characteristics observed in the patients and older adults and the absence of an effect of dopamine, these results raise the possibility that aging and Parkinson's disease affect motor memory in similar ways.

Restrictive eating across a spectrum from healthy to unhealthy: behavioral and neural mechanisms.

BACKGROUND: Restriction of food intake is a central feature of anorexia nervosa (AN) and other eating disorders, yet also occurs in the absence of psychopathology. The neural mechanisms of restrictive eating in health and disease are unclear. METHODS: This study examined behavioral and neural mechanisms associated with restrictive eating among individuals with and without eating disorders. Dietary restriction was examined in four groups of women (n = 110): healthy controls, dieting healthy controls, patients with subthreshold (non-low weight) AN, and patients with AN. A Food Choice Task was administered during fMRI scanning to examine neural activation associated with food choices, and a laboratory meal was conducted. RESULTS: Behavioral findings distinguished between healthy and ill participants. Healthy individuals, both dieting and non-dieting, chose significantly more high-fat foods than patients with AN or subthreshold AN. Among healthy individuals, choice was primarily influenced by tastiness, whereas, among both patient groups, healthiness played a larger role. Dorsal striatal activation associated with choice was most pronounced among individuals with AN and was significantly associated with selecting fewer high-fat choices in the task and lower caloric intake in the meal the following day. CONCLUSIONS: A continuous spectrum of behavior was suggested by the increasing amount of weight loss across groups. Yet, data from this Food Choice Task with fMRI suggest there is a behavioral distinction between illness and health, and that the neural mechanisms underlying food choice in AN are distinct. These behavioral and neural mechanisms of restrictive eating may be useful targets for treatment development.

Neural Representations of Food-Related Attributes in the Human Orbitofrontal Cortex during Choice Deliberation in Anorexia Nervosa.

Decisions about what to eat recruit the orbitofrontal cortex (OFC) and involve the evaluation of food-related attributes such as taste and health. These attributes are used differently by healthy individuals and patients with disordered eating behavior, but it is unclear whether these attributes are decodable from activity in the OFC in both groups and whether neural representations of these attributes are differentially related to decisions about food. We used fMRI combined with behavioral tasks to investigate the representation of taste and health attributes in the human OFC and the role of these representations in food choices in healthy women and women with anorexia nervosa (AN). We found that subjective ratings of tastiness and healthiness could be decoded from patterns of activity in the OFC in both groups. However, health-related patterns of activity in the OFC were more related to the magnitude of choice preferences among patients with AN than healthy individuals. These findings suggest that maladaptive decision-making in AN is associated with more consideration of health information represented by the OFC during deliberation about what to eat.SIGNIFICANCE STATEMENT An open question about the OFC is whether it supports the evaluation of food-related attributes during deliberation about what to eat. We found that healthiness and tastiness information was decodable from patterns of neural activity in the OFC in both patients with AN and healthy controls. Critically, neural representations of health were more strongly related to choices in patients with AN, suggesting that maladaptive overconsideration of healthiness during deliberation about what to eat is related to activity in the OFC. More broadly, these results show that activity in the human OFC is associated with the evaluation of relevant attributes during value-based decision-making. These findings may also guide future research into the development of treatments for AN.

Learned temporal statistics guide information seeking and shape memory.

Curiosity drives information seeking and promotes learning. Prior work has focused on how curiosity is elicited by intrinsic qualities of information, leaving open questions about how curiosity, exploration, and learning are shaped by the environment. Here we examine how temporal dynamics of the learning environment shape curiosity and learning. Participants (n = 71) foraged for the answer to trivia questions in two conditions that differed only in their temporal statistics. In one condition, the timing of information delivery followed a uniform distribution, while in another it followed a heavy-tailed distribution. We found that the two conditions elicited distinct responses in both behavior and pupil dilation: participants were more likely to wait for information and to later remember it in the uniform distribution. By contrast, participants showed greater surprise, evidenced in a spike in pupil dilation, when presented with the answers in the heavy-tailed distribution. Furthermore, pupil dilation was inversely related to curiosity and memory, suggesting that temporal uncertainty may interfere with the positive effects of curiosity on learning. Our findings demonstrate that the predicted timing of information delivery influences information seeking, memory, and physiological arousal, suggesting that information is best learned when it is both intrinsically interesting and presented within a temporally predictable environment. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Memory for individual items is related to nonreinforced preference change.

It is commonly assumed that memories contribute to value-based decisions. Nevertheless, most theories of value-based decision-making do not account for memory influences on choice. Recently, new interest has emerged in the interactions between these two fundamental processes, mainly using reinforcement-based paradigms. Here, we aimed to study the role memory processes play in preference change following the nonreinforced cue-approach training (CAT) paradigm. In CAT, the mere association of cued items with a speeded motor response influences choices. Previous studies with this paradigm showed that a single training session induces a long-lasting effect of enhanced preferences for high-value trained stimuli, that is maintained for several months. We hypothesized that CAT increases memory of trained items, leading to enhanced accessibility of their positive associative memories and in turn to preference changes. In two preregistered experiments, we found evidence that memory is enhanced for trained items and that better memory is correlated with enhanced preferences at the individual item level, both immediately and 1 mo following CAT. Our findings suggest that memory plays a central role in value-based decision-making following CAT, even in the absence of external reinforcements. These findings contribute to new theories relating memory and value-based decision-making and set the groundwork for the implementation of novel nonreinforced behavioral interventions that lead to long-lasting behavioral change.

Memory and decision making interact to shape the value of unchosen options.

The goal of deliberation is to separate between options so that we can commit to one and leave the other behind. However, deliberation can, paradoxically, also form an association in memory between the chosen and unchosen options. Here, we consider this possibility and examine its consequences for how outcomes affect not only the value of the options we chose, but also, by association, the value of options we did not choose. In five experiments (total n = 612), including a preregistered experiment (n = 235), we found that the value assigned to unchosen options is inversely related to their chosen counterparts. Moreover, this inverse relationship was associated with participants' memory of the pairs they chose between. Our findings suggest that deciding between options does not end the competition between them. Deliberation binds choice options together in memory such that the learned value of one can affect the inferred value of the other.

Changes in brain and behavior during food-based decision-making following treatment of anorexia nervosa.

BACKGROUND: Anorexia nervosa is a severe illness with a high mortality rate, driven in large part by severe and persistent restriction of food intake. A critical challenge is to identify brain mechanisms associated with maladaptive eating behavior and whether they change with treatment. This study tested whether food choice-related caudate activation in anorexia nervosa changes with treatment. METHODS: Healthy women (n = 29) and women hospitalized with anorexia nervosa (n = 24), ages 18 to 40 years, completed a Food Choice Task during fMRI scanning at two timepoints. Among patients, procedures occurred upon hospital admission (Time 1) and again after patients had gained to normal weight (Time 2). Healthy controls were tested twice at an interval group-matched to patients. Choice-related caudate activation was assessed at each timepoint, using parametric analyses in an a priori region of interest. RESULTS: Among patients, the proportion of high-fat foods selected did not change over time (p's > 0.47), but decreased neural activity in the caudate after treatment was associated with increased selection of high-fat foods (r23 = - 0.43, p = 0.037). Choice-related caudate activation differed among women with anorexia nervosa vs healthy control women at Time 1 (healthy control: M = 0.15 ± 0.87, anorexia nervosa: M = 0.70 ± 1.1, t51 = - 2.05, p = 0.045), but not at Time 2 (healthy control: M = 0.18 ± 1.0, anorexia nervosa: M = 0.37 ± 0.99, t51 = - 0.694, p = 0.49). CONCLUSIONS: Caudate activity was more strongly associated with decisions about food among individuals with anorexia nervosa relative to healthy comparison individuals prior to treatment, and decreases in caudate engagement among individuals with anorexia nervosa undergoing treatment were associated with increases in high-fat food choices. The findings underscore the need for treatment development that more successfully alters both eating behavior and the neural mechanisms that guide it.

Deficient Goal-Directed Control in a Population Characterized by Extreme Goal Pursuit.

Research in computational psychiatry has sought to understand the basis of compulsive behavior by relating it to basic psychological and neural mechanisms: specifically, goal-directed versus habitual control. These psychological categories have been further identified with formal computational algorithms, model-based and model-free learning, which helps to provide quantitative tools to distinguish them. Computational psychiatry may be particularly useful for examining phenomena in individuals with anorexia nervosa (AN), whose self-starvation appears both excessively goal directed and habitual. However, these laboratory-based studies have not aimed to examine complex behavior, as seen outside the laboratory, in contexts that extend beyond monetary rewards. We therefore assessed (1) whether behavior in AN was characterized by enhanced or diminished model-based behavior, (2) the domain specificity of any abnormalities by comparing learning in a food-specific (i.e., illness-relevant) context as well as in a monetary context, and (3) whether impairments were secondary to starvation by comparing learning before and after initial treatment. Across all conditions, individuals with AN, relative to healthy controls, showed an impairment in model-based, but not model-free, learning, suggesting a general and persistent contribution of habitual over goal-directed control, across domains and time points. Thus, eating behavior in individuals with AN that appears very goal-directed may be under more habitual than goal-directed control, and this is not remediated by achieving weight restoration.

Dopamine is associated with prioritization of reward-associated memories in Parkinson's disease.

Patients with Parkinson's disease have reduced reward sensitivity related to dopaminergic neuron loss, which is associated with impairments in reinforcement learning. Increasingly, however, dopamine-dependent reward signals are recognized to play an important role beyond reinforcement learning. In particular, it has been shown that reward signals mediated by dopamine help guide the prioritization of events for long-term memory consolidation. Meanwhile, studies of memory in patients with Parkinson's disease have focused on overall memory capacity rather than what is versus what isn't remembered, leaving open questions about the effect of dopamine replacement on the prioritization of memories by reward and the time-dependence of this effect. The current study sought to fill this gap by testing the effect of reward and dopamine on memory in patients with Parkinson's disease. We tested the effect of dopamine modulation and reward on two forms of long-term memory: episodic memory for neutral objects and memory for stimulus-value associations. We measured both forms of memory in a single task, adapting a standard task of reinforcement learning with incidental episodic encoding events of trial-unique objects. Objects were presented on each trial at the time of feedback, which was either rewarding or not. Memory for the trial-unique images and for the stimulus-value associations, and the influence of reward on both, was tested immediately after learning and 2 days later. We measured performance in Parkinson's disease patients tested either ON or OFF their dopaminergic medications and in healthy older control subjects. We found that dopamine was associated with a selective enhancement of memory for reward-associated images, but that it did not influence overall memory capacity. Contrary to predictions, this effect did not differ between the immediate and delayed memory tests. We also found that while dopamine had an effect on reward-modulated episodic memory, there was no effect of dopamine on memory for stimulus-value associations. Our results suggest that impaired prioritization of cognitive resource allocation may contribute to the early cognitive deficits of Parkinson's disease.

A comparison of food-based decision-making between restricting and binge-eating/purging subtypes of anorexia nervosa.

OBJECTIVE: By definition, restricting (ANR) and binge-eating/purging (ANBP) subtypes of anorexia nervosa (AN) differ in some manifestations of maladaptive eating behavior. This study aimed to determine whether the groups differ in the choices they make about what to eat, and whether there are differences in valuation related to food choice, using an experimental paradigm. METHOD: Inpatients with ANR (n = 40) and ANBP (n = 46) participated in a Food Choice Task. During the task, participants rated 76 food images for healthiness and tastiness, and choice preferences. Groups were compared in percent selection of high-fat and low-fat foods, value ratings of foods, and engagement of self-control in food choice. RESULTS: There were no differences between AN subtypes in healthiness or tastiness ratings, or in tendency to limit choice of high-fat foods. There was no difference between the groups in measures of self-control in food choice. DISCUSSION: Individuals with ANR and ANBP similarly manifest reduced choices of high-fat foods, with similar tendencies to undervalue the tastiness of high-fat foods. These results suggest that while individuals with ANR and ANBP differ across a range of clinical characteristics, the decision-making process associated with the maladaptive restriction of high-fat foods characteristic of AN is shared by both subtypes.

An objective evaluation of the beholder's response to abstract and figurative art based on construal level theory.

Does abstract art evoke a different cognitive state than figurative art? To address this question empirically, we bridged art theory and cognitive research and designed an experiment leveraging construal level theory (CLT). CLT is based on experimental data showing that psychologically distant events (i.e., occurring farther away in space or time) are represented more abstractly than closer events. We measured construal level elicited by abstract vs. representational art and asked subjects to assign abstract/representational paintings by the same artist to a situation that was temporally/spatially near or distant. Across three experiments, we found that abstract paintings were assigned to the distant situation significantly more often than representational paintings, indicating that abstract art was evocative of greater psychological distance. Our data demonstrate that different levels of artistic abstraction evoke different levels of mental abstraction and suggest that CLT provides an empirical approach to the analysis of cognitive states evoked by different levels of artistic abstraction.