The successor representation subserves hierarchical abstraction for goal-directed behavior.

Humans have the ability to craft abstract, temporally extended and hierarchically organized plans. For instance, when considering how to make spaghetti for dinner, we typically concern ourselves with useful "subgoals" in the task, such as cutting onions, boiling pasta, and cooking a sauce, rather than particulars such as how many cuts to make to the onion, or exactly which muscles to contract. A core question is how such decomposition of a more abstract task into logical subtasks happens in the first place. Previous research has shown that humans are sensitive to a form of higher-order statistical learning named "community structure". Community structure is a common feature of abstract tasks characterized by a logical ordering of subtasks. This structure can be captured by a model where humans learn predictions of upcoming events multiple steps into the future, discounting predictions of events further away in time. One such model is the "successor representation", which has been argued to be useful for hierarchical abstraction. As of yet, no study has convincingly shown that this hierarchical abstraction can be put to use for goal-directed behavior. Here, we investigate whether participants utilize learned community structure to craft hierarchically informed action plans for goal-directed behavior. Participants were asked to search for paintings in a virtual museum, where the paintings were grouped together in "wings" representing community structure in the museum. We find that participants' choices accord with the hierarchical structure of the museum and that their response times are best predicted by a successor representation. The degree to which the response times reflect the community structure of the museum correlates with several measures of performance, including the ability to craft temporally abstract action plans. These results suggest that successor representation learning subserves hierarchical abstractions relevant for goal-directed behavior.

Task-level value affects trial-level reward processing.

Despite disagreement about how anterior cingulate cortex (ACC) supports decision making, a recent hypothesis suggests that activity in this region is best understood in the context of a task or series of tasks. One important task-level variable is average reward because it is both a known driver of effortful behaviour and an important determiner of the tasks in which we choose to engage. Here we asked how average task value affects reward-related ACC activity. To answer this question, we measured a reward-related signal said to be generated in ACC called the reward positivity (RewP) while participants gambled in three tasks of differing average value. The RewP was reduced in the high-value task, an effect that was not explainable by either reward magnitude or outcome expectancy. This result suggests that ACC does not evaluate outcomes and cues in isolation, but in the context of the value of the current task.

Human midcingulate cortex encodes distributed representations of task progress.

The function of midcingulate cortex (MCC) remains elusive despite decades of investigation and debate. Complicating matters, individual MCC neurons respond to highly diverse task-related events, and MCC activation is reported in most human neuroimaging studies employing a wide variety of task manipulations. Here we investigate this issue by applying a model-based cognitive neuroscience approach involving neural network simulations, functional magnetic resonance imaging, and representational similarity analysis. We demonstrate that human MCC encodes distributed, dynamically evolving representations of extended, goal-directed action sequences. These representations are uniquely sensitive to the stage and identity of each sequence, indicating that MCC sustains contextual information necessary for discriminating between task states. These results suggest that standard univariate approaches for analyzing MCC function overlook the major portion of task-related information encoded by this brain area and point to promising new avenues for investigation.

Smoking Decisions: Altered Reinforcement Learning Signals Induced by Nicotine State.

INTRODUCTION: Alterations in dopamine signaling play a key role in reinforcement learning and nicotine addiction, but the relationship between these two processes has not been well characterized. We investigated this relationship in young adult smokers using a combination of behavioral and computational measures of reinforcement learning. METHODS: We asked moderately dependent smokers to engage in a reinforcement learning task three times: smoking as usual, smoking abstinence, and cigarette consumption. Participants' trial-to-trial training choices were modeled using a reinforcement learning model that calculates separate learning rates associated with positive and negative prediction errors. RESULTS: We found that learning from positive prediction error signals is reduced during smoking abstinence and enhanced following cigarette consumption. By contrast, learning from negative prediction error signals was enhanced during smoking abstinence and reduced following cigarette consumption. Finally, when tested with novel pairs of stimuli, participants were relatively better at selecting the positive feedback predicting stimuli than avoiding the negative feedback predicting stimuli during the smoking as usual session, a pattern that reversed following cigarette consumption. CONCLUSIONS: These findings provide a specific computational account of altered reinforcement learning induced by smoking state (abstinence and consumption) and may represent a unique target for treatment of nicotine addiction. IMPLICATIONS: This study illustrates the potential of computational psychiatry for understanding reinforcement learning deficits associated with substance use disorders in general and nicotine addiction in particular. We found that learning from positive prediction error signals is reduced during smoking abstinence and enhanced following cigarette consumption. By contrast, learning from negative prediction error signals was enhanced during smoking abstinence and reduced following cigarette consumption. By highlighting important computational differences between three states of smoking, these findings hold out promise for integrating experimental, computational, and theoretical analyses of decision-making function together with research on addiction-related disorders.

Subgoal- and Goal-related Reward Prediction Errors in Medial Prefrontal Cortex.

A longstanding view of the organization of human and animal behavior holds that behavior is hierarchically organized-in other words, directed toward achieving superordinate goals through the achievement of subordinate goals or subgoals. However, most research in neuroscience has focused on tasks without hierarchical structure. In past work, we have shown that negative reward prediction error (RPE) signals in medial prefrontal cortex (mPFC) can be linked not only to superordinate goals but also to subgoals. This suggests that mPFC tracks impediments in the progression toward subgoals. Using fMRI of human participants engaged in a hierarchical navigation task, here we found that mPFC also processes positive prediction errors at the level of subgoals, indicating that this brain region is sensitive to advances in subgoal completion. However, when subgoal RPEs were elicited alongside with goal-related RPEs, mPFC responses reflected only the goal-related RPEs. These findings suggest that information from different levels of hierarchy is processed selectively, depending on the task context.

What you give is what you get: Payment of one randomly selected trial induces risk-aversion and decreases brain responses to monetary feedback.

In economic studies, it is standard practice to pay out the reward of only one randomly selected trial (pay-one) instead of the total reward accumulated across trials (pay-all), assuming that both methods are equivalent. We tested this assumption by recording electrophysiological activity to reward feedback from participants engaged in a decision-making task under both a pay-one and a pay-all condition. We show that participants are approximately 12% more risk averse in the pay-one condition than in the pay-all condition. Furthermore, we observed that the electrophysiological response to monetary rewards, the reward positivity, is significantly reduced in the pay-one condition relative to the pay-all condition. The difference of brain responses is associated with the difference in risky behavior across conditions. We concluded that the two payment methods lead to significantly different results and are therefore not equivalent.

Brain mechanisms underlying apathy.

The past few decades have seen growing interest in the neuropsychiatric syndrome of apathy, conceptualised as a loss of motivation manifesting as a reduction of goal-directed behaviour. Apathy occurs frequently, and with substantial impact on quality of life, in a broad range of neurological and psychiatric conditions. Apathy is also consistently associated with neuroimaging changes in specific medial frontal cortex and subcortical structures, suggesting that disruption of a common systems-level mechanism may underlie its development, irrespective of the condition that causes it. In parallel with this growing recognition of the clinical importance of apathy, significant advances have been made in understanding normal motivated behaviour in humans and animals. These developments have occurred at several different conceptual levels, from work linking neural structures and neuromodulatory systems to specific aspects of motivated behaviour, to higher order computational models that aim to unite these findings within frameworks for normal goal-directed behaviour. In this review we develop a conceptual framework for understanding pathological apathy based on this current understanding of normal motivated behaviour. We first introduce prominent theories of motivated behaviour-which often involves sequences of actions towards a goal that needs to be maintained across time. Next, we outline the behavioural effects of disrupting these processes in animal models, highlighting the specific effects of these manipulations on different components of motivated behaviour. Finally, we relate these findings to clinical apathy, demonstrating the homologies between this basic neuroscience work and emerging behavioural and physiological evidence from patient studies of this syndrome.

Electrophysiological responses of medial prefrontal cortex to feedback at different levels of hierarchy.

Recent advances in computational reinforcement learning suggest that humans and animals can learn from different types of reinforcers in a hierarchically organised fashion. According to this theoretical framework, while humans learn to coordinate subroutines based on external reinforcers such as food rewards, simple actions within those subroutines are reinforced by an internal reinforcer called a pseudo-reward. Although the neural mechanisms underlying these processes are unknown, recent empirical evidence suggests that the medial prefrontal cortex (MPFC) is involved. To elucidate this issue, we measured a component of the human event-related brain potential, called the reward positivity, that is said to reflect a reward prediction error signal generated in the MPFC. Using a task paradigm involving reinforcers at two levels of hierarchy, we show that reward positivity amplitude is sensitive to the valence of low-level pseudo-rewards but, contrary to our expectation, is not modulated by high-level rewards. Further, reward positivity amplitude to low-level feedback is modulated by the goals of the higher level. These results, which were further replicated in a control experiment, suggest that the MPFC is involved in the processing of rewards at multiple levels of hierarchy.

Electrophysiological measures reveal the role of anterior cingulate cortex in learning from unreliable feedback.

Although a growing number of studies have investigated the neural mechanisms of reinforcement learning, it remains unclear how the brain responds to feedback that is unreliable. A recent theory proposes that the reward positivity (RewP) component of the event-related brain potential (ERP) and frontal midline theta (FMT) power reflect separate feedback-related processing functions of anterior cingulate cortex (ACC). In the present study, the electroencephalogram (EEG) was recorded from participants as they engaged in a time estimation task in which feedback reliability was manipulated across conditions. After each response, they received a cue that indicated that the following feedback stimulus was 100%, 75%, or 50% reliable. The results showed that participants' time estimates adjusted linearly according to the feedback reliability. Moreover, presentation of the cue indicating 100% reliability elicited a larger RewP-like ERP component than the other cues did, and feedback presentation elicited a RewP of approximately equal amplitude for all of the three reliability conditions. By contrast, FMT power elicited by negative feedback decreased linearly from the 100% condition to 75% and 50% condition, and only FMT power predicted behavioral adjustments on the following trials. In addition, an analysis of Beta power and cross-frequency coupling (CFC) of Beta power with FMT phase suggested that Beta-FMT communication modulated motor areas for the purpose of adjusting behavior. We interpreted these findings in terms of the hierarchical reinforcement learning account of ACC, in which the RewP and FMT are proposed to reflect reward processing and control functions of ACC, respectively.

Reward-based contextual learning supported by anterior cingulate cortex.

The anterior cingulate cortex (ACC) is commonly associated with cognitive control and decision making, but its specific function is highly debated. To explore a recent theory that the ACC learns the reward values of task contexts (Holroyd & McClure in Psychological Review, 122, 54-83, 2015; Holroyd & Yeung in Trends in Cognitive Sciences, 16, 122-128, 2012), we recorded the event-related brain potentials (ERPs) from participants as they played a novel gambling task. The participants were first required to select from among three games in one "virtual casino," and subsequently they were required to select from among three different games in a different virtual casino; unbeknownst to them, the payoffs for the games were higher in one casino than in the other. Analysis of the reward positivity, an ERP component believed to reflect reward-related signals carried to the ACC by the midbrain dopamine system, revealed that the ACC is sensitive to differences in the reward values associated with both the casinos and the games inside the casinos, indicating that participants learned the values of the contexts in which rewards were delivered. These results highlight the importance of the ACC in learning the reward values of task contexts in order to guide action selection.

Reward Sensitivity of ACC as an Intermediate Phenotype between DRD4-521T and Substance Misuse.

The development and expression of the midbrain dopamine system is determined in part by genetic factors that vary across individuals such that dopamine-related genes are partly responsible for addiction vulnerability. However, a complete account of how dopamine-related genes predispose individuals to drug addiction remains to be developed. Adopting an intermediate phenotype approach, we investigated whether reward-related electrophysiological activity of ACC-a cortical region said to utilize dopamine reward signals to learn the value of extended, context-specific sequences of goal-directed behaviors-mediates the influence of multiple dopamine-related functional polymorphisms over substance use. We used structural equation modeling to examine whether two related electrophysiological phenomena associated with the control and reinforcement learning functions of ACC-theta power and the reward positivity-mediated the relationship between the degree of substance misuse and genetic polymorphisms that regulate dopamine processing in frontal cortex. Substance use data were collected from 812 undergraduate students. One hundred ninety-six returned on a subsequent day to participate in an electrophysiological experiment and to provide saliva samples for DNA analysis. We found that these electrophysiological signals mediated a relationship between the DRD4-521T dopamine receptor genotype and substance misuse. Our results provide a theoretical framework that bridges the gap between genes and behavior in drug addiction and illustrate how future interventions might be individually tailored for specific genetic and neurocognitive profiles.

Task-specific effects of reward on task switching.

Although cognitive control and reinforcement learning have been researched extensively over the last few decades, only recently have studies investigated their interrelationship. An important unanswered question concerns how the control system decides what task to execute and how vigorously to carry out the task once selected. Based on a recent theory of control formulated according to principles of hierarchical reinforcement learning, we asked whether rewards can affect top-down control over task performance at the level of task representation. Participants were rewarded for correctly performing only one of two tasks in a standard task-switching experiment. Reaction times and error rates were lower for the reinforced task compared to the non-reinforced task. Moreover, the switch cost in error rates for the non-reinforced task was significantly larger compared to the reinforced task, especially for trials in which the imperative stimulus afforded different responses for the two tasks, resulting in a "non-paradoxical" asymmetric switch cost. These findings suggest that reinforcement at the task level resulted in greater application of top-down control rather than in stronger stimulus-response pathways for the rewarded task.

Impaired reward processing by anterior cingulate cortex in children with attention deficit hyperactivity disorder.

Decades of research have examined the neurocognitive mechanisms of cognitive control, but the motivational factors underlying task selection and performance remain to be elucidated. We recently proposed that anterior cingulate cortex (ACC) utilizes reward prediction error signals carried by the midbrain dopamine system to learn the value of tasks according to the principles of hierarchical reinforcement learning. According to this position, disruption of the ACC-dopamine interface can disrupt the selection and execution of extended, task-related behaviors. To investigate this issue, we recorded the event-related brain potential (ERP) from children with attention deficit hyperactivity disorder (ADHD), which is strongly associated with ACC-dopamine dysfunction, and from typically developing children while they navigated a simple "virtual T-maze" to find rewards. Depending on the condition, the feedback stimuli on each trial indicated that the children earned or failed to earn either money or points. We found that the reward positivity, an ERP component proposed to index the impact of dopamine-related reward signals on ACC, was significantly larger with money feedback than with points feedback for the children with ADHD, but not for the typically developing children. These results suggest that disruption of the ACC-dopamine interface may underlie the impairments in motivational control observed in childhood ADHD.

The controllosphere: The neural origin of cognitive effort.

Why do some mental activities feel harder than others? The answer to this question is surprisingly controversial. Current theories propose that cognitive effort affords a computational benefit, such as instigating a switch from an activity with low reward value to a different activity with higher reward value. By contrast, in this article, I relate cognitive effort to the fact that brain neuroanatomy and neurophysiology render some neural states more energy-efficient than others. I introduce the concept of the "controllosphere," an energy-inefficient region of neural state space associated with high control, which surrounds the better known "intrinsic manifold," an energy-efficient subspace associated with low control. Integration of control-theoretic principles with classic neurocomputational models of cognitive control suggests that dorsolateral prefrontal cortex (DLPFC) implements a controller that can drive the system state into the controllosphere, anterior cingulate cortex (ACC) implements an observer that monitors changes of state of the controlled system, and cognitive effort reflects a mismatch between DLPFC and ACC energies for control and observation. On this account, cognitive effort scales with the energetic demands of the DLPFC control signal, especially when the consequences of the control are unobservable by ACC. Further, I propose that neural transitions through the controllosphere lead to a buildup of neural waste. Cognitive effort therefore prevents against neural damage by discouraging extended periods of high control. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Hierarchical control over foraging behavior by anterior cingulate cortex.

Foraging is a natural behavior that involves making sequential decisions to maximize rewards while minimizing the costs incurred when doing so. The prevalence of foraging across species suggests that a common brain computation underlies its implementation. Although anterior cingulate cortex is believed to contribute to foraging behavior, its specific role has been contentious, with predominant theories arguing either that it encodes environmental value or choice difficulty. Additionally, recent attempts to characterize foraging have taken place within the reinforcement learning framework, with increasingly complex models scaling with task complexity. Here we review reinforcement learning foraging models, highlighting the hierarchical structure of many foraging problems. We extend this literature by proposing that ACC guides foraging according to principles of model-based hierarchical reinforcement learning. This idea holds that ACC function is organized hierarchically along a rostral-caudal gradient, with rostral structures monitoring the status and completion of high-level task goals (like finding food), and midcingulate structures overseeing the execution of task options (subgoals, like harvesting fruit) and lower-level actions (such as grabbing an apple).

Dynamic changes in task preparation in a multi-task environment: The task transformation paradigm.

A key element of human flexible behavior concerns the ability to continuously predict and prepare for sudden changes in tasks or actions. Here, we tested whether people can dynamically modulate task preparation processes and decision-making strategies when the identity of a to-be-performed task becomes uncertain. To this end, we developed a new paradigm where participants need to prepare for one of nine tasks on each trial. Crucially, in some blocks, the task being prepared could suddenly shift to a different task after a longer cue-target interval, by changing either the stimulus category or categorization rule that defined the initial task. We found that participants were able to dynamically modulate task preparation in the face of this task uncertainty. A second experiment shows that these changes in behavior were not simply a function of decreasing task expectancy, but rather of increasing switch expectancy. Finally, in the third and fourth experiment, we demonstrate that these dynamic modulations can be applied in a compositional manner, depending on whether either only the stimulus category or categorization rule would be expected to change.

High temporal discounters overvalue immediate rewards rather than undervalue future rewards: an event-related brain potential study.

Impulsivity is characterized in part by heightened sensitivity to immediate relative to future rewards. Although previous research has suggested that "high discounters" in intertemporal choice tasks tend to prefer immediate over future rewards because they devalue the latter, it remains possible that they instead overvalue immediate rewards. To investigate this question, we recorded the reward positivity, a component of the event-related brain potential (ERP) associated with reward processing, with participants engaged in a task in which they received both immediate and future rewards and nonrewards. The participants also completed a temporal discounting task without ERP recording. We found that immediate but not future rewards elicited the reward positivity. High discounters also produced larger reward positivities to immediate rewards than did low discounters, indicating that high discounters relatively overvalued immediate rewards. These findings suggest that high discounters may be more motivated than low discounters to work for monetary rewards, irrespective of the time of arrival of the incentives.

Constraints on decision making: implications from genetics, personality, and addiction.

An influential neurocomputational theory of the biological mechanisms of decision making, the "basal ganglia go/no-go model," holds that individual variability in decision making is determined by differences in the makeup of a striatal system for approach and avoidance learning. The model has been tested empirically with the probabilistic selection task (PST), which determines whether individuals learn better from positive or negative feedback. In accordance with the model, in the present study we examined whether an individual's ability to learn from positive and negative reinforcement can be predicted by genetic factors related to the midbrain dopamine system. We also asked whether psychiatric and personality factors related to substance dependence and dopamine affect PST performance. Although we found characteristics that predicted individual differences in approach versus avoidance learning, these observations were qualified by additional findings that appear inconsistent with the predictions of the go/no-go model. These results highlight a need for future research to validate the PST as a measure of basal ganglia reward learning.

Theories of anterior cingulate cortex function: opportunity cost.

The target article highlights the role of the anterior cingulate cortex (ACC) in conflict monitoring, but ACC function may be better understood in terms of the hierarchical organization of behavior. This proposal suggests that the ACC selects extended goal-directed actions according to their learned costs and benefits and executes those behaviors subject to depleting resources.

Electrophysiological measures of conflict and reward processing are associated with decisions to engage in physical effort.

Anterior cingulate cortex (ACC), a key brain region involved in cognitive control and decision making, is suggested to mediate effort- and value-based decision making, but the specific role of ACC in this process remains debated. Here we used frontal midline theta (FMT) and the reward positivity (RewP) to examine ACC function in a value-based decision making task requiring physical effort. We investigated whether (1) FMT power is sensitive to the difficulty of the decision or to selecting effortful actions, and (2) RewP is sensitive to the subjective value of reward outcomes as a function of effort investment. On each trial, participants chose to execute a low-effort or a high-effort behavior (that required squeezing a hand-dynamometer) to obtain smaller or larger rewards, respectively, while their brainwaves were recorded. We replicated prior findings that tonic FMT increased over the course of the hour-long task, which suggests increased application of control in the face of growing fatigue. RewP amplitude also increased following execution of high-effort compared to low-effort behavior, consistent with increased valuation of reward outcomes by ACC. Although neither phasic nor tonic FMT were associated with decision difficulty or effort selection per se, an exploratory analysis revealed that the interaction of phasic FMT and expected value of choice predicted effort choice. This interaction suggests that phasic FMT increases specifically under situations of decision difficulty when participants ultimately select a high-effort choice. These results point to a unique role for ACC in motivating and persisting at effortful behavior when decision conflict is high.