Active reinforcement learning versus action bias and hysteresis: control with a mixture of experts and nonexperts.

Active reinforcement learning enables dynamic prediction and control, where one should not only maximize rewards but also minimize costs such as of inference, decisions, actions, and time. For an embodied agent such as a human, decisions are also shaped by physical aspects of actions. Beyond the effects of reward outcomes on learning processes, to what extent can modeling of behavior in a reinforcement-learning task be complicated by other sources of variance in sequential action choices? What of the effects of action bias (for actions per se) and action hysteresis determined by the history of actions chosen previously? The present study addressed these questions with incremental assembly of models for the sequential choice data from a task with hierarchical structure for additional complexity in learning. With systematic comparison and falsification of computational models, human choices were tested for signatures of parallel modules representing not only an enhanced form of generalized reinforcement learning but also action bias and hysteresis. We found evidence for substantial differences in bias and hysteresis across participants-even comparable in magnitude to the individual differences in learning. Individuals who did not learn well revealed the greatest biases, but those who did learn accurately were also significantly biased. The direction of hysteresis varied among individuals as repetition or, more commonly, alternation biases persisting from multiple previous actions. Considering that these actions were button presses with trivial motor demands, the idiosyncratic forces biasing sequences of action choices were robust enough to suggest ubiquity across individuals and across tasks requiring various actions. In light of how bias and hysteresis function as a heuristic for efficient control that adapts to uncertainty or low motivation by minimizing the cost of effort, these phenomena broaden the consilient theory of a mixture of experts to encompass a mixture of expert and nonexpert controllers of behavior.

Temporally organized representations of reward and risk in the human brain.

The value and uncertainty associated with choice alternatives constitute critical features relevant for decisions. However, the manner in which reward and risk representations are temporally organized in the brain remains elusive. Here we leverage the spatiotemporal precision of intracranial electroencephalography, along with a simple card game designed to elicit the unfolding computation of a set of reward and risk variables, to uncover this temporal organization. Reward outcome representations across wide-spread regions follow a sequential order along the anteroposterior axis of the brain. In contrast, expected value can be decoded from multiple regions at the same time, and error signals in both reward and risk domains reflect a mixture of sequential and parallel encoding. We further highlight the role of the anterior insula in generalizing between reward prediction error and risk prediction error codes. Together our results emphasize the importance of neural dynamics for understanding value-based decisions under uncertainty.

Encoding of Predictive Associations in Human Prefrontal and Medial Temporal Neurons During Pavlovian Appetitive Conditioning.

Pavlovian conditioning is thought to involve the formation of learned associations between stimuli and values, and between stimuli and specific features of outcomes. Here, we leveraged human single neuron recordings in ventromedial prefrontal, dorsomedial frontal, hippocampus, and amygdala while patients of both sexes performed an appetitive Pavlovian conditioning task probing both stimulus-value and stimulus-stimulus associations. Ventromedial prefrontal cortex encoded predictive value along with the amygdala, and also encoded predictions about the identity of stimuli that would subsequently be presented, suggesting a role for neurons in this region in encoding predictive information beyond value. Unsigned error signals were found in dorsomedial frontal areas and hippocampus, potentially supporting learning of non-value related outcome features. Our findings implicate distinct human prefrontal and medial temporal neuronal populations in mediating predictive associations which could partially support model-based mechanisms during Pavlovian conditioning.

Neural substrates of parallel devaluation-sensitive and devaluation-insensitive Pavlovian learning in humans.

We aim to differentiate the brain regions involved in the learning and encoding of Pavlovian associations sensitive to changes in outcome value from those that are not sensitive to such changes by combining a learning task with outcome devaluation, eye-tracking, and functional magnetic resonance imaging in humans. Contrary to theoretical expectation, voxels correlating with reward prediction errors in the ventral striatum and subgenual cingulate appear to be sensitive to devaluation. Moreover, regions encoding state prediction errors appear to be devaluation insensitive. We can also distinguish regions encoding predictions about outcome taste identity from predictions about expected spatial location. Regions encoding predictions about taste identity seem devaluation sensitive while those encoding predictions about an outcome's spatial location seem devaluation insensitive. These findings suggest the existence of multiple and distinct associative mechanisms in the brain and help identify putative neural correlates for the parallel expression of both devaluation sensitive and insensitive conditioned behaviors.

Novelty and uncertainty differentially drive exploration across development.

Across the lifespan, individuals frequently choose between exploiting known rewarding options or exploring unknown alternatives. A large body of work has suggested that children may explore more than adults. However, because novelty and reward uncertainty are often correlated, it is unclear how they differentially influence decision-making across development. Here, children, adolescents, and adults (ages 8-27 years, N = 122) completed an adapted version of a recently developed value-guided decision-making task that decouples novelty and uncertainty. In line with prior studies, we found that exploration decreased with increasing age. Critically, participants of all ages demonstrated a similar bias to select choice options with greater novelty, whereas aversion to reward uncertainty increased into adulthood. Computational modeling of participant choices revealed that whereas adolescents and adults demonstrated attenuated uncertainty aversion for more novel choice options, children's choices were not influenced by reward uncertainty.

Temporally organized representations of reward and risk in the human brain.

The value and uncertainty associated with choice alternatives constitute critical features along which decisions are made. While the neural substrates supporting reward and risk processing have been investigated, the temporal organization by which these computations are encoded remains elusive. Here we leverage the high spatiotemporal precision of intracranial electroencephalography (iEEG) to uncover how representations of decision-related computations unfold in time. We present evidence of locally distributed representations of reward and risk variables that are temporally organized across multiple regions of interest. Reward outcome representations across wide-spread regions follow a temporally cascading order along the anteroposterior axis of the brain. In contrast, expected value can be decoded from multiple regions at the same time, and error signals in both reward and risk domains reflect a mixture of sequential and parallel encoding. We highlight the role of the anterior insula in generalizing between reward prediction error (RePE) and risk prediction error (RiPE), within which the encoding of RePE in the distributed iEEG signal predicts RiPE. Together our results emphasize the utility of uncovering temporal dynamics in the human brain for understanding how computational processes critical for value-based decisions under uncertainty unfold.

Can the Brain Strategically Go on Automatic Pilot? The Effect of If-Then Planning on Behavioral Flexibility.

People often have good intentions but fail to adhere to them. Implementation intentions, a form of strategic planning, can help people to close this intention-behavior gap. Their effectiveness has been proposed to depend on the mental formation of a stimulus-response association between a trigger and target behavior, thereby creating an "instant habit." If implementation intentions do indeed lead to reliance on habitual control, then this may come at the cost of reduced behavioral flexibility. Furthermore, we would expect a shift from recruitment of corticostriatal brain regions implicated in goal-directed control toward habit regions. To test these ideas, we performed a fMRI study in which participants received instrumental training supported by either implementation or goal intentions, followed by an outcome revaluation to test reliance on habitual versus goal-directed control. We found that implementation intentions led to increased efficiency early in training, as reflected by higher accuracy, faster RTs, and decreased anterior caudate engagement. However, implementation intentions did not reduce behavioral flexibility when goals changed during the test phase, nor did it affect the underlying corticostriatal pathways. In addition, this study showed that "slips of action" toward devalued outcomes are associated with reduced activity in brain regions implicated in goal-directed control (ventromedial prefrontal cortex and lateral orbitofrontal cortex) and increased activity of the fronto-parietal salience network (including the insula, dorsal anterior cingulate cortex, and SMA). In conclusion, our behavioral and neuroimaging findings suggest that strategic if-then planning does not lead to a shift from goal-directed toward habitual control.

Neurons in human pre-supplementary motor area encode key computations for value-based choice.

Adaptive behaviour in real-world environments requires that choices integrate several variables, including the novelty of the options under consideration, their expected value and uncertainty in value estimation. Here, to probe how integration over decision variables occurs during decision-making, we recorded neurons from the human pre-supplementary motor area (preSMA), ventromedial prefrontal cortex and dorsal anterior cingulate. Unlike the other areas, preSMA neurons not only represented separate pre-decision variables for each choice option but also encoded an integrated utility signal for each choice option and, subsequently, the decision itself. Post-decision encoding of variables for the chosen option was more widely distributed and especially prominent in the ventromedial prefrontal cortex. Our findings position the human preSMA as central to the implementation of value-based decisions.

Neural substrates of parallel devaluation-sensitive and devaluation-insensitive Pavlovian learning in humans.

Pavlovian learning depends on multiple and parallel associations leading to distinct classes of conditioned responses that vary in their flexibility following changes in the value of an associated outcome. Here, we aimed to differentiate brain areas involved in learning and encoding associations that are sensitive to changes in the value of an outcome from those that are not sensitive to such changes. To address this question, we combined a Pavlovian learning task with outcome devaluation, eye-tracking and functional magnetic resonance imaging. We used computational modeling to identify brain regions involved in learning stimulus-reward associations and stimulus-stimulus associations, by testing for brain areas correlating with reward-prediction errors and state-prediction errors, respectively. We found that, contrary to theoretical predictions about reward prediction errors being exclusively model-free, voxels correlating with reward prediction errors in the ventral striatum and subgenual anterior cingulate cortex were sensitive to devaluation. On the other hand, brain areas correlating with state prediction errors were found to be devaluation insensitive. In a supplementary analysis, we distinguished brain regions encoding predictions about outcome taste identity from those involved in encoding predictions about its expected spatial location. A subset of regions involved in taste identity predictions were devaluation sensitive while those involved in encoding predictions about spatial location were devaluation insensitive. These findings provide insights into the role of multiple associative mechanisms in the brain in mediating Pavlovian conditioned behavior - illustrating how distinct neural pathways can in parallel produce both devaluation sensitive and devaluation insensitive behaviors.

Neural mechanisms underlying the hierarchical construction of perceived aesthetic value.

Little is known about how the brain computes the perceived aesthetic value of complex stimuli such as visual art. Here, we used computational methods in combination with functional neuroimaging to provide evidence that the aesthetic value of a visual stimulus is computed in a hierarchical manner via a weighted integration over both low and high level stimulus features contained in early and late visual cortex, extending into parietal and lateral prefrontal cortices. Feature representations in parietal and lateral prefrontal cortex may in turn be utilized to produce an overall aesthetic value in the medial prefrontal cortex. Such brain-wide computations are not only consistent with a feature-based mechanism for value construction, but also resemble computations performed by a deep convolutional neural network. Our findings thus shed light on the existence of a general neurocomputational mechanism for rapidly and flexibly producing value judgements across an array of complex novel stimuli and situations.

Novelty and uncertainty regulate the balance between exploration and exploitation through distinct mechanisms in the human brain.

Both novelty and uncertainty are potent features guiding exploration; however, they are often experimentally conflated, and an understanding of how they interact to regulate the balance between exploration and exploitation has proved elusive. Using a task designed to decouple the influence of novelty and uncertainty, we identify separable mechanisms through which exploration is directed. We show that uncertainty-directed exploration is sensitive to the prospective benefit offered by new information, whereas novelty-directed exploration is maintained regardless of its potential advantage. Using a computational framework in conjunction with fMRI, we show that uncertainty-directed choice is rooted in an adaptive bias indexing the prospective utility of exploration. In contrast, novelty persistently promotes exploration by optimistically inflating reward expectations while simultaneously dampening uncertainty signals. Our results identify separable neural substrates charged with balancing the explore/exploit trade-off to foster a manageable decomposition of an otherwise intractable problem.

Reinforcement learning with associative or discriminative generalization across states and actions: fMRI at 3 T and 7 T.

The model-free algorithms of "reinforcement learning" (RL) have gained clout across disciplines, but so too have model-based alternatives. The present study emphasizes other dimensions of this model space in consideration of associative or discriminative generalization across states and actions. This "generalized reinforcement learning" (GRL) model, a frugal extension of RL, parsimoniously retains the single reward-prediction error (RPE), but the scope of learning goes beyond the experienced state and action. Instead, the generalized RPE is efficiently relayed for bidirectional counterfactual updating of value estimates for other representations. Aided by structural information but as an implicit rather than explicit cognitive map, GRL provided the most precise account of human behavior and individual differences in a reversal-learning task with hierarchical structure that encouraged inverse generalization across both states and actions. Reflecting inference that could be true, false (i.e., overgeneralization), or absent (i.e., undergeneralization), state generalization distinguished those who learned well more so than action generalization. With high-resolution high-field fMRI targeting the dopaminergic midbrain, the GRL model's RPE signals (alongside value and decision signals) were localized within not only the striatum but also the substantia nigra and the ventral tegmental area, including specific effects of generalization that also extend to the hippocampus. Factoring in generalization as a multidimensional process in value-based learning, these findings shed light on complexities that, while challenging classic RL, can still be resolved within the bounds of its core computations.

Neurocircuit dynamics of arbitration between decision-making strategies across obsessive-compulsive and related disorders.

Obsessions and compulsions are central components of obsessive-compulsive disorder (OCD) and obsessive-compulsive related disorders such as body dysmorphic disorder (BDD). Compulsive behaviours may result from an imbalance of habitual and goal-directed decision-making strategies. The relationship between these symptoms and the neural circuitry underlying habitual and goal-directed decision-making, and the arbitration between these strategies, remains unknown. This study examined resting state effective connectivity between nodes of these systems in two cohorts with obsessions and compulsions, each compared with their own corresponding healthy controls: OCD (nOCD = 43; nhealthy = 24) and BDD (nBDD = 21; nhealthy = 16). In individuals with OCD, the left ventrolateral prefrontal cortex, a node of the arbitration system, exhibited more inhibitory causal influence over the left posterolateral putamen, a node of the habitual system, compared with controls. Inhibitory causal influence in this connection showed a trend for a similar pattern in individuals with BDD compared with controls. Those with stronger negative connectivity had lower obsession and compulsion severity in both those with OCD and those with BDD. These relationships were not evident within the habitual or goal-directed circuits, nor were they associated with depressive or anxious symptomatology. These results suggest that abnormalities in the arbitration system may represent a shared neural phenotype across these two related disorders that is specific to obsessive-compulsive symptoms. In addition to nosological implications, these results identify potential targets for novel, circuit-specific treatments.

Neural, physiological, and psychological markers of appetitive conditioning in anorexia nervosa: a study protocol.

BACKGROUND: Anorexia nervosa (AN) is a chronic and disabling psychiatric condition characterized by low hedonic drive towards food, and is thought to be inclusive of altered dimensions of reward processing. Whether there exists a fundamental aberrancy in the capacity to acquire and maintain de novo hedonic associations-a critical component of hedonic responding-has never been studied in AN. METHODS: This multi-modal study will employ a 2-day Pavlovian appetitive conditioning paradigm to interrogate the (1) acquisition, (2) extinction, (3) spontaneous recovery and (4) reinstatement of appetitive learning in adolescents and young adults with AN. Participants will be 30 currently ill, underweight individuals with AN; 30 weight-restored individuals with AN; and 30 age-matched healthy controls, all aged 12-22 years. All subjects will undergo clinical assessment, followed by the 2-day appetitive conditioning task during which fMRI, pupillometry, heart rate deceleration, and subjective ratings will be acquired. DISCUSSION: This study will be the first to interrogate appetitive conditioning in AN-a disorder characterized by altered hedonic responding to food. Results will help establish objective biomarkers of appetitive conditioning in AN and lay the groundwork for developing novel lines of treatment for AN and other psychiatric disorders involving diminished ability to experience pleasure and reward. TRIAL REGISTRATION: Pending. INTENDED REGISTRY: Clinicaltrials.gov.

Mesolimbic Neurobehavioral Mechanisms of Reward Motivation in Anorexia Nervosa: A Multimodal Imaging Study.

Diminished motivation to pursue and obtain primary and secondary rewards has been demonstrated in anorexia nervosa (AN). However, the neurobehavioral mechanisms underlying the behavioral activation component of aberrant reward motivation remains incompletely understood. This work aims to explore this underexplored facet of reward motivation in AN. We recruited female adolescents with AN, restricting type (n = 32) and a healthy control group (n = 28). All participants underwent functional magnetic resonance imaging (fMRI) while performing a monetary reward task. Diffusion MRI data was also collected to examine the reward motivation circuit's structural connectivity. Behavioral results demonstrated slower speed of reward-seeking behavior in those with AN compared with controls. Accompanying this was lower functional connectivity and reduced white matter structural integrity of the connection between the ventral tegmental area/substantia nigra pars compacta and the nucleus accumbens within the mesolimbic circuit. Further, there was evidence of neurobehavioral decoupling in AN between reward-seeking behavior and mesolimbic regional activation and functional connectivity. Aberrant activity of the bed nucleus of the stria terminalis (BNST) and its connectivity with the mesolimbic system was also evident in AN during the reward motivation period. Our findings suggest functional and structural dysconnectivity within a mesolimbic reward circuit, neurofunctional decoupling from reward-seeking behavior, and abnormal BNST function and circuit interaction with the mesolimbic system. These results show behavioral indicators of aberrant reward motivation in AN, particularly in its activational component. This is mediated neuronally by mesolimbic reward circuit functional and structural dysconnectivity as well as neurobehavioral decoupling. Based on these findings, we suggest a novel circuit-based mechanism of impaired reward processing in AN, with the potential for translation to developing more targeted and effective treatments in this difficult-to-treat psychiatric condition.

Reinforcement-learning in fronto-striatal circuits.

We review the current state of knowledge on the computational and neural mechanisms of reinforcement-learning with a particular focus on fronto-striatal circuits. We divide the literature in this area into five broad research themes: the target of the learning-whether it be learning about the value of stimuli or about the value of actions; the nature and complexity of the algorithm used to drive the learning and inference process; how learned values get converted into choices and associated actions; the nature of state representations, and of other cognitive machinery that support the implementation of various reinforcement-learning operations. An emerging fifth area focuses on how the brain allocates or arbitrates control over different reinforcement-learning sub-systems or "experts". We will outline what is known about the role of the prefrontal cortex and striatum in implementing each of these functions. We then conclude by arguing that it will be necessary to build bridges from algorithmic level descriptions of computational reinforcement-learning to implementational level models to better understand how reinforcement-learning emerges from multiple distributed neural networks in the brain.

Determining the effects of training duration on the behavioral expression of habitual control in humans: a multilaboratory investigation.

It has been suggested that there are two distinct and parallel mechanisms for controlling instrumental behavior in mammals: goal-directed actions and habits. To gain an understanding of how these two systems interact to control behavior, it is essential to characterize the mechanisms by which the balance between these systems is influenced by experience. Studies in rodents have shown that the amount of training governs the relative expression of these two systems: Behavior is goal-directed following moderate training, but the more extensively an instrumental action is trained, the more it becomes habitual. It is less clear whether humans exhibit similar training effects on the expression of goal-directed and habitual behavior, as human studies have reported contradictory findings. To tackle these contradictory findings, we formed a consortium, where four laboratories undertook a preregistered experimental induction of habits by manipulating the amount of training. There was no statistical evidence for a main effect of the amount of training on the formation and expression of habits. However, exploratory analyses suggest a moderating effect of the affective component of stress on the impact of training over habit expression. Participants who were lower in affective stress appeared to be initially goal-directed, but became habitual with increased training, whereas participants who were high in affective stress were already habitual even after moderate training, thereby manifesting insensitivity to overtraining effects. Our findings highlight the importance of the role of moderating variables such as individual differences in stress and anxiety when studying the experimental induction of habits in humans.

White matter tracts characteristics in habitual decision-making circuit underlie ritual behaviors in anorexia nervosa.

Anorexia nervosa (AN) is a difficult to treat, pernicious psychiatric disorder that has been linked to decision-making abnormalities. We examined the structural characteristics of habitual and goal-directed decision-making circuits and their connecting white matter tracts in 32 AN and 43 healthy controls across two independent data sets of adults and adolescents as an explanatory sub-study. Total bilateral premotor/supplementary motor area-putamen tracts in the habit circuit had a significantly higher volume in adults with AN, relative to controls. Positive correlations were found between both the number of tracts and white matter volume (WMV) in the habit circuit, and the severity of ritualistic/compulsive behaviors in adults and adolescents with AN. Moreover, we found a significant influence of the habit circuit WMV on AN ritualistic/compulsive symptom severity, depending on the preoccupations symptom severity levels. These findings suggest that AN is associated with white matter plasticity alterations in the habit circuit. The association between characteristics of habit circuit white matter tracts and AN behavioral symptoms provides support for a circuit based neurobiological model of AN, and identifies the habit circuit as a focus for further investigation to aid in development of novel and more effective treatments based on brain-behavior relationships.

How representative are neuroimaging samples? Large-scale evidence for trait anxiety differences between fMRI and behaviour-only research participants.

Over the past three decades, functional magnetic resonance imaging (fMRI) has become crucial to study how cognitive processes are implemented in the human brain. However, the question of whether participants recruited into fMRI studies differ from participants recruited into other study contexts has received little to no attention. This is particularly pertinent when effects fail to generalize across study contexts: for example, a behavioural effect discovered in a non-imaging context not replicating in a neuroimaging environment. Here, we tested the hypothesis, motivated by preliminary findings (N = 272), that fMRI participants differ from behaviour-only participants on one fundamental individual difference variable: trait anxiety. Analysing trait anxiety scores and possible confounding variables from healthy volunteers across multiple institutions (N = 3317), we found robust support for lower trait anxiety in fMRI study participants, consistent with a sampling or self-selection bias. The bias was larger in studies that relied on phone screening (compared with full in-person psychiatric screening), recruited at least partly from convenience samples (compared with community samples), and in pharmacology studies. Our findings highlight the need for surveying trait anxiety at recruitment and for appropriate screening procedures or sampling strategies to mitigate this bias.

Aesthetic preference for art can be predicted from a mixture of low- and high-level visual features.

It is an open question whether preferences for visual art can be lawfully predicted from the basic constituent elements of a visual image. Here, we developed and tested a computational framework to investigate how aesthetic values are formed. We show that it is possible to explain human preferences for a visual art piece based on a mixture of low- and high-level features of the image. Subjective value ratings could be predicted not only within but also across individuals, using a regression model with a common set of interpretable features. We also show that the features predicting aesthetic preference can emerge hierarchically within a deep convolutional neural network trained only for object recognition. Our findings suggest that human preferences for art can be explained at least in part as a systematic integration over the underlying visual features of an image.