Blunted Expected Reward Value Signals in Binge Alcohol Drinkers.

Alcohol-related morbidities and mortality are highly prevalent, increasing the burden to societies and health systems with 3 million deaths globally each year in young adults directly attributable to alcohol. Cue-induced alcohol craving has been formulated as a type of aberrant associative learning, modeled using temporal difference theory with an expected reward value (ERV) linked to craving. Clinically, although harmful use of alcohol is associated with increased time spent obtaining and using alcohol, it is also associated with self-neglect. The latter implies that the motivational aspects of nonalcohol stimuli are blunted. Using an instrumental learning task with non-alcohol-related stimuli, here, we tested hypotheses that the encoding of cue signals (ERV) predicting reward delivery would be blunted in binge alcohol drinkers in both sexes. We also predicted that for the binge drinking group alone, ratings of problematic alcohol use would correlate with abnormal ERV signals consistent with between groups (i.e., binge drinkers vs controls) abnormalities. Our results support our hypotheses with the ERV (nonalcohol cue) signal blunted in binge drinkers and with the magnitude of the abnormality correlating with ratings of problematic alcohol use. This implies that consistent with hypotheses, the motivational aspects of non-alcohol-related stimuli are blunted in binge drinkers. A better understanding of the mechanisms of harmful alcohol use will, in time, facilitate the development of more effective interventions, which should aim to decrease the motivational value of alcohol and increase the motivational value of non-alcohol-related stimuli.SIGNIFICANCE STATEMENT Allostasis theory predicts specific abnormalities in brain function and subjective experiences that occur when people develop drug problems including addiction. Cue-induced alcohol craving has been formulated as a type of aberrant associative learning, modeled using temporal difference theory with ERV linked to craving. Here, we used an instrumental learning task with non-alcohol-associated stimuli to test hypotheses that the encoding of nonalcohol cue signals (ERV) and reward prediction error signals showed blunting in binge alcohol drinkers. We conclude that fMRI can be used to noninvasively test allostasis and associative learning theory predictions in binge drinkers.

Neural regions associated with gain-loss frequency and average reward in older and younger adults.

Research on the biological basis of reinforcement-learning has focused on how brain regions track expected value based on average reward. However, recent work suggests that humans are more attuned to reward frequency. Furthermore, older adults are less likely to use expected values to guide choice than younger adults. This raises the question of whether brain regions assumed to be sensitive to average reward, like the medial and lateral PFC, also track reward frequency, and whether there are age-based differences. Older (60-81 years) and younger (18-30 years) adults performed the Soochow Gambling task, which separates reward frequency from average reward, while undergoing fMRI. Overall, participants preferred options that provided negative net payoffs, but frequent gains. Older adults improved less over time, were more reactive to recent negative outcomes, and showed greater frequency-related activation in several regions, including DLPFC. We also found broader recruitment of prefrontal and parietal regions associated with frequency value and reward prediction errors in older adults, which may indicate compensation. The results suggest greater reliance on average reward for younger adults than older adults.

Acute stress blunts prediction error signals in the dorsal striatum during reinforcement learning.

Acute stress is pervasive in everyday modern life and is thought to affect how people make choices and learn from them. Reinforcement learning, which implicates learning from the unexpected rewarding and punishing outcomes of our choices (i.e., prediction errors), is critical for adjusted behaviour and seems to be affected by acute stress. However, the neural mechanisms by which acute stress disrupts this type of learning are still poorly understood. Here, we investigate whether and how acute stress blunts neural signalling of prediction errors during reinforcement learning using model-based functional magnetic resonance imaging. Male participants completed a well-established reinforcement-learning task involving monetary gains and losses whilst under stress and control conditions. Acute stress impaired participants' (n = 23) behavioural performance towards obtaining monetary gains (p < 0.001), but not towards avoiding losses (p = 0.57). Importantly, acute stress blunted signalling of prediction errors during gain and loss trials in the dorsal striatum (p = 0.040) - with subsidiary analyses suggesting that acute stress preferentially blunted signalling of positive prediction errors. Our results thus reveal a neurocomputational mechanism by which acute stress may impair reward learning.

Neural basis of corruption in power-holders.

Corruption often involves bribery, when a briber suborns a power-holder to gain advantages usually at a cost of moral transgression. Despite its wide presence in human societies, the neurocomputational basis of bribery remains elusive. Here, using model-based fMRI, we investigated the neural substrates of how a power-holder decides to accept or reject a bribe. Power-holders considered two types of moral cost brought by taking bribes: the cost of conniving with a fraudulent briber, encoded in the anterior insula, and the harm brought to a third party, represented in the right temporoparietal junction. These moral costs were integrated into a value signal in the ventromedial prefrontal cortex. The dorsolateral prefrontal cortex was selectively engaged to guide anti-corrupt behaviors when a third party would be harmed. Multivariate and connectivity analyses further explored how these neural processes depend on individual differences. These findings advance our understanding of the neurocomputational mechanisms underlying corrupt behaviors.

Computational and Neurobiological Substrates of Cost-Benefit Integration in Altruistic Helping Decision.

Although altruistic behaviors, e.g., sacrificing one's own interests to alleviate others' suffering, are widely observed in human society, altruism varies greatly across individuals. Such individual differences in altruistic preference have been hypothesized to arise from both individuals' dispositional empathic concern for others' welfare and context-specific cost-benefit integration processes. However, how cost-benefit integration is implemented in the brain and how it is linked to empathy remain unclear. Here, we combine a novel paradigm with the model-based functional magnetic resonance imaging (fMRI) approach to examine the neurocomputational basis of altruistic behaviors. Thirty-seven adults (16 females) were tested. Modeling analyses suggest that individuals are likely to integrate their own monetary costs with nonlinearly transformed recipients' benefits. Neuroimaging results demonstrate the involvement of an extended common currency system during decision-making by showing that selfish and other-regarding motives were processed in dorsal anterior cingulate cortex (ACC) and right inferior parietal lobe in a domain-general manner. Importantly, a functional dissociation of adjacent but different subregions within anterior insular cortex (aINS) was observed for different subprocesses underlying altruistic behaviors. While dorsal aINS (daINS) and inferior frontal gyrus (IFG) were involved in valuation of benefactors' costs, ventral aINS and middle INS (vaINS/mINS), as empathy-related regions, reflected individual variations in valuating recipients' benefits. Multivariate analyses further suggest that both vaINS/mINS and dorsolateral prefrontal cortex (DLPFC) reflect individual variations in general altruistic preferences which account for both dispositional empathy and context-specific other-regarding tendency. Together, these findings provide valuable insights into our understanding of psychological and neurobiological basis of altruistic behaviors.SIGNIFICANCE STATEMENT Altruistic behaviors play a crucial role in facilitating solidarity and development of human society, but the mechanisms of the cost-benefit integration underlying these behaviors are still unclear. Using model-based neuroimaging approaches, we clarify that people integrate personal costs and non-linearly transformed other's benefits during altruistic decision-making and the implementations of the integration processes are supported by an extended common currency neural network. Importantly, multivariate analyses reveal that both empathy-related and cognitive control-related brain regions are involved in modulating individual variations of altruistic preference, which implicate complex psychological and computational processes. Our results provide a neurocomputational account of how people weigh between different attributes to make altruistic decisions and why altruistic preference varies to a great extent across individuals.

Using reinforcement learning models in social neuroscience: frameworks, pitfalls and suggestions of best practices.

The recent years have witnessed a dramatic increase in the use of reinforcement learning (RL) models in social, cognitive and affective neuroscience. This approach, in combination with neuroimaging techniques such as functional magnetic resonance imaging, enables quantitative investigations into latent mechanistic processes. However, increased use of relatively complex computational approaches has led to potential misconceptions and imprecise interpretations. Here, we present a comprehensive framework for the examination of (social) decision-making with the simple Rescorla-Wagner RL model. We discuss common pitfalls in its application and provide practical suggestions. First, with simulation, we unpack the functional role of the learning rate and pinpoint what could easily go wrong when interpreting differences in the learning rate. Then, we discuss the inevitable collinearity between outcome and prediction error in RL models and provide suggestions of how to justify whether the observed neural activation is related to the prediction error rather than outcome valence. Finally, we suggest posterior predictive check is a crucial step after model comparison, and we articulate employing hierarchical modeling for parameter estimation. We aim to provide simple and scalable explanations and practical guidelines for employing RL models to assist both beginners and advanced users in better implementing and interpreting their model-based analyses.

Breaking human social decision making into multiple components and then putting them together again.

Most of our waking time as human beings is spent interacting with other individuals. In order to make good decisions in this social milieu, it is often necessary to make inferences about the internal states, traits and intentions of others. Recently, some progress has been made toward uncovering the neural computations underlying human social decision-making by combining functional magnetic resonance neuroimaging (fMRI) with computational modeling of behavior. Modeling of behavioral data allows us to identify the key computations necessary for social decision-making and to determine how these computations are integrated. Furthermore, by correlating these variables against neuroimaging data, it has become possible to elucidate where in the brain various computations are implemented. Here we review the current state of knowledge in the domain of social computational neuroscience. Findings to date have emphasized that social decisions are driven by multiple computations conducted in parallel, and implemented in distinct brain regions. We suggest that further progress is going to depend on identifying how and where such variables get integrated in order to yield a coherent behavioral output.

Neurocomputational mechanisms underlying immoral decisions benefiting self or others.

Immoral behavior often consists of weighing transgression of a moral norm against maximizing personal profits. One important question is to understand why immoral behaviors vary based on who receives specific benefits and what are the neurocomputational mechanisms underlying such moral flexibility. Here, we used model-based functional magnetic resonance imaging to investigate how immoral behaviors change when benefiting oneself or someone else. Participants were presented with offers requiring a tradeoff between a moral cost (i.e. profiting a morally bad cause) and a benefit for either oneself or a charity. Participants were more willing to obtain ill-gotten profits for themselves than for a charity, driven by a devaluation of the moral cost when deciding for their own interests. The subjective value of an immoral offer, computed as a linear summation of the weighed monetary gain and moral cost, recruited the ventromedial prefrontal cortex (PFC) regardless of beneficiaries. Moreover, paralleling the behavioral findings, this region enhanced its functional coupling with mentalizing-related regions while deciding whether to gain morally tainted profits for oneself vs charity. Finally, individual differences in moral preference differentially modulated choice-specific signals in the dorsolateral PFC according to who benefited from the decisions. These findings provide insights for understanding the neurobiological basis of moral flexibility.

Neural Mechanisms for Adaptive Learned Avoidance of Mental Effort.

Humans tend to avoid mental effort. Previous studies have demonstrated this tendency using various demand-selection tasks; participants generally avoid options associated with higher cognitive demand. However, it remains unclear whether humans avoid mental effort adaptively in uncertain and nonstationary environments. If so, it also remains unclear what neural mechanisms underlie such learned avoidance and whether they remain the same regardless of cognitive-demand types. We addressed these issues by developing novel demand-selection tasks where associations between choice options and cognitive-demand levels change over time, with two variations using mental arithmetic and spatial reasoning problems (males/females: 29:4 and 18:2). Most participants showed avoidance, and their choices depended on the demand experienced on multiple preceding trials. We assumed that participants updated the expected cost of mental effort through experience, and fitted their choices by reinforcement learning models, comparing several possibilities. Model-based fMRI analyses revealed that activity in the dorsomedial and lateral frontal cortices was positively correlated with the trial-by-trial expected cost for the chosen option commonly across the different types of cognitive demand. Analyses also revealed a trend of negative correlation in the ventromedial prefrontal cortex. We further identified correlates of cost-prediction error at time of problem presentation or answering the problem, the latter of which partially overlapped with or were proximal to the correlates of expected cost at time of choice cue in the dorsomedial frontal cortex. These results suggest that humans adaptively learn to avoid mental effort, having neural mechanisms to represent expected cost and cost-prediction error, and the same mechanisms operate for various types of cognitive demand.SIGNIFICANCE STATEMENT In daily life, humans encounter various cognitive demands and tend to avoid high-demand options. However, it remains unclear whether humans avoid mental effort adaptively under dynamically changing environments. If so, it also remains unclear what the underlying neural mechanisms are and whether they operate regardless of cognitive-demand types. To address these issues, we developed novel tasks where participants could learn to avoid high-demand options under uncertain and nonstationary environments. Through model-based fMRI analyses, we found regions whose activity was correlated with the expected mental effort cost, or cost-prediction error, regardless of demand type. These regions overlap, or are adjacent with each other, in the dorsomedial frontal cortex. This finding helps clarify the mechanisms for cognitive-demand avoidance, and provides empirical building blocks for the emerging computational theory of mental effort.

Revealing Neurocomputational Mechanisms of Reinforcement Learning and Decision-Making With the hBayesDM Package.

Reinforcement learning and decision-making (RLDM) provide a quantitative framework and computational theories with which we can disentangle psychiatric conditions into the basic dimensions of neurocognitive functioning. RLDM offer a novel approach to assessing and potentially diagnosing psychiatric patients, and there is growing enthusiasm for both RLDM and computational psychiatry among clinical researchers. Such a framework can also provide insights into the brain substrates of particular RLDM processes, as exemplified by model-based analysis of data from functional magnetic resonance imaging (fMRI) or electroencephalography (EEG). However, researchers often find the approach too technical and have difficulty adopting it for their research. Thus, a critical need remains to develop a user-friendly tool for the wide dissemination of computational psychiatric methods. We introduce an R package called hBayesDM (hierarchical Bayesian modeling of Decision-Making tasks), which offers computational modeling of an array of RLDM tasks and social exchange games. The hBayesDM package offers state-of-the-art hierarchical Bayesian modeling, in which both individual and group parameters (i.e., posterior distributions) are estimated simultaneously in a mutually constraining fashion. At the same time, the package is extremely user-friendly: users can perform computational modeling, output visualization, and Bayesian model comparisons, each with a single line of coding. Users can also extract the trial-by-trial latent variables (e.g., prediction errors) required for model-based fMRI/EEG. With the hBayesDM package, we anticipate that anyone with minimal knowledge of programming can take advantage of cutting-edge computational-modeling approaches to investigate the underlying processes of and interactions between multiple decision-making (e.g., goal-directed, habitual, and Pavlovian) systems. In this way, we expect that the hBayesDM package will contribute to the dissemination of advanced modeling approaches and enable a wide range of researchers to easily perform computational psychiatric research within different populations.

Congruence of Inherent and Acquired Values Facilitates Reward-Based Decision-Making.

UNLABELLED: Most real-life cues exhibit certain inherent values that may interfere with or facilitate the acquisition of new expected values during associative learning. In particular, when inherent and acquired values are congruent, learning may progress more rapidly. Here we investigated such an influence through a 2 × 2 factorial design, using attractiveness (high/low) of the facial picture as a proxy for the inherent value of the cue and its reward probability (high/low) as a surrogate for the acquired value. Each picture was paired with a monetary win or loss either congruently or incongruently. Behavioral results from 32 human participants indicated both faster response time and faster learning rate for value-congruent cue-outcome pairings. Model-based fMRI analysis revealed a fractionation of reinforcement learning (RL) signals in the ventral striatum, including a strong and novel correlation between the cue-specific decaying learning rate and BOLD activity in the ventral caudate. Additionally, we detected a functional link between neural signals of both learning rate and reward prediction error in the ventral striatum, and the signal of expected value in the ventromedial prefrontal cortex, showing a novel confirmation of the mathematical RL model via functional connectivity. SIGNIFICANCE STATEMENT: Most real-world decisions require the integration of inherent value and sensitivity to outcomes to facilitate adaptive learning. Inherent value is drawing increasing interest from decision scientists because it influences decisions in contexts ranging from advertising to investing. This study provides novel insight into how inherent value influences the acquisition of new expected value during associative learning. Specifically, we find that the congruence between the inherent value and the acquired reward influences the neural coding of learning rate. We also show for the first time that neuroimaging signals coding the learning rate, prediction error, and acquired value follow the multiplicative Rescorla-Wagner learning rule, a finding predicted by reinforcement learning theory.

Motivational Context Modulates Prediction Error Response in Schizophrenia.

BACKGROUND: Recent findings demonstrate that patients with schizophrenia are worse at learning to predict rewards than losses, suggesting that motivational context modulates learning in this disease. However, these findings derive from studies in patients treated with antipsychotic medications, D2 receptor antagonists that may interfere with the neural systems that underlie motivation and learning. Thus, it remains unknown how motivational context affects learning in schizophrenia, separate from the effects of medication. METHODS: To examine the impact of motivational context on learning in schizophrenia, we tested 16 unmedicated patients with schizophrenia and 23 matched controls on a probabilistic learning task while they underwent functional magnetic resonance imaging (fMRI) under 2 conditions: one in which they pursued rewards, and one in which they avoided losses. Computational models were used to derive trial-by-trial prediction error responses to feedback. RESULTS: Patients performed worse than controls on the learning task overall, but there were no behavioral effects of condition. FMRI revealed an attenuated prediction error response in patients in the medial prefrontal cortex, striatum, and medial temporal lobe when learning to predict rewards, but not when learning to avoid losses. CONCLUSIONS: Patients with schizophrenia showed differences in learning-related brain activity when learning to predict rewards, but not when learning to avoid losses. Together with prior work, these results suggest that motivational deficits related to learning in schizophrenia are characteristic of the disease and not solely a result of antipsychotic treatment.

Altered behavioral and neural responsiveness to counterfactual gains in the elderly.

Counterfactual information processing refers to the consideration of events that did not occur in comparison to those actually experienced, in order to determine optimal actions, and can be formulated as computational learning signals, referred to as fictive prediction errors. Decision making and the neural circuitry for counterfactual processing are altered in healthy elderly adults. This experiment investigated age differences in neural systems for decision making with knowledge of counterfactual outcomes. Two groups of healthy adult participants, young (N = 30; ages 19-30 years) and elderly (N = 19; ages 65-80 years), were scanned with fMRI during 240 trials of a strategic sequential investment task in which a particular strategy of differentially weighting counterfactual gains and losses during valuation is associated with more optimal performance. Elderly participants earned significantly less than young adults, differently weighted counterfactual consequences and exploited task knowledge, and exhibited altered activity in a fronto-striatal circuit while making choices, compared to young adults. The degree to which task knowledge was exploited was positively correlated with modulation of neural activity by expected value in the vmPFC for young adults, but not in the elderly. These findings demonstrate that elderly participants' poor task performance may be related to different counterfactual processing.

The algorithmic level is the bridge between computation and brain.

Every scientist chooses a preferred level of analysis and this choice shapes the research program, even determining what counts as evidence. This contribution revisits Marr's (1982) three levels of analysis (implementation, algorithmic, and computational) and evaluates the prospect of making progress at each individual level. After reviewing limitations of theorizing within a level, two strategies for integration across levels are considered. One is top-down in that it attempts to build a bridge from the computational to algorithmic level. Limitations of this approach include insufficient theoretical constraint at the computation level to provide a foundation for integration, and that people are suboptimal for reasons other than capacity limitations. Instead, an inside-out approach is forwarded in which all three levels of analysis are integrated via the algorithmic level. This approach maximally leverages mutual data constraints at all levels. For example, algorithmic models can be used to interpret brain imaging data, and brain imaging data can be used to select among competing models. Examples of this approach to integration are provided. This merging of levels raises questions about the relevance of Marr's tripartite view.

Changes in corticostriatal connectivity during reinforcement learning in humans.

Many computational models assume that reinforcement learning relies on changes in synaptic efficacy between cortical regions representing stimuli and striatal regions involved in response selection, but this assumption has thus far lacked empirical support in humans. We recorded hemodynamic signals with fMRI while participants navigated a virtual maze to find hidden rewards. We fitted a reinforcement-learning algorithm to participants' choice behavior and evaluated the neural activity and the changes in functional connectivity related to trial-by-trial learning variables. Activity in the posterior putamen during choice periods increased progressively during learning. Furthermore, the functional connections between the sensorimotor cortex and the posterior putamen strengthened progressively as participants learned the task. These changes in corticostriatal connectivity differentiated participants who learned the task from those who did not. These findings provide a direct link between changes in corticostriatal connectivity and learning, thereby supporting a central assumption common to several computational models of reinforcement learning.

Neural evidence for adaptive strategy selection in value-based decision-making.

In everyday life, humans often encounter complex environments in which multiple sources of information can influence their decisions. We propose that in such situations, people select and apply different strategies representing different cognitive models of the decision problem. Learning advances by evaluating the success of using a strategy and eventually by switching between strategies. To test our strategy selection model, we investigated how humans solve a dynamic learning task with complex auditory and visual information, and assessed the underlying neural mechanisms with functional magnetic resonance imaging. Using the model, we were able to capture participants' choices and to successfully attribute expected values and reward prediction errors to activations in the dopaminoceptive system (e.g., ventral striatum [VS]) as well as decision conflict to signals in the anterior cingulate cortex. The model outperformed an alternative approach that did not update decision strategies, but the relevance of information itself. Activation of sensory areas depended on whether the selected strategy made use of the respective source of information. Selection of a strategy also determined how value-related information influenced effective connectivity between sensory systems and the VS. Our results suggest that humans can structure their search for and use of relevant information by adaptively selecting between decision strategies.

The neural substrate of prior information in perceptual decision making: a model-based analysis.

PRIOR INFORMATION BIASES THE DECISION PROCESS: actions consistent with prior information are executed swiftly, whereas actions inconsistent with prior information are executed slowly. How is this bias implemented in the brain? To address this question we conducted an experiment in which people had to decide quickly whether a cloud of dots moved coherently to the left or to the right. Cues provided probabilistic information about the upcoming stimulus. Behavioral data were analyzed with the linear ballistic accumulator (LBA) model, confirming that people used the cue to bias their decisions. The functional magnetic resonance imaging (fMRI) data showed that presentation of the cue differentially activated orbitofrontal cortex, hippocampus, and the putamen. Directional cues selectively activated the contralateral putamen. The fMRI analysis yielded results only when the LBA bias parameter was included as a covariate, highlighting the practical benefits of formal modeling. Our results suggest that the human brain uses prior information by increasing cortico-striatal activation to selectively disinhibit preferred responses.