Brain mechanism of foraging: Reward-dependent synaptic plasticity versus neural integration of values.

During foraging behavior, action values are persistently encoded in neural activity and updated depending on the history of choice outcomes. What is the neural mechanism for action value maintenance and updating? Here, we explore two contrasting network models: synaptic learning of action value versus neural integration. We show that both models can reproduce extant experimental data, but they yield distinct predictions about the underlying biological neural circuits. In particular, the neural integrator model but not the synaptic model requires that reward signals are mediated by neural pools selective for action alternatives and their projections are aligned with linear attractor axes in the valuation system. We demonstrate experimentally observable neural dynamical signatures and feasible perturbations to differentiate the two contrasting scenarios, suggesting that the synaptic model is a more robust candidate mechanism. Overall, this work provides a modeling framework to guide future experimental research on probabilistic foraging.

Acetylcholine and noradrenaline enhance foraging optimality in humans.

Foraging theory prescribes when optimal foragers should leave the current option for more rewarding alternatives. Actual foragers often exploit options longer than prescribed by the theory, but it is unclear how this foraging suboptimality arises. We investigated whether the upregulation of cholinergic, noradrenergic, and dopaminergic systems increases foraging optimality. In a double-blind, between-subject design, participants (N = 160) received placebo, the nicotinic acetylcholine receptor agonist nicotine, a noradrenaline reuptake inhibitor reboxetine, or a preferential dopamine reuptake inhibitor methylphenidate, and played the role of a farmer who collected milk from patches with different yield. Across all groups, participants on average overharvested. While methylphenidate had no effects on this bias, nicotine, and to some extent also reboxetine, significantly reduced deviation from foraging optimality, which resulted in better performance compared to placebo. Concurring with amplified goal-directedness and excluding heuristic explanations, nicotine independently also improved trial initiation and time perception. Our findings elucidate the neurochemical basis of behavioral flexibility and decision optimality and open unique perspectives on psychiatric disorders affecting these functions.

A special role for anterior cingulate cortex, but not orbitofrontal cortex or basolateral amygdala, in choices involving information.

Subjects are often willing to pay a cost for information. In a procedure that promotes paradoxical choices, animals choose between a richer option followed by a cue that is rewarded 50% of the time (No Info) vs. a leaner option followed by one of two cues that signal certain outcomes: one always rewarded (100%) and the other never rewarded, 0% (Info). Since decisions involve comparing the subjective value of options after integrating all their features, preference for information may rely on cortico-amygdalar circuitry. To test this, male and female rats were prepared with bilateral inhibitory Designer Receptors Exclusively Activated by Designer Drugs (DREADDs) in the anterior cingulate cortex, orbitofrontal cortex, basolateral amygdala, or null virus (control). We inhibited these regions after stable preference was acquired. We found that inhibition of the anterior cingulate cortex destabilized choice preference in female rats without affecting latency to choose or response rate to cues. A logistic regression fit revealed that previous choice predicted current choice in all conditions, however previously rewarded Info trials strongly predicted preference in all conditions except in female rats following anterior cingulate cortex inhibition. The results reveal a causal, sex-dependent role for the anterior cingulate cortex in decisions involving information.

Mechanisms of adjustments to different types of uncertainty in the reward environment across mice and monkeys.

Despite being unpredictable and uncertain, reward environments often exhibit certain regularities, and animals navigating these environments try to detect and utilize such regularities to adapt their behavior. However, successful learning requires that animals also adjust to uncertainty associated with those regularities. Here, we analyzed choice data from two comparable dynamic foraging tasks in mice and monkeys to investigate mechanisms underlying adjustments to different types of uncertainty. In these tasks, animals selected between two choice options that delivered reward probabilistically, while baseline reward probabilities changed after a variable number (block) of trials without any cues to the animals. To measure adjustments in behavior, we applied multiple metrics based on information theory that quantify consistency in behavior, and fit choice data using reinforcement learning models. We found that in both species, learning and choice were affected by uncertainty about reward outcomes (in terms of determining the better option) and by expectation about when the environment may change. However, these effects were mediated through different mechanisms. First, more uncertainty about the better option resulted in slower learning and forgetting in mice, whereas it had no significant effect in monkeys. Second, expectation of block switches accompanied slower learning, faster forgetting, and increased stochasticity in choice in mice, whereas it only reduced learning rates in monkeys. Overall, while demonstrating the usefulness of metrics based on information theory in examining adaptive behavior, our study provides evidence for multiple types of adjustments in learning and choice behavior according to uncertainty in the reward environment.

Cognitive effort exertion enhances electrophysiological responses to rewarding outcomes.

Recent work has highlighted neural mechanisms underlying cognitive effort-related discounting of anticipated rewards. However, findings on whether effort exertion alters the subjective value of obtained rewards are inconsistent. Here, we provide a more nuanced account of how cognitive effort affects subsequent reward processing in a novel task designed to assess effort-induced modulations of the Reward Positivity, an event-related potential indexing reward-related neural activity. We found that neural responses to both gains and losses were significantly elevated in trials requiring more versus less cognitive effort. Moreover, time-frequency analysis revealed that these effects were mirrored in gain-related delta, but not in loss-related theta band activity, suggesting that people ascribed more value to high-effort outcomes. In addition, we also explored whether individual differences in behavioral effort discounting rates and reward sensitivity in the absence of effort may affect the relationship between effort exertion and subsequent reward processing. Together, our findings provide evidence that cognitive effort exertion can increase the subjective value of subsequent outcomes and that this effect may primarily rely on modulations of delta band activity.

Attempts to Influence the Value of Alcohol by Manipulating Social Influence and Context.

Background: Recent cognitive neuroscience models of value-based decision-making suggest value-based choices for alcohol are sensitive to various inputs, such as context and social influence. In two online experiments, we tested whether manipulating these inputs influenced proxies for alcohol value. Experiment 1: 157 social drinkers were presented with 4 hypothetical scenarios (drinking alone, with friends who are also drinking, with friends but trying to "cut-down" for health reasons, with friends who aren't drinking) in a within-subjects design, and completed the Brief Assessment of Alcohol Demand after each as a measure of value. Value for alcohol (number of drinks purchased) was greatest when drinking with friends who were also drinking compared to drinking alone (d = 0.95), friends not drinking (d = 1.49) and friends drinking/health related (d = 1.59). Value for alcohol was also greater when drinking alone compared to with friends who were not drinking (d = 0.55), and also with friends drinking/health related (d = 0.62). Experiment 2: 241 participants were randomly allocated to see one of four categories of images in a 2 (context: bar vs house) x 2 (social influence: enjoy vs not enjoy) design, before completing a Concurrent Choice Task for alcohol and Visual Analog Scales. There were no significant effects found on either task, both taken as proxies for value. Conclusion: There was inconclusive evidence that the value for alcohol could be manipulated by social context. This could be explained by greater saliency of the manipulation in asking participants to imagine themselves in a hypothetical situation as opposed to presenting images depicting drinking scenarios.

Contribution of the sensorimotor beta oscillations and the cortico-basal ganglia-thalamic circuitry during value-based decision making: A simultaneous EEG-fMRI investigation.

In decision neuroscience, the motor system has primarily been considered to be involved in executing choice actions. However, a competing perspective suggests its engagement in the evaluation of options, traditionally considered to be performed by the brain's valuation system. Here, we investigate the role of the motor system in value-based decision making by determining the neural circuitries associated with the sensorimotor beta oscillations previously identified to encode decision options. In a simultaneous EEG-fMRI study, participants evaluated reward and risk associated with a forthcoming action. A significant sensorimotor beta desynchronization was identified prior to and independent of response. The level of beta desynchronization showed evidence of encoding the reward levels. This beta desynchronization covaried, on a trial-by-trial level, with BOLD activity in the cortico-basal ganglia-thalamic circuitry. In contrast, there was only a weak covariation within the valuation network, despite significant modulation of its BOLD activity by reward levels. These results suggest that the way in which decision variables are processed differs in the valuation network and in the cortico-basal ganglia-thalamic circuitry. We propose that sensorimotor beta oscillations indicate incentive motivational drive towards a choice action computed from the decision variables even prior to making a response, and it arises from the cortico-basal ganglia-thalamic circuitry.

Disinvestment in healthcare: a scoping review of systematic reviews.

OBJECTIVES: Disinvestment from low value health technologies is growing globally. Diverse evidence gathering and assessment methods were used to implement disinvestment initiatives, however, less than half of the empirical studies report reduced use of the low-value services. This scoping review aimed to synthesize the information from available reviews on the concepts and purposes of disinvestment in healthcare, the approaches and methods used, the role of stakeholders and facilitators and barriers in its implementation. METHODS: This scoping review was guided by the Joanna Briggs Institute Manual for Evidence Synthesis and PRISMA statement for scoping review. Published reviews on disinvestment were identified from scientific databases including health technology assessment (HTA) Web sites using the terms "disinvestment," "health technology reassessment," and "healthcare." The data obtained was synthesized narratively to identify similarities and differences across the approaches based on the prespecified categories. RESULTS: Seventeen reviews were included with thirty-four initiatives identified across sixteen countries at various levels of implementation and responsible agencies for the activities. Two most used methods to facilitate disinvestment decisions are Programme Budgeting and Marginal Analysis (PBMA) and HTA. Stakeholder involvement is the most important aspect to be addressed, as it acts as both facilitator and barrier in disinvestment initiatives implementation. CONCLUSIONS: Disinvestment programs have been implemented at multilevel, involving multistakeholders and using multiple methods such as PBMA and HTA. However, there is a lack of clarity on the additional dimensions of technical analysis related to these tools. Further research could focus on technology optimization in healthcare as part of overall health technology management.

Real-Life Self-Control is Predicted by Parietal Activity During Preference Decision Making: A Brain Decoding Analysis.

Despite its relevance for health and education, the neurocognitive mechanism of real-life self-control is largely unknown. While recent research revealed a prominent role of the ventromedial prefrontal cortex in the computation of an integrative value signal, the contribution and relevance of other brain regions for real-life self-control remains unclear. To investigate neural correlates of decisions in line with long-term consequences and to assess the potential of brain decoding methods for the individual prediction of real-life self-control, we combined functional magnetic resonance imaging during preference decision making with ecological momentary assessment of daily self-control in a large community sample (N = 266). Decisions in line with long-term consequences were associated with increased activity in bilateral angular gyrus and precuneus, regions involved in different forms of perspective taking, such as imagining one's own future and the perspective of others. Applying multivariate pattern analysis to the same clusters revealed that individual patterns of activity predicted the probability of real-life self-control. Brain activations are discussed in relation to episodic future thinking and mentalizing as potential mechanisms mediating real-life self-control.

Corticostriatal White Matter Integrity and Dopamine D1 Receptor Availability Predict Age Differences in Prefrontal Value Signaling during Reward Learning.

Probabilistic reward learning reflects the ability to adapt choices based on probabilistic feedback. The dopaminergically innervated corticostriatal circuit in the brain plays an important role in supporting successful probabilistic reward learning. Several components of the corticostriatal circuit deteriorate with age, as it does probabilistic reward learning. We showed previously that D1 receptor availability in NAcc predicts the strength of anticipatory value signaling in vmPFC, a neural correlate of probabilistic learning that is attenuated in older participants and predicts probabilistic reward learning performance. We investigated how white matter integrity in the pathway between nucleus accumbens (NAcc) and ventromedial prefrontal cortex (vmPFC) relates to the strength of anticipatory value signaling in vmPFC in younger and older participants. We found that in a sample of 22 old and 23 young participants, fractional anisotropy in the pathway between NAcc and vmPFC predicted the strength of value signaling in vmPFC independently from D1 receptor availability in NAcc. These findings provide tentative evidence that integrity in the dopaminergic and white matter pathways of corticostriatal circuitry supports the expression of value signaling in vmPFC which supports reward learning, however, the limited sample size calls for independent replication. These and future findings could add to the improved understanding of how corticostriatal integrity contributes to reward learning ability.

Neural underpinnings of value-guided choice during auction tasks: An eye-fixation related potentials study.

Values are attributed to goods during free viewing of objects which entails multi- and trans-saccadic cognitive processes. Using electroencephalographic eye-fixation related potentials, the present study investigated how neural signals related to value-guided choice evolved over time when viewing household and office products during an auction task. Participants completed a Becker-DeGroot-Marschak auction task whereby half of the stimuli were presented in either a free or forced bid protocol to obtain willingness-to-pay. Stimuli were assigned to three value categories of low, medium and high value based on subjective willingness-to-pay. Eye fixations were organised into five 800 ms time-bins spanning the objects total viewing time. Independent component analysis was applied to eye-fixation related potentials. One independent component (IC) was found to represent fixations for high value products with increased activation over the left parietal region of the scalp. An IC with a spatial maximum over a fronto-central region of the scalp coded the intermediate values. Finally, one IC displaying activity that extends over the right frontal scalp region responded to intermediate- and low-value items. Each of these components responded early on during viewing an object and remained active over the entire viewing period, both during free and forced bid trials. Results suggest that the subjective value of goods are encoded using sets of brain activation patterns which are tuned to respond uniquely to either low, medium, or high values. Data indicates that the right frontal region of the brain responds to low and the left frontal region to high values. Values of goods are determined at an early point in the decision making process and carried for the duration of the decision period via trans-saccadic processes.

Action in auctions: neural and computational mechanisms of bidding behaviour.

Competition for resources is a fundamental characteristic of evolution. Auctions have been widely used to model competition of individuals for resources, and bidding behaviour plays a major role in social competition. Yet, how humans learn to bid efficiently remains an open question. We used model-based neuroimaging to investigate the neural mechanisms of bidding behaviour under different types of competition. Twenty-seven subjects (nine male) played a prototypical bidding game: a double action, with three "market" types, which differed in the number of competitors. We compared different computational learning models of bidding: directional learning models (DL), where the model bid is "nudged" depending on whether it was accepted or rejected, along with standard reinforcement learning models (RL). We found that DL fit the behaviour best and resulted in higher payoffs. We found the binary learning signal associated with DL to be represented by neural activity in the striatum distinctly posterior to a weaker reward prediction error signal. We posited that DL is an efficient heuristic for valuation when the action (bid) space is continuous. Indeed, we found that the posterior parietal cortex represents the continuous action space of the task, and the frontopolar prefrontal cortex distinguishes among conditions of social competition. Based on our findings, we proposed a conceptual model that accounts for a sequence of processes that are required to perform successful and flexible bidding under different types of competition.

Disturbance of approach-avoidance behaviors in non-human primates by stimulation of the limbic territories of basal ganglia and anterior insula.

The basal ganglia (BG) are involved in motivation and goal-directed behavior. Recent studies suggest that limbic territories of BG not only support reward seeking (appetitive approach) but also the encoding of aversive conditioned stimuli (CS) and the production of aversive-related behaviors (avoidance or escape). This study aimed to identify inside two BG nuclei, the striatum and pallidum, the territories involved in aversive behaviors and to compare the effects of stimulating these territories to those resulting from stimulation of the anterior Insula (aIns), a region that is well-known to be involved in aversive encoding and associated behaviors. Two monkeys performed an approach/avoidance task in which they had to choose a behavior (approach or avoidance) in an appetitive (reward) or aversive (air-puff) context. During this task, either one (single-cue) or two (dual-cue) CS provided essential information about which context-adapted behavior should be selected. Microstimulation was applied during the CS presentation. Stimulation generally reduced approaches in the appetitive contexts and increased escape behaviors (premature responses) and/or passive avoidance (noninitiated action) in aversive context. These effects were more pronounced in ventral parts of all examined structures, with significant differences observed between stimulated structures. Thresholds to induce effects were lowest in the pallidum. Striatal stimulation led to the largest diversity of effects, with a subregion even leading to enhanced active avoidance. Finally, aIns stimulations produced stronger effects in the dual-cue context. These results provide causal evidence that limbic territories of BG, like aIns, play crucial roles in the selection of context-motivated behaviors.

The Neuro-Computational Architecture of Value-Based Selection in the Human Brain.

Current neural models of value-based decision-making consider choices as a 2-stage process, proceeding from the "valuation" of each option under consideration to the "selection" of the best option on the basis of their subjective values. However, little is known about the computational mechanisms at play at the selection stage and its implementation in the human brain. Here, we used drift-diffusion models combined with model-based functional magnetic resonance imaging, effective connectivity, and multivariate pattern analysis to characterize the neuro-computational architecture of value-based decisions. We found that 2 key drift-diffusion computations at the selection stage, namely integration and choice readout, engage distinct brain regions, with the dorsolateral prefrontal cortex integrating a decision value signal computed in the ventromedial prefrontal cortex, and the posterior parietal cortex reading out choice outcomes. Our findings suggest that this prefronto-parietal network acts as a hub implementing behavioral selection through a distributed drift-diffusion process.

The Attraction Effect Modulates Reward Prediction Errors and Intertemporal Choices.

Classical economic theory contends that the utility of a choice option should be independent of other options. This view is challenged by the attraction effect, in which the relative preference between two options is altered by the addition of a third, asymmetrically dominated option. Here, we leveraged the attraction effect in the context of intertemporal choices to test whether both decisions and reward prediction errors (RPE) in the absence of choice violate the independence of irrelevant alternatives principle. We first demonstrate that intertemporal decision making is prone to the attraction effect in humans. In an independent group of participants, we then investigated how this affects the neural and behavioral valuation of outcomes using a novel intertemporal lottery task and fMRI. Participants' behavioral responses (i.e., satisfaction ratings) were modulated systematically by the attraction effect and this modulation was correlated across participants with the respective change of the RPE signal in the nucleus accumbens. Furthermore, we show that, because exponential and hyperbolic discounting models are unable to account for the attraction effect, recently proposed sequential sampling models might be more appropriate to describe intertemporal choices. Our findings demonstrate for the first time that the attraction effect modulates subjective valuation even in the absence of choice. The findings also challenge the prospect of using neuroscientific methods to measure utility in a context-free manner and have important implications for theories of reinforcement learning and delay discounting. SIGNIFICANCE STATEMENT: Many theories of value-based decision making assume that people first assess the attractiveness of each option independently of each other and then pick the option with the highest subjective value. The attraction effect, however, shows that adding a new option to a choice set can change the relative value of the existing options, which is a violation of the independence principle. Using an intertemporal choice framework, we tested whether such violations also occur when the brain encodes the difference between expected and received rewards (i.e., the reward prediction error). Our results suggest that neither intertemporal choice nor valuation without choice adhere to the independence principle.

Risk seeking for losses modulates the functional connectivity of the default mode and left frontoparietal networks in young males.

Value-based decision making (VBDM) is a principle that states that humans and other species adapt their behavior according to the dynamic subjective values of the chosen or unchosen options. The neural bases of this process have been extensively investigated using task-based fMRI and lesion studies. However, the growing field of resting-state functional connectivity (RSFC) may shed light on the organization and function of brain connections across different decision-making domains. With this aim, we used independent component analysis to study the brain network dynamics in a large cohort of young males (N = 145) and the relationship of these dynamics with VBDM. Participants completed a battery of behavioral tests that evaluated delay aversion, risk seeking for losses, risk aversion for gains, and loss aversion, followed by an RSFC scan session. We identified a set of large-scale brain networks and conducted our analysis only on the default mode network (DMN) and networks comprising cognitive control, appetitive-driven, and reward-processing regions. Higher risk seeking for losses was associated with increased connectivity between medial temporal regions, frontal regions, and the DMN. Higher risk seeking for losses was also associated with increased coupling between the left frontoparietal network and occipital cortices. These associations illustrate the participation of brain regions involved in prospective thinking, affective decision making, and visual processing in participants who are greater risk-seekers, and they demonstrate the sensitivity of RSFC to detect brain connectivity differences associated with distinct VBDM parameters.

Transcranial Stimulation Over the Dorsolateral Prefrontal Cortex Increases the Impact of Past Expenses on Decision-Making.

Goal-directed choices should be guided by the expected value of the available options. However, people are often influenced by past costs in their decisions, thus succumbing to a bias known as the "sunk-cost effect." Recent functional magnetic resonance imaging data show that the sunk-cost effect is associated with increased activity in dorsolateral prefrontal cortex (dlPFC) and altered crosstalk of the dlPFC with other prefrontal areas. Are these correlated neural processes causally involved in the sunk-cost effect? Here, we employed transcranial direct current stimulation (tDCS) to examine the role of the dlPFC for biasing choices in line with the cost of past expenses. Specifically, we applied different types of tDCS over the right dlPFC while participants performed an investment task designed to assess the impact of past investments on current choices. Our results show a pronounced sunk-cost effect that was significantly increased by anodal tDCS, but left unaltered by cathodal or sham stimulation. Importantly, choices were not affected by stimulation when no prior investments had been made, underlining the specificity of the obtained effect. Our findings suggest a critical role of the dlPFC in the sunk-cost effect and thus elucidate neural mechanisms by which past investments may influence current decision-making.

Neural mechanisms of cue-approach training.

Biasing choices may prove a useful way to implement behavior change. Previous work has shown that a simple training task (the cue-approach task), which does not rely on external reinforcement, can robustly influence choice behavior by biasing choice toward items that were targeted during training. In the current study, we replicate previous behavioral findings and explore the neural mechanisms underlying the shift in preferences following cue-approach training. Given recent successes in the development and application of machine learning techniques to task-based fMRI data, which have advanced understanding of the neural substrates of cognition, we sought to leverage the power of these techniques to better understand neural changes during cue-approach training that subsequently led to a shift in choice behavior. Contrary to our expectations, we found that machine learning techniques applied to fMRI data during non-reinforced training were unsuccessful in elucidating the neural mechanism underlying the behavioral effect. However, univariate analyses during training revealed that the relationship between BOLD and choices for Go items increases as training progresses compared to choices of NoGo items primarily in lateral prefrontal cortical areas. This new imaging finding suggests that preferences are shifted via differential engagement of task control networks that interact with value networks during cue-approach training.

Interaction of insular cortex and ventral striatum mediates the effect of incentive memory on choice between goal-directed actions.

The anterior insular cortex (IC) and the nucleus accumbens (NAc) core have been separately implicated in the selection and performance of actions based on the incentive value of the instrumental outcome. Here, we examined the role of connections between the IC and the NAc core in the performance of goal-directed actions. Rats were trained on two actions for distinct outcomes, after which one of the two outcomes was devalued by specific satiety immediately before a choice extinction test. We first confirmed the projection from the IC to the NAc core and then disconnected these structures via asymmetrical excitotoxic lesions before training. Contralateral, but not ipsilateral, disconnection of the IC and NAc core disrupted outcome devaluation. We hypothesized that communication between the IC and NAc core is necessary for the retrieval of incentive value at test. To test this, we infused the GABAA agonist muscimol into the IC and the µ-opioid receptor antagonist CTAP into the contralateral NAc before the choice extinction test. As expected, inactivation of the IC in one hemisphere and blocking µ-opioid receptors in the contralateral NAc core abolished outcome-selective devaluation. These results suggest that the IC and NAc core form part of a circuit mediating the retrieval of outcome values and the subsequent choice between goal-directed actions based on those values.

Neuronal variability in orbitofrontal cortex during economic decisions.

Neuroeconomic models assume that economic decisions are based on the activity of offer value cells in the orbitofrontal cortex (OFC), but testing this assertion has proven difficult. In principle, the decision made on a given trial should correlate with the stochastic fluctuations of these cells. However, this correlation, measured as a choice probability (CP), is small. Importantly, a neuron's CP reflects not only its individual contribution to the decision (termed readout weight), but also the intensity and the structure of correlated variability across the neuronal population (termed noise correlation). A precise mathematical relation between CPs, noise correlations, and readout weights was recently derived by Haefner and colleagues (Haefner RM, Gerwinn S, Macke JH, Bethge M. Nat Neurosci 16: 235-242, 2013) for a linear decision model. In this framework, concurrent measurements of noise correlations and CPs can provide quantitative information on how a population of cells contributes to a decision. Here we examined neuronal variability in the OFC of rhesus monkeys during economic decisions. Noise correlations had similar structure but considerably lower strength compared with those typically measured in sensory areas during perceptual decisions. In contrast, variability in the activity of individual cells was high and comparable to that recorded in other cortical regions. Simulation analyses based on Haefner's equation showed that noise correlations measured in the OFC combined with a plausible readout of offer value cells reproduced the experimental measures of CPs. In other words, the results obtained for noise correlations and those obtained for CPs taken together support the hypothesis that economic decisions are primarily based on the activity of offer value cells.