Brain mechanism of foraging: Reward-dependent synaptic plasticity versus neural integration of values.

During foraging behavior, action values are persistently encoded in neural activity and updated depending on the history of choice outcomes. What is the neural mechanism for action value maintenance and updating? Here, we explore two contrasting network models: synaptic learning of action value versus neural integration. We show that both models can reproduce extant experimental data, but they yield distinct predictions about the underlying biological neural circuits. In particular, the neural integrator model but not the synaptic model requires that reward signals are mediated by neural pools selective for action alternatives and their projections are aligned with linear attractor axes in the valuation system. We demonstrate experimentally observable neural dynamical signatures and feasible perturbations to differentiate the two contrasting scenarios, suggesting that the synaptic model is a more robust candidate mechanism. Overall, this work provides a modeling framework to guide future experimental research on probabilistic foraging.

Acetylcholine and noradrenaline enhance foraging optimality in humans.

Foraging theory prescribes when optimal foragers should leave the current option for more rewarding alternatives. Actual foragers often exploit options longer than prescribed by the theory, but it is unclear how this foraging suboptimality arises. We investigated whether the upregulation of cholinergic, noradrenergic, and dopaminergic systems increases foraging optimality. In a double-blind, between-subject design, participants (N = 160) received placebo, the nicotinic acetylcholine receptor agonist nicotine, a noradrenaline reuptake inhibitor reboxetine, or a preferential dopamine reuptake inhibitor methylphenidate, and played the role of a farmer who collected milk from patches with different yield. Across all groups, participants on average overharvested. While methylphenidate had no effects on this bias, nicotine, and to some extent also reboxetine, significantly reduced deviation from foraging optimality, which resulted in better performance compared to placebo. Concurring with amplified goal-directedness and excluding heuristic explanations, nicotine independently also improved trial initiation and time perception. Our findings elucidate the neurochemical basis of behavioral flexibility and decision optimality and open unique perspectives on psychiatric disorders affecting these functions.

A special role for anterior cingulate cortex, but not orbitofrontal cortex or basolateral amygdala, in choices involving information.

Subjects are often willing to pay a cost for information. In a procedure that promotes paradoxical choices, animals choose between a richer option followed by a cue that is rewarded 50% of the time (No Info) vs. a leaner option followed by one of two cues that signal certain outcomes: one always rewarded (100%) and the other never rewarded, 0% (Info). Since decisions involve comparing the subjective value of options after integrating all their features, preference for information may rely on cortico-amygdalar circuitry. To test this, male and female rats were prepared with bilateral inhibitory Designer Receptors Exclusively Activated by Designer Drugs (DREADDs) in the anterior cingulate cortex, orbitofrontal cortex, basolateral amygdala, or null virus (control). We inhibited these regions after stable preference was acquired. We found that inhibition of the anterior cingulate cortex destabilized choice preference in female rats without affecting latency to choose or response rate to cues. A logistic regression fit revealed that previous choice predicted current choice in all conditions, however previously rewarded Info trials strongly predicted preference in all conditions except in female rats following anterior cingulate cortex inhibition. The results reveal a causal, sex-dependent role for the anterior cingulate cortex in decisions involving information.

Mechanisms of adjustments to different types of uncertainty in the reward environment across mice and monkeys.

Despite being unpredictable and uncertain, reward environments often exhibit certain regularities, and animals navigating these environments try to detect and utilize such regularities to adapt their behavior. However, successful learning requires that animals also adjust to uncertainty associated with those regularities. Here, we analyzed choice data from two comparable dynamic foraging tasks in mice and monkeys to investigate mechanisms underlying adjustments to different types of uncertainty. In these tasks, animals selected between two choice options that delivered reward probabilistically, while baseline reward probabilities changed after a variable number (block) of trials without any cues to the animals. To measure adjustments in behavior, we applied multiple metrics based on information theory that quantify consistency in behavior, and fit choice data using reinforcement learning models. We found that in both species, learning and choice were affected by uncertainty about reward outcomes (in terms of determining the better option) and by expectation about when the environment may change. However, these effects were mediated through different mechanisms. First, more uncertainty about the better option resulted in slower learning and forgetting in mice, whereas it had no significant effect in monkeys. Second, expectation of block switches accompanied slower learning, faster forgetting, and increased stochasticity in choice in mice, whereas it only reduced learning rates in monkeys. Overall, while demonstrating the usefulness of metrics based on information theory in examining adaptive behavior, our study provides evidence for multiple types of adjustments in learning and choice behavior according to uncertainty in the reward environment.

Cognitive effort exertion enhances electrophysiological responses to rewarding outcomes.

Recent work has highlighted neural mechanisms underlying cognitive effort-related discounting of anticipated rewards. However, findings on whether effort exertion alters the subjective value of obtained rewards are inconsistent. Here, we provide a more nuanced account of how cognitive effort affects subsequent reward processing in a novel task designed to assess effort-induced modulations of the Reward Positivity, an event-related potential indexing reward-related neural activity. We found that neural responses to both gains and losses were significantly elevated in trials requiring more versus less cognitive effort. Moreover, time-frequency analysis revealed that these effects were mirrored in gain-related delta, but not in loss-related theta band activity, suggesting that people ascribed more value to high-effort outcomes. In addition, we also explored whether individual differences in behavioral effort discounting rates and reward sensitivity in the absence of effort may affect the relationship between effort exertion and subsequent reward processing. Together, our findings provide evidence that cognitive effort exertion can increase the subjective value of subsequent outcomes and that this effect may primarily rely on modulations of delta band activity.

Attempts to Influence the Value of Alcohol by Manipulating Social Influence and Context.

Background: Recent cognitive neuroscience models of value-based decision-making suggest value-based choices for alcohol are sensitive to various inputs, such as context and social influence. In two online experiments, we tested whether manipulating these inputs influenced proxies for alcohol value. Experiment 1: 157 social drinkers were presented with 4 hypothetical scenarios (drinking alone, with friends who are also drinking, with friends but trying to "cut-down" for health reasons, with friends who aren't drinking) in a within-subjects design, and completed the Brief Assessment of Alcohol Demand after each as a measure of value. Value for alcohol (number of drinks purchased) was greatest when drinking with friends who were also drinking compared to drinking alone (d = 0.95), friends not drinking (d = 1.49) and friends drinking/health related (d = 1.59). Value for alcohol was also greater when drinking alone compared to with friends who were not drinking (d = 0.55), and also with friends drinking/health related (d = 0.62). Experiment 2: 241 participants were randomly allocated to see one of four categories of images in a 2 (context: bar vs house) x 2 (social influence: enjoy vs not enjoy) design, before completing a Concurrent Choice Task for alcohol and Visual Analog Scales. There were no significant effects found on either task, both taken as proxies for value. Conclusion: There was inconclusive evidence that the value for alcohol could be manipulated by social context. This could be explained by greater saliency of the manipulation in asking participants to imagine themselves in a hypothetical situation as opposed to presenting images depicting drinking scenarios.

Contribution of the sensorimotor beta oscillations and the cortico-basal ganglia-thalamic circuitry during value-based decision making: A simultaneous EEG-fMRI investigation.

In decision neuroscience, the motor system has primarily been considered to be involved in executing choice actions. However, a competing perspective suggests its engagement in the evaluation of options, traditionally considered to be performed by the brain's valuation system. Here, we investigate the role of the motor system in value-based decision making by determining the neural circuitries associated with the sensorimotor beta oscillations previously identified to encode decision options. In a simultaneous EEG-fMRI study, participants evaluated reward and risk associated with a forthcoming action. A significant sensorimotor beta desynchronization was identified prior to and independent of response. The level of beta desynchronization showed evidence of encoding the reward levels. This beta desynchronization covaried, on a trial-by-trial level, with BOLD activity in the cortico-basal ganglia-thalamic circuitry. In contrast, there was only a weak covariation within the valuation network, despite significant modulation of its BOLD activity by reward levels. These results suggest that the way in which decision variables are processed differs in the valuation network and in the cortico-basal ganglia-thalamic circuitry. We propose that sensorimotor beta oscillations indicate incentive motivational drive towards a choice action computed from the decision variables even prior to making a response, and it arises from the cortico-basal ganglia-thalamic circuitry.

Disinvestment in healthcare: a scoping review of systematic reviews.

OBJECTIVES: Disinvestment from low value health technologies is growing globally. Diverse evidence gathering and assessment methods were used to implement disinvestment initiatives, however, less than half of the empirical studies report reduced use of the low-value services. This scoping review aimed to synthesize the information from available reviews on the concepts and purposes of disinvestment in healthcare, the approaches and methods used, the role of stakeholders and facilitators and barriers in its implementation. METHODS: This scoping review was guided by the Joanna Briggs Institute Manual for Evidence Synthesis and PRISMA statement for scoping review. Published reviews on disinvestment were identified from scientific databases including health technology assessment (HTA) Web sites using the terms "disinvestment," "health technology reassessment," and "healthcare." The data obtained was synthesized narratively to identify similarities and differences across the approaches based on the prespecified categories. RESULTS: Seventeen reviews were included with thirty-four initiatives identified across sixteen countries at various levels of implementation and responsible agencies for the activities. Two most used methods to facilitate disinvestment decisions are Programme Budgeting and Marginal Analysis (PBMA) and HTA. Stakeholder involvement is the most important aspect to be addressed, as it acts as both facilitator and barrier in disinvestment initiatives implementation. CONCLUSIONS: Disinvestment programs have been implemented at multilevel, involving multistakeholders and using multiple methods such as PBMA and HTA. However, there is a lack of clarity on the additional dimensions of technical analysis related to these tools. Further research could focus on technology optimization in healthcare as part of overall health technology management.

Real-Life Self-Control is Predicted by Parietal Activity During Preference Decision Making: A Brain Decoding Analysis.

Despite its relevance for health and education, the neurocognitive mechanism of real-life self-control is largely unknown. While recent research revealed a prominent role of the ventromedial prefrontal cortex in the computation of an integrative value signal, the contribution and relevance of other brain regions for real-life self-control remains unclear. To investigate neural correlates of decisions in line with long-term consequences and to assess the potential of brain decoding methods for the individual prediction of real-life self-control, we combined functional magnetic resonance imaging during preference decision making with ecological momentary assessment of daily self-control in a large community sample (N = 266). Decisions in line with long-term consequences were associated with increased activity in bilateral angular gyrus and precuneus, regions involved in different forms of perspective taking, such as imagining one's own future and the perspective of others. Applying multivariate pattern analysis to the same clusters revealed that individual patterns of activity predicted the probability of real-life self-control. Brain activations are discussed in relation to episodic future thinking and mentalizing as potential mechanisms mediating real-life self-control.

Corticostriatal White Matter Integrity and Dopamine D1 Receptor Availability Predict Age Differences in Prefrontal Value Signaling during Reward Learning.

Probabilistic reward learning reflects the ability to adapt choices based on probabilistic feedback. The dopaminergically innervated corticostriatal circuit in the brain plays an important role in supporting successful probabilistic reward learning. Several components of the corticostriatal circuit deteriorate with age, as it does probabilistic reward learning. We showed previously that D1 receptor availability in NAcc predicts the strength of anticipatory value signaling in vmPFC, a neural correlate of probabilistic learning that is attenuated in older participants and predicts probabilistic reward learning performance. We investigated how white matter integrity in the pathway between nucleus accumbens (NAcc) and ventromedial prefrontal cortex (vmPFC) relates to the strength of anticipatory value signaling in vmPFC in younger and older participants. We found that in a sample of 22 old and 23 young participants, fractional anisotropy in the pathway between NAcc and vmPFC predicted the strength of value signaling in vmPFC independently from D1 receptor availability in NAcc. These findings provide tentative evidence that integrity in the dopaminergic and white matter pathways of corticostriatal circuitry supports the expression of value signaling in vmPFC which supports reward learning, however, the limited sample size calls for independent replication. These and future findings could add to the improved understanding of how corticostriatal integrity contributes to reward learning ability.

Neural underpinnings of value-guided choice during auction tasks: An eye-fixation related potentials study.

Values are attributed to goods during free viewing of objects which entails multi- and trans-saccadic cognitive processes. Using electroencephalographic eye-fixation related potentials, the present study investigated how neural signals related to value-guided choice evolved over time when viewing household and office products during an auction task. Participants completed a Becker-DeGroot-Marschak auction task whereby half of the stimuli were presented in either a free or forced bid protocol to obtain willingness-to-pay. Stimuli were assigned to three value categories of low, medium and high value based on subjective willingness-to-pay. Eye fixations were organised into five 800 ms time-bins spanning the objects total viewing time. Independent component analysis was applied to eye-fixation related potentials. One independent component (IC) was found to represent fixations for high value products with increased activation over the left parietal region of the scalp. An IC with a spatial maximum over a fronto-central region of the scalp coded the intermediate values. Finally, one IC displaying activity that extends over the right frontal scalp region responded to intermediate- and low-value items. Each of these components responded early on during viewing an object and remained active over the entire viewing period, both during free and forced bid trials. Results suggest that the subjective value of goods are encoded using sets of brain activation patterns which are tuned to respond uniquely to either low, medium, or high values. Data indicates that the right frontal region of the brain responds to low and the left frontal region to high values. Values of goods are determined at an early point in the decision making process and carried for the duration of the decision period via trans-saccadic processes.

Action in auctions: neural and computational mechanisms of bidding behaviour.

Competition for resources is a fundamental characteristic of evolution. Auctions have been widely used to model competition of individuals for resources, and bidding behaviour plays a major role in social competition. Yet, how humans learn to bid efficiently remains an open question. We used model-based neuroimaging to investigate the neural mechanisms of bidding behaviour under different types of competition. Twenty-seven subjects (nine male) played a prototypical bidding game: a double action, with three "market" types, which differed in the number of competitors. We compared different computational learning models of bidding: directional learning models (DL), where the model bid is "nudged" depending on whether it was accepted or rejected, along with standard reinforcement learning models (RL). We found that DL fit the behaviour best and resulted in higher payoffs. We found the binary learning signal associated with DL to be represented by neural activity in the striatum distinctly posterior to a weaker reward prediction error signal. We posited that DL is an efficient heuristic for valuation when the action (bid) space is continuous. Indeed, we found that the posterior parietal cortex represents the continuous action space of the task, and the frontopolar prefrontal cortex distinguishes among conditions of social competition. Based on our findings, we proposed a conceptual model that accounts for a sequence of processes that are required to perform successful and flexible bidding under different types of competition.

The Attraction Effect Modulates Reward Prediction Errors and Intertemporal Choices.

Classical economic theory contends that the utility of a choice option should be independent of other options. This view is challenged by the attraction effect, in which the relative preference between two options is altered by the addition of a third, asymmetrically dominated option. Here, we leveraged the attraction effect in the context of intertemporal choices to test whether both decisions and reward prediction errors (RPE) in the absence of choice violate the independence of irrelevant alternatives principle. We first demonstrate that intertemporal decision making is prone to the attraction effect in humans. In an independent group of participants, we then investigated how this affects the neural and behavioral valuation of outcomes using a novel intertemporal lottery task and fMRI. Participants' behavioral responses (i.e., satisfaction ratings) were modulated systematically by the attraction effect and this modulation was correlated across participants with the respective change of the RPE signal in the nucleus accumbens. Furthermore, we show that, because exponential and hyperbolic discounting models are unable to account for the attraction effect, recently proposed sequential sampling models might be more appropriate to describe intertemporal choices. Our findings demonstrate for the first time that the attraction effect modulates subjective valuation even in the absence of choice. The findings also challenge the prospect of using neuroscientific methods to measure utility in a context-free manner and have important implications for theories of reinforcement learning and delay discounting. SIGNIFICANCE STATEMENT: Many theories of value-based decision making assume that people first assess the attractiveness of each option independently of each other and then pick the option with the highest subjective value. The attraction effect, however, shows that adding a new option to a choice set can change the relative value of the existing options, which is a violation of the independence principle. Using an intertemporal choice framework, we tested whether such violations also occur when the brain encodes the difference between expected and received rewards (i.e., the reward prediction error). Our results suggest that neither intertemporal choice nor valuation without choice adhere to the independence principle.

Risk seeking for losses modulates the functional connectivity of the default mode and left frontoparietal networks in young males.

Value-based decision making (VBDM) is a principle that states that humans and other species adapt their behavior according to the dynamic subjective values of the chosen or unchosen options. The neural bases of this process have been extensively investigated using task-based fMRI and lesion studies. However, the growing field of resting-state functional connectivity (RSFC) may shed light on the organization and function of brain connections across different decision-making domains. With this aim, we used independent component analysis to study the brain network dynamics in a large cohort of young males (N = 145) and the relationship of these dynamics with VBDM. Participants completed a battery of behavioral tests that evaluated delay aversion, risk seeking for losses, risk aversion for gains, and loss aversion, followed by an RSFC scan session. We identified a set of large-scale brain networks and conducted our analysis only on the default mode network (DMN) and networks comprising cognitive control, appetitive-driven, and reward-processing regions. Higher risk seeking for losses was associated with increased connectivity between medial temporal regions, frontal regions, and the DMN. Higher risk seeking for losses was also associated with increased coupling between the left frontoparietal network and occipital cortices. These associations illustrate the participation of brain regions involved in prospective thinking, affective decision making, and visual processing in participants who are greater risk-seekers, and they demonstrate the sensitivity of RSFC to detect brain connectivity differences associated with distinct VBDM parameters.

Transcranial Stimulation Over the Dorsolateral Prefrontal Cortex Increases the Impact of Past Expenses on Decision-Making.

Goal-directed choices should be guided by the expected value of the available options. However, people are often influenced by past costs in their decisions, thus succumbing to a bias known as the "sunk-cost effect." Recent functional magnetic resonance imaging data show that the sunk-cost effect is associated with increased activity in dorsolateral prefrontal cortex (dlPFC) and altered crosstalk of the dlPFC with other prefrontal areas. Are these correlated neural processes causally involved in the sunk-cost effect? Here, we employed transcranial direct current stimulation (tDCS) to examine the role of the dlPFC for biasing choices in line with the cost of past expenses. Specifically, we applied different types of tDCS over the right dlPFC while participants performed an investment task designed to assess the impact of past investments on current choices. Our results show a pronounced sunk-cost effect that was significantly increased by anodal tDCS, but left unaltered by cathodal or sham stimulation. Importantly, choices were not affected by stimulation when no prior investments had been made, underlining the specificity of the obtained effect. Our findings suggest a critical role of the dlPFC in the sunk-cost effect and thus elucidate neural mechanisms by which past investments may influence current decision-making.

Neural mechanisms of cue-approach training.

Biasing choices may prove a useful way to implement behavior change. Previous work has shown that a simple training task (the cue-approach task), which does not rely on external reinforcement, can robustly influence choice behavior by biasing choice toward items that were targeted during training. In the current study, we replicate previous behavioral findings and explore the neural mechanisms underlying the shift in preferences following cue-approach training. Given recent successes in the development and application of machine learning techniques to task-based fMRI data, which have advanced understanding of the neural substrates of cognition, we sought to leverage the power of these techniques to better understand neural changes during cue-approach training that subsequently led to a shift in choice behavior. Contrary to our expectations, we found that machine learning techniques applied to fMRI data during non-reinforced training were unsuccessful in elucidating the neural mechanism underlying the behavioral effect. However, univariate analyses during training revealed that the relationship between BOLD and choices for Go items increases as training progresses compared to choices of NoGo items primarily in lateral prefrontal cortical areas. This new imaging finding suggests that preferences are shifted via differential engagement of task control networks that interact with value networks during cue-approach training.

Interaction of insular cortex and ventral striatum mediates the effect of incentive memory on choice between goal-directed actions.

The anterior insular cortex (IC) and the nucleus accumbens (NAc) core have been separately implicated in the selection and performance of actions based on the incentive value of the instrumental outcome. Here, we examined the role of connections between the IC and the NAc core in the performance of goal-directed actions. Rats were trained on two actions for distinct outcomes, after which one of the two outcomes was devalued by specific satiety immediately before a choice extinction test. We first confirmed the projection from the IC to the NAc core and then disconnected these structures via asymmetrical excitotoxic lesions before training. Contralateral, but not ipsilateral, disconnection of the IC and NAc core disrupted outcome devaluation. We hypothesized that communication between the IC and NAc core is necessary for the retrieval of incentive value at test. To test this, we infused the GABAA agonist muscimol into the IC and the µ-opioid receptor antagonist CTAP into the contralateral NAc before the choice extinction test. As expected, inactivation of the IC in one hemisphere and blocking µ-opioid receptors in the contralateral NAc core abolished outcome-selective devaluation. These results suggest that the IC and NAc core form part of a circuit mediating the retrieval of outcome values and the subsequent choice between goal-directed actions based on those values.

Characterizing Human Habits in the Lab.

Habits pose a fundamental puzzle for those aiming to understand human behavior. They pervade our everyday lives and dominate some forms of psychopathology but are extremely hard to elicit in the lab. In this Registered Report, we developed novel experimental paradigms grounded in computational models, which suggest that habit strength should be proportional to the frequency of behavior and, in contrast to previous research, independent of value. Specifically, we manipulated how often participants performed responses in two tasks varying action repetition without, or separately from, variations in value. Moreover, we asked how this frequency-based habitization related to value-based operationalizations of habit and self-reported propensities for habitual behavior in real life. We find that choice frequency during training increases habit strength at test and that this form of habit shows little relation to value-based operationalizations of habit. Our findings empirically ground a novel perspective on the constituents of habits and suggest that habits may arise in the absence of external reinforcement. We further find no evidence for an overlap between different experimental approaches to measuring habits and no associations with self-reported real-life habits. Thus, our findings call for a rigorous reassessment of our understanding and measurement of human habitual behavior in the lab.

Dual-tasking modulates movement speed but not value-based choices during walking.

Value-based decision-making often occurs in multitasking scenarios relying on both cognitive and motor processes. Yet, laboratory experiments often isolate these processes, thereby neglecting potential interactions. This isolated approach reveals a dichotomy: the cognitive process by which reward influences decision-making is capacity-limited, whereas the influence of motor cost is free of such constraints. If true, dual-tasking should predominantly impair reward processing but not affect the impact of motor costs. To test this hypothesis, we designed a decision-making task in which participants made choices to walk toward targets for rewards while navigating past an obstacle. The motor cost to reach these rewards varied in real-time. Participants either solely performed the decision-making task, or additionally performed a secondary pitch-recall task. Results revealed that while both reward and motor costs influenced decision-making, the secondary task did not affect these factors. Instead, dual-tasking slowed down participants' walking, thereby reducing the overall reward rate. Hence, contrary to the prediction that the added cognitive demand would affect the weighing of reward or motor cost differentially, these processes seem to be maintained at the expense of slowing down the motor system. This slowdown may be indicative of interference at the locomotor level, thereby underpinning motor-cognitive interactions during decision-making.

Value of freedom to choose encoded by the human brain.

Humans and animals value the opportunity to choose by preferring alternatives that offer more rather than fewer choices. This preference for choice may arise not only from an increased probability of obtaining preferred outcomes but also from the freedom it provides. We used human neuroimaging to investigate the neural basis of the preference for choice as well as for the items that could be chosen. In each trial, participants chose between two options, a monetary amount option and a "choice option." The latter consisted of a number that corresponded to the number of everyday items participants would subsequently be able to choose from. We found that the opportunity to choose from a larger number of items was equivalent to greater amounts of money, indicating that participants valued having more choice; moreover, participants varied in the degree to which they valued having the opportunity to choose, with some valuing it more than the increased probability of obtaining preferred items. Neural activations in the mid striatum increased with the value of the opportunity to choose. The same region also coded the value of the items. Conversely, activation in the dorsolateral striatum was not related to the value of the items but was elevated when participants were offered more choices, particularly in those participants who overvalued the opportunity to choose. These data suggest a functional dissociation of value representations within the striatum, with general representations in mid striatum and specific representations of the value of freedom provided by the opportunity to choose in dorsolateral striatum.