The Causes and Consequences of Drifting Expectations.

Awaiting news of uncertain outcomes is distressing because the news might be disappointing. To prevent such disappointments, people often "brace for the worst," pessimistically lowering expectations before news arrives to decrease the possibility of surprising disappointment (a negative prediction error, or PE). Computational decision-making research commonly assumes that expectations do not drift within trials, yet it is unclear whether expectations pessimistically drift in real-world, high-stakes settings, what factors influence expectation drift, and whether it effectively buffers emotional responses to goal-relevant outcomes. Moreover, individuals learn from PEs to accurately anticipate future outcomes, but it is unknown whether expectation drift also impedes PE-based learning. In a sample of students awaiting exam grades (N = 625), we found that expectations often drift and tend to drift pessimistically. We demonstrate that bracing is preferentially modulated by uncertainty; it transiently buffers the initial emotional impact of negative PEs but impairs PE-based learning, counterintuitively sustaining uncertainty into the future.

More than a feeling: physiological measures of affect index the integration of effort costs and rewards during anticipatory effort evaluation.

The notion that humans avoid effortful action is one of the oldest and most persistent in psychology. Influential theories of effort propose that effort valuations are made according to a cost-benefit trade-off: we tend to invest mental effort only when the benefits outweigh the costs. While these models provide a useful conceptual framework, the affective components of effort valuation remain poorly understood. Here, we examined whether primitive components of affective response-positive and negative valence, captured via facial electromyography (fEMG)-can be used to better understand valuations of cognitive effort. Using an effortful arithmetic task, we find that fEMG activity in the corrugator supercilii-thought to index negative valence-1) tracks the anticipation and exertion of cognitive effort and 2) is attenuated in the presence of high rewards. Together, these results suggest that activity in the corrugator reflects the integration of effort costs and rewards during effortful decision-making.

Cognitive effort exertion enhances electrophysiological responses to rewarding outcomes.

Recent work has highlighted neural mechanisms underlying cognitive effort-related discounting of anticipated rewards. However, findings on whether effort exertion alters the subjective value of obtained rewards are inconsistent. Here, we provide a more nuanced account of how cognitive effort affects subsequent reward processing in a novel task designed to assess effort-induced modulations of the Reward Positivity, an event-related potential indexing reward-related neural activity. We found that neural responses to both gains and losses were significantly elevated in trials requiring more versus less cognitive effort. Moreover, time-frequency analysis revealed that these effects were mirrored in gain-related delta, but not in loss-related theta band activity, suggesting that people ascribed more value to high-effort outcomes. In addition, we also explored whether individual differences in behavioral effort discounting rates and reward sensitivity in the absence of effort may affect the relationship between effort exertion and subsequent reward processing. Together, our findings provide evidence that cognitive effort exertion can increase the subjective value of subsequent outcomes and that this effect may primarily rely on modulations of delta band activity.

The Average Reward Rate Modulates Behavioral and Neural Indices of Effortful Control Allocation.

People tend to avoid exerting cognitive effort, and findings from recent behavioral studies suggest that effort allocation is in part determined by the opportunity cost of slothful responding-operationalized as the average reward rate per unit time. When the average rate of reward is high, individuals make more errors in cognitive control tasks, presumably owing to a withdrawal of costly cognitive processing. An open question remains whether the presumed modulations of cognitively effortful control processes are observable at the neural level. Here, we measured EEG while participants completed the Simon task, a well-known response conflict task, while the experienced average reward rate fluctuated across trials. We examined neural activity associated with the opportunity cost of time by applying generalized eigendecomposition, a hypothesis-driven source separation technique, to identify a midfrontal component associated with the average reward rate. Fluctuations in average reward rate modulated not only component amplitude but also, most importantly, component theta power (4-8 Hz). Higher average reward rate was associated with reduced theta power, suggesting that the opportunity of time modulates effort allocation. These neural results provide evidence for the idea that people strategically modulate the amount of cognitive effort they exert based on the opportunity cost of time.

Dopaminergic medication increases motivation to exert cognitive control by reducing subjective effort costs in Parkinson's patients.

Engaging in demanding mental activities requires the allocation of cognitive control, which can be effortful and aversive. Individuals thus tend to avoid exerting cognitive effort if less demanding behavioral options are available. Recent accounts propose a key role for dopamine in motivating behavior by increasing the sensitivity to rewards associated with effort exertion. Whether dopamine additionally plays a specific role in modulating the sensitivity to the costs of cognitive effort, even in the absence of any incentives, is much less clear. To address this question, we assessed cognitive effort avoidance in patients (n = 38) with Parkinson's disease, a condition characterized by loss of midbrain dopaminergic neurons, both ON and OFF dopaminergic medication and compared them to healthy controls (n = 24). Effort avoidance was assessed using the Demand Selection Task (DST), in which participants could freely choose between performing a high-demand or a low-demand version of a task-switching paradigm. Critically, participants were not offered any incentives to choose the more effortful option, nor for good performance. While healthy controls and patients OFF their dopaminergic medications consistently preferred the low-demand option, effort avoidance in patients ON dopaminergic medications was reduced compared to patients OFF, a difference which seems to lessen over trials. These differences in preference could not be explained by altered task-switching performance. Although patients ON were less accurate at detecting the different effort levels, as measured during instructed forced-choice blocks, their detection ability was not associated with effort avoidance, unlike in the healthy controls and the patients OFF. Our findings provide evidence that dopamine replacement in Parkinson's patients increases the willingness to engage in cognitively demanding behavior, and that this cannot be explained by possible effects of dopamine replacement on performance nor on the ability to detect effort demands. These results suggest that dopamine plays a role in reducing the sensitivity to effort costs that is independent of its role in enhancing the sensitivity to the benefits of effort exertion.

Task-evoked pupillary responses track effort exertion: Evidence from task-switching.

A spate of research has examined how individuals regulate effortful processing in service of goal-directed behaviors. One key challenge in developing an account of this regulation is quantifying the momentary amount of cognitive effort exerted by an individual in service of their goals. A growing body of literature has suggested using task-evoked pupil dilations as a potential psychophysiological index of cognitive effort; however, it remains unclear whether pupil diameter indexes effort exertion or merely reflects task load, as both are tightly intertwined. Here, we attempt to disentangle these disparate accounts of pupil diameter by leveraging individual differences in executive function (as measured by Stroop interference) and a motivational manipulation (i.e., monetary incentives) while participants complete a task-switching paradigm. In line with both the effort and demand accounts, we observed larger task-evoked pupillary responses (TEPRs) for trials in which there was a task switch versus a task repetition. Additionally, we found that larger phasic pupillary responses at baseline (without reward incentives) predicted smaller switch costs. Mirroring this pattern, individual differences in reward-induced switch cost reductions were predicted by reward-induced increases in phasic pupil diameter. Finally, we observed that the interrelationship between effort and pupil diameter at baseline was modulated by individual differences in Stroop interference costs. Together, these findings provide support for an effort account of TEPRs, and suggest that pupillometry is a viable index of cognitive effort.

Acute Psychosocial Stress Increases Cognitive-Effort Avoidance.

Adverse effects following acute stress are traditionally thought to reflect functional impairments of central executive-dependent cognitive-control processes. However, recent evidence demonstrates that cognitive-control application is perceived as effortful and aversive, indicating that stress-related decrements in cognitive performance could denote decreased motivation to expend effort instead. To investigate this hypothesis, we tested 40 young, healthy individuals (20 female, 20 male) under both stress and control conditions in a 2-day study that had a within-subjects design. Cognitive-effort avoidance was assessed using the demand-selection task, in which participants chose between performing low-demand and high-demand variants of a task-switching paradigm. We found that acute stress indeed increased participants' preference for less demanding behavior, whereas task-switching performance remained intact. Additional Bayesian and multiverse analyses confirmed the robustness of this effect. Our findings provide novel insights into how stressful experiences shape behavior by modulating our motivation to employ cognitive control.

Stress promotes generalization of older but not recent threat memories.

Stress broadly affects the ability to regulate emotions and may contribute to generalization of threat-related behaviors to harmless stimuli. Behavioral generalization also tends to increase over time as memory precision for recent events gives way to more gist-like representations. Thus, acute stress coupled with a delay in time from a negative experience may be a strong predictor of the transition from normal to generalized fear expression. Here, we investigated the effect of a single-episode acute stressor on generalization of aversive learning when stress is administered either immediately after an aversive learning event or following a delay. In a between-subjects design, healthy adult volunteers underwent threat (fear) conditioning using a tone-conditioned stimulus paired with an electric shock to the wrist and another tone not paired with shock. Behavioral generalization was tested to a range of novel tones either on the same day (experiment 1) or 24 h later (experiment 2) and was preceded by either an acute stress induction or a control task. Anticipatory sympathetic arousal [i.e., skin conductance responses (SCRs)] and explicit measures of shock expectancy served as dependent measures. Stress administered shortly after threat conditioning did not affect behavioral generalization. In contrast, stress administered following a delay led to heightened arousal and increased generalization of SCRs and explicit measures of shock expectancy. These findings show that acute stress increases generalization of older but not recent threat memories and have clinical relevance to understanding overgeneralization characteristics of anxiety and stress-related disorders.

Substance use is associated with reduced devaluation sensitivity.

Substance use has been linked to impairments in reward processing and decision-making, yet empirical research on the relationship between substance use and devaluation of reward in humans is limited. We report findings from two studies that tested whether individual differences in substance use behavior predicted reward learning strategies and devaluation sensitivity in a nonclinical sample. Participants in Experiment 1 (N = 66) and Experiment 2 (N = 91) completed subscales of the Externalizing Spectrum Inventory and then performed a two-stage reinforcement learning task that included a devaluation procedure. Spontaneous eye blink rate was used as an indirect proxy for dopamine functioning. In Experiment 1, correlational analysis revealed a negative relationship between substance use and devaluation sensitivity. In Experiment 2, regression modeling revealed that while spontaneous eyeblink rate moderated the relationship between substance use and reward learning strategies, substance use alone was related to devaluation sensitivity. These results suggest that once reward-action associations are established during reinforcement learning, substance use predicted reduced sensitivity to devaluation independently of variation in eyeblink rate. Thus, substance use is not only related to increased habit formation but also to difficulty disengaging from learned habits. Implications for the role of the dopaminergic system in habitual responding in individuals with substance use problems are discussed.

Modulating Episodic Memory Alters Risk Preference during Decision-making.

When choosing between options that vary in risk, we often rely on our experience with options-our episodic memories-to make that choice. Although episodic memory has been demonstrated to be critically involved in value-based decision-making, it is not clear how these memory processes contribute to decision-making that involves risk. To investigate this issue, we tested a group of participants on a repeated-choice risky decision-making task. Before completing this task, half of the participants were given a well-validated episodic induction task-a brief training procedure in recollecting the details of a past experience-known to engage episodic memory processes, and the other half were given a general impressions induction task. Our main finding was that risk-taking following the general impressions induction task was significantly lower than following the episodic induction task. In a follow-up experiment, we tested risk-taking in another group of participants without any prior induction task and found that risk-taking from this no-induction (baseline) group was more similar to the episodic induction than to the general impression group. Overall, these findings suggest engaging episodic memory processes when learning about decision outcomes can alter apparent risk-taking behavior in decision-making from experience.

Unexpected but Incidental Positive Outcomes Predict Real-World Gambling.

Positive mood can affect a person's tendency to gamble, possibly because positive mood fosters unrealistic optimism. At the same time, unexpected positive outcomes, often called prediction errors, influence mood. However, a linkage between positive prediction errors-the difference between expected and obtained outcomes-and consequent risk taking has yet to be demonstrated. Using a large data set of New York City lottery gambling and a model inspired by computational accounts of reward learning, we found that people gamble more when incidental outcomes in the environment (e.g., local sporting events and sunshine) are better than expected. When local sports teams performed better than expected, or a sunny day followed a streak of cloudy days, residents gambled more. The observed relationship between prediction errors and gambling was ubiquitous across the city's socioeconomically diverse neighborhoods and was specific to sports and weather events occurring locally in New York City. Our results suggest that unexpected but incidental positive outcomes influence risk taking.

From Creatures of Habit to Goal-Directed Learners: Tracking the Developmental Emergence of Model-Based Reinforcement Learning.

Theoretical models distinguish two decision-making strategies that have been formalized in reinforcement-learning theory. A model-based strategy leverages a cognitive model of potential actions and their consequences to make goal-directed choices, whereas a model-free strategy evaluates actions based solely on their reward history. Research in adults has begun to elucidate the psychological mechanisms and neural substrates underlying these learning processes and factors that influence their relative recruitment. However, the developmental trajectory of these evaluative strategies has not been well characterized. In this study, children, adolescents, and adults performed a sequential reinforcement-learning task that enabled estimation of model-based and model-free contributions to choice. Whereas a model-free strategy was apparent in choice behavior across all age groups, a model-based strategy was absent in children, became evident in adolescents, and strengthened in adults. These results suggest that recruitment of model-based valuation systems represents a critical cognitive component underlying the gradual maturation of goal-directed behavior.

Working-memory capacity protects model-based learning from stress.

Accounts of decision-making have long posited the operation of separate, competing valuation systems in the control of choice behavior. Recent theoretical and experimental advances suggest that this classic distinction between habitual and goal-directed (or more generally, automatic and controlled) choice may arise from two computational strategies for reinforcement learning, called model-free and model-based learning. Popular neurocomputational accounts of reward processing emphasize the involvement of the dopaminergic system in model-free learning and prefrontal, central executive-dependent control systems in model-based choice. Here we hypothesized that the hypothalamic-pituitary-adrenal (HPA) axis stress response--believed to have detrimental effects on prefrontal cortex function--should selectively attenuate model-based contributions to behavior. To test this, we paired an acute stressor with a sequential decision-making task that affords distinguishing the relative contributions of the two learning strategies. We assessed baseline working-memory (WM) capacity and used salivary cortisol levels to measure HPA axis stress response. We found that stress response attenuates the contribution of model-based, but not model-free, contributions to behavior. Moreover, stress-induced behavioral changes were modulated by individual WM capacity, such that low-WM-capacity individuals were more susceptible to detrimental stress effects than high-WM-capacity individuals. These results enrich existing accounts of the interplay between acute stress, working memory, and prefrontal function and suggest that executive function may be protective against the deleterious effects of acute stress.

Cognitive control predicts use of model-based reinforcement learning.

Accounts of decision-making and its neural substrates have long posited the operation of separate, competing valuation systems in the control of choice behavior. Recent theoretical and experimental work suggest that this classic distinction between behaviorally and neurally dissociable systems for habitual and goal-directed (or more generally, automatic and controlled) choice may arise from two computational strategies for reinforcement learning (RL), called model-free and model-based RL, but the cognitive or computational processes by which one system may dominate over the other in the control of behavior is a matter of ongoing investigation. To elucidate this question, we leverage the theoretical framework of cognitive control, demonstrating that individual differences in utilization of goal-related contextual information--in the service of overcoming habitual, stimulus-driven responses--in established cognitive control paradigms predict model-based behavior in a separate, sequential choice task. The behavioral correspondence between cognitive control and model-based RL compellingly suggests that a common set of processes may underpin the two behaviors. In particular, computational mechanisms originally proposed to underlie controlled behavior may be applicable to understanding the interactions between model-based and model-free choice behavior.

Model-based learning protects against forming habits.

Studies in humans and rodents have suggested that behavior can at times be "goal-directed"-that is, planned, and purposeful-and at times "habitual"-that is, inflexible and automatically evoked by stimuli. This distinction is central to conceptions of pathological compulsion, as in drug abuse and obsessive-compulsive disorder. Evidence for the distinction has primarily come from outcome devaluation studies, in which the sensitivity of a previously learned behavior to motivational change is used to assay the dominance of habits versus goal-directed actions. However, little is known about how habits and goal-directed control arise. Specifically, in the present study we sought to reveal the trial-by-trial dynamics of instrumental learning that would promote, and protect against, developing habits. In two complementary experiments with independent samples, participants completed a sequential decision task that dissociated two computational-learning mechanisms, model-based and model-free. We then tested for habits by devaluing one of the rewards that had reinforced behavior. In each case, we found that individual differences in model-based learning predicted the participants' subsequent sensitivity to outcome devaluation, suggesting that an associative mechanism underlies a bias toward habit formation in healthy individuals.

Physiological and behavioral signatures of reflective exploratory choice.

Physiological arousal, a marker of emotional response, has been demonstrated to accompany human decision making under uncertainty. Anticipatory emotions have been portrayed as basic and rapid evaluations of chosen actions. Instead, could these arousal signals stem from a "cognitive" assessment of value that utilizes the full environment structure, as opposed to merely signaling a coarse, reflexive assessment of the possible consequences of choices? Combining an exploration-exploitation task, computational modeling, and skin conductance measurements, we find that physiological arousal manifests a reflective assessment of the benefit of the chosen action, mirroring observed behavior. Consistent with the level of computational sophistication evident in these signals, a follow-up experiment demonstrates that anticipatory arousal is modulated by current environment volatility, in accordance with the predictions of our computational account. Finally, we examine the cognitive costs of the exploratory choice behavior these arousal signals accompany by manipulating concurrent cognitive demand. Taken together, these results demonstrate that the arousal that accompanies choice under uncertainty arises from a more reflective and "cognitive" assessment of the chosen action's consequences than has been revealed previously.

Proximity to rewards modulates parameters of effortful control exertion.

The now-classic goal-gradient hypothesis posits that organisms increase effort expenditure as a function of their proximity to a goal. Despite nearly a century having passed since its original formulation, goal-gradient-like behavior in human cognitive performance remains poorly understood: Are we more willing to engage in costly cognitive processing when we are near, versus far, from a goal state? Moreover, the computational mechanisms underpinning these potential goal-gradient effects-for example, whether goal proximity affects fidelity of stimulus encoding, response caution, or other identifiable mechanisms governing speed and accuracy-are unclear. Here, in two experiments, we examine the effect of goal proximity, operationalized as progress toward the completion of a rewarded task block, upon task performance in an attentionally demanding oddball task. Supporting the goal-gradient hypothesis, we found that participants responded more quickly, but not less accurately, when rewards were proximal than when they were distal. Critically, this effect was only observed when participants were given information about goal proximity. Using hierarchical drift diffusion modeling, we found that these apparent goal-gradient performance effects were best explained by a collapsing bound model, in which proximity to a goal reduced response caution and increased information processing. Taken together, these results suggest that goal gradients could help explain the oft-observed fluctuations in engagement of cognitively effortful processing, extending the scope of the goal-gradient hypothesis to the domain of cognitive tasks. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Distinguishing between intrinsic and instrumental sources of the value of choice.

Considerable evidence suggests that people value the freedom to choose. However, it is unclear whether this preference for choice stems purely from choice's intrinsic value, or whether people prefer to choose because it tends to provide instrumental information about desirable outcomes. To address this question, participants completed a novel choice task in which they could freely choose to exert choice or not, manipulating the level of instrumental contingency between participants' choices and eventual outcomes, which we operationalized using the information-theoretic concept of mutual information. Across two experiments (N = 100 each), we demonstrate a marked preference for choice, but importantly found that participants' preference for free choice is weakened when actions are decoupled from outcomes. Taken together, our results demonstrate that a significant factor in people's preference for choice is an assumption about the instrumental value of choice, suggesting against a purely intrinsic value of choice.

Greater reliance on model-free learning in adolescent anorexia nervosa: An examination of dual-system reinforcement learning.

Alterations in learning and decision-making systems are thought to contribute to core features of anorexia nervosa (AN), a psychiatric disorder characterized by persistent dietary restriction and weight loss. Instrumental learning theory identifies a dual-system of habit and goal-directed decision-making, linked to model-free and model-based reinforcement learning algorithms. Difficulty arbitrating between these systems, resulting in an over-reliance on one strategy over the other, has been implicated in compulsivity and extreme goal pursuit, both of which are observed in AN. Characterizing alterations in model-free and model-based systems, and their neural correlates, in AN may clarify mechanisms contributing to symptom heterogeneity (e.g., binge/purge symptoms). This study tested whether adolescents with restricting AN (AN-R; n = 36) and binge/purge AN (AN-BP; n = 20) differentially utilized model-based and model-free learning systems compared to a healthy control group (HC; n = 28) during a Markov two-step decision-making task under conditions of reward and punishment. Associations between model-free and model-based learning and resting-state functional connectivity between neural regions of interest, including orbitofrontal cortex (OFC), nucleus accumbens (NAcc), putamen, and sensory motor cortex (SMC) were examined. AN-R showed higher utilization of model-free learning compared to HC for reward, but attenuated model-free and model-based learning for punishment. In AN-R only, higher model-based learning was associated with stronger OFC-to-left NAcc functional connectivity, regions linked to goal-directed behavior. Greater utilization of model-free learning for reward in AN-R may differentiate this group, particularly during adolescence, and facilitate dietary restriction by prioritizing habitual control in rewarding contexts.