Lesions to different regions of frontal cortex have dissociable effects on voluntary persistence.

Deciding how long to keep waiting for uncertain future rewards is a complex problem. Previous research has shown that choosing to stop waiting results from an evaluative process that weighs the subjective value of the awaited reward against the opportunity cost of waiting. In functional neuroimaging data, activity in ventromedial prefrontal cortex (vmPFC) tracks the dynamics of this evaluation, while activation in the dorsomedial prefrontal cortex (dmPFC) and anterior insula (AI) ramps up before a decision to quit is made. Here, we provide causal evidence of the necessity of these brain regions for successful performance in a willingness-to-wait task. 28 participants with frontal lobe lesions were tested on their ability to adaptively calibrate how long they waited for monetary rewards. We grouped the participants based on the location of their lesions, which were primarily in ventromedial, dorsomedial, or lateral parts of their prefrontal cortex (vmPFC, dmPFC, and lPFC, respectively), or in the anterior insula. We compared the performance of each subset of lesion participants to behavior in a control group without lesions (n=18). Finally, we fit a newly developed computational model to the data to glean a more mechanistic understanding of how lesions affect the cognitive processes underlying choice. We found that participants with lesions to the vmPFC waited less overall, while participants with lesions to the dmPFC and anterior insula were specifically impaired at calibrating their level of persistence to the environment. These behavioral effects were accounted for by systematic differences in parameter estimates from a computational model of task performance: while the vmPFC group showed reduced initial willingness to wait, lesions to the dmPFC/anterior insula were associated with slower learning from negative feedback. These findings corroborate the notion that failures of persistence can be driven by sophisticated cost-benefit analyses rather than lapses in self-control. They also support the functional specialization of different parts of the prefrontal cortex in service of voluntary persistence.

Statistical information about reward timing is insufficient for promoting optimal persistence decisions.

When deciding how long to keep waiting for delayed rewards that will arrive at an uncertain time, different distributions of possible reward times dictate different optimal strategies for maximizing reward. When reward timing distributions are heavy-tailed (e.g., waiting on hold) there is a point at which waiting is no longer advantageous because the opportunity cost of waiting is too high. Alternatively, when reward timing distributions have more predictable timing (e.g., uniform), it is advantageous to wait as long as necessary for the reward. Although people learn to approximate optimal strategies, little is known about how this learning occurs. One possibility is that people learn a general cognitive representation of the probability distribution that governs reward timing and then infer a strategy from that model of the environment. Another possibility is that they learn an action policy in a way that depends more narrowly on direct task experience, such that general knowledge of the reward timing distribution is insufficient for expressing the optimal strategy. Here, in a series of studies in which participants decided how long to persist for delayed rewards before quitting, we provided participants with information about the reward timing distribution in several ways. Whether the information was provided through counterfactual feedback (Study 1), previous exposure (Studies 2a and 2b), or description (Studies 3a and 3b), it did not obviate the need for direct, feedback-driven learning in a decision context. Therefore, learning when to quit waiting for delayed rewards might depend on task-specific experience, not solely on probabilistic reasoning.

Assessing evidence accumulation and rule learning in humans with an online game.

Evidence accumulation, an essential component of perception and decision making, is frequently studied with psychophysical tasks involving noisy or ambiguous stimuli. In these tasks, participants typically receive verbal or written instructions that describe the strategy that should be used to guide decisions. Although convenient and effective, explicit instructions can influence learning and decision making strategies and can limit comparisons with animal models, in which behaviors are reinforced through feedback. Here, we developed an online video game and nonverbal training pipeline, inspired by pulse-based tasks for rodents, as an alternative to traditional psychophysical tasks used to study evidence accumulation. Using this game, we collected behavioral data from hundreds of participants trained with an explicit description of the decision rule or with experiential feedback. Participants trained with feedback alone learned the game rules rapidly and used strategies and displayed biases similar to those who received explicit instructions. Finally, by leveraging data across hundreds of participants, we show that perceptual judgments were well described by an accumulation process in which noise scaled nonlinearly with evidence, consistent with previous animal studies but inconsistent with diffusion models widely used to describe perceptual decisions in humans. These results challenge the conventional description of the accumulation process and suggest that online games provide a valuable platform to examine perceptual decision making and learning in humans. In addition, the feedback-based training pipeline developed for this game may be useful for evaluating perceptual decision making in human populations with difficulty following verbal instructions.NEW & NOTEWORTHY Perceptual uncertainty sets critical constraints on our ability to accumulate evidence and make decisions; however, its sources remain unclear. We developed a video game, and feedback-based training pipeline, to study uncertainty during decision making. Leveraging choices from hundreds of subjects, we demonstrate that human choices are inconsistent with popular diffusion models of human decision making and instead are best fit by models in which perceptual uncertainty scales nonlinearly with the strength of sensory evidence.

Experience-driven recalibration of learning from surprising events.

Different environments favor different patterns of adaptive learning. A surprising event that in one context would accelerate belief updating might, in another context, be downweighted as a meaningless outlier. Here, we investigated whether people would spontaneously regulate the influence of surprise on learning in response to event-by-event experiential feedback. Across two experiments, we examined whether participants performing a perceptual judgment task under spatial uncertainty (n = 29, n = 63) adapted their patterns of predictive gaze according to the informativeness or uninformativeness of surprising events in their current environment. Uninstructed predictive eye movements exhibited a form of metalearning in which surprise came to modulate event-by-event learning rates in opposite directions across contexts. Participants later appropriately readjusted their patterns of adaptive learning when the statistics of the environment underwent an unsignaled reversal. Although significant adjustments occurred in both directions, performance was consistently superior in environments in which surprising events reflected meaningful change, potentially reflecting a bias towards interpreting surprise as informative and/or difficulty ignoring salient outliers. Our results provide evidence for spontaneous, context-appropriate recalibration of the role of surprise in adaptive learning.

Arousal-based pupil modulation is dictated by luminance.

Pupillometry has become a standard measure for assessing arousal state. However, environmental factors such as luminance, a primary dictator of pupillary responses, often vary across studies. To what degree does luminance interact with arousal-driven pupillary changes? Here, we parametrically assessed luminance-driven pupillary responses across a wide-range of luminances, while concurrently manipulating cognitive arousal using auditory math problems of varying difficulty. At the group-level, our results revealed that the modulatory effect of cognitive arousal on pupil size interacts multiplicatively with luminance, with the largest effects occurring at low and mid-luminances. However, at the level of individuals, there were qualitatively distinct individual differences in the modulatory effect of cognitive arousal on luminance-driven pupillary responses. Our findings suggest that pupillometry as a measure for assessing arousal requires more careful consideration: there are ranges of luminance levels that are more ideal in observing pupillary differences between arousal conditions than others.

Choices favoring cognitive effort in a foraging environment decrease when multiple forms of effort and delay are interleaved.

Cognitive and physical effort are typically regarded as costly, but demands for effort also seemingly boost the appeal of prospects under certain conditions. One contextual factor that might influence choices for or against effort is the mix of different types of demand a decision maker encounters in a given environment. In two foraging experiments, participants encountered prospective rewards that required equally long intervals of cognitive effort, physical effort, or unfilled delay. Monetary offers varied per trial, and the two experiments differed in whether the type of effort or delay cost was the same on every trial, or varied across trials. When each participant faced only one type of cost, cognitive effort persistently produced the highest acceptance rate compared to trials with an equivalent period of either physical effort or unfilled delay. We theorized that if cognitive effort were intrinsically rewarding, we would observe the same pattern of preferences when participants foraged for varying cost types in addition to rewards. Contrary to this prediction, in the second experiment, an initially higher acceptance rate for cognitive effort trials disappeared over time amid an overall decline in acceptance rates as participants gained experience with all three conditions. Our results indicate that cognitive demands may reduce the discounting effect of delays, but not because decision makers assign intrinsic value to cognitive effort. Rather, the results suggest that a cognitive effort requirement might influence contextual factors such as subjective delay duration estimates, which can be recalibrated if multiple forms of demand are interleaved.

Dynamic Network Analysis Demonstrates the Formation of Stable Functional Networks During Rule Learning.

Variations in the functional connectivity of large-scale cortical brain networks may explain individual differences in learning ability. We used a dynamic network analysis of fMRI data to identify changes in functional brain networks that are associated with context-dependent rule learning. During fMRI scanning, naïve subjects performed a cognitive task designed to test their ability to learn context-dependent rules. Notably, subjects were given minimal instructions about the task prior to scanning. We identified several key network characteristics associated with fast and accurate rule learning. First, consistent with the formation of stable functional networks, a dynamic community detection analysis revealed regionally specific reductions in flexible switching between different functional communities in successful learners. Second, successful rule learners showed decreased centrality of ventral attention regions and increased assortative mixing of cognitive control regions as the rules were learned. Finally, successful subjects showed greater decoupling of default and attention communities throughout the entire task, whereas ventral attention and cognitive control regions became more connected during learning. Overall, the results support a framework by which a stable ventral attention community and more flexible cognitive control community support sustained attention and the formation of rule representations in successful learners.

Eye movements reflect adaptive predictions and predictive precision.

Successful decision-making depends on the ability to form predictions about uncertain future events. Existing evidence suggests predictive representations are not limited to point estimates but also include information about the associated level of predictive uncertainty. Estimates of predictive uncertainty have an important role in governing the rate at which beliefs are updated in response to new observations. It is not yet known, however, whether the same form of uncertainty-modulated learning occurs naturally and spontaneously when there is no task requirement to express predictions explicitly. Here, we used a gaze-based predictive inference paradigm to show that (a) predictive inference manifested in spontaneous gaze dynamics, (b) feedback-driven updating of spontaneous gaze-based predictions reflected adaptation to environmental statistics, and (c) anticipatory gaze variability tracked predictive uncertainty in an event-by-event manner. Our results demonstrate that sophisticated predictive inference can occur spontaneously and that oculomotor behavior can provide a multidimensional readout of internal predictive beliefs. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Functional brain network reconfiguration during learning in a dynamic environment.

When learning about dynamic and uncertain environments, people should update their beliefs most strongly when new evidence is most informative, such as when the environment undergoes a surprising change or existing beliefs are highly uncertain. Here we show that modulations of surprise and uncertainty are encoded in a particular, temporally dynamic pattern of whole-brain functional connectivity, and this encoding is enhanced in individuals that adapt their learning dynamics more appropriately in response to these factors. The key feature of this whole-brain pattern of functional connectivity is stronger connectivity, or functional integration, between the fronto-parietal and other functional systems. Our results provide new insights regarding the association between dynamic adjustments in learning and dynamic, large-scale changes in functional connectivity across the brain.

Time-conjunctive representations of future events.

It is widely accepted that people can predict the relative imminence of future events. However, it is unknown whether the timing of future events is represented using only a "strength-like" estimate or if future events are represented conjunctively with their position on a mental timeline. We examined how people judge temporal relationships among anticipated future events using the novel Judgment of Anticipated Co-Occurrence (JACO) task. Participants were initially trained on a stream of letters sampled from a probabilistically repeating sequence. During test trials, the stream was interrupted with pairs of probe letters and the participants' task was to choose the probe letter they expected to appear in the stream during a lagged target window 4-6 items (4.3-8.5 s) in the future. Participants performed above chance as they gained experience with the task. Because the correct item was sometimes the more imminent probe letter and other times the less imminent probe letter, these results rule out the possibility that participants relied solely on thresholding a strength-like estimate of temporal imminence. Rather, these results suggest that participants held (1) temporally organized predictions of the future letters in the stream, (2) a temporal estimate of the lagged target window, and (3) some means to compare the two and evaluate their temporal alignment. Response time increased with the lag to the more imminent probe letter, suggesting that participants accessed the future sequentially in a manner that mirrors scanning processes previously proposed to operate on memory representations in the short-term judgment of recency task.

Spectral partitioning identifies individual heterogeneity in the functional network topography of ventral and anterior medial prefrontal cortex.

Regions of human medial prefrontal cortex (mPFC) and posterior cingulate cortex (PCC) are part of the default network (DN), and additionally are implicated in diverse cognitive functions ranging from autobiographical memory to subjective valuation. Our ability to interpret the apparent co-localization of task-related effects with DN-regions is constrained by a limited understanding of the individual-level heterogeneity in mPFC/PCC functional organization. Here we used cortical surface-based meta-analysis to identify a parcel in human PCC that was more strongly associated with the DN than with valuation effects. We then used resting-state fMRI data and a data-driven network analysis algorithm, spectral partitioning, to partition mPFC and PCC into "DN" and "non-DN" subdivisions in individual participants (n = 100 from the Human Connectome Project). The spectral partitioning algorithm identified individual-level cortical subdivisions that varied markedly across individuals, especially in mPFC, and were reliable across test/retest datasets. Our results point toward new strategies for assessing whether distinct cognitive functions engage common or distinct mPFC subregions at the individual level.

Dissociable Forms of Uncertainty-Driven Representational Change Across the Human Brain.

Environmental change can lead decision makers to shift rapidly among different behavioral regimes. These behavioral shifts can be accompanied by rapid changes in the firing pattern of neural networks. However, it is unknown what the populations of neurons that participate in such "network reset" phenomena are representing. Here, we investigated the following: (1) whether and where rapid changes in multivariate activity patterns are observable with fMRI during periods of rapid behavioral change and (2) what types of representations give rise to these phenomena. We did so by examining fluctuations in multivoxel patterns of BOLD activity from male and female human subjects making sequential inferences about the state of a partially observable and discontinuously changing variable. We found that, within the context of this sequential inference task, the multivariate patterns of activity in a number of cortical regions contain representations that change more rapidly during periods of uncertainty following a change in behavioral context. In motor cortex, this phenomenon was indicative of discontinuous change in behavioral outputs, whereas in visual regions, the same basic phenomenon was evoked by tracking of salient environmental changes. In most other cortical regions, including dorsolateral prefrontal and anterior cingulate cortex, the phenomenon was most consistent with directly encoding the degree of uncertainty. However, in a few other regions, including orbitofrontal cortex, the phenomenon was best explained by representations of a shifting context that evolve more rapidly during periods of rapid learning. These representations may provide a dynamic substrate for learning that facilitates rapid disengagement from learned responses during periods of change.SIGNIFICANCE STATEMENT Brain activity patterns tend to change more rapidly during periods of uncertainty and behavioral adjustment, yet the computational role of such rapid transitions is poorly understood. Here, we identify brain regions with fMRI BOLD activity patterns that change more rapidly during periods of behavioral adjustment and use computational modeling to attribute the phenomenon to specific causes. We demonstrate that the phenomenon emerges in different brain regions for different computational reasons, the most common being the representation of uncertainty itself, but that, in a selective subset of regions including orbitofrontal cortex, the phenomenon was best explained as a shifting latent state signal that may serve to control the degree to which recent temporal context affects ongoing expectations.

The effects of acute stress on the calibration of persistence.

People frequently fail to wait for delayed rewards after choosing them. These preference reversals are sometimes thought to reflect self-control failure. Other times, however, continuing to wait for a delayed reward may be counterproductive (e.g., when reward timing uncertainty is high). Research has demonstrated that people can calibrate how long to wait for rewards in a given environment. Thus, the role of self-control might be to integrate information about the environment to flexibly adapt behavior, not merely to promote waiting. Here we tested effects of acute stress, which has been shown to tax control processes, on persistence, and the calibration of persistence, in young adult human participants. Half the participants (n = 60) performed a task in which persistence was optimal, and the other half (n = 60) performed a task in which it was optimal to quit waiting for reward soon after each trial began. Each participant completed the task either after cold pressor stress or no stress. Stress did not influence persistence or optimal calibration of persistence. Nevertheless, an exploratory analysis revealed an "inverted-U" relationship between cortisol increase and performance in the stress groups, suggesting that choosing the adaptive waiting policy may be facilitated with some stress and impaired with severe stress.

Do Political and Economic Choices Rely on Common Neural Substrates? A Systematic Review of the Emerging Neuropolitics Literature.

The methods of cognitive neuroscience are beginning to be applied to the study of political behavior. The neural substrates of value-based decision-making have been extensively examined in economic contexts; this might provide a powerful starting point for understanding political decision-making. Here, we asked to what extent the neuropolitics literature to date has used conceptual frameworks and experimental designs that make contact with the reward-related approaches that have dominated decision neuroscience. We then asked whether the studies of political behavior that can be considered in this light implicate the brain regions that have been associated with subjective value related to "economic" reward. We performed a systematic literature review to identify papers addressing the neural substrates of political behavior and extracted the fMRI studies reporting behavioral measures of subjective value as defined in decision neuroscience studies of reward. A minority of neuropolitics studies met these criteria and relatively few brain activation foci from these studies overlapped with regions where activity has been related to subjective value. These findings show modest influence of reward-focused decision neuroscience on neuropolitics research to date. Whether the neural substrates of subjective value identified in economic choice paradigms generalize to political choice thus remains an open question. We argue that systematically addressing the commonalities and differences in these two classes of value-based choice will be important in developing a more comprehensive model of the brain basis of human decision-making.

Medial prefrontal cortical activity reflects dynamic re-evaluation during voluntary persistence.

Deciding how long to keep waiting for future rewards is a nontrivial problem, especially when the timing of rewards is uncertain. We carried out an experiment in which human decision makers waited for rewards in two environments in which reward-timing statistics favored either a greater or lesser degree of behavioral persistence. We found that decision makers adaptively calibrated their level of persistence for each environment. Functional neuroimaging revealed signals that evolved differently during physically identical delays in the two environments, consistent with a dynamic and context-sensitive reappraisal of subjective value. This effect was observed in a region of ventromedial prefrontal cortex that is sensitive to subjective value in other contexts, demonstrating continuity between valuation mechanisms involved in discrete choice and in temporally extended decisions analogous to foraging. Our findings support a model in which voluntary persistence emerges from dynamic cost/benefit evaluation rather than from a control process that overrides valuation mechanisms.

Functionally dissociable influences on learning rate in a dynamic environment.

Maintaining accurate beliefs in a changing environment requires dynamically adapting the rate at which one learns from new experiences. Beliefs should be stable in the face of noisy data but malleable in periods of change or uncertainty. Here we used computational modeling, psychophysics, and fMRI to show that adaptive learning is not a unitary phenomenon in the brain. Rather, it can be decomposed into three computationally and neuroanatomically distinct factors that were evident in human subjects performing a spatial-prediction task: (1) surprise-driven belief updating, related to BOLD activity in visual cortex; (2) uncertainty-driven belief updating, related to anterior prefrontal and parietal activity; and (3) reward-driven belief updating, a context-inappropriate behavioral tendency related to activity in ventral striatum. These distinct factors converged in a core system governing adaptive learning. This system, which included dorsomedial frontal cortex, responded to all three factors and predicted belief updating both across trials and across individuals.

Neural and behavioral evidence for an intrinsic cost of self-control.

The capacity for self-control is critical to adaptive functioning, yet our knowledge of the underlying processes and mechanisms is presently only inchoate. Theoretical work in economics has suggested a model of self-control centering on two key assumptions: (1) a division within the decision-maker between two 'selves' with differing preferences; (2) the idea that self-control is intrinsically costly. Neuroscience has recently generated findings supporting the 'dual-self' assumption. The idea of self-control costs, in contrast, has remained speculative. We report the first independent evidence for self-control costs. Through a neuroimaging meta-analysis, we establish an anatomical link between self-control and the registration of cognitive effort costs. This link predicts that individuals who strongly avoid cognitive demand should also display poor self-control. To test this, we conducted a behavioral experiment leveraging a measure of demand avoidance along with two measures of self-control. The results obtained provide clear support for the idea of self-control costs.

Rational temporal predictions can underlie apparent failures to delay gratification.

An important category of seemingly maladaptive decisions involves failure to postpone gratification. A person pursuing a desirable long-run outcome may abandon it in favor of a short-run alternative that has been available all along. Here we present a theoretical framework in which this seemingly irrational behavior emerges from stable preferences and veridical judgments. Our account recognizes that decision makers generally face uncertainty regarding the time at which future outcomes will materialize. When timing is uncertain, the value of persistence depends crucially on the nature of a decision maker's prior temporal beliefs. Certain forms of temporal beliefs imply that a delay's predicted remaining length increases as a function of time already waited. In this type of situation, the rational, utility-maximizing strategy is to persist for a limited amount of time and then give up. We show empirically that people's explicit predictions of remaining delay lengths indeed increase as a function of elapsed time in several relevant domains, implying that temporal judgments offer a rational basis for limiting persistence. We then develop our framework into a simple working model and show how it accounts for individual differences in a laboratory task (the well-known "marshmallow test"). We conclude that delay-of-gratification failure, generally viewed as a manifestation of limited self-control capacity, can instead arise as an adaptive response to the perceived statistics of one's environment.

The valuation system: a coordinate-based meta-analysis of BOLD fMRI experiments examining neural correlates of subjective value.

Numerous experiments have recently sought to identify neural signals associated with the subjective value (SV) of choice alternatives. Theoretically, SV assessment is an intermediate computational step during decision making, in which alternatives are placed on a common scale to facilitate value-maximizing choice. Here we present a quantitative, coordinate-based meta-analysis of 206 published fMRI studies investigating neural correlates of SV. Our results identify two general patterns of SV-correlated brain responses. In one set of regions, both positive and negative effects of SV on BOLD are reported at above-chance rates across the literature. Areas exhibiting this pattern include anterior insula, dorsomedial prefrontal cortex, dorsal and posterior striatum, and thalamus. The mixture of positive and negative effects potentially reflects an underlying U-shaped function, indicative of signal related to arousal or salience. In a second set of areas, including ventromedial prefrontal cortex and anterior ventral striatum, positive effects predominate. Positive effects in the latter regions are seen both when a decision is confronted and when an outcome is delivered, as well as for both monetary and primary rewards. These regions appear to constitute a "valuation system," carrying a domain-general SV signal and potentially contributing to value-based decision making.