Motor System-Dependent Effects of Amygdala and Ventral Striatum Lesions on Explore-Exploit Behaviors.

Deciding whether to forego immediate rewards or explore new opportunities is a key component of flexible behavior and is critical for the survival of the species. Although previous studies have shown that different cortical and subcortical areas, including the amygdala and ventral striatum (VS), are implicated in representing the immediate (exploitative) and future (explorative) value of choices, the effect of the motor system used to make choices has not been examined. Here, we tested male rhesus macaques with amygdala or VS lesions on two versions of a three-arm bandit task where choices were registered with either a saccade or an arm movement. In both tasks we presented the monkeys with explore-exploit tradeoffs by periodically replacing familiar options with novel options that had unknown reward probabilities. We found that monkeys explored more with saccades but showed better learning with arm movements. VS lesions caused the monkeys to be more explorative with arm movements and less explorative with saccades, although this may have been due to an overall decrease in performance. VS lesions affected the monkeys' ability to learn novel stimulus-reward associations in both tasks, while after amygdala lesions this effect was stronger when choices were made with saccades. Further, on average, VS and amygdala lesions reduced the monkeys' ability to choose better options only when choices were made with a saccade. These results show that learning reward value associations to manage explore-exploit behaviors is motor system dependent and they further define the contributions of amygdala and VS to reinforcement learning.

Mice exhibit stochastic and efficient action switching during probabilistic decision making.

In probabilistic and nonstationary environments, individuals must use internal and external cues to flexibly make decisions that lead to desirable outcomes. To gain insight into the process by which animals choose between actions, we trained mice in a task with time-varying reward probabilities. In our implementation of such a two-armed bandit task, thirsty mice use information about recent action and action­outcome histories to choose between two ports that deliver water probabilistically. Here we comprehensively modeled choice behavior in this task, including the trial-to-trial changes in port selection, i.e., action switching behavior. We find that mouse behavior is, at times, deterministic and, at others, apparently stochastic. The behavior deviates from that of a theoretically optimal agent performing Bayesian inference in a hidden Markov model (HMM). We formulate a set of models based on logistic regression, reinforcement learning, and sticky Bayesian inference that we demonstrate are mathematically equivalent and that accurately describe mouse behavior. The switching behavior of mice in the task is captured in each model by a stochastic action policy, a history-dependent representation of action value, and a tendency to repeat actions despite incoming evidence. The models parsimoniously capture behavior across different environmental conditionals by varying the stickiness parameter, and like the mice, they achieve nearly maximal reward rates. These results indicate that mouse behavior reaches near-maximal performance with reduced action switching and can be described by a set of equivalent models with a small number of relatively fixed parameters.

Developmental differences in children and adults' enforcement of explore versus exploit search strategies in the United States and Turkey.

Across development, as children acquire a deeper understanding of their environment, they explore less and take advantage, or "exploit," what they already know. Here, we test whether children also enforce exploration-oriented search behaviors onto others. Specifically, we ask whether children are more likely to encourage a search agent to explore versus exploit their environment, and whether this pattern varies across childhood (between 3 and 6 years). We also ask whether this pattern differs between children and adults, and generalizes across two different sociocultural contexts-Turkey and the United States-that differ on dimensions that might relate to children's decisions about exploration (e.g., curiosity-focused educational practices, attitudes toward uncertainty avoidance). Participants (N = 358) watched an agent search for rewards and were asked at various points whether the agent should "stay" (exploit) in their current location, or "go" (explore) to a new location. At all points in the experiment, children enforced exploration significantly more often than adults. Early in the agent's search, children in the US enforced exploration more often than children in Turkey; later in the search, younger children (from both sociocultural contexts) were more likely to continue enforcing exploration compared to older children. These findings highlight that children are not only highly exploratory themselves, but also enforce exploration onto others-underscoring the central role that exploration plays in driving early cognitive development across diverse sociocultural contexts. RESEARCH HIGHLIGHTS: The current study examined developmental and cross-cultural differences in children and adults' enforcement of explore-exploit search strategies. Children in the US and Turkey enforced exploration more than adults, who enforced exploitation more often; results were generally consistent across cultures with small differences. Mirroring developmental changes in children's own search behavior; the tendency to enforce exploration decreased between 3- to 6-years of age. Findings underscore the central role of an "exploration mindset" in children's early decision-making-even when exploration has no direct benefits to the child themselves.

The explore/exploit trade-off: An ecologically valid and translational framework that can advance mechanistic understanding of eating disorders.

The explore/exploit trade-off is a decision-making process that is conserved across species and balances exploring unfamiliar choices of unknown value with choosing familiar options of known value to maximize reward. This framework is rooted in behavioral ecology and has traditionally been used to study maladaptive versus adaptive non-human animal foraging behavior. Researchers have begun to recognize the potential utility of understanding human decision-making and psychopathology through the explore/exploit trade-off. In this article, we propose that explore/exploit trade-off holds promise for advancing our mechanistic understanding of decision-making processes that confer vulnerability for and maintain eating pathology due to its neurodevelopmental bases, conservation across species, and ability to be mathematically modeled. We present a model for how suboptimal explore/exploit decision-making can promote disordered eating and present recommendations for future research applying this framework to eating pathology. Taken together, the explore/exploit trade-off provides a translational framework for expanding etiologic and maintenance models of eating pathology, given developmental changes in explore/exploit decision-making that coincide in time with the emergence of eating pathology and evidence of biased explore/exploit decision-making in psychopathology. Additionally, understanding explore/exploit decision-making in eating disorders may improve knowledge of their underlying pathophysiology, informing targeted clinical interventions such as neuromodulation and pharmacotherapy. PUBLIC SIGNIFICANCE STATEMENT: The explore/exploit trade-off is a cross-species decision-making process whereby organisms choose between a known option with a known reward or sampling unfamiliar options. We hypothesize that imbalanced explore/exploit decision-making can promote disordered eating and present preliminary data. We propose that explore/exploit trade-off has significant potential to advance understanding of the neurocognitive and neurodevelopmental mechanisms of eating pathology, which could ultimately guide revisions of etiologic models and inform novel interventions.

El balance entre explorar y explotar es un proceso de toma de decisiones que se conserva a través de las especies y equilibra la exploración de opciones desconocidas de valor desconocido con la elección de opciones familiares de valor conocido para maximizar la recompensa. Este marco está arraigado en la ecología del comportamiento y tradicionalmente se ha utilizado para estudiar el comportamiento de forrajeo no adaptativo versus adaptativo en animales no humanos. Los investigadores han comenzado a reconocer la utilidad potencial de entender la toma de decisiones humanas y la psicopatología a través del balance entre explorar y explotar. En este artículo, proponemos que el balance entre explorar y explotar ofrece promesas para avanzar en nuestra comprensión mecanicista de los procesos de toma de decisiones que confieren vulnerabilidad y mantienen la patología alimentaria debido a sus bases neurodesarrolladoras, su conservación a través de las especies y su capacidad de ser modelado matemáticamente. Presentamos un modelo de cómo la toma de decisiones subóptima entre explorar y explotar puede promover la alimentación disfuncional y presentamos recomendaciones para futuras investigaciones que apliquen este marco a la patología alimentaria. En conjunto, el balance entre explorar y explotar proporciona un marco translacional para expandir los modelos etiológicos y de mantenimiento de la patología alimentaria, dadas los cambios en el desarrollo de la toma de decisiones entre explorar y explotar que coinciden en el tiempo con la aparición de la patología alimentaria y la evidencia de una toma de decisiones entre explorar y explotar sesgada en la psicopatología. Además, comprender la toma de decisiones entre explorar y explotar en los trastornos alimentarios puede mejorar el conocimiento de su fisiopatología subyacente, informando intervenciones clínicas dirigidas como la neuromodulación y la farmacoterapia.

What Does the Frontopolar Cortex Contribute to Goal-Directed Cognition and Action?

Understanding the unique functions of different subregions of primate prefrontal cortex has been a longstanding goal in cognitive neuroscience. Yet, the anatomy and function of one of its largest subregions (the frontopolar cortex) remain enigmatic and underspecified. Our Society for Neuroscience minisymposium Primate Frontopolar Cortex: From Circuits to Complex Behaviors will comprise a range of new anatomic and functional approaches that have helped to clarify the basic circuit anatomy of the frontal pole, its functional involvement during performance of cognitively demanding behavioral paradigms in monkeys and humans, and its clinical potential as a target for noninvasive brain stimulation in patients with brain disorders. This review consolidates knowledge about the anatomy and connectivity of frontopolar cortex and provides an integrative summary of its function in primates. We aim to answer the question: what, if anything, does frontopolar cortex contribute to goal-directed cognition and action?

Computational mechanisms underpinning greater exploratory behaviour in excess weight relative to healthy weight adolescents.

Obesity in adolescence is associated with cognitive changes that lead to difficulties in shifting unhealthy habits in favour of alternative healthy behaviours, similar to addictive behaviours. An outstanding question is whether this shift in goal-directed behaviour is driven by over-exploitation or over-exploration of rewarding outcomes. Here, we addressed this question by comparing explore/exploit behaviour on the Iowa Gambling Task in 43 adolescents with excess weight against 38 adolescents with healthy weight. We computationally modelled both exploitation behaviour (e.g., reinforcement sensitivity and inverse decay parameters), and explorative behaviour (e.g., maximum directed exploration value). We found that overall, adolescents with excess weight displayed more behavioural exploration than their healthy-weight counterparts - specifically, demonstrating greater overall switching behaviour. Computational models revealed that this behaviour was driven by a higher maximum directed exploration value in the excess-weight group (U = 520.00, p = .005, BF10 = 5.11). Importantly, however, we found substantial evidence that groups did not differ in reinforcement sensitivity (U = 867.00, p = .641, BF10 = 0.30). Overall, our study demonstrates a preference for exploratory behaviour in adolescents with excess weight, independent of sensitivity to reward. This pattern could potentially underpin an intrinsic desire to explore energy-dense unhealthy foods - an as-yet untapped mechanism that could be targeted in future treatments of obesity in adolescents.

Computational models of exploration and exploitation characterise onset and efficacy of treatment in methamphetamine use disorder.

People with Methamphetamine Use Disorder (PwMUD) spend substantial time and resources on substance use, which hinders their ability to explore alternate reinforcers. Gold-standard behavioural treatments attempt to remedy this by encouraging action towards non-drug reinforcers, but substance use often persists. We aimed to unravel the mechanistic drivers of this behaviour by applying a computational model of explore/exploit behaviour to decision-making data (Iowa Gambling Task) from 106 PwMUD and 48 controls. We then examined the longitudinal link between explore/exploit mechanisms and changes in methamphetamine use 6 weeks later. Exploitation parameters included reinforcement sensitivity and inverse decay (i.e., number of past outcomes used to guide choices). Exploration parameters included maximum directed exploration value (i.e., value of trying novel actions). The Timeline Follow Back measured changes in methamphetamine use. Compared to controls, PwMUD showed deficits in exploitative decision-making, characterised by reduced reinforcement sensitivity, U = 3065, p = 0.009, and less use of previous choice outcomes, U = 3062, p = 0.010. This was accompanied by a behavioural pattern of frequent shifting between choices, which appeared consistent with random exploration. Furthermore, PwMUD with greater reductions of methamphetamine use at 6 weeks had increased directed exploration (ß = 0.22, p = 0.045); greater use of past choice outcomes (ß = -0.39, p = 0.002) and greater choice consistency (ß = -0.39, p = 0.002). Therefore, limited computational exploitation and increased behavioural exploration characterise PwMUD's presentation to treatment, while increased directed exploration, use of past choice outcomes and choice consistency predict greater reductions of methamphetamine use.

Primate Orbitofrontal Cortex Codes Information Relevant for Managing Explore-Exploit Tradeoffs.

Reinforcement learning (RL) refers to the behavioral process of learning to obtain reward and avoid punishment. An important component of RL is managing explore-exploit tradeoffs, which refers to the problem of choosing between exploiting options with known values and exploring unfamiliar options. We examined correlates of this tradeoff, as well as other RL related variables, in orbitofrontal cortex (OFC) while three male monkeys performed a three-armed bandit learning task. During the task, novel choice options periodically replaced familiar options. The values of the novel options were unknown, and the monkeys had to explore them to see if they were better than other currently available options. The identity of the chosen stimulus and the reward outcome were strongly encoded in the responses of single OFC neurons. These two variables define the states and state transitions in our model that are relevant to decision-making. The chosen value of the option and the relative value of exploring that option were encoded at intermediate levels. We also found that OFC value coding was stimulus specific, as opposed to coding value independent of the identity of the option. The location of the option and the value of the current environment were encoded at low levels. Therefore, we found encoding of the variables relevant to learning and managing explore-exploit tradeoffs in OFC. These results are consistent with findings in the ventral striatum and amygdala and show that this monosynaptically connected network plays an important role in learning based on the immediate and future consequences of choices.SIGNIFICANCE STATEMENT Orbitofrontal cortex (OFC) has been implicated in representing the expected values of choices. Here we extend these results and show that OFC also encodes information relevant to managing explore-exploit tradeoffs. Specifically, OFC encodes an exploration bonus, which characterizes the relative value of exploring novel choice options. OFC also strongly encodes the identity of the chosen stimulus, and reward outcomes, which are necessary for computing the value of novel and familiar options.

Increasing the gradient of energetic cost does not initiate adaptation in human walking.

When in a new situation, the nervous system may benefit from adapting its control policy. In determining whether or not to initiate this adaptation, the nervous system may rely on some features of the new situation. Here, we tested whether one such feature is salient cost savings. We changed cost saliency by manipulating the gradient of participants' energetic cost landscape during walking. We hypothesized that steeper gradients would cause participants to spontaneously adapt their step frequency to lower costs. To manipulate the gradient, a mechatronic system applied controlled fore-aft forces to the waist of participants as a function of their step frequency as they walked on a treadmill. These forces increased the energetic cost of walking at high step frequencies and reduced it at low step frequencies. We successfully created three cost landscapes of increasing gradients, where the natural variability in participants' step frequency provided cost changes of 3.6% (shallow), 7.2% (intermediate), and 10.2% (steep). Participants did not spontaneously initiate adaptation in response to any of the gradients. Using metronome-guided walking-a previously established protocol for eliciting initiation of adaptation-participants next experienced a step frequency with a lower cost. Participants then adapted by -1.41 ± 0.81 (P = 0.007) normalized units away from their originally preferred step frequency obtaining cost savings of 4.80% ± 3.12%. That participants would adapt under some conditions, but not in response to steeper cost gradients, suggests that the nervous system does not solely rely on the gradient of energetic cost to initiate adaptation in novel situations.NEW & NOTEWORTHY People can adapt to novel conditions but often require cues to initiate the adaptation. Using a mechatronic system to reshape energetic cost gradients during treadmill walking, we tested whether the nervous system can use information present in the cost gradient to spontaneously initiate adaptation. We found that our participants did not spontaneously initiate adaptation even in the steepest gradient. The nervous system does not rely solely on the cost gradient when initiating adaptation.

Learning and choosing in an uncertain world: An investigation of the explore-exploit dilemma in static and dynamic environments.

How do people solve the explore-exploit trade-off in a changing environment? In this paper we present experimental evidence from an "observe or bet" task, in which people have to determine when to engage in information-seeking behavior and when to switch to reward-taking actions. In particular we focus on the comparison between people's behavior in a changing environment and their behavior in an unchanging one. Our experimental work is motivated by rational analysis of the problem that makes strong predictions about information search and reward seeking in static and changeable environments. Our results show a striking agreement between human behavior and the optimal policy, but also highlight a number of systematic differences. In particular, we find that while people often employ suboptimal strategies the first time they encounter the learning problem, most people are able to approximate the correct strategy after minimal experience. In order to describe both the manner in which people's choices are similar to but slightly different from an optimal standard, we introduce four process models for the observe or bet task and evaluate them as potential theories of human behavior.

Children Use Teachers' Beliefs About Their Abilities to Calibrate Explore-Exploit Decisions.

Models of the explore-exploit problem have explained how children's decision making is weighed by a bias for information (directed exploration), randomness, and generalization. These behaviors are often tested in domains where a choice to explore (or exploit) is guaranteed to reveal an outcome. An often overlooked but critical component of the assessment of explore-exploit decisions lies in the expected success of taking actions in the first place-and, crucially, how such decisions might be carried out when learning from others. Here, we examine how children consider an informal teacher's beliefs about the child's competence when deciding how difficult a task they want to pursue. We present a simple model of this problem that predicts that while learners should follow the recommendation of an accurate teacher, they should exploit easier games when a teacher overestimates their abilities, and explore harder games when she underestimates them. We tested these predictions in two experiments with adults (Experiment 1) and 6- to 8-year-old children (Experiment 2). In our task, participants' performance on a picture-matching game was either overestimated, underestimated, or accurately represented by a confederate (the "Teacher"), who then presented three new matching games of varying assessed difficulty (too easy, too hard, just right) at varying potential reward (low, medium, high). In line with our model's predictions, we found that both adults and children calibrated their choices to the teacher's representation of their competence. That is, to maximize expected reward, when she underestimated them, participants chose games the teacher evaluated as being too hard for them; when she overestimated them, they chose games she evaluated as being too easy; and when she was accurate, they chose games she assessed as being just right. This work provides insight into the early-emerging ability to calibrate explore-exploit decisions to others' knowledge when learning in informal pedagogical contexts.

Electrophysiological Markers of Aberrant Cue-Specific Exploration in Hazardous Drinkers.

Background: Hazardous drinking is associated with maladaptive alcohol-related decision-making. Existing studies have often focused on how participants learn to exploit familiar cues based on prior reinforcement, but little is known about the mechanisms that drive hazardous drinkers to explore novel alcohol cues when their value is not known. Methods: We investigated exploration of novel alcohol and non-alcohol cues in hazardous drinkers (N = 27) and control participants (N = 26) during electroencephalography (EEG). A normative computational model with two free parameters was fit to estimate participants' weighting of the future value of exploration and immediate value of exploitation. Results: Hazardous drinkers demonstrated increased exploration of novel alcohol cues, and conversely, increased probability of exploiting familiar alternatives instead of exploring novel non-alcohol cues. The motivation to explore novel alcohol stimuli in hazardous drinkers was driven by an elevated relative future valuation of uncertain alcohol cues. P3a predicted more exploratory decision policies driven by an enhanced relative future valuation of novel alcohol cues. P3b did not predict choice behavior, but computational parameter estimates suggested that hazardous drinkers with enhanced P3b to alcohol cues were likely to learn to exploit their immediate expected value. Conclusions: Hazardous drinkers did not display atypical choice behavior, different P3a/P3b amplitudes, or computational estimates to novel non-alcohol cues-diverging from previous studies in addiction showing atypical generalized explore-exploit decisions with non-drug-related cues. These findings reveal that cue-specific neural computations may drive aberrant alcohol-related decision-making in hazardous drinkers-highlighting the importance of drug-relevant cues in studies of decision-making in addiction.

Pallidal neuromodulation of the explore/exploit trade-off in decision-making.

Every decision that we make involves a conflict between exploiting our current knowledge of an action's value or exploring alternative courses of action that might lead to a better, or worse outcome. The sub-cortical nuclei that make up the basal ganglia have been proposed as a neural circuit that may contribute to resolving this explore-exploit 'dilemma'. To test this hypothesis, we examined the effects of neuromodulating the basal ganglia's output nucleus, the globus pallidus interna, in patients who had undergone deep brain stimulation (DBS) for isolated dystonia. Neuromodulation enhanced the number of exploratory choices to the lower value option in a two-armed bandit probabilistic reversal-learning task. Enhanced exploration was explained by a reduction in the rate of evidence accumulation (drift rate) in a reinforcement learning drift diffusion model. We estimated the functional connectivity profile between the stimulating DBS electrode and the rest of the brain using a normative functional connectome derived from heathy controls. Variation in the extent of neuromodulation induced exploration between patients was associated with functional connectivity from the stimulation electrode site to a distributed brain functional network. We conclude that the basal ganglia's output nucleus, the globus pallidus interna, can adaptively modify decision choice when faced with the dilemma to explore or exploit.

Prolonged Physiological Stress Is Associated With a Lower Rate of Exploratory Learning That Is Compounded by Depression.

BACKGROUND: Stress is a major risk factor for depression, and both are associated with important changes in decision-making patterns. However, decades of research have only weakly connected physiological measurements of stress to the subjective experience of depression. Here, we examined the relationship between prolonged physiological stress, mood, and explore-exploit decision making in a population navigating a dynamic environment under stress: health care workers during the COVID-19 pandemic. METHODS: We measured hair cortisol levels in health care workers who completed symptom surveys and performed an explore-exploit restless-bandit decision-making task; 32 participants were included in the final analysis. Hidden Markov and reinforcement learning models assessed task behavior. RESULTS: Participants with higher hair cortisol exhibited less exploration (r = -0.36, p = .046). Higher cortisol levels predicted less learning during exploration (ß = -0.42, false discovery rate [FDR]-corrected p [pFDR] = .022). Importantly, mood did not independently correlate with cortisol concentration, but rather explained additional variance (ß = 0.46, pFDR = .022) and strengthened the relationship between higher cortisol and lower levels of exploratory learning (ß = -0.47, pFDR = .022) in a joint model. These results were corroborated by a reinforcement learning model, which revealed less learning with higher hair cortisol and low mood (ß = -0.67, pFDR = .002). CONCLUSIONS: These results imply that prolonged physiological stress may limit learning from new information and lead to cognitive rigidity, potentially contributing to burnout. Decision-making measures link subjective mood states to measured physiological stress, suggesting that they should be incorporated into future biomarker studies of mood and stress conditions.

What's in the Box? Preschoolers Consider Ambiguity, Expected Value, and Information for Future Decisions in Explore-Exploit Tasks.

Self-directed exploration in childhood appears driven by a desire to resolve uncertainties in order to learn more about the world. However, in adult decision-making, the choice to explore new information rather than exploit what is already known takes many factors beyond uncertainty (such as expected utilities and costs) into account. The evidence for whether young children are sensitive to complex, contextual factors in making exploration decisions is limited and mixed. Here, we investigate whether modifying uncertain options influences explore-exploit behavior in preschool-aged children (48-68 months). Over the course of three experiments, we manipulate uncertain options' ambiguity, expected value, and potential to improve epistemic state for future exploration in a novel forced-choice design. We find evidence that young children are influenced by each of these factors, suggesting that early, self-directed exploration involves sophisticated, context-sensitive decision-making under uncertainty.

Active learning impairments in substance use disorders when resolving the explore-exploit dilemma: A replication and extension of previous computational modeling results.

BACKGROUND: Substance use disorders (SUDs) represent a major public health risk. Yet, our understanding of the mechanisms that maintain these disorders remains incomplete. In a recent computational modeling study, we found initial evidence that SUDs are associated with slower learning rates from negative outcomes and less value-sensitive choice (low "action precision"), which could help explain continued substance use despite harmful consequences. METHODS: Here we aimed to replicate and extend these results in a pre-registered study with a new sample of 168 individuals with SUDs and 99 healthy comparisons (HCs). We performed the same computational modeling and group comparisons as in our prior report (doi: 10.1016/j.drugalcdep.2020.108208) to confirm previously observed effects. After completing all pre-registered replication analyses, we then combined the previous and current datasets (N = 468) to assess whether differences were transdiagnostic or driven by specific disorders. RESULTS: Replicating prior results, SUDs showed slower learning rates for negative outcomes in both Bayesian and frequentist analyses (partial η2=.02). Previously observed differences in action precision were not confirmed. Learning rates for positive outcomes were also similar between groups. Logistic regressions including all computational parameters as predictors in the combined datasets could differentiate several specific disorders from HCs, but could not differentiate most disorders from each other. CONCLUSIONS: These results provide robust evidence that individuals with SUDs adjust behavior more slowly in the face of negative outcomes than HCs. They also suggest this effect is common across several different SUDs. Future research should examine its neural basis and whether learning rates could represent a new treatment target or moderator of treatment outcome.

Children are more exploratory and learn more than adults in an approach-avoid task.

Intuitively, children appear to be more exploratory than adults, and this exploration seems to help children learn,. However, there have been few clear tests of these ideas. We test whether exploration and learning change across development using a task that presents a "learning trap." In this task, exploitation-maximizing immediate reward and avoiding costs-may lead the learner to draw incorrect conclusions, while exploration may lead to better learning but be more costly. In Studies 1, 2, and 3 we find that preschoolers and early school-aged children explore more than adults and learn the true structure of the environment better. Study 3 demonstrates that children explore more than adults even though they, like adults, predict that exploration will be costly, and it shows that exploration and learning are correlated. Study 4 shows that children's and adults' learning depends on the evidence they generate during exploration: children exposed to adult-like evidence learn like adults, and adults exposed to child-like evidence learn like children. Together, these studies support the idea that children may be more exploratory than adults, and this increased exploration influences learning.

The neurocomputational bases of explore-exploit decision-making.

Flexible decision-making requires animals to forego immediate rewards (exploitation) and try novel choice options (exploration) to discover if they are preferable to familiar alternatives. Using the same task and a partially observable Markov decision process (POMDP) model to quantify the value of choices, we first determined that the computational basis for managing explore-exploit tradeoffs is conserved across monkeys and humans. We then used fMRI to identify where in the human brain the immediate value of exploitative choices and relative uncertainty about the value of exploratory choices were encoded. Consistent with prior neurophysiological evidence in monkeys, we observed divergent encoding of reward value and uncertainty in prefrontal and parietal regions, including frontopolar cortex, and parallel encoding of these computations in motivational regions including the amygdala, ventral striatum, and orbitofrontal cortex. These results clarify the interplay between prefrontal and motivational circuits that supports adaptive explore-exploit decisions in humans and nonhuman primates.

The experimental tobacco marketplace: Effects of low-ventilated cigarette exposure.

OBJECTIVES: Regulating filter ventilation will change the relative reinforcing value of products resulting in nicotine/tobacco users facing the explore/exploit dilemma (ie, choice between unfamiliar and familiar options). This study examined the effects of price increases in higher-ventilated cigarettes (HVCs) and exposure to lower-ventilated cigarettes (LVCs) on explore/exploit patterns of tobacco-product purchasing in the Experimental Tobacco Marketplace (ETM). METHODS: HVC smokers (N = 20) completed one assessment session and 3 ETM sessions separated by weeks of at-home LVC exposure. In each ETM session, participants made 7-days of tobacco-product purchases as HVCs price increased across trials. RESULTS: Prohibitive prices of HVC decreased the likelihood of HVCs purchases and increased the likelihood of LVC purchases. Initial exposure (week 1) to LVC reduced the number of cigarettes purchased when HVC prices were high and increased exploration of alternative tobacco products. Successive exposure to LVC (repeated access in weeks 2,5,6,9,10) decreased likelihood of HVCs and alternative product purchases and increased the likelihood of LVCs purchases. CONCLUSIONS: Regulating filter ventilation may initially increase exploration of alternative tobacco products but lead to exploitation of LVCs over time. Tobacco control strategies should take advantage of this transition period when smokers seek information on unfamiliar products to implement harm reduction strategies.

Heterogeneity of decision-making strategies for preschoolers on a variant of the IGT.

Adaptive decision-making strategies are critical for dealing with the complexity of the social world. The present study investigated the use of decision-making strategies in preschoolers and their association to prosocial behavior and peer problems. Eighty-six preschoolers aged 3- and 4-years completed the preschool decision-making task (PGT), a child variant of the Iowa Gambling task . Win-stay/lose-shift responses along with exploration (consecutive choices from the advantageous deck) and exploitation (shifting between options) were examined. Preschoolers showed a range of strategies, with 4-year-olds adapting their approach as the game progressed and making better use of feedback in comparison to 3-year-olds. Children who differed in terms of choices from the advantageous deck were distinguished by different combinations of exploration and exploitation. Furthermore, unique combinations of decision-making strategies also distinguished children who were rated as high versus low in prosocial behavior as well as children rated as having a high versus low level of peer problems. The findings suggest that consideration of strategies used in decision-making tasks could provide useful insight in a clinical setting, particularly for populations with social difficulties.