Adolescents flexibly adapt action selection based on controllability inferences.

From early in life, we encounter both controllable environments, in which our actions can causally influence the reward outcomes we experience, and uncontrollable environments, in which they cannot. Environmental controllability is theoretically proposed to organize our behavior. In controllable contexts, we can learn to proactively select instrumental actions that bring about desired outcomes. In uncontrollable environments, Pavlovian learning enables hard-wired, reflexive reactions to anticipated, motivationally salient events, providing "default" behavioral responses. Previous studies characterizing the balance between Pavlovian and instrumental learning systems across development have yielded divergent findings, with some studies observing heightened expression of Pavlovian learning during adolescence and others observing a reduced influence of Pavlovian learning during this developmental stage. In this study, we aimed to investigate whether a theoretical model of controllability-dependent arbitration between learning systems might explain these seemingly divergent findings in the developmental literature, with the specific hypothesis that adolescents' action selection might be particularly sensitive to environmental controllability. To test this hypothesis, 90 participants, aged 8-27, performed a probabilistic-learning task that enables estimation of Pavlovian influence on instrumental learning, across both controllable and uncontrollable conditions. We fit participants' data with a reinforcement-learning model in which controllability inferences adaptively modulate the dominance of Pavlovian versus instrumental control. Relative to children and adults, adolescents exhibited greater flexibility in calibrating the expression of Pavlovian bias to the degree of environmental controllability. These findings suggest that sensitivity to environmental reward statistics that organize motivated behavior may be heightened during adolescence.

Sensitivity to the Instrumental Value of Choice Increases Across Development.

Across development, people tend to demonstrate a preference for contexts in which they have the opportunity to make choices. However, it is not clear how children, adolescents, and adults learn to calibrate this preference based on the costs and benefits of agentic choice. Here, in both a primary, in-person, reinforcement-learning experiment (N = 92; age range = 10-25 years) and a preregistered online replication study (N = 150; age range = 8-25 years), we found that participants overvalued agentic choice but also calibrated their agency decisions to the reward structure of the environment, increasingly selecting agentic choice when choice had greater instrumental value. Regression analyses and computational modeling of participant choices revealed that participants' bias toward agentic choice-reflecting its intrinsic value-remained consistent across age, whereas sensitivity to the instrumental value of agentic choice increased from childhood to early adulthood.

Reward Enhances Memory via Age-Varying Online and Offline Neural Mechanisms across Development.

Reward motivation enhances memory through interactions between mesolimbic, hippocampal, and cortical systems, both during and after encoding. Developmental changes in these distributed neural circuits may lead to age-related differences in reward-motivated memory and the underlying neural mechanisms. Converging evidence from cross-species studies suggests that subcortical dopamine signaling is increased during adolescence, which may lead to stronger memory representations of rewarding, relative to mundane, events and changes in the contributions of underlying subcortical and cortical brain mechanisms across age. Here, we used fMRI to examine how reward motivation influences the "online" encoding and "offline" postencoding brain mechanisms that support long-term associative memory from childhood to adulthood in human participants of both sexes. We found that reward motivation led to both age-invariant enhancements and nonlinear age-related differences in associative memory after 24 h. Furthermore, reward-related memory benefits were linked to age-varying neural mechanisms. During encoding, interactions between the prefrontal cortex (PFC) and ventral tegmental area (VTA) were associated with better high-reward memory to a greater degree with increasing age. Preencoding to postencoding changes in functional connectivity between the anterior hippocampus and VTA were also associated with better high-reward memory, but more so at younger ages. Our findings suggest that there may be developmental differences in the contributions of offline subcortical and online cortical brain mechanisms supporting reward-motivated memory.SIGNIFICANCE STATEMENT A substantial body of research has examined the neural mechanisms through which reward influences memory formation in adults. However, despite extensive evidence that both reward processing and associative memory undergo dynamic change across development, few studies have examined age-related changes in these processes. We found both age-invariant and nonlinear age-related differences in reward-motivated memory. Moreover, our findings point to developmental differences in the processes through which reward modulates the prioritization of information in long-term memory, with greater early reliance on offline subcortical consolidation mechanisms and increased contribution of systems-level online encoding circuitry with increasing age. These results highlight dynamic developmental changes in the cognitive and neural mechanisms through which motivationally salient information is prioritized in memory from childhood to adulthood.

Developmental shifts in computations used to detect environmental controllability.

Accurate assessment of environmental controllability enables individuals to adaptively adjust their behavior-exploiting rewards when desirable outcomes are contingent upon their actions and minimizing costly deliberation when their actions are inconsequential. However, it remains unclear how estimation of environmental controllability changes from childhood to adulthood. Ninety participants (ages 8-25) completed a task that covertly alternated between controllable and uncontrollable conditions, requiring them to explore different actions to discover the current degree of environmental controllability. We found that while children were able to distinguish controllable and uncontrollable conditions, accuracy of controllability assessments improved with age. Computational modeling revealed that whereas younger participants' controllability assessments relied on evidence gleaned through random exploration, older participants more effectively recruited their task structure knowledge to make highly informative interventions. Age-related improvements in working memory mediated this qualitative shift toward increased use of an inferential strategy. Collectively, these findings reveal an age-related shift in the cognitive processes engaged to assess environmental controllability. Improved detection of environmental controllability may foster increasingly adaptive behavior over development by revealing when actions can be leveraged for one's benefit.

Neural effects of controllability as a key dimension of stress exposure.

Cross-species evidence suggests that the ability to exert control over a stressor is a key dimension of stress exposure that may sensitize frontostriatal-amygdala circuitry to promote more adaptive responses to subsequent stressors. The present study examined neural correlates of stressor controllability in young adults. Participants (N = 56; Mage = 23.74, range = 18-30 years) completed either the controllable or uncontrollable stress condition of the first of two novel stressor controllability tasks during functional magnetic resonance imaging (fMRI) acquisition. Participants in the uncontrollable stress condition were yoked to age- and sex-matched participants in the controllable stress condition. All participants were subsequently exposed to uncontrollable stress in the second task, which is the focus of fMRI analyses reported here. A whole-brain searchlight classification analysis revealed that patterns of activity in the right dorsal anterior insula (dAI) during subsequent exposure to uncontrollable stress could be used to classify participants' initial exposure to either controllable or uncontrollable stress with a peak of 73% accuracy. Previous experience of exerting control over a stressor may change the computations performed within the right dAI during subsequent stress exposure, shedding further light on the neural underpinnings of stressor controllability.

Reward-motivated memories influence new learning across development.

Previously rewarding experiences can influence choices in new situations. Past work has demonstrated that existing reward associations can either help or hinder future behaviors and that there is substantial individual variability in the transfer of value across contexts. Developmental changes in reward sensitivity may also modulate the impact of prior reward associations on later goal-directed behavior. The current study aimed to characterize how reward associations formed in the past affected learning in the present from childhood to adulthood. Participants completed a reinforcement learning paradigm using high- and low-reward stimuli from a task completed 24 h earlier, as well as novel stimuli, as choice options. We found that prior high-reward associations impeded learning across all ages. We then assessed how individual differences in the prioritization of high- versus low-reward associations in memory impacted new learning. Greater high-reward memory prioritization was associated with worse learning performance for previously high-reward relative to low-reward stimuli across age. Adolescents also showed impeded early learning regardless of individual differences in high-reward memory prioritization. Detrimental effects of previous reward on choice behavior did not persist beyond learning. These findings indicate that prior reward associations proactively interfere with future learning from childhood to adulthood and that individual differences in reward-related memory prioritization influence new learning across age.

Reinforcement learning with associative or discriminative generalization across states and actions: fMRI at 3 T and 7 T.

The model-free algorithms of "reinforcement learning" (RL) have gained clout across disciplines, but so too have model-based alternatives. The present study emphasizes other dimensions of this model space in consideration of associative or discriminative generalization across states and actions. This "generalized reinforcement learning" (GRL) model, a frugal extension of RL, parsimoniously retains the single reward-prediction error (RPE), but the scope of learning goes beyond the experienced state and action. Instead, the generalized RPE is efficiently relayed for bidirectional counterfactual updating of value estimates for other representations. Aided by structural information but as an implicit rather than explicit cognitive map, GRL provided the most precise account of human behavior and individual differences in a reversal-learning task with hierarchical structure that encouraged inverse generalization across both states and actions. Reflecting inference that could be true, false (i.e., overgeneralization), or absent (i.e., undergeneralization), state generalization distinguished those who learned well more so than action generalization. With high-resolution high-field fMRI targeting the dopaminergic midbrain, the GRL model's RPE signals (alongside value and decision signals) were localized within not only the striatum but also the substantia nigra and the ventral tegmental area, including specific effects of generalization that also extend to the hippocampus. Factoring in generalization as a multidimensional process in value-based learning, these findings shed light on complexities that, while challenging classic RL, can still be resolved within the bounds of its core computations.

Real-World Exploration Increases Across Adolescence and Relates to Affect, Risk Taking, and Social Connectivity.

Cross-species research suggests that exploratory behaviors increase during adolescence and relate to the social, affective, and risky behaviors characteristic of this developmental stage. However, how these typical adolescent behaviors manifest and relate in real-world settings remains unclear. Using geolocation tracking to quantify exploration-variability in daily movement patterns-over a 3-month period in 58 adolescents and adults (ages 13-27) in New York City, we investigated whether daily exploration varied with age and whether exploration related to social connectivity, risk taking, and momentary positive affect. In our cross-sectional sample, we found an association between daily exploration and age, with individuals near the transition to legal adulthood exhibiting the highest exploration levels. Days of higher exploration were associated with greater positive affect irrespective of age. Higher mean exploration was associated with greater social connectivity in all participants but was linked to higher risk taking selectively among adolescents. Our results highlight the interplay of exploration and socioemotional behaviors across development and suggest that societal norms may modulate their expression in naturalistic contexts.

Flexibility in valenced reinforcement learning computations across development.

Optimal integration of positive and negative outcomes during learning varies depending on an environment's reward statistics. The present study investigated the extent to which children, adolescents, and adults (N = 142 8-25 year-olds, 55% female, 42% White, 31% Asian, 17% mixed race, and 8% Black; data collected in 2021) adapt their weighting of better-than-expected and worse-than-expected outcomes when learning from reinforcement. Participants made choices across two contexts: one in which weighting positive outcomes more heavily than negative outcomes led to better performance, and one in which the reverse was true. Reinforcement learning modeling revealed that across age, participants shifted their valence biases in accordance with environmental structure. Exploratory analyses revealed strengthening of context-dependent flexibility with increasing age.

Associative memory persistence in 3- to 5-year-olds.

Adults struggle to recollect episodic memories from early life. This phenomenon-referred to as "infantile" and "childhood amnesia"-has been widely observed across species and is characterized by rapid forgetting from birth until early childhood. While a number of studies have focused on infancy, few studies have examined the persistence of memory for newly learned associations during the putative period of childhood amnesia. In this study, we investigated forgetting in 137 children ages 3-5 years old by using an interactive storybook task. We assessed associative memory between subjects after 5-min, 24-h, and 1-week delay periods. Across all delays, we observed a significant increase in memory performance with age. While all ages demonstrated above-chance memory performance after 5-min and 24-h delays, we observed chance-level memory accuracy in 3-year-olds following a 1-week delay. The observed age differences in associative memory support the proposal that hippocampal-dependent memory systems undergo rapid development during the preschool years. These data have the potential to inform future work translating memory persistence and malleability research from rodent models to humans by establishing timescales at which we expect young children to forget newly learned associations.

Stress attenuates the flexible updating of aversive value.

In a dynamic environment, sources of threat or safety can unexpectedly change, requiring the flexible updating of stimulus-outcome associations that promote adaptive behavior. However, aversive contexts in which we are required to update predictions of threat are often marked by stress. Acute stress is thought to reduce behavioral flexibility, yet its influence on the modulation of aversive value has not been well characterized. Given that stress exposure is a prominent risk factor for anxiety and trauma-related disorders marked by persistent, inflexible responses to threat, here we examined how acute stress affects the flexible updating of threat responses. Participants completed an aversive learning task, in which one stimulus was probabilistically associated with an electric shock, while the other stimulus signaled safety. A day later, participants underwent an acute stress or control manipulation before completing a reversal learning task during which the original stimulus-outcome contingencies switched. Skin conductance and neuroendocrine responses provided indices of sympathetic arousal and stress responses, respectively. Despite equivalent initial learning, stressed participants showed marked impairments in reversal learning relative to controls. Additionally, reversal learning deficits across participants were related to heightened levels of alpha-amylase, a marker of noradrenergic activity. Finally, fitting arousal data to a computational reinforcement learning model revealed that stress-induced reversal learning deficits emerged from stress-specific changes in the weight assigned to prediction error signals, disrupting the adaptive adjustment of learning rates. Our findings provide insight into how stress renders individuals less sensitive to changes in aversive reinforcement and have implications for understanding clinical conditions marked by stress-related psychopathology.

Aversive learning strengthens episodic memory in both adolescents and adults.

Adolescence is often filled with positive and negative emotional experiences that may change how individuals remember and respond to stimuli in their environment. In adults, aversive events can both enhance memory for associated stimuli as well as generalize to enhance memory for unreinforced but conceptually related stimuli. The present study tested whether learned aversive associations similarly lead to better memory and generalization across a category of stimuli in adolescents. Participants completed an olfactory Pavlovian category conditioning task in which trial-unique exemplars from one of two categories were partially reinforced with an aversive odor. Participants then returned 24 h later to complete a recognition memory test. We found better corrected recognition memory for the reinforced versus the unreinforced category of stimuli in both adults and adolescents. Further analysis revealed that enhanced recognition memory was driven specifically by better memory for the reinforced exemplars. Autonomic arousal during learning was also related to subsequent memory. These findings build on previous work in adolescent and adult humans and rodents showing comparable acquisition of aversive Pavlovian conditioned responses across age groups and demonstrate that memory for stimuli with an acquired aversive association is enhanced in both adults and adolescents.

Neurocognitive Development of Motivated Behavior: Dynamic Changes across Childhood and Adolescence.

The ability to anticipate and respond appropriately to the challenges and opportunities present in our environments is critical for adaptive behavior. Recent methodological innovations have led to substantial advances in our understanding of the neurocircuitry supporting such motivated behavior in adulthood. However, the neural circuits and cognitive processes that enable threat- and reward-motivated behavior undergo substantive changes over the course of development, and these changes are less well understood. In this article, we highlight recent research in human and animal models demonstrating how developmental changes in prefrontal-subcortical neural circuits give rise to corresponding changes in the processing of threats and rewards from infancy to adulthood. We discuss how these developmental trajectories are altered by experiential factors, such as early-life stress, and highlight the relevance of this research for understanding the developmental onset and treatment of psychiatric disorders characterized by dysregulation of motivated behavior.

Active Avoidance: Neural Mechanisms and Attenuation of Pavlovian Conditioned Responding.

Patients with anxiety disorders often experience a relapse in symptoms after exposure therapy. Similarly, threat responses acquired during Pavlovian threat conditioning often return after extinction learning. Accordingly, there is a need for alternative methods to persistently reduce threat responding. Studies in rodents have suggested that exercising behavioral control over an aversive stimulus can persistently diminish threat responses, and that these effects are mediated by the amygdala, ventromedial prefrontal cortex, and striatum. In this fMRI study, we attempted to translate these findings to humans. Subjects first underwent threat conditioning. We then contrasted two forms of safety learning: active avoidance, in which participants could prevent the shock through an action, and yoked extinction, with shock presentation matched to the active condition, but without instrumental control. The following day, we assessed subjects' threat responses (measured by skin conductance) to the conditioned stimuli without shock. Subjects next underwent threat conditioning with novel stimuli. Yoked extinction subjects showed an increase in conditioned response to stimuli from the previous day, but the active avoidance group did not. Additionally, active avoidance subjects showed reduced conditioned responding during novel threat conditioning, but the extinction group did not. We observed between-group differences in striatal BOLD responses to shock omission in Avoidance/Extinction. These findings suggest a differential role for the striatum in human active avoidance versus extinction learning, and indicate that active avoidance may be more effective than extinction in persistently diminishing threat responses.SIGNIFICANCE STATEMENT Extinguished threat responses often reemerge with time, highlighting the importance of identifying more enduring means of attenuation. We compared the effects of active avoidance learning and yoked extinction on threat responses in humans and contrasted the neural circuitry engaged by these two processes. Subjects who learned to prevent a shock through an action maintained low threat responses after safety learning and showed attenuated threat conditioning with novel stimuli, in contrast to those who underwent yoked extinction. The results suggest that experiences of active control over threat engage the striatum and promote a shift from expression of innate defensive responses toward more adaptive behavioral responses to threatening stimuli.

From Creatures of Habit to Goal-Directed Learners: Tracking the Developmental Emergence of Model-Based Reinforcement Learning.

Theoretical models distinguish two decision-making strategies that have been formalized in reinforcement-learning theory. A model-based strategy leverages a cognitive model of potential actions and their consequences to make goal-directed choices, whereas a model-free strategy evaluates actions based solely on their reward history. Research in adults has begun to elucidate the psychological mechanisms and neural substrates underlying these learning processes and factors that influence their relative recruitment. However, the developmental trajectory of these evaluative strategies has not been well characterized. In this study, children, adolescents, and adults performed a sequential reinforcement-learning task that enabled estimation of model-based and model-free contributions to choice. Whereas a model-free strategy was apparent in choice behavior across all age groups, a model-based strategy was absent in children, became evident in adolescents, and strengthened in adults. These results suggest that recruitment of model-based valuation systems represents a critical cognitive component underlying the gradual maturation of goal-directed behavior.

Experiential reward learning outweighs instruction prior to adulthood.

Throughout our lives, we face the important task of distinguishing rewarding actions from those that are best avoided. Importantly, there are multiple means by which we acquire this information. Through trial and error, we use experiential feedback to evaluate our actions. We also learn which actions are advantageous through explicit instruction from others. Here, we examined whether the influence of these two forms of learning on choice changes across development by placing instruction and experience in competition in a probabilistic-learning task. Whereas inaccurate instruction markedly biased adults' estimations of a stimulus's value, children and adolescents were better able to objectively estimate stimulus values through experience. Instructional control of learning is thought to recruit prefrontal-striatal brain circuitry, which continues to mature into adulthood. Our behavioral data suggest that this protracted neurocognitive maturation may cause the motivated actions of children and adolescents to be less influenced by explicit instruction than are those of adults. This absence of a confirmation bias in children and adolescents represents a paradoxical developmental advantage of youth over adults in the unbiased evaluation of actions through positive and negative experience.

Individual differences in learning predict the return of fear.

Using a laboratory analogue of learned fear (Pavlovian fear conditioning), we show that there is substantial heterogeneity across individuals in spontaneous recovery of fear following extinction training. We propose that this heterogeneity might stem from qualitative individual differences in the nature of extinction learning. Whereas some individuals tend to form a new memory during extinction, leaving their fear memory intact, others update the original threat association with new safety information, effectively unlearning the fear memory. We formalize this account in a computational model of fear learning and show that individuals who, according to the model, are more likely to form new extinction memories tend to show greater spontaneous recovery compared to individuals who appear to only update a single memory. This qualitative variation in fear and extinction learning may have important implications for understanding vulnerability and resilience to fear-related psychiatric disorders.

Serotonin transporter polyadenylation polymorphism modulates the retention of fear extinction memory.

Growing evidence suggests serotonin's role in anxiety and depression is mediated by its effects on learned fear associations. Pharmacological and genetic manipulations of serotonin signaling in mice alter the retention of fear extinction learning, which is inversely associated with anxious temperament in mice and humans. Here, we test whether genetic variation in serotonin signaling in the form of a common human serotonin transporter polyadenylation polymorphism (STPP/rs3813034) is associated with spontaneous fear recovery after extinction. We show that the risk allele of this polymorphism is associated with impaired retention of fear extinction memory and heightened anxiety and depressive symptoms. These STPP associations in humans mirror the phenotypic effects of serotonin transporter knockout in mice, highlighting the STPP as a potential genetic locus underlying interindividual differences in serotonin transporter function in humans. Furthermore, we show that the serotonin transporter polyadenylation profile associated with the STPP risk allele is altered through the chronic administration of fluoxetine, a treatment that also facilitates retention of extinction learning. The propensity to form persistent fear associations due to poor extinction recall may be an intermediate phenotype mediating the effects of genetic variation in serotonergic function on anxiety and depression. The consistency and specificity of these data across species provide robust support for this hypothesis and suggest that the little-studied STPP may be an important risk factor for mood and anxiety disorders in humans.

Altered fear learning across development in both mouse and human.

The only evidence-based behavioral treatment for anxiety and stress-related disorders involves desensitization techniques that rely on principles of extinction learning. However, 40% of patients do not respond to this treatment. Efforts have focused on individual differences in treatment response, but have not examined when, during development, such treatments may be most effective. We examined fear-extinction learning across development in mice and humans. Parallel behavioral studies revealed attenuated extinction learning during adolescence. Probing neural circuitry in mice revealed altered synaptic plasticity of prefrontal cortical regions implicated in suppression of fear responses across development. The results suggest a lack of synaptic plasticity in the prefrontal regions, during adolescence, is associated with blunted regulation of fear extinction. These findings provide insight into optimizing treatment outcomes for when, during development, exposure therapies may be most effective.

Interoceptive ability predicts aversion to losses.

Emotions have been proposed to inform risky decision-making through the influence of affective physiological responses on subjective value. The ability to perceive internal body states, or "interoception" may influence this relationship. Here, we examined whether interoception predicts participants' degree of loss aversion, which has been previously linked to choice-related arousal responses. Participants performed both a heartbeat-detection task indexing interoception and a risky monetary decision-making task, from which loss aversion, risk attitudes and choice consistency were parametrically measured. Interoceptive ability correlated selectively with loss aversion and was unrelated to the other value parameters. This finding suggests that specific and separable component processes underlying valuation are shaped not only by our physiological responses, as shown in previous findings, but also by our interoceptive access to such signals.