Influence of surprise on reinforcement learning in younger and older adults.

Surprise is a key component of many learning experiences, and yet its precise computational role, and how it changes with age, remain debated. One major challenge is that surprise often occurs jointly with other variables, such as uncertainty and outcome probability. To assess how humans learn from surprising events, and whether aging affects this process, we studied choices while participants learned from bandits with either Gaussian or bi-modal outcome distributions, which decoupled outcome probability, uncertainty, and surprise. A total of 102 participants (51 older, aged 50-73; 51 younger, 19-30 years) chose between three bandits, one of which had a bimodal outcome distribution. Behavioral analyses showed that both age-groups learned the average of the bimodal bandit less well. A trial-by-trial analysis indicated that participants performed choice reversals immediately following large absolute prediction errors, consistent with heightened sensitivity to surprise. This effect was stronger in older adults. Computational models indicated that learning rates in younger as well as older adults were influenced by surprise, rather than uncertainty, but also suggested large interindividual variability in the process underlying learning in our task. Our work bridges between behavioral economics research that has focused on how outcomes with low probability affect choice in older adults, and reinforcement learning work that has investigated age differences in the effects of uncertainty and suggests that older adults overly adapt to surprising events, even when accounting for probability and uncertainty effects.

Reduction of Aversive Learning Rates in Pavlovian Conditioning by Angiotensin II Antagonist Losartan: A Randomized Controlled Trial.

BACKGROUND: Angiotensin receptor blockade has been linked to aspects of aversive learning and memory formation and to the prevention of posttraumatic stress disorder symptom development. METHODS: We investigated the influence of the angiotensin receptor blocker losartan on aversive Pavlovian conditioning using a probabilistic learning paradigm. In a double-blind, randomized, placebo-controlled design, we tested 45 (18 female) healthy volunteers during a baseline session, after application of losartan or placebo (drug session), and during a follow-up session. During each session, participants engaged in a task in which they had to predict the probability of an electrical stimulation on every trial while the true shock contingencies switched repeatedly between phases of high and low shock threat. Computational reinforcement learning models were used to investigate learning dynamics. RESULTS: Acute administration of losartan significantly reduced participants' adjustment during both low-to-high and high-to-low threat changes. This was driven by reduced aversive learning rates in the losartan group during the drug session compared with baseline. The 50-mg drug dose did not induce reduction of blood pressure or change in reaction times, ruling out a general reduction in attention and engagement. Decreased adjustment of aversive expectations was maintained at a follow-up session 24 hours later. CONCLUSIONS: This study shows that losartan acutely reduces Pavlovian learning in aversive environments, thereby highlighting a potential role of the renin-angiotensin system in anxiety development.

Better late than never: sleep still supports memory consolidation after prolonged periods of wakefulness.

While the benefits of sleep for associative memory are well established, it is unclear whether single-item memories profit from overnight consolidation to the same extent. We addressed this question in a preregistered, online study and also investigated how the temporal proximity between learning and sleep influences overnight retention. Sleep relative to wakefulness improved retention of item and associative memories to similar extents irrespective of whether sleep occurred soon after learning or following a prolonged waking interval. Our findings highlight the far-reaching influences of sleep on memory that can arise even after substantial periods of wakefulness.

Trait anxiety is associated with hidden state inference during aversive reversal learning.

Updating beliefs in changing environments can be driven by gradually adapting expectations or by relying on inferred hidden states (i.e. contexts), and changes therein. Previous work suggests that increased reliance on context could underly fear relapse phenomena that hinder clinical treatment of anxiety disorders. We test whether trait anxiety variations in a healthy population influence how much individuals rely on hidden-state inference. In a Pavlovian learning task, participants observed cues that predicted an upcoming electrical shock with repeatedly changing probability, and were asked to provide expectancy ratings on every trial. We show that trait anxiety is associated with steeper expectation switches after contingency reversals and reduced oddball learning. Furthermore, trait anxiety is related to better fit of a state inference, compared to a gradual learning, model when contingency changes are large. Our findings support previous work suggesting hidden-state inference as a mechanism behind anxiety-related to fear relapse phenomena.

Impaired adaptation of learning to contingency volatility in internalizing psychopathology.

Using a contingency volatility manipulation, we tested the hypothesis that difficulty adapting probabilistic decision-making to second-order uncertainty might reflect a core deficit that cuts across anxiety and depression and holds regardless of whether outcomes are aversive or involve reward gain or loss. We used bifactor modeling of internalizing symptoms to separate symptom variance common to both anxiety and depression from that unique to each. Across two experiments, we modeled performance on a probabilistic decision-making under volatility task using a hierarchical Bayesian framework. Elevated scores on the common internalizing factor, with high loadings across anxiety and depression items, were linked to impoverished adjustment of learning to volatility regardless of whether outcomes involved reward gain, electrical stimulation, or reward loss. In particular, high common factor scores were linked to dampened learning following better-than-expected outcomes in volatile environments. No such relationships were observed for anxiety- or depression-specific symptom factors.

Stratification of MDD and GAD patients by resting state brain connectivity predicts cognitive bias.

Patients with Generalized Anxiety Disorder (GAD) and Major Depressive Disorder (MDD) show between-group comorbidity and symptom overlap, and within-group heterogeneity. Resting state functional connectivity might provide an alternate, biologically informed means by which to stratify patients with GAD or MDD. Resting state functional magnetic resonance imaging data were acquired from 23 adults with GAD, 21 adults with MDD, and 27 healthy adult control participants. We investigated whether within- or between-network connectivity indices from five resting state networks predicted scores on continuous measures of depression and anxiety. Successful predictors were used to stratify participants into two new groups. We examined whether this stratification predicted attentional bias towards threat and whether this varied between patients and controls. Depression scores were linked to elevated connectivity within a limbic network including the amygdala, hippocampus, VMPFC and subgenual ACC. Patients with GAD or MDD with high limbic connectivity showed poorer performance on an attention-to-threat task than patients with low limbic connectivity. No parallel effect was observed for control participants, resulting in an interaction of clinical status by resting state group. Our findings provide initial evidence for the external validity of stratification of MDD and GAD patients by functional connectivity markers. This stratification cuts across diagnostic boundaries and might valuably inform future intervention studies. Our findings also highlight that biomarkers of interest can have different cognitive correlates in individuals with versus without clinically significant symptomatology. This might reflect protective influences leading to resilience in some individuals but not others.

Second language feedback abolishes the "hot hand" effect during even-probability gambling.

Research into language-emotion interactions has revealed intriguing cognitive inhibition effects by emotionally negative words in bilinguals. Here, we turn to the domain of human risk taking and show that the experience of positive recency in games of chance-the "hot hand" effect-is diminished when game outcomes are provided in a second language rather than the native language. We engaged late Chinese-English bilinguals with "play" or "leave" decisions upon presentation of equal-odds bets while manipulating language of feedback and outcome value. When positive game outcomes were presented in their second language, English, participants subsequently took significantly fewer gambles and responded slower compared with the trials in which equivalent feedback was provided in Chinese, their native language. Positive feedback was identified as driving the cross-language difference in preference for risk over certainty: feedback for previous winning outcomes presented in Chinese increased subsequent risk taking, whereas in the English context no such effect was observed. Complementing this behavioral effect, event-related brain potentials elicited by feedback words showed an amplified response to Chinese relative to English in the feedback-related negativity window, indicating a stronger impact in the native than in the second language. We also observed a main effect of language on P300 amplitude and found it correlated with the cross-language difference in risk selections, suggesting that the greater the difference in attention between languages, the greater the difference in risk-taking behavior. These results provide evidence that the hot hand effect is at least attenuated when an individual operates in a non-native language.

Characterizing the associative content of brain structures involved in habitual and goal-directed actions in humans: a multivariate FMRI study.

While there is accumulating evidence for the existence of distinct neural systems supporting goal-directed and habitual action selection in the mammalian brain, much less is known about the nature of the information being processed in these different brain regions. Associative learning theory predicts that brain systems involved in habitual control, such as the dorsolateral striatum, should contain stimulus and response information only, but not outcome information, while regions involved in goal-directed action, such as ventromedial and dorsolateral prefrontal cortex and dorsomedial striatum, should be involved in processing information about outcomes as well as stimuli and responses. To test this prediction, human participants underwent fMRI while engaging in a binary choice task designed to enable the separate identification of these different representations with a multivariate classification analysis approach. Consistent with our predictions, the dorsolateral striatum contained information about responses but not outcomes at the time of an initial stimulus, while the regions implicated in goal-directed action selection contained information about both responses and outcomes. These findings suggest that differential contributions of these regions to habitual and goal-directed behavioral control may depend in part on basic differences in the type of information that these regions have access to at the time of decision making.