Different brain systems support learning from received and avoided pain during human pain-avoidance learning.

Both unexpected pain and unexpected pain absence can drive avoidance learning, but whether they do so via shared or separate neural and neurochemical systems is largely unknown. To address this issue, we combined an instrumental pain-avoidance learning task with computational modeling, functional magnetic resonance imaging (fMRI), and pharmacological manipulations of the dopaminergic (100 mg levodopa) and opioidergic (50 mg naltrexone) systems (N = 83). Computational modeling provided evidence that untreated participants learned more from received than avoided pain. Our dopamine and opioid manipulations negated this learning asymmetry by selectively increasing learning rates for avoided pain. Furthermore, our fMRI analyses revealed that pain prediction errors were encoded in subcortical and limbic brain regions, whereas no-pain prediction errors were encoded in frontal and parietal cortical regions. However, we found no effects of our pharmacological manipulations on the neural encoding of prediction errors. Together, our results suggest that human pain-avoidance learning is supported by separate threat- and safety-learning systems, and that dopamine and endogenous opioids specifically regulate learning from successfully avoided pain.

A multistudy analysis reveals that evoked pain intensity representation is distributed across brain systems.

Information is coded in the brain at multiple anatomical scales: locally, distributed across regions and networks, and globally. For pain, the scale of representation has not been formally tested, and quantitative comparisons of pain representations across regions and networks are lacking. In this multistudy analysis of 376 participants across 11 studies, we compared multivariate predictive models to investigate the spatial scale and location of evoked heat pain intensity representation. We compared models based on (a) a single most pain-predictive region or resting-state network; (b) pain-associated cortical-subcortical systems developed from prior literature ("multisystem models"); and (c) a model spanning the full brain. We estimated model accuracy using leave-one-study-out cross-validation (CV; 7 studies) and subsequently validated in 4 independent holdout studies. All spatial scales conveyed information about pain intensity, but distributed, multisystem models predicted pain 20% more accurately than any individual region or network and were more generalizable to multimodal pain (thermal, visceral, and mechanical) and specific to pain. Full brain models showed no predictive advantage over multisystem models. These findings show that multiple cortical and subcortical systems are needed to decode pain intensity, especially heat pain, and that representation of pain experience may not be circumscribed by any elementary region or canonical network. Finally, the learner generalization methods we employ provide a blueprint for evaluating the spatial scale of information in other domains.

Individual variability in brain representations of pain.

Characterizing cerebral contributions to individual variability in pain processing is crucial for personalized pain medicine, but has yet to be done. In the present study, we address this problem by identifying brain regions with high versus low interindividual variability in their relationship with pain. We trained idiographic pain-predictive models with 13 single-trial functional MRI datasets (n = 404, discovery set) and quantified voxel-level importance for individualized pain prediction. With 21 regions identified as important pain predictors, we examined the interindividual variability of local pain-predictive weights in these regions. Higher-order transmodal regions, such as ventromedial and ventrolateral prefrontal cortices, showed larger individual variability, whereas unimodal regions, such as somatomotor cortices, showed more stable pain representations across individuals. We replicated this result in an independent dataset (n = 124). Overall, our study identifies cerebral sources of individual differences in pain processing, providing potential targets for personalized assessment and treatment of pain.

Impaired learning to dissociate advantageous and disadvantageous risky choices in adolescents.

Adolescence is characterized by a surge in maladaptive risk-taking behaviors, but whether and how this relates to developmental changes in experience-based learning is largely unknown. In this preregistered study, we addressed this issue using a novel task that allowed us to separate the learning-driven optimization of risky choice behavior over time from overall risk-taking tendencies. Adolescents (12-17 years old) learned to dissociate advantageous from disadvantageous risky choices less well than adults (20-35 years old), and this impairment was stronger in early than mid-late adolescents. Computational modeling revealed that adolescents' suboptimal performance was largely due to an inefficiency in core learning and choice processes. Specifically, adolescents used a simpler, suboptimal, expectation-updating process and a more stochastic choice policy. In addition, the modeling results suggested that adolescents, but not adults, overvalued the highest rewards. Finally, an exploratory latent-mixture model analysis indicated that a substantial proportion of the participants in each age group did not engage in experience-based learning but used a gambler's fallacy strategy, stressing the importance of analyzing individual differences. Our results help understand why adolescents tend to make more, and more persistent, maladaptive risky decisions than adults when the values of these decisions have to be learned from experience.

Effect sizes and test-retest reliability of the fMRI-based neurologic pain signature.

Identifying biomarkers that predict mental states with large effect sizes and high test-retest reliability is a growing priority for fMRI research. We examined a well-established multivariate brain measure that tracks pain induced by nociceptive input, the Neurologic Pain Signature (NPS). In N = 295 participants across eight studies, NPS responses showed a very large effect size in predicting within-person single-trial pain reports (d = 1.45) and medium effect size in predicting individual differences in pain reports (d = 0.49). The NPS showed excellent short-term (within-day) test-retest reliability (ICC = 0.84, with average 69.5 trials/person). Reliability scaled with the number of trials within-person, with ≥60 trials required for excellent test-retest reliability. Reliability was tested in two additional studies across 5-day (N = 29, ICC = 0.74, 30 trials/person) and 1-month (N = 40, ICC = 0.46, 5 trials/person) test-retest intervals. The combination of strong within-person correlations and only modest between-person correlations between the NPS and pain reports indicate that the two measures have different sources of between-person variance. The NPS is not a surrogate for individual differences in pain reports but can serve as a reliable measure of pain-related physiology and mechanistic target for interventions.

Effects of advice on experienced-based learning in adolescents and adults.

Recent studies that compared effects of pre-learning advice on experience-based learning in adolescents and adults have yielded mixed results. Previous studies on this topic used choice tasks in which age-related differences in advice-related learning bias and exploratory choice behavior are difficult to dissociate. Moreover, these studies did not examine whether effects of advice depend on working memory load. In this preregistered study (in adolescents [13-15 years old] and adults [18-31 years old]), we addressed these issues by factorially combining advice and working memory load manipulations in an estimation task that does not require choices and hence eliminates the influence of known age-related differences in exploration. We found that advice guided participants' initial estimates in both age groups. When advice was correct, this improved estimation performance, especially in adolescents when working memory load was high. When advice was incorrect, it had a longer-lasting effect on adolescents' performance than on adults' performance. In contrast to previous findings in choice tasks, we found no evidence that advice biased learning in either age group. Taken together, our results suggest that learning in an estimation task improves between adolescence and adulthood but that the effects of advice on learning do not differ substantially between adolescents and adults.

Uncertainty-driven regulation of learning and exploration in adolescents: A computational account.

Healthy adults flexibly adapt their learning strategies to ongoing changes in uncertainty, a key feature of adaptive behaviour. However, the developmental trajectory of this ability is yet unknown, as developmental studies have not incorporated trial-to-trial variation in uncertainty in their analyses or models. To address this issue, we compared adolescents' and adults' trial-to-trial dynamics of uncertainty, learning rate, and exploration in two tasks that assess learning in noisy but otherwise stable environments. In an estimation task-which provides direct indices of trial-specific learning rate-both age groups reduced their learning rate over time, as self-reported uncertainty decreased. Accordingly, the estimation data in both groups was better explained by a Bayesian model with dynamic learning rate (Kalman filter) than by conventional reinforcement-learning models. Furthermore, adolescents' learning rates asymptoted at a higher level, reflecting an over-weighting of the most recent outcome, and the estimated Kalman-filter parameters suggested that this was due to an overestimation of environmental volatility. In a choice task, both age groups became more likely to choose the higher-valued option over time, but this increase in choice accuracy was smaller in the adolescents. In contrast to the estimation task, we found no evidence for a Bayesian expectation-updating process in the choice task, suggesting that estimation and choice tasks engage different learning processes. However, our modeling results of the choice task suggested that both age groups reduced their degree of exploration over time, and that the adolescents explored overall more than the adults. Finally, age-related differences in exploration parameters from fits to the choice data were mediated by participants' volatility parameter from fits to the estimation data. Together, these results suggest that adolescents overestimate the rate of environmental change, resulting in elevated learning rates and increased exploration, which may help understand developmental changes in learning and decision-making.

Pain-Evoked Reorganization in Functional Brain Networks.

Recent studies indicate that a significant reorganization of cerebral networks may occur in patients with chronic pain, but how immediate pain experience influences the organization of large-scale functional networks is not yet well characterized. To investigate this question, we used functional magnetic resonance imaging in 106 participants experiencing both noxious and innocuous heat. Painful stimulation caused network-level reorganization of cerebral connectivity that differed substantially from organization during innocuous stimulation and standard resting-state networks. Noxious stimuli increased somatosensory network connectivity with (a) frontoparietal networks involved in context representation, (b) "ventral attention network" regions involved in motivated action selection, and (c) basal ganglia and brainstem regions. This resulted in reduced "small-worldness," modularity (fewer networks), and global network efficiency and in the emergence of an integrated "pain supersystem" (PS) whose activity predicted individual differences in pain sensitivity across 5 participant cohorts. Network hubs were reorganized ("hub disruption") so that more hubs were localized in PS, and there was a shift from "connector" hubs linking disparate networks to "provincial" hubs connecting regions within PS. Our findings suggest that pain reorganizes the network structure of large-scale brain systems. These changes may prioritize responses to painful events and provide nociceptive systems privileged access to central control of cognition and action during pain.

Different brain networks mediate the effects of social and conditioned expectations on pain.

Information about others' experiences can strongly influence our own feelings and decisions. But how does such social information affect the neural generation of affective experience, and are the brain mechanisms involved distinct from those that mediate other types of expectation effects? Here, we used fMRI to dissociate the brain mediators of social influence and associative learning effects on pain. Participants viewed symbolic depictions of other participants' pain ratings (social information) and classically conditioned pain-predictive cues before experiencing painful heat. Social information and conditioned stimuli each had significant effects on pain ratings, and both effects were mediated by self-reported expectations. Yet, these effects were mediated by largely separable brain activity patterns, involving different large-scale functional networks. These results show that learned versus socially instructed expectations modulate pain via partially different mechanisms-a distinction that should be accounted for by theories of predictive coding and related top-down influences.

Noradrenergic and Cholinergic Modulation of Belief Updating.

To make optimal predictions in a dynamic environment, the impact of new observations on existing beliefs-that is, the learning rate-should be guided by ongoing estimates of change and uncertainty. Theoretical work has proposed specific computational roles for various neuromodulatory systems in the control of learning rate, but empirical evidence is still sparse. The aim of the current research was to examine the role of the noradrenergic and cholinergic systems in learning rate regulation. First, we replicated our recent findings that the centroparietal P3 component of the EEG-an index of phasic catecholamine release in the cortex-predicts trial-to-trial variability in learning rate and mediates the effects of surprise and belief uncertainty on learning rate (Study 1, n = 17). Second, we found that pharmacological suppression of either norepinephrine or acetylcholine activity produced baseline-dependent effects on learning rate following nonobvious changes in an outcome-generating process (Study 1). Third, we identified two genes, coding for α2A receptor sensitivity (ADRA2A) and norepinephrine reuptake (NET), as promising targets for future research on the genetic basis of individual differences in learning rate (Study 2, n = 137). Our findings suggest a role for the noradrenergic and cholinergic systems in belief updating and underline the importance of studying interactions between different neuromodulatory systems.

Behavioural and neural evidence for self-reinforcing expectancy effects on pain.

Beliefs and expectations often persist despite evidence to the contrary. Here we examine two potential mechanisms underlying such 'self-reinforcing' expectancy effects in the pain domain: modulation of perception and biased learning. In two experiments, cues previously associated with symbolic representations of high or low temperatures preceded painful heat. We examined trial-to-trial dynamics in participants' expected pain, reported pain and brain activity. Subjective and neural pain responses assimilated towards cue-based expectations, and pain responses in turn predicted subsequent expectations, creating a positive dynamic feedback loop. Furthermore, we found evidence for a confirmation bias in learning: higher- and lower-than-expected pain triggered greater expectation updating for high- and low-pain cues, respectively. Individual differences in this bias were reflected in the updating of pain-anticipatory brain activity. Computational modelling provided converging evidence that expectations influence both perception and learning. Together, perceptual assimilation and biased learning promote self-reinforcing expectations, helping to explain why beliefs can be resistant to change.

What's in a word? How instructions, suggestions, and social information change pain and emotion.

Instructions, suggestions, and other types of social information can have powerful effects on pain and emotion. Prominent examples include observational learning, social influence, placebo, and hypnosis. These different phenomena and their underlying brain mechanisms have been studied in partially separate literatures, which we discuss, compare, and integrate in this review. Converging findings from these literatures suggest that (1) instructions and social information affect brain systems associated with the generation of pain and emotion, and with reinforcement learning, and that (2) these changes are mediated by alterations in prefrontal systems responsible for top-down control and the generation of affective meaning. We argue that changes in expectation and appraisal, a process of assessing personal meaning and implications for wellbeing, are two potential key mediators of the effects of instructions and social information on affective experience. Finally, we propose a tentative model of how prefrontal regions, especially dorsolateral and ventromedial prefrontal cortex may regulate affective processing based on instructions and socially transmitted expectations more broadly.

Quantifying cerebral contributions to pain beyond nociception.

Cerebral processes contribute to pain beyond the level of nociceptive input and mediate psychological and behavioural influences. However, cerebral contributions beyond nociception are not yet well characterized, leading to a predominant focus on nociception when studying pain and developing interventions. Here we use functional magnetic resonance imaging combined with machine learning to develop a multivariate pattern signature-termed the stimulus intensity independent pain signature-1 (SIIPS1)-that predicts pain above and beyond nociceptive input in four training data sets (Studies 1-4, N=137). The SIIPS1 includes patterns of activity in nucleus accumbens, lateral prefrontal and parahippocampal cortices, and other regions. In cross-validated analyses of Studies 1-4 and in two independent test data sets (Studies 5-6, N=46), SIIPS1 responses explain variation in trial-by-trial pain ratings not captured by a previous fMRI-based marker for nociceptive pain. In addition, SIIPS1 responses mediate the pain-modulating effects of three psychological manipulations of expectations and perceived control. The SIIPS1 provides an extensible characterization of cerebral contributions to pain and specific brain targets for interventions.

Spared internal but impaired external reward prediction error signals in major depressive disorder during reinforcement learning.

BACKGROUND: Major depressive disorder (MDD) creates debilitating effects on a wide range of cognitive functions, including reinforcement learning (RL). In this study, we sought to assess whether reward processing as such, or alternatively the complex interplay between motivation and reward might potentially account for the abnormal reward-based learning in MDD. METHODS: A total of 35 treatment resistant MDD patients and 44 age matched healthy controls (HCs) performed a standard probabilistic learning task. RL was titrated using behavioral, computational modeling and event-related brain potentials (ERPs) data. RESULTS: MDD patients showed comparable learning rate compared to HCs. However, they showed decreased lose-shift responses as well as blunted subjective evaluations of the reinforcers used during the task, relative to HCs. Moreover, MDD patients showed normal internal (at the level of error-related negativity, ERN) but abnormal external (at the level of feedback-related negativity, FRN) reward prediction error (RPE) signals during RL, selectively when additional efforts had to be made to establish learning. CONCLUSIONS: Collectively, these results lend support to the assumption that MDD does not impair reward processing per se during RL. Instead, it seems to alter the processing of the emotional value of (external) reinforcers during RL, when additional intrinsic motivational processes have to be engaged.

Group-regularized individual prediction: theory and application to pain.

Multivariate pattern analysis (MVPA) has become an important tool for identifying brain representations of psychological processes and clinical outcomes using fMRI and related methods. Such methods can be used to predict or 'decode' psychological states in individual subjects. Single-subject MVPA approaches, however, are limited by the amount and quality of individual-subject data. In spite of higher spatial resolution, predictive accuracy from single-subject data often does not exceed what can be accomplished using coarser, group-level maps, because single-subject patterns are trained on limited amounts of often-noisy data. Here, we present a method that combines population-level priors, in the form of biomarker patterns developed on prior samples, with single-subject MVPA maps to improve single-subject prediction. Theoretical results and simulations motivate a weighting based on the relative variances of biomarker-based prediction-based on population-level predictive maps from prior groups-and individual-subject, cross-validated prediction. Empirical results predicting pain using brain activity on a trial-by-trial basis (single-trial prediction) across 6 studies (N=180 participants) confirm the theoretical predictions. Regularization based on a population-level biomarker-in this case, the Neurologic Pain Signature (NPS)-improved single-subject prediction accuracy compared with idiographic maps based on the individuals' data alone. The regularization scheme that we propose, which we term group-regularized individual prediction (GRIP), can be applied broadly to within-person MVPA-based prediction. We also show how GRIP can be used to evaluate data quality and provide benchmarks for the appropriateness of population-level maps like the NPS for a given individual or study.

Catecholaminergic Regulation of Learning Rate in a Dynamic Environment.

Adaptive behavior in a changing world requires flexibly adapting one's rate of learning to the rate of environmental change. Recent studies have examined the computational mechanisms by which various environmental factors determine the impact of new outcomes on existing beliefs (i.e., the 'learning rate'). However, the brain mechanisms, and in particular the neuromodulators, involved in this process are still largely unknown. The brain-wide neurophysiological effects of the catecholamines norepinephrine and dopamine on stimulus-evoked cortical responses suggest that the catecholamine systems are well positioned to regulate learning about environmental change, but more direct evidence for a role of this system is scant. Here, we report evidence from a study employing pharmacology, scalp electrophysiology and computational modeling (N = 32) that suggests an important role for catecholamines in learning rate regulation. We found that the P3 component of the EEG-an electrophysiological index of outcome-evoked phasic catecholamine release in the cortex-predicted learning rate, and formally mediated the effect of prediction-error magnitude on learning rate. P3 amplitude also mediated the effects of two computational variables-capturing the unexpectedness of an outcome and the uncertainty of a preexisting belief-on learning rate. Furthermore, a pharmacological manipulation of catecholamine activity affected learning rate following unanticipated task changes, in a way that depended on participants' baseline learning rate. Our findings provide converging evidence for a causal role of the human catecholamine systems in learning-rate regulation as a function of environmental change.

What is in the feedback? Effect of induced happiness vs. sadness on probabilistic learning with vs. without exploration.

According to dominant neuropsychological theories of affect, emotions signal salience of events and in turn facilitate a wide spectrum of response options or action tendencies. Valence of an emotional experience is pivotal here, as it alters reward and punishment processing, as well as the balance between safety and risk taking, which can be translated into changes in the exploration-exploitation trade-off during reinforcement learning (RL). To test this idea, we compared the behavioral performance of three groups of participants that all completed a variant of a standard probabilistic learning task, but who differed regarding which mood state was actually induced and maintained (happy, sad or neutral). To foster a change from an exploration to an exploitation-based mode, we removed feedback information once learning was reliably established. Although changes in mood were successful, learning performance was balanced between the three groups. Critically, when focusing on exploitation-driven learning only, they did not differ either. Moreover, mood valence did not alter the learning rate or exploration per se, when titrated using complementing computational modeling. By comparing systematically these results to our previous study (Bakic et al., 2014), we found that arousal levels did differ between studies, which might account for limited modulatory effects of (positive) mood on RL in the present case. These results challenge the assumption that mood valence alone is enough to create strong shifts in the way exploitation or exploration is eventually carried out during (probabilistic) learning. In this context, we discuss the possibility that both valence and arousal are actually necessary components of the emotional mood state to yield changes in the use and exploration of incentives cues during RL.

Conceptual Conditioning: Mechanisms Mediating Conditioning Effects on Pain.

Classical conditioning can profoundly modify subsequent pain responses, but the mechanisms that drive this effect are unresolved. In pain-conditioning studies, cues are typically conditioned to primary aversive reinforcers; hence, subsequent pain modulation could reflect learned precognitive associations (i.e., those involving neural plasticity independent of expectations and other forms of conceptual thought) or conceptual expectancies. We isolated conceptual contributions using a thermal pain-conditioning procedure in which different conditioned stimulus (CS) cues were repeatedly paired with symbolic representations of high and low noxious heat. In a subsequent test phase, identical noxious stimuli evoked larger skin conductance responses (SCRs) and pain ratings when preceded by CS cues associated with high temperature than by those associated with low temperature. These effects were mediated by participants' self-reported expectancies. CS cues associated with high temperature also evoked larger anticipatory SCRs than did CS cues associated with low temperature, but larger anticipatory SCRs predicted smaller subsequent heat-evoked SCRs. These results provide novel evidence that conditioned modulation of pain physiology can be acquired through purely conceptual processes, and that self-reported expectancies and physiological threat responses have opposing effects on pain.

Representation of aversive prediction errors in the human periaqueductal gray.

Pain is a primary driver of learning and motivated action. It is also a target of learning, as nociceptive brain responses are shaped by learning processes. We combined an instrumental pain avoidance task with an axiomatic approach to assessing fMRI signals related to prediction errors (PEs), which drive reinforcement-based learning. We found that pain PEs were encoded in the periaqueductal gray (PAG), a structure important for pain control and learning in animal models. Axiomatic tests combined with dynamic causal modeling suggested that ventromedial prefrontal cortex, supported by putamen, provides an expected value-related input to the PAG, which then conveys PE signals to prefrontal regions important for behavioral regulation, including orbitofrontal, anterior mid-cingulate and dorsomedial prefrontal cortices. Thus, pain-related learning involves distinct neural circuitry, with implications for behavior and pain dynamics.

Effects of positive mood on probabilistic learning: behavioral and electrophysiological correlates.

Whether positive mood can change reinforcement learning or not remains an open question. In this study, we used a probabilistic learning task and explored whether positive mood could alter the way positive versus negative feedback was used to guide learning. This process was characterized both at the behavioral and electro-encephalographic levels. Thirty two participants were randomly allocated either to a positive or a neutral (control) mood condition. Behavioral results showed that while learning performance was balanced between the two groups, participants in the positive mood group had a higher learning rate than participants in the neutral mood group. At the electrophysiological level, we found that positive mood increased the error-related negativity when the stimulus-response associations were deterministic, selectively (as opposed to random or probabilistic). However, it did not influence the feedback-related negativity. These new findings are discussed in terms of an enhanced internal reward prediction error signal after the induction of positive mood when the probability of getting a reward is high.