The effects of extinction and an explicitly unpaired treatment on the reinforcing properties of a Pavlovian conditioned stimulus.

This series of experiments examined the effects of extinction and an explicitly unpaired treatment on the ability of a conditioned stimulus (CS) to function as a reinforcer. Rats were trained to lever press for food, exposed to pairings of a noise CS and food, and, finally, tested for their willingness to lever press for the CS in the absence of the food. Experiment 1 provided a demonstration of conditioned reinforcement (using controls that were only exposed to unpaired presentations of the CS and food) and showed that it was equivalent after one or four sessions of CS-food pairings. Experiments 2 and 3 showed that, after one session of CS-food pairings, repeated presentations of the CS alone reduced its reinforcing properties; but after four sessions of CS-food pairings, repeated presentations of the CS alone had no effect on these properties. Experiment 4 showed that, after four sessions of CS-food pairings, explicitly unpaired presentations of the CS and food completely undermined conditioned reinforcement. Finally, Experiment 5 provided within-experiment evidence that, after four sessions of CS-food pairings, the reinforcing properties of the CS were disrupted by explicitly unpaired presentations of the CS and food but spared by repeated presentations of the CS alone. Together, these findings indicate that the effectiveness of extinction in undermining the reinforcing properties of a CS depends on its level of conditioning; and that, where extinction fails to disrupt these properties, they are successfully undermined by an explicitly unpaired treatment. They are discussed with respect to findings in the literature on Pavlovian-to-instrumental transfer; and the Rescorla-Wagner model, which anticipates that an explicitly unpaired treatment will be more effective than extinction in reversing the effects of conditioning.

Why is the Rescorla-Wagner model so influential?

The influence of the Rescorla-Wagner model cannot be overestimated, despite that (1) the model does not differ much computationally from its predecessors and competitors, and (2) its shortcomings are well-known in the learning community. Here we discuss the reasons behind its widespread influence in the cognitive and neural sciences, and argue that it is the constant search for general-process theories by learning scholars which eventually produced a model whose application spans many different areas of research to this day. We focus on the theoretical and empirical background of the model, the theoretical connections that it has with later developments across Marr's levels of analysis, as well as the broad variety of research that it has guided and inspired.

Aberrant reward prediction errors in young adult at-risk alcohol users.

Previous studies suggest that individuals with substance use disorder have abnormally large responses to unexpected outcomes (reward prediction errors [RPEs]). However, there is much less information on RPE in individuals at risk of alcohol misuse, prior to neurobiological adaptations that might result from sustained alcohol use. Here, participants (mean age 23.77 years, range 18-32 years) performed the electrophysiological monetary incentive delay task. This task involved responding to a target stimulus following reward incentive cues to win, or avoid losing, the cued reward while brain activity was recorded under 64-channel EEG. The Alcohol Use Disorders Identification Test (AUDIT) was used to quantify at-risk alcohol use, with high (n = 22, mean AUDIT score: 13.82) and low (n = 22, mean AUDIT score: 5.77) alcohol use groups. Trial-by-trial RPEs were estimated using a Rescorla-Wagner reinforcement model based on behavioral data. A single-trial analysis revealed that the feedback-related negativity (FRN) and feedback P3 (fb-P3) event-related potential components were significantly modulated by RPEs. There was increased RPE-related fb-P3 amplitude for those in the high alcohol use group. Next, the mean amplitude of ERPs elicited by positive and negative RPEs were compared between groups. We found that high alcohol use participants had attenuated FRN amplitude in contrast with low alcohol use participants for both positive and negative RPEs but enhanced fb-P3 for both positive and negative RPE. These results, with differences in RPE in an at-risk group, suggest that RPE a potential vulnerability marker for alcohol use disorder.

Graded error signals in eyeblink conditioning.

Minimizing errors is an important aspect of learning. However, it is not enough merely to record if an error occurred. For efficient learning, information about the magnitude of errors is critical. Did my tennis swing completely miss the target or did I hit the ball, but not quite in the sweet spot? How can neurons - which have traditionally been thought of as binary units - signal the magnitude of an error? Here I review evidence that eyeblink conditioning - a basic form of motor learning - depends on graded signals from the inferior olive which guides plasticity in the cerebellum and ultimately tunes behavior. Specifically, evidence suggests that: (1) Error signals are conveyed to the cerebellum via the inferior olive; (2) Signals from the inferior olive are graded; (3) The strength of the olivary signal affects learning; (4) Cerebellar feedback influences the strength of the olivary signal. I end the review by exploring how graded error signals might explain some behavioral learning phenomena.

Rescorla-Wagner Models with Sparse Dynamic Attention.

The Rescorla-Wagner (R-W) model describes human associative learning by proposing that an agent updates associations between stimuli, such as events in their environment or predictive cues, proportionally to a prediction error. While this model has proven informative in experiments, it has been posited that humans selectively attend to certain cues to overcome a problem with the R-W model scaling to large cue dimensions. We formally characterize this scaling problem and provide a solution that involves limiting attention in a R-W model to a sparse set of cues. Given the universal difficulty in selecting features for prediction, sparse attention faces challenges beyond those faced by the R-W model. We demonstrate several ways in which a naive attention model can fail explain those failures and leverage that understanding to produce a Sparse Attention R-W with Inference framework (SAR-WI). The SAR-WI framework not only satisfies a constraint on the number of attended cues, it also performs as well as the R-W model on a number of natural learning tasks, can correctly infer associative strengths, and focuses attention on predictive cues while ignoring uninformative cues. Given the simplicity of proposed alterations, we hope this work informs future development and empirical validation of associative learning models that seek to incorporate sparse attention.

Learning about things that never happened: A critique and refinement of the Rescorla-Wagner update rule when many outcomes are possible.

A vector-based model of discriminative learning is presented. It is demonstrated to learn association strengths identical to the Rescorla-Wagner model under certain parameter settings (Rescorla & Wagner, 1972, Classical Conditioning II: Current Research and Theory, 2, 64-99). For other parameter settings, it approximates the association strengths learned by the Rescorla-Wagner model. I argue that the Rescorla-Wagner model has conceptual details that exclude it as an algorithmically plausible model of learning. The vector learning model, however, does not suffer from the same conceptual issues. Finally, we demonstrate that the vector learning model provides insight into how animals might learn the semantics of stimuli rather than just their associations. Results for simulations of language processing experiments are reported.

The neuroscience of perceptual categorization in pigeons: A mechanistic hypothesis.

We are surrounded by an endless variation of objects. The ability to categorize these objects represents a core cognitive competence of humans and possibly all vertebrates. Research on category learning in nonhuman animals started with the seminal studies of Richard Herrnstein on the category "human" in pigeons. Since then, we have learned that pigeons are able to categorize a large number of stimulus sets, ranging from Cubist paintings to English orthography. Strangely, this prolific field has largely neglected to also study the avian neurobiology of categorization. Here, we present a hypothesis that combines experimental results and theories from categorization research in pigeons with neurobiological insights on visual processing and dopamine-mediated learning in primates. We conclude that in both fields, similar conclusions on the mechanisms of perceptual categorization have been drawn, despite very little cross-reference or communication between these two areas to date. We hypothesize that perceptual categorization is a two-component process in which stimulus features are first rapidly extracted in a feed-forward process, thereby enabling a fast subdivision along multiple category borders. In primates this seems to happen in the inferotemporal cortex, while pigeons may primarily use a cluster of associative visual forebrain areas. The second process rests on dopaminergic error-prediction learning that enables prefrontal areas to connect top down the relevant visual category dimension to the appropriate action dimension.

Purkinje cell activity during classical conditioning with different conditional stimuli explains central tenet of Rescorla­Wagner model [corrected].

A central tenet of Rescorla and Wagner's model of associative learning is that the reinforcement value of a paired trial diminishes as the associative strength between the presented stimuli increases. Despite its fundamental importance to behavioral sciences, the neural mechanisms underlying the model have not been fully explored. Here, we present findings that, taken together, can explain why a stronger association leads to a reduced reinforcement value, within the context of eyeblink conditioning. Specifically, we show that learned pause responses in Purkinje cells, which trigger adaptively timed conditioned eyeblinks, suppress the unconditional stimulus (US) signal in a graded manner. Furthermore, by examining how Purkinje cells respond to two distinct conditional stimuli and to a compound stimulus, we provide evidence that could potentially help explain the somewhat counterintuitive overexpectation phenomenon, which was derived from the Rescorla-Wagner model.

Cognitive emotion regulation enhances aversive prediction error activity while reducing emotional responses.

Cognitive emotion regulation is a powerful way of modulating emotional responses. However, despite the vital role of emotions in learning, it is unknown whether the effect of cognitive emotion regulation also extends to the modulation of learning. Computational models indicate prediction error activity, typically observed in the striatum and ventral tegmental area, as a critical neural mechanism involved in associative learning. We used model-based fMRI during aversive conditioning with and without cognitive emotion regulation to test the hypothesis that emotion regulation would affect prediction error-related neural activity in the striatum and ventral tegmental area, reflecting an emotion regulation-related modulation of learning. Our results show that cognitive emotion regulation reduced emotion-related brain activity, but increased prediction error-related activity in a network involving ventral tegmental area, hippocampus, insula and ventral striatum. While the reduction of response activity was related to behavioral measures of emotion regulation success, the enhancement of prediction error-related neural activity was related to learning performance. Furthermore, functional connectivity between the ventral tegmental area and ventrolateral prefrontal cortex, an area involved in regulation, was specifically increased during emotion regulation and likewise related to learning performance. Our data, therefore, provide first-time evidence that beyond reducing emotional responses, cognitive emotion regulation affects learning by enhancing prediction error-related activity, potentially via tegmental dopaminergic pathways.

Psychological and neural mechanisms of experimental extinction: a selective review.

The present review examines key psychological concepts in the study of experimental extinction and implications these have for an understanding of the underlying neurobiology of extinction learning. We suggest that many of the signature characteristics of extinction learning (spontaneous recovery, renewal, reinstatement, rapid reacquisition) can be accommodated by the standard associative learning theory assumption that extinction results in partial erasure of the original learning together with new inhibitory learning. Moreover, we consider recent behavioral and neural evidence that supports the partial erasure view of extinction, but also note shortcomings in our understanding of extinction circuits as these relate to the negative prediction error concept. Recent work suggests that common prediction error and stimulus-specific prediction error terms both may be required to explain neural plasticity both in acquisition and extinction learning. In addition, we suggest that many issues in the content of extinction learning have not been fully addressed in current research, but that neurobiological approaches should be especially helpful in addressing such issues. These include questions about the nature of extinction learning (excitatory CS-No US, inhibitory CS-US learning, occasion setting processes), especially as this relates to studies of the micro-circuitry of extinction, as well as its representational content (sensory, motivational, response). An additional understudied problem in extinction research is the role played by attention processes and their underlying neural networks, although some research and theory converge on the idea that extinction is accompanied by attention decrements (i.e., habituation-like processes).

Flexible learning in complex worlds.

Cognitive flexibility can enhance the ability to adjust to changing environments. Here, we use learning simulations to investigate the possible advantages of flexible learning in volatile (changing) environments. We compare two established learning mechanisms, one with constant learning rates and one with rates that adjust to volatility. We study an ecologically relevant case of volatility, based on observations of developing cleaner fish Labroides dimidiatus that experience a transition from a simpler to a more complex foraging environment. There are other similar transitions in nature, such as migrating to a new and different habitat. We also examine two traditional approaches to volatile environments in experimental psychology and behavioral ecology: reversal learning, and learning set formation (consisting of a sequence of different discrimination tasks). These provide experimental measures of cognitive flexibility. Concerning transitions to a complex world, we show that both constant and flexible learning rates perform well, losing only a small proportion of available rewards in the period after a transition, but flexible rates perform better than constant rates. For reversal learning, flexible rates improve the performance with each successive reversal because of increasing learning rates, but this does not happen for constant rates. For learning set formation, we find no improvement in performance with successive shifts to new stimuli to discriminate for either flexible or constant learning rates. Flexible learning rates might thus explain increasing performance in reversal learning but not in learning set formation, and this can shed light on the nature of cognitive flexibility in a given system.

The neurocomputational signature of decision-making for unfair offers in females under acute psychological stress.

Stress is a crucial factor affecting social decision-making. However, its impacts on the behavioral and neural processes of females' unfairness decision-making remain unclear. Combining computational modeling and functional near-infrared spectroscopy (fNIRS), this study attempted to illuminate the neurocomputational signature of unfairness decision-making in females. We also considered the effect of trait stress coping styles. Forty-four healthy young females (20.98 ± 2.89 years) were randomly assigned to the stress group (n = 21) and the control group (n = 23). Acute psychosocial stress was induced by the Trier Social Stress Test (TSST), and participants then completed the one-shot ultimatum game (UG) as responders. The results showed that acute psychosocial stress reduced the adaptability to fairness and lead to more random decision-making responses. Moreover, in the stress group, a high level of negative coping style predicted more deterministic decision. fNIRS results showed that stress led to an increase of oxy-hemoglobin (HbO) peak in the right temporoparietal junction (rTPJ), while decreased the activation of left middle temporal gyrus (lMTG) when presented the moderately unfair (MU) offers. This signified more involvement of the mentalization and the inhibition of moral processing. Moreover, individuals with higher negative coping scores showed more deterministic decision behaviors under stress. Taken together, our study emphasizes the role of acute psychosocial stress in affecting females' unfairness decision-making mechanisms in social interactions, and provides evidences for the "tend and befriend" pattern based on a cognitive neuroscience perspec.

What do you learn from a single cue? Dimensional reweighting and cue reassociation from experience with a newly unreliable phonetic cue.

In language comprehension, we use perceptual cues to infer meanings. Some of these cues reside on perceptual dimensions. For example, the difference between bear and pear is cued by a difference in voice onset time (VOT), which is a continuous perceptual dimension. The present paper asks whether, and when, experience with a single value on a dimension behaving unexpectedly is used by the learner to reweight the whole dimension. We show that learners reweight the whole VOT dimension when exposed to a single VOT value (e.g., 45 ms) and provided with feedback indicating that the speaker intended to produce a /b/ 50% of the time and a /p/ the other 50% of the time. Importantly, dimensional reweighting occurs only if 1) the 50/50 feedback is unexpected for the VOT value, and 2) there is another dimension that is predictive of feedback. When no predictive dimension is available, listeners reassociate the experienced VOT value with the more surprising outcome but do not downweight the entire VOT dimension. These results provide support for perceptual representations of speech sounds that combine cues and dimensions, for viewing perceptual learning in speech as a combination of error-driven cue reassociation and dimensional reweighting, and for considering dimensional reweighting to be reallocation of attention that occurs only when there is evidence that reallocating attention would improve prediction accuracy (Harmon, Z., Idemaru, K., & Kapatsinski, V. 2019. Learning mechanisms in cue reweighting. Cognition, 189, 76-88.).

Reaction Time "Mismatch Costs" Change with the Likelihood of Stimulus-Response Compatibility.

Dyadic interactions require dynamic correspondence between one's own movements and those of the other agent. This mapping is largely viewed as imitative, with the behavioural hallmark being a reaction-time cost for mismatched actions. Yet the complex motor patterns humans enact together extend beyond direct-matching, varying adaptively between imitation, complementary movements, and counter-imitation. Optimal behaviour requires an agent to predict not only what is likely to be observed but also how that observed action will relate to their own motor planning. In 28 healthy adults, we examined imitation and counter-imitation in a task that varied the likelihood of stimulus-response congruence from highly predictable, to moderately predictable, to unpredictable. To gain mechanistic insights into the statistical learning of stimulus-response compatibility, we compared two computational models of behaviour: (1) a classic fixed learning-rate model (Rescorla-Wagner reinforcement [RW]) and (2) a hierarchical model of perceptual-behavioural processes in which the learning rate adapts to the inferred environmental volatility (hierarchical Gaussian filter [HGF]). Though more complex and hence penalized by model selection, the HGF provided a more likely model of the participants' behaviour. Matching motor responses were only primed (faster) in the most experimentally volatile context. This bias was reversed so that mismatched actions were primed when beliefs about volatility were lower. Inferential statistics indicated that matching responses were only primed in unpredictable contexts when stimuli-response congruence was at 50:50 chance. Outside of these unpredictable blocks the classic stimulus-response compatibility effect was reversed: Incongruent responses were faster than congruent ones. We show that hierarchical Bayesian learning of environmental statistics may underlie response priming during dyadic interactions.

Threat learning in space: how stimulus-outcome spatial compatibility modulates conditioned skin conductance response.

A central question in Pavlovian conditioning concerns the critical conditions that drive the acquisition and maintenance of the stimulus-outcome association. The spatial relationship between the conditioned (CS) and unconditioned (US) stimuli is considered to exert strong effects on learning. However, how spatial information modulates Pavlovian learning remains mostly unexplored in humans. Here, we test how the compatibility between the CS and the US location influences the acquisition, extinction, and recovery (following reinstatement) of Pavlovian conditioned threat. Participants (N = 20) completed a differential threat conditioning task in which visual CSs appeared on the same (compatible) or opposite (incompatible) hemispace as the US delivery (aversive shock to one hand), while their skin conductance response served as an index of learning. Results show that initial threat expectations were biased in favor of compatible CSs before conditioning. Nevertheless, this bias was revised during acquisition to reflect current stimulus-outcome contingencies. Computational modeling suggested that this effect occurred through a higher reliance on positive aversive prediction errors for incompatible CSs, thereby facilitating learning of their association with the US. Additionally, the conditioned response to incompatible CSs was associated with initially slower extinction and a greater recovery after threat reinstatement. These findings suggest that spatial information conveyed by stimuli and outcomes can be flexibly used to enact defensive responses to the current source of danger, highlighting the adaptive nature of Pavlovian learning.

Prediction error in models of adaptive behavior.

Pavlovian conditioning is evident in every species in which it has been assessed, and there is a consensus about its interpretation across behavioral,1,2 brain,3,4,5,6 and computational analyses7,8,9,10,11: conditioned behavior reflects the formation of a directional associative link from the memory of one stimulus (e.g., a visual stimulus) to another (e.g., food), with learning stopping when there is no error between the prediction generated by the visual stimulus and what happens next (e.g., food). This consensus fails to anticipate the results that we report here. In our experiments with rats, we find that arranging predictive (visual stimulus→food) and nonpredictive (food→visual stimulus) relationships produces marked and sustained changes in conditioned behaviors when the visual stimulus is presented alone. Moreover, the type of relationship affects (1) the distribution of conditioned behaviors related to the properties of both food (called goal-tracking) and the visual stimulus (called sign-tracking) and (2) when in the visual stimulus, these two behaviors are evident. These results represent an impetus for a fundamental shift in how Pavlovian conditioning is interpreted: animals learn about the relationship between two stimuli irrespective of the order in which they are presented, but they exhibit this knowledge in different ways. This interpretation and our new results are captured by a recent model of Pavlovian conditioning,12,13 HeiDI, and both are consistent with the need for animals to represent the fact that the impact of a cause (e.g., the ingestion of nutrients or the bite of a predator) can be felt before or after the cause has been perceived.

Oscillatory brain activity links experience to expectancy during associative learning.

Associating a novel situation with a specific outcome involves a cascade of cognitive processes, including selecting relevant stimuli, forming predictions regarding expected outcomes, and updating memorized predictions based on experience. The present manuscript uses computational modeling and machine learning to test the hypothesis that alpha-band (8-12 Hz) oscillations are involved in the updating of expectations based on experience. Participants learned that a visual cue predicted an aversive loud noise with a probability of 50%. The Rescorla-Wagner model of associative learning explained trial-wise changes in self-reported noise expectancy as well as alpha power changes. Experience in the past trial and self-reported expectancy for the subsequent trial were accurately decoded based on the topographical distribution of alpha power at specific latencies. Decodable information during initial association formation and contingency report recurred when viewing the conditioned cue. Findings support the idea that alpha oscillations have multiple, temporally specific, roles in the formation of associations between cues and outcomes.

Prediction and error in early infant speech learning: A speech acquisition model.

In the last two decades, statistical clustering models have emerged as a dominant model of how infants learn the sounds of their language. However, recent empirical and computational evidence suggests that purely statistical clustering methods may not be sufficient to explain speech sound acquisition. To model early development of speech perception, the present study used a two-layer network trained with Rescorla-Wagner learning equations, an implementation of discriminative, error-driven learning. The model contained no a priori linguistic units, such as phonemes or phonetic features. Instead, expectations about the upcoming acoustic speech signal were learned from the surrounding speech signal, with spectral components extracted from an audio recording of child-directed speech as both inputs and outputs of the model. To evaluate model performance, we simulated infant responses in the high-amplitude sucking paradigm using vowel and fricative pairs and continua. The simulations were able to discriminate vowel and consonant pairs and predicted the infant speech perception data. The model also showed the greatest amount of discrimination in the expected spectral frequencies. These results suggest that discriminative error-driven learning may provide a viable approach to modelling early infant speech sound acquisition.

Dopamine release in the nucleus accumbens core signals perceived saliency.

A large body of work has aimed to define the precise information encoded by dopaminergic projections innervating the nucleus accumbens (NAc). Prevailing models are based on reward prediction error (RPE) theory, in which dopamine updates associations between rewards and predictive cues by encoding perceived errors between predictions and outcomes. However, RPE cannot describe multiple phenomena to which dopamine is inextricably linked, such as behavior driven by aversive and neutral stimuli. We combined a series of behavioral tasks with direct, subsecond dopamine monitoring in the NAc of mice, machine learning, computational modeling, and optogenetic manipulations to describe behavior and related dopamine release patterns across multiple contingencies reinforced by differentially valenced outcomes. We show that dopamine release only conforms to RPE predictions in a subset of learning scenarios but fits valence-independent perceived saliency encoding across conditions. Here, we provide an extended, comprehensive framework for accumbal dopamine release in behavioral control.

The redundancy effect is related to a lack of conditioned inhibition: Evidence from a task in which excitation and inhibition are symmetrical.

Rescorla and Wagner's model of learning describes excitation and inhibition as symmetrical opposites. However, tasks used in human causal learning experiments, such as the allergist task, generally involve learning about cues leading to the presence or absence of the outcome, which may not reflect this assumption. This is important when considering learning effects which provide a challenge to this model, such as the redundancy effect. The redundancy effect describes higher causal ratings for the blocked cue X than for the uncorrelated cue Y in the design A+/AX+/BY+/CY-, the opposite pattern to that predicted by the Rescorla-Wagner model, which predicts higher associative strength for Y than for X. Crucially, this prediction depends on cue C gaining some inhibitory associative strength. In this article, we used a task in which cues could have independent inhibitory effects on the outcome, to investigate whether a lack of inhibition was related to the redundancy effect. In Experiment 1, inhibition for C was not detected in the allergist task, supporting this possibility. Three further experiments using the alternative task showed that a lack of inhibition was related to the redundancy effect: the redundancy effect was smaller when C was rated as inhibitory. Individual variation in the strength of inhibition for C also determined the size of the redundancy effect. Given that weak inhibition was detected in the alternative scenario but not in the allergist task, we recommend carefully choosing the type of task used to investigate associative learning phenomena, as it may influence results.