Why is the Rescorla-Wagner model so influential?

The influence of the Rescorla-Wagner model cannot be overestimated, despite that (1) the model does not differ much computationally from its predecessors and competitors, and (2) its shortcomings are well-known in the learning community. Here we discuss the reasons behind its widespread influence in the cognitive and neural sciences, and argue that it is the constant search for general-process theories by learning scholars which eventually produced a model whose application spans many different areas of research to this day. We focus on the theoretical and empirical background of the model, the theoretical connections that it has with later developments across Marr's levels of analysis, as well as the broad variety of research that it has guided and inspired.

Ambiguity drives higher-order Pavlovian learning.

In the natural world, stimulus-outcome associations are often ambiguous, and most associations are highly complex and situation-dependent. Learning to disambiguate these complex associations to identify which specific outcomes will occur in which situations is critical for survival. Pavlovian occasion setters are stimuli that determine whether other stimuli will result in a specific outcome. Occasion setting is a well-established phenomenon, but very little investigation has been conducted on how occasion setters are disambiguated when they themselves are ambiguous (i.e., when they do not consistently signal whether another stimulus will be reinforced). In two preregistered studies, we investigated the role of higher-order Pavlovian occasion setting in humans. We developed and tested the first computational model predicting direct associative learning, traditional occasion setting (i.e., 1st-order occasion setting), and 2nd-order occasion setting. This model operationalizes stimulus ambiguity as a mechanism to engage in higher-order Pavlovian learning. Both behavioral and computational modeling results suggest that 2nd-order occasion setting was learned, as evidenced by lack and presence of transfer of occasion setting properties when expected and the superior fit of our 2nd-order occasion setting model compared to the 1st-order occasion setting or direct associations models. These results provide a controlled investigation into highly complex associative learning and may ultimately lead to improvements in the treatment of Pavlovian-based mental health disorders (e.g., anxiety disorders, substance use).

Subsampling of cues in associative learning.

Theories of learning distinguish between elemental and configural stimulus processing depending on whether stimuli are processed independently or as whole configurations. Evidence for elemental processing comes from findings of summation in animals where a compound of two dissimilar stimuli is deemed to be more predictive than each stimulus alone, whereas configural processing is supported by experiments using similar stimuli in which summation is not found. However, in humans the summation effect is robust and impervious to similarity manipulations. In three experiments in human predictive learning, we show that summation can be obliterated when partially reinforced cues are added to the summands in training and tests. This lack of summation only holds when the partially reinforced cues are similar to the reinforced cues (experiment 1) and seems to depend on participants sampling only the most salient cue in each trial (experiments 2a and 2b) in a sequential visual search process. Instead of attributing our and others' instances of lack of summation to the customary idea of configural processing, we offer a formal subsampling rule that might be applied to situations in which the stimuli are hard to parse from each other.

Determining the effects of training duration on the behavioral expression of habitual control in humans: a multilaboratory investigation.

It has been suggested that there are two distinct and parallel mechanisms for controlling instrumental behavior in mammals: goal-directed actions and habits. To gain an understanding of how these two systems interact to control behavior, it is essential to characterize the mechanisms by which the balance between these systems is influenced by experience. Studies in rodents have shown that the amount of training governs the relative expression of these two systems: Behavior is goal-directed following moderate training, but the more extensively an instrumental action is trained, the more it becomes habitual. It is less clear whether humans exhibit similar training effects on the expression of goal-directed and habitual behavior, as human studies have reported contradictory findings. To tackle these contradictory findings, we formed a consortium, where four laboratories undertook a preregistered experimental induction of habits by manipulating the amount of training. There was no statistical evidence for a main effect of the amount of training on the formation and expression of habits. However, exploratory analyses suggest a moderating effect of the affective component of stress on the impact of training over habit expression. Participants who were lower in affective stress appeared to be initially goal-directed, but became habitual with increased training, whereas participants who were high in affective stress were already habitual even after moderate training, thereby manifesting insensitivity to overtraining effects. Our findings highlight the importance of the role of moderating variables such as individual differences in stress and anxiety when studying the experimental induction of habits in humans.

Action sequences, habits, and attention in copying strategies.

Understanding how culture evolves in society is an extremely difficult task. The bifocal stance theory (BST) deploys two copying strategies which can be linked to dual-system theories of behavior. BST would benefit from incorporating results from these theories, such as the evolution of attention to goals or steps of a behavioral sequence, and the role of the environment in prompting different copying strategies.

Dual-system free-operant avoidance: Extension of a theory.

Our theory of positively reinforced free-operant behavior (Perez & Dickinson, 2020) assumes that responding is controlled by two systems. One system is sensitive to the correlation between response and reinforcement rates and controls goal-directed behavior, whereas a habitual system learns by reward prediction error. We present an extension of this theory to the aversive domain that explains why free-operant avoidance responding increases with both the experienced rate of negative reinforcement and the difference between this rate and that programmed by the avoidance schedule. The theory also assumes that the habitual component is reinforced by the acquisition of aversive inhibitory properties by the feedback stimuli generated by responding, which then act as safety signals that reinforce habit performance. Our analysis suggests that the distinction between habitual and goal-directed control of rewarded behavior can also be applied to the aversive domain. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Design and selection of Toca 511 for clinical use: modified retroviral replicating vector with improved stability and gene expression.

Retroviral replicating vectors (RRVs) are a nonlytic alternative to oncolytic replicating viruses as anticancer agents, being selective both for dividing cells and for cells that have defects in innate immunity and interferon responsiveness. Tumor cells fit both these descriptions. Previous publications have described a prototype based on an amphotropic murine leukemia virus (MLV), encoding yeast cytosine deaminase (CD) that converts the prodrug 5-fluorocytosine (5-FC) to the potent anticancer drug, 5-fluorouracil (5-FU) in an infected tumor. We report here the selection of one lead clinical candidate based on a general design goal to optimize the genetic stability of the virus and the CD activity produced by the delivered transgene. Vectors were tested for titer, genetic stability, CD protein and enzyme activity, ability to confer susceptibility to 5-FC, and preliminary in vivo antitumor activity and stability. One vector, Toca 511, (aka T5.0002) encoding an optimized CD, shows a threefold increased specific activity in infected cells over infection with the prototype RRV and shows markedly higher genetic stability. Animal testing demonstrated that Toca 511 replicates stably in human tumor xenografts and, after 5-FC administration, causes complete regression of such xenografts. Toca 511 (vocimagene amiretrorepvec) has been taken forward to preclinical and clinical trials.

Instrumental behavior in humans is sensitive to the correlation between response rate and reward rate.

Recent theories of instrumental behavior postulate that the correlation between response and reward rate is a critical factor in instrumental goal-directed performance. However, it is still not clear whether human actions can be sensitive to rate correlation. Using a novel within-subject design, participants were trained under ratio and interval contingencies of reinforcement matching both reward probabilities and reward rates between conditions. The impact of rate correlation on performance was evident in the higher performance observed under ratio contingencies for both types of matching. Moreover, there was no difference in performance between two classes of interval schedules with equivalent correlational properties but different reward probabilities. These results are discussed in terms of a recent dual-system model of instrumental behavior.

Pavlovian occasion setting in human fear and appetitive conditioning: Effects of trait anxiety and trait depression.

Contexts and discrete stimuli often hierarchically influence the association between a stimulus and outcome. This phenomenon, called occasion setting, is central to modulation-based Pavlovian learning. We conducted two experiments with humans in fear and appetitive conditioning paradigms, training stimuli in differential conditioning, feature-positive discriminations, and feature-negative discriminations. We also investigated the effects of trait anxiety and trait depression on these forms of learning. Results from both experiments showed that participants were able to successfully learn which stimuli predicted the electric shock and monetary reward outcomes. Additionally, as hypothesized, the stimuli trained as occasion setters had little-to-no effect on simple reinforced or non-reinforced stimuli, suggesting the former were indeed occasion setters. Lastly, in fear conditioning, trait anxiety was associated with increases in fear of occasion setter/conditional stimulus compounds; in appetitive conditioning, trait depression was associated with lower expectations of monetary reward for the trained negative occasion setting compound and transfer of the negative occasion setter to the simple reinforced stimulus. These results suggest that clinically anxious individuals may have enhanced fear of occasion setting compounds, and clinically depressed individuals may expect less reward with compounds involving the negative occasion setter.

Fibrocyte activation in rheumatoid arthritis.

OBJECTIVES: RA is a common, relapsing autoimmune disease primarily affecting the joints. Fibroblast-like synovial (FLS) cells are thought to be responsible for pannus formation and secretion of factors that recruit leucocytes to affected joints, thereby promoting bone and cartilage destruction. Fibrocytes are multipotent circulating stem cells that may have a role in RA pathogenesis, perhaps as the precursors of the FLS cells, or by regulating FLS cell function. METHODS: We utilized multidimensional phospho-specific flow cytometry to characterize the activation status of peripheral blood (PB) fibrocytes derived from human RA patients at different stages of disease and from mice with CIA. RESULTS: Human PB fibrocytes from RA patients exhibited phosporylation activation of the p44/42 and p38 MAP kinases (MAPKs), and STAT3 (signal transducer and activator of transcription) and STAT-5 early in disease, within the first year of diagnosis. Similarly, in murine CIA, an increase in the total number of PB phosphoSTAT5-positive fibrocytes was observed at early time points in disease. Notably, in the affected paws of mice with CIA, we identified an increased number of fibrocytes, in contrast to the paws of control mice. CONCLUSIONS: These data suggest that activated fibrocytes may influence the disease process in RA and may serve as surrogate markers for disease in the PB of affected patients.

A theory of actions and habits: The interaction of rate correlation and contiguity systems in free-operant behavior.

Contemporary theories of instrumental performance assume that responding can be controlled by 2 behavioral systems, 1 goal-directed that encodes the outcome of an action, and 1 habitual that reinforces the response strength of the same action. Here we present a model of free-operant behavior in which goal-directed control is determined by the correlation between the rates of the action and the outcome whereas the total prediction error generated by contiguous reinforcement by the outcome controls habitual response strength. The outputs of these two systems summate to generate a total response strength. This cooperative model addresses the difference in the behavioral impact of ratio and interval schedules, the transition from goal-directed to habitual control with extended training, the persistence of goal-directed control under choice procedures and following extinction, among other phenomena. In these respects, this dual-system model is unique in its account of free-operant behavior. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Evidence for a dissociation between causal beliefs and instrumental actions.

Human experiments have demonstrated that instrumental performance of an action and the causal beliefs of the effectiveness of an action in producing a reward are correlated and controlled by the probability of an action leading to a reward. The animal literature, however, shows that instrumental performance under free-operant training differs even when the reward probabilities are matched while subjects undergo training under ratio or interval schedules of reward. In two experiments, we investigated whether causal beliefs would correlate with instrumental performance under ratio and interval schedules for matched reward probabilities. In both experiments, we found that performance was higher under ratio than under interval training. However, causal beliefs were similar between these two conditions despite these differences in instrumental performance. When reward probabilities were increased by experimental manipulations in Experiment 2, the causal beliefs increased but performance decreased with respect to Experiment 1. This is evidence that instrumental performance and causal action-reward attribution may not follow from the same psychological process under free-operant training.

Human instrumental performance in ratio and interval contingencies: A challenge for associative theory.

Associative learning theories regard the probability of reinforcement as the critical factor determining responding. However, the role of this factor in instrumental conditioning is not completely clear. In fact, free-operant experiments show that participants respond at a higher rate on variable ratio than on variable interval schedules even though the reinforcement probability is matched between the schedules. This difference has been attributed to the differential reinforcement of long inter-response times (IRTs) by interval schedules, which acts to slow responding. In the present study, we used a novel experimental design to investigate human responding under random ratio (RR) and regulated probability interval (RPI) schedules, a type of interval schedule that sets a reinforcement probability independently of the IRT duration. Participants responded on each type of schedule before a final choice test in which they distributed responding between two schedules similar to those experienced during training. Although response rates did not differ during training, the participants responded at a lower rate on the RPI schedule than on the matched RR schedule during the choice test. This preference cannot be attributed to a higher probability of reinforcement for long IRTs and questions the idea that similar associative processes underlie classical and instrumental conditioning.

Exploring the Effect of Stimulus Similarity on the Summation Effect in Causal Learning.

Several contemporary models anticipate that the summation effect is modulated by the similarity between the cues forming a compound. Here, we explore this hypothesis in a series of causal learning experiments. Participants were presented with two visual cues that separately predicted a common outcome and later asked for the outcome predicted by the compound of the two cues. Similarity was varied between groups through changes in shape, spatial position, color, configuration, and rotation. In variance with the predictions of these models, we observed similar and strong levels of summation in both groups across all manipulations of similarity. The effect, however, was significantly reduced by manipulations intended to impact assumptions about the causal independence of the cues forming the compound, but this reduction was independent of stimulus similarity. These results are problematic for similarity-based models and can be more readily explained by rational approaches to causal learning.

A re-examination of responding on ratio and regulated-probability interval schedules.

The higher response rates observed on ratio than on matched interval reward schedules has been attributed to the differential reinforcement of longer inter-response times (IRTs) on the interval contingency. Some data, however, seem to contradict this hypothesis, showing that the difference is still observed when the role of IRT reinforcement is neutralized by using a regulated-probability interval schedule (RPI). Given the mixed evidence for these predictions, we re-examined this hypothesis by training three groups of rats to lever press under ratio, interval and RPI schedules across two phases while matching reward rates within triads. At the end of the first phase, the master ratio and RPI groups responded at similar rates. In the second phase, an interval group yoked to the same master ratio group of the first phase responded at a lower rate than the RPI group. Post-hoc analysis showed comparable reward rates for master and yoked schedules. The experienced response-outcome rate correlations were likewise similar and approached zero as training progressed. We discuss these results in terms of a contemporary dual-system model of instrumental conditioning.

Differential role of ICAM ligands in determination of human memory T cell differentiation.

BACKGROUND: Leukocyte Function Antigen-1 (LFA-1) is a primary adhesion molecule that plays important roles in T cell activation, leukocyte recirculation, and trans-endothelial migration. By applying a multivariate intracellular phospho-proteomic analysis, we demonstrate that LFA-1 differentially activates signaling molecules. RESULTS: Signal intensity was dependent on both ICAM ligand and LFA-1 concentration. In the presence of CD3 and CD28 stimulation, ICAM-2 and ICAM-3 decreased TGFbeta1 production more than ICAM-1. In long-term differentiation experiments, stimulation with ICAM-3, CD3, and CD28 generated IFNgamma producing CD4+CD45RO+CD62L-CD11aBrightCD27- cells that had increased expression of intracellular BCL2, displayed distinct chemokine receptor profiles, and exhibited distinct migratory characteristics. Only CD3/CD28 with ICAM-3 generated CD4+CD45RO+CD62L-CD11aBrightCD27- cells that were functionally responsive to chemotaxis and exhibited higher frequencies of cells that signaled to JNK and ERK1/2 upon stimulation with MIP3alpha. Furthermore, these reports identify that the LFA-1 receptor, when presented with multiple ligands, can result in distinct T cell differentiation states and suggest that the combinatorial integration of ICAM ligand interactions with LFA-1 have functional consequences for T cell biology. CONCLUSION: Thus, the ICAM ligands, differentially modulate LFA-1 signaling in T cells and potentiate the development of memory human T cells in vitro. These findings are of importance in a mechanistic understanding of memory cell differentiation and ex vivo generation of memory cell subsets for therapeutic applications.

Simultaneous measurement of multiple active kinase states using polychromatic flow cytometry.

Intracellular assays of signaling systems have been limited by an inability to correlate functional subsets of cells in complex populations on the basis of active kinase states. Such correlations could be important in distinguishing changes in signaling status that arise in rare cell subsets during functional activation or in disease manifestation. Here we demonstrate the ability to simultaneously detect activated kinase members of the mitogen-activated protein kinases family (p38 MAPK, p44/42 MAPK, JNK/SAPK), members of cell survival pathways (AKT/PKB), and members of T-cell activation pathways (TYK2), among others, in subpopulations of complex cell populations by multiparameter flow-cytometric analysis. We demonstrate the utility of these probes in identifying distinct signaling cascades for (1) both artificial and physiological stimulatory conditions of peripheral blood mononuclear cells (PBMCs), (2) cytokine stimulation in human memory and naïve lymphocyte subsets as identified by five differentiation markers, and (3) ordering of kinase activation in potential signaling hierarchies. Polychromatic flow-cytometric active kinase measurements demonstrate that multidimensional analysis of signaling pathways can provide functional signaling pathway assessment on a single-cell level and allow for potential correlation with biological and clinical parameters.

Multiplexed transcriptome analysis to detect ALK, ROS1 and RET rearrangements in lung cancer.

ALK, ROS1 and RET gene fusions are important predictive biomarkers for tyrosine kinase inhibitors in lung cancer. Currently, the gold standard method for gene fusion detection is Fluorescence In Situ Hybridization (FISH) and while highly sensitive and specific, it is also labour intensive, subjective in analysis, and unable to screen a large numbers of gene fusions. Recent developments in high-throughput transcriptome-based methods may provide a suitable alternative to FISH as they are compatible with multiplexing and diagnostic workflows. However, the concordance between these different methods compared with FISH has not been evaluated. In this study we compared the results from three transcriptome-based platforms (Nanostring Elements, Agena LungFusion panel and ThermoFisher NGS fusion panel) to those obtained from ALK, ROS1 and RET FISH on 51 clinical specimens. Overall agreement of results ranged from 86-96% depending on the platform used. While all platforms were highly sensitive, both the Agena panel and Thermo Fisher NGS fusion panel reported minor fusions that were not detectable by FISH. Our proof-of-principle study illustrates that transcriptome-based analyses are sensitive and robust methods for detecting actionable gene fusions in lung cancer and could provide a robust alternative to FISH testing in the diagnostic setting.

Appreciating the heterogeneity in autoimmune disease: multiparameter assessment of intracellular signaling mechanisms.

Autoimmune disease pathologies are multifactorial with complex interactive networks of cells and chemical messengers that initiate cascades of aberrant cellular activity. Rheumatoid arthritis (RA) is a chronic inflammatory disease that is characterized by systemic inflammation, destruction of the joints, and production of autoantibodies recognizing dozens of putative autoantigens. The presence of autoreactive T cells in individuals leads to pathological autoimmunity by activating additional cellular constituents to mediate inflammation and joint destruction. The etiology of RA is unknown, and knowledge is lacking of the molecular mechanisms underlying the production and subsequent regulation of autoreactive T cells and predicting patient responses to treatments. Biochemical investigations into mechanisms of the disease have relied on animal models that are helpful in dissecting elements of the disease but that are not necessarily reflective of human RA development. The study of multiple activated signaling pathways in complex populations of cells, such as peripheral blood, at the single-cell level has not previously been possible. This article describes how intracellular phosphoepitope staining methodology in conjunction with surface-cell immunophenotyping can be used to deconvolute cellular subsets and allow functional characterization of patient-derived material. Multiparameter flow cytometric analysis allows for small subpopulations-representing different cellular subsets and differentiation or activation states-to be discerned and simultaneously assessed for intracellular biochemical activities. This article also describes how single-cell signal network analysis can be used to stratify patients and may be useful for understanding mechanisms of disease progression, treatment resistance, and development of diagnostic indicators.

Inhibition and reversal of myogenic differentiation by purine-based microtubule assembly inhibitors.

Using a muscle cell differentiation screen, we have identified myoseverin from a 2,6,9-trisubsituted purine library as a purine-based microtubule binding molecule [1]. Structure-activity relation studies of myoseverin identify positions N2 and N6 to be critical for inhibiting muscle differentiation. Inhibition of microtubule polymerization induced the reversion of terminally differentiated myotubes to mononucleated cells that were responsive to both growth and differentiation conditions, without any observable cytotoxicity. Comparison of myoseverin derivatives to taxol, vinblastine, nocodazole, and colchicine identify myoseverin's effect as being selectively reversible in addition to lacking the cytotoxic effects of these non-purine-based microtubule-disrupting molecules. Myoseverin, as a purine-based microtubule inhibitor, reverted terminal muscle-differentiated cells to a state that was responsive to environmental cues. These results suggest that myoseverin may have applications in muscle regeneration and stem cell differentiation.