How do animals weigh conflicting information about reward sources over time? Comparing dynamic averaging models.

Optimal foraging theory suggests that animals make decisions which maximize their food intake per unit time when foraging, but the mechanisms animals use to track the value of behavioral alternatives and choose between them remain unclear. Several models for how animals integrate past experience have been suggested. However, these models make differential predictions for the occurrence of spontaneous recovery of choice: a behavioral phenomenon in which a hiatus from the experimental environment results in animals reverting to a behavioral allocation consistent with a reward distribution from the more distant past, rather than one consistent with their most recently experienced distribution. To explore this phenomenon and compare these models, three free-operant experiments with rats were conducted using a serial reversal design. In Phase 1, two responses (A and B) were baited with pellets on concurrent variable interval schedules, favoring option A. In Phase 2, lever baiting was reversed to favor option B. Rats then entered a delay period, where they were maintained at weight in their home cages and no experimental sessions took place. Following this delay, preference was assessed using initial responding in test sessions where levers were presented, but not baited. Models were compared in performance, including an exponentially weighted moving average, the Temporal Weighting Rule, and variants of these models. While the data provided strong evidence of spontaneous recovery of choice, the form and extent of recovery was inconsistent with the models under investigation. Potential interpretations are discussed in relation to both the decision rule and valuation functions employed.

Examination of alternative-response discrimination training and resurgence in rats.

Resurgence is an increase in a previously reinforced behavior following a worsening of conditions for a more recently reinforced behavior. Discrimination training is incorporated into treatment for problem behavior to prevent treatment adherence failures that may result in resurgence. There is evidence that resurgence may be reduced when a stimulus that signals alternative-response extinction is present compared with absent; however, the generality of this effect is unknown given the limited testing conditions. The goal of the present experiments was to further examine the effects of such stimuli in a reverse-translational evaluation using rats. Target responding was reinforced in baseline and then placed on extinction in the following discrimination-training phase. An alternative response was differentially reinforced in a two-component multiple schedule where one stimulus (i.e., SD) signaled alternative-response reinforcement and the other (i.e., SΔ) signaled extinction. Experiment 1 assessed resurgence in both the SΔ and SD when alternative reinforcement was removed. Experiment 2 evaluated resurgence under conditions that better approximated those used in the clinic in which the alternative-response SΔ was present or absent. The SΔ failed to suppress target responding during resurgence testing in both experiments. These findings suggest that the conditions under which an alternative-response SΔ will successfully mitigate resurgence may be limited and require further research.

Moving Beyond Reinforcement and Response Strength.

Behavior analysis has often simultaneously depended upon and denied an implicit, hypothetical process of reinforcement as response strengthening. I discuss what I see as problematic about the use of such an implicit, possibly inaccurate, and likely unfalsifiable theory and describe issues to consider with respect to an alternative view without response strengthening. In my take on such an approach, important events (i.e., "reinforcers") provide a means to measure learning about predictive relations in the environment by modulating (i.e., inducing) performance dependent upon what is predicted and the relevant motivational mode or behavioral system active at that time (i.e., organismic state). Important events might be phylogenetically important, or they might acquire importance by being useful as signals for guiding an organism to where, when, or how currently relevant events might be obtained (or avoided). Given the role of learning predictive relations in such an approach, it is suggested that a potentially useful first step is to work toward formal descriptions of the structure of the predictive relations embodied in common facets of operant behavior (e.g., response-reinforcer contingencies, conditioned reinforcement, and stimulus control). Ultimately, the success of such an approach will depend upon how well it integrates formal characterizations of predictive relations (and how they are learned without response strengthening) and the relevant concomitant changes in organismic state across time. I also consider how thinking about the relevant processes in such a way might improve both our basic science and our technology of behavior.

Impulsive choice predicts anxiety-like behavior, but not alcohol or sucrose consumption, in male Long-Evans rats.

BACKGROUND: Prior human research indicates robust, positive relations between impulsive choice (i.e., preference for smaller, immediate over larger, delayed rewards) and alcohol use disorders. However, varied findings in the nonhuman literature reveal a relatively ambiguous relation between impulsive choice and alcohol consumption in rodents. In addition, few rodent studies have investigated potential relations between impulsive choice and common covariates of alcohol consumption (e.g., avidity for sweet substances or anxiety-like behavior). METHODS: Ninety-two male Long-Evans rats completed an impulsive-choice task. From this larger sample, extreme high- and low-impulsive groups (n = 30 each) were retained for further testing. In separate tests, subsequent open-field behavior and consumption of oral alcohol (12% w/v) and isocaloric sucrose were examined. Impulsive choice was then retested to examine whether behavior remained stable over the course of the experiment. RESULTS: No significant relations emerged between impulsive choice and either alcohol or sucrose consumption. However, impulsive choice predicted greater anxiety-like behavior (avoidance of the center field, defecation) in the open-field test. In turn, greater anxiety predicted lower alcohol and sucrose consumption. Finally, choice remained generally stable across the experiment, although high-impulsive rats tended toward less impulsive choice in the retest. CONCLUSIONS: Although impulsive choice and alcohol consumption appear to share some variance with anxiety-like behavior, the present data offer no support for a relation between impulsive choice and alcohol consumption in Long-Evans rats. Together with mixed rodent data from prior reports, these findings attenuate cross-species comparisons to human relations between impulsive choice and alcohol use disorders.

The impact of D-amphetamine and SCH23390 on behavioral momentum of food seeking and reinstatement in rats.

Although the environmental determinants of context-specific behavioral persistence have been extensively studied within behavioral momentum theory, little is known about the neurobiological determinants. The present experiment assessed the impact of indirect dopamine agonism with D-amphetamine or dopamine D1 receptor antagonism with SCH23390 on context-specific persistence of behavior. Two groups of rats were trained to make operant responses for equal rates of food delivery in two alternating contexts arranged across sessions. Following baseline, rats received either drug (D-amphetamine or SCH23390) or saline injections before each context. Resistance to extinction and reinstatement in each context were subsequently tested in the absence of drug. Previous exposure to D-amphetamine increased resistance to extinction and reinstatement of behavior, whereas SCH23390 had little impact on resistance to extinction and enhanced reinstatement in the saline context. Quantitative analyses based on behavioral momentum theory suggested that previous treatment with D-amphetamine within a stimulus context may have resulted in a shift to more habitual stimulus-driven behavior that was impacted less by reinforcer omission during subsequent extinction. These results suggest that the persistence of behavior is greater in a context associated with dopamine receptor agonism and that the activity at D1 receptors may differentially modulate extinction and reinstatement performance. These findings may serve as a starting point for future examination and integration of the biological and environmental determinants of behavioral persistence within the framework of behavioral momentum theory.

Temporal integration and instrumental conditioned reinforcement.

Stimuli associated with primary reinforcement for instrumental behavior are widely believed to acquire the capacity to function as conditioned reinforcers via Pavlovian conditioning. Some Pavlovian conditioning studies suggest that animals learn the important temporal relations between stimuli and integrate such temporal information over separate experiences to form a temporal map. The present experiment examined whether Pavlovian conditioning can establish a positive instrumental conditioned reinforcer through such temporal integration. Two groups of rats received either delay or trace appetitive conditioning in which a neutral stimulus predicted response-independent food deliveries (CS1→US). Both groups then experienced one session of backward second-order conditioning of the training CS1 and a novel CS2 (CS1-CS2 pairing). Finally, the ability of CS2 to function as a conditioned reinforcer for a new instrumental response (leverpressing) was assessed. Consistent with the previous demonstrations of temporal integration in fear conditioning, a CS2 previously trained in a trace-conditioning protocol served as a better instrumental conditioned reinforcer after backward second-order conditioning than did a CS2 previously trained in a delay protocol. These results suggest that an instrumental conditioned reinforcer can be established via temporal integration and raise challenges for existing quantitative accounts of instrumental conditioned reinforcement.

Ketamine produces no detectable long-term positive or negative effects on cognitive flexibility or reinforcement learning of male rats.

RATIONALE: Patients with major depressive disorder (MDD) often experience abnormalities in behavioral adaptation following environmental changes (i.e., cognitive flexibility) and tend to undervalue positive outcomes but overvalue negative outcomes. The probabilistic reversal learning task (PRL) is used to study these deficits across species and to explore drugs that may have therapeutic value. Selective serotonin-reuptake inhibitors (SSRIs) have limited effectiveness in treating MDD and produce inconsistent effects in non-human versions of the PRL. As such, ketamine, a novel and potentially rapid-acting therapeutic, has begun to be examined using the PRL. Two previous studies examining the effects of ketamine in the PRL have shown conflicting results and only examined short-term effects of ketamine. OBJECTIVE: This experiment examined PRL performance across a 2-week period following a single exposure to a ketamine dose that varied across groups. METHODS: After five sessions of PRL training, groups of rats received an injection of either 0, 10, 20 or 30 mg/kg ketamine. One-hour post-injection, rats engaged in the PRL, and subsequently sessions continued daily for 2 weeks. Traditional behavioral and computational reinforcement learning-derived measures were examined. RESULTS: Results showed that ketamine had acute effects 1-h post-injection, including a significant decrease in the value of the punishment learning rate. Beyond 1 h, ketamine produced no detectable improvements nor decrements in performance across 2 weeks. CONCLUSION: Overall, the present results suggest that the range of ketamine doses examined do not have long-term positive or negative effects on cognitive flexibility or reward processing in healthy rats as measured by the PRL.

Resurgence Following Higher or Lower Quality Alternative Reinforcement.

Resurgence is a temporary increase in a previously suppressed target behavior following a worsening in reinforcement conditions. Previous studies have examined how higher rates or magnitudes of alternative reinforcement affect suppression of the target behavior and subsequent resurgence. However, there has been no investigation of the effects of higher versus lower qualities of alternative reinforcement on resurgence. Using a three-phase resurgence preparation with rats, the present experiments examined the effects of an alternative reinforcer that was of higher (Experiment 1) or lower (Experiment 2) quality than the reinforcer that had previously maintained the target behavior. The results of both experiments showed greater reductions in target behavior with a higher quality alternative reinforcer and larger increases in target responding when a higher quality alternative reinforcer was removed. Along with prior findings with higher rates and magnitudes of alternative reinforcement, these findings suggest that variations in reinforcer dimensions that increase the efficacy of alternative reinforcement also tend to increase resurgence when alternative reinforcement is removed. The results are discussed in terms of the resurgence as choice in context model and in terms of potential clinical implications.

Resurgence of destructive behavior following decreases in alternative reinforcement: A prospective analysis.

Basic and retrospective translational research has shown that the magnitude of resurgence is determined by the size of the decrease in alternative reinforcement, with larger decreases producing more resurgence. However, this finding has not been evaluated prospectively with a clinical population. In Experiment 1, five participants experienced a fixed progression of reinforcement schedule-thinning steps during treatment of their destructive behavior. Resurgence occurred infrequently across steps and participants, and when resurgence did occur, its clinical meaningfulness was often minimal. In Experiment 2, five new participants experienced these same schedule-thinning steps but in a counterbalanced order. Resurgence occurred most often and was generally largest with larger decreases in alternative reinforcement programmed earlier in the evaluation. Large decreases in alternative reinforcement may be more problematic clinically when they occur earlier in treatment. Whether larger transitions can be recommended in the clinic following the success of smaller ones will require additional research.

Loss of nondrug reinforcement in one context produces alcohol seeking in another context.

In the resurgence preparation, extinguished alcohol-maintained responding increases when food reinforcement introduced into the same context during extinction is also subsequently removed. However, drug and nondrug reinforcers may often be obtained in separate specific contexts. Accordingly, we aimed to determine whether extinguished behavior previously maintained by alcohol would increase upon elimination of nondrug reinforcement within a multiple schedule arranging distinct discriminative stimulus contexts of food and alcohol availability. In Experiment 1, rats earned food or alcohol in alternating stimulus contexts within a multiple schedule. First, alcohol-maintained responding was extinguished and then food deliveries in the alternating component were also discontinued. Extinguished alcohol-maintained responding increased upon discontinuation of food deliveries. However, periods of alcohol availability alternated with periods of extinction during a portion of training; thus, elimination of food reinforcers during the resurgence test may have inadvertently served as a cue for alcohol availability. In Experiment 2, the training phase that complicated interpretation of the results of Experiment 1 was eliminated. Alcohol-maintained responding again increased when food-maintained responding was placed on extinction in the other component. The present results indicate that loss of nondrug reinforcement in one discriminative context can increase extinguished alcohol seeking in another context and that multiple schedules of reinforcement might be useful for examining such effects.

Behavioral momentum and resurgence: Effects of time in extinction and repeated resurgence tests.

Resurgence is an increase in a previously extinguished operant response that occurs if an alternative reinforcement introduced during extinction is removed. Shahan and Sweeney (2011) developed a quantitative model of resurgence based on behavioral momentum theory that captures existing data well and predicts that resurgence should decrease as time in extinction and exposure to the alternative reinforcement increases. Two experiments tested this prediction. The data from Experiment 1 suggested that without a return to baseline, resurgence decreases with increased exposure to alternative reinforcement and to extinction of the target response. Experiment 2 tested the predictions of the model across two conditions, one with constant alternative reinforcement for five sessions, and the other with alternative reinforcement removed three times. In both conditions, the alternative reinforcement was removed for the final test session. Experiment 2 again demonstrated a decrease in relapse across repeated resurgence tests. Furthermore, comparably little resurgence was observed at the same time point in extinction in the final test, despite dissimilar previous exposures to alternative reinforcement removal. The quantitative model provided a good description of the observed data in both experiments. More broadly, these data suggest that increased exposure to extinction may be a successful strategy to reduce resurgence. The relationship between these data and existing tests of the effect of time in extinction on resurgence is discussed.

Examining resurgence in rats following expanded-operant treatments.

Resurgence of previously reinforced behavior represents a challenge to otherwise successful interventions based on differential reinforcement of alternative behavior (DRA). Expanded-operant treatments seek to increase the number of functional alternative behaviors through DRA, thereby potentially mitigating resurgence. However, the few studies that have directly examined these methods as a tool for resurgence mitigation have provided limited and unclear results. Thus, the present experiments were designed to investigate the effect of expanded-operant DRA methods on resurgence of previously reinforced behavior using rat subjects. In two experiments, following a baseline phase in which a target response was trained, groups of rats experienced concurrent (i.e., five simultaneous alternative responses), serial (i.e., five sequentially available alternative responses), or single DRA interventions arranging similar rates of alternative reinforcement in order to examine potential differences in resurgence. Both experiments showed that neither serial nor concurrent DRA expanded-operant treatments reduced resurgence compared with single DRA regardless of whether stimuli associated with previously reinforced alternative responses were removed (Experiment 1) or remained present (Experiment 2) for the serial-DRA group. Further, a primacy effect in resurgence was obtained for the serial-DRA group in both experiments. Overall, these results suggest that expanded-operant treatments may not help to reduce resurgence.

Aversive control versus stimulus control by punishment.

Estes (1944) reported that adding electric shock punishment to extinction hastened response suppression but that responding increased when shock was removed. This result contributed to a view that reinforcement and punishment are asymmetrical processes because punishment has only indirect and temporary suppressive effects. Azrin and Holz (1966) suggested the result might be interpreted instead as shock serving as a discriminative stimulus for the absence of reinforcement. Here, to further examine potential stimulus control by punishment in a similar preparation, two groups of rats initially responded for food plus punishment and a third group for food alone. Reinforcement was then removed for all groups for the remaining three phases. With P and N denoting punishment and no punishment, the four phases for the three groups were: P-P-N-N, P-N-P-N, and N-P-N-N. We found some evidence for stimulus control by shock deliveries for group N-P-N-N (as suggested by Azrin and Holz), but all other changes in responding appeared due to introduction or removal of the aversive properties of shock. Although punishment may indeed have temporary effects under many circumstances, we argue that the view that this implies asymmetrical reinforcement and punishment processes was based on the flawed assumption that reinforcement has direct strengthening effects.

Basic and applied research on extinction bursts.

Discontinuation of the contingency between a response and its reinforcer sometimes produces a temporary increase in the response before its rate decreases, a phenomenon called the extinction burst. Prior clinical and basic studies on the prevalence of the extinction burst provide highly disparate estimates. Existing theories on the extinction burst fail to account for the dynamic nature of this phenomenon, and the basic behavioral processes that control response bursting remain poorly understood. In this paper, we first review the basic and applied literature on the extinction burst. We then describe a recent refinement of the concatenated matching law called the temporally weighted matching law that appears to resolve the above-mentioned issues regarding the extinction burst. We present illustrative translational data based conceptually on the model. Finally, we discuss specific recommendations derived from the temporally weighted matching law regarding procedures clinicians could implement to potentially mitigate or prevent extinction bursts.

Further evaluation of treatment duration on the resurgence of destructive behavior.

Translation of promising procedures for mitigating treatment relapse has received considerable attention recently from researchers across the basic-applied continuum. One procedure that has demonstrated mixed support involves increasing the duration of treatment as a strategy for blunting resurgence. In a recent translational study, Greer et al. (2020) failed to detect a mitigation effect of increased treatment duration on the resurgence of destructive behavior. However, design limitations may have been responsible. The present study corrected these limitations by (a) employing a sequential design to decrease the possibility of multiple-treatment interference, (b) evaluating more treatment durations, (c) arranging treatments of fixed durations, and (d) conducting treatments of more extreme duration in a different clinical sample. Despite these improvements in experimental rigor and the testing of more extreme boundary conditions, the present study also failed to detect a mitigation effect of increased treatment duration. Likely explanations are discussed.

Differential reinforcement and resistance to change of divided-attention performance.

Behavioral momentum theory provides a framework for understanding how conditions of reinforcement influence instrumental response strength under conditions of disruption (i.e., resistance to change). The present experiment examined resistance to change of divided-attention performance when different overall probabilities of reinforcement were arranged across two components of a multiple schedule. Pigeons responded in a delayed-matching-to-sample procedure with compound samples (color + line orientation) and element comparisons (two colors or two line orientations). Reinforcement ratios of 1:9, 1:1, and 9:1 for accurate matches on the two types of comparison trials were examined across conditions using reinforcement probabilities (color/lines) of .9/.1, .5/.5, and .1/.9 in the rich component and .18/.02, .1/.1, and .02/.18 in the lean component. Relative accuracy with color and line comparisons was an orderly function of relative reinforcement, but this relation did not depend on the overall rate of reinforcement between components. The resistance to change of divided-attention performance was greater for both trial types in the rich component with presession feeding and extinction, but not with decreases in sample duration. These findings suggest promise for the applicability of quantitative models of operant behavior to divided-attention performance, but they highlight the need to further explore conditions impacting the resistance to change of attending.

Delayed matching to sample: reinforcement has opposite effects on resistance to change in two related procedures.

The effects of reinforcement on delayed matching to sample (DMTS) have been studied in two within-subjects procedures. In one, reinforcer magnitudes or probabilities vary from trial to trial and are signaled within trials (designated signaled DMTS trials). In the other, reinforcer probabilities are consistent for a series of trials produced by responding on variable-interval (VI) schedules within multiple-schedule components (designated multiple VI DMTS). In both procedures, forgetting functions in rich trials or components are higher than and roughly parallel to those in lean trials or components. However, during disruption, accuracy has been found to decrease more in rich than in lean signaled DMTS trials and, conversely, to decrease more in lean than in rich multiple VI DMTS components. In the present study, we compared these procedures in two groups of pigeons. In baseline, forgetting functions in rich trials or components were higher than and roughly parallel to those in lean trials or components, and were similar between the procedures. During disruption by prefeeding or extinction, accuracy decreased more in rich signaled DMTS trials, whereas accuracy decreased more in lean multiple VI DMTS components. These results replicate earlier studies and are predicted by a model of DMTS from Nevin, Davison, Odum, and Shahan (2007).

A Theory of the Extinction Burst.

A preliminary theory of a temporary increase in the rate of an operant response with the transition to extinction (i.e., the extinction burst) is proposed. The theory assumes reinforcers are events permitting access to some valuable activity, and that such activity can compete for allocation with the target response under some conditions (e.g., very high reinforcement rates). With the transition to extinction, elimination of this competition for allocation can produce an increase in the the target response, but the increase is transient because the value of the target response decreases with exposure to extinction. The theory provides a way to understand why the extinction burst is not ubiquitous, seems more common following very small ratio schedules, occurs for a short period of time following the transition to extinction, and may be eliminated with the availability of alternative reinforcement. It appears to provide a reasonable starting point for a theory of the extinction burst that does not necessarily require inclusion of invigorating effects of frustration, and it is closely aligned with Resurgence as Choice theory. Additional research on factors modulating reinforcement-related activities and how they affect the extinction burst could help to further evaluate the theory.

Effects of repeated exposure to escalating versus constant punishment intensity on response allocation.

The present experiment investigated the effects of 1) repeated exposures to escalating punishment intensities and 2) repeated exposure to punishment after periods of vacation on response allocation between punished and unpunished responding in three groups of rats. The first group (intensity + vacation) experienced repeated exposures to escalating punishment intensities after a period of vacation (i.e., return to baseline) from punishment. The second group (intensity-only) experienced repeated exposures to escalating punishment intensities without vacation from punishment. The third group (vacation-only) experienced repeated exposures to a constant punishment intensity after a period of vacation from punishment. Results showed that superimposition of punishment on one of two concurrently available responses decreased allocation toward the punished response and increased allocation toward the unpunished response. Furthermore, greater changes in allocation were observed with the introduction of a moderate constant intensity than with the introduction of a low intensity that increased across sessions. Reexposure to punishment had different effects between the groups. Although there was evidence that high shock intensities can enhance the efficacy of lower intensities to shift allocation away from the punished response and toward the unpunished response, there was little evidence of changes in response allocation with reintroduction of punishment after a period of vacation.

Nondrug reinforcers contingent on alternative behavior or abstinence increase resistance to extinction and reinstatement of ethanol-maintained behavior.

The effects of delivering nondrug alternative reinforcement on resistance to extinction and reinstatement of rats' ethanol-maintained lever pressing were evaluated in two experiments. In both, rats self-administered ethanol by lever pressing in a two-component multiple schedule during baseline. In the Rich component, alternative food reinforcement was made available for performing an alternative response (Experiment 1) or according to a differential-reinforcement-of-other-behavior schedule for lever pressing (Experiment 2). In the Lean component, only ethanol was available. Moreover, the frequency of alternative reinforcement was manipulated across conditions in Experiment 1. Following baseline, lever pressing was extinguished in both components by suspending ethanol reinforcement, and alternative food reinforcers were discontinued. Finally, to test for reinstatement, ethanol reinforcers were delivered independently of lever pressing in both components. In both experiments, proportion-of-baseline response rates were higher during extinction and reinstatement testing in the Rich component than in the Lean component (although differentiation was not observed at the lowest frequency of alternative reinforcement in Experiment 1). Thus, alternative nondrug reinforcers increased resistance to extinction and reinstatement of rats' ethanol-maintained lever pressing, even when those reinforcers were delivered contingently on an alternative response or on abstinence from lever pressing.