Engineered highs: Reward variability and frequency as potential prerequisites of behavioural addiction.

Influential learning-based accounts of substance addictions posit the attribution of incentive salience to drug-associated cues, and its escalation by the direct dopaminergic effects of drugs. In translating this account to disordered gambling, we have noted how the intermittent nature of monetary rewards in gambling (i.e. the variable ratio) may allow for analogous learning processes, via effects on dopaminergic signalling. The aim of the present article is to consider how multiple sources of reward variability operate within modern gambling products, and how similar sources of variability, as well as some novel sources of variability, also apply to other digital products implicated in behavioural addictions, including gaming, shopping, social media and online pornography. Online access to these activities facilitates not only unparalleled accessibility but also introduces novel forms of reward variability, as seen in the effects of infinite scrolls and personalized recommendations. We use the term uncertainty to refer to the subjective experience of reward variability. We further highlight two psychological factors that appear to moderate the effects of uncertainty: 1) the timecourse of uncertainty, especially with regard to its resolution, 2) the frequency of exposure, allowing temporal compression. Collectively, the evidence illustrates how qualitative and quantitative variability of reward can confer addictive potential to non-drug reinforcers by exploiting the psychological and neural processes that rely on predictability to guide reward seeking behaviour.

Choice between food and cocaine or fentanyl reinforcers under fixed and variable schedules in female and male rhesus monkeys.

RATIONALE: Illicit drugs may be unpredictable in terms of the time and effort required to obtain them, and this can be modeled with variable- (VR) vs. fixed-ratio (FR) schedules. In a recent experiment (Zamarripa et al. 2023), the potency of cocaine to maintain choice was greatest under a VR (compared with a FR) when food was available under a FR schedule. OBJECTIVES: The goal of the current study was to extend prior choice results with VR vs. FR schedules to a more efficient procedure with cocaine or fentanyl vs. food. Furthermore, the FR schedule of food delivery was manipulated to determine whether increased drug choice under a VR (compared with a FR) schedule depends on the size of the schedule of nondrug reinforcement. METHODS: Adult female (n = 2) and male (n = 4) monkeys chose between cocaine (0-30 µg/kg/injection) or fentanyl (0-1.0 µg/kg/injection) and food (2 pellets/delivery) under a 5-component procedure. In different conditions, food was available under a FR 25, 50, or 100 and cocaine or fentanyl were available under FR or VR 100 schedules. RESULTS: Cocaine's potency to maintain choice was greatest under a VR 100 (compared with FR 100) when food was available under a FR 50 or 100, and fentanyl's potency to maintain choice was generally greatest under a VR 100 (compared with FR 100) when food was available under a FR 25 or 100. However, outcomes between FR and VR schedules with fentanyl were less robust compared with cocaine. CONCLUSION: Variability in the time and effort required to obtain illicit drugs could contribute to excessive allocation of behavior toward drug use at the expense of more predictable nondrug alternatives, supporting treatment or policies aimed at making drug access more predictable through agonist medications or a safe supply. The impact of variable requirements on drug choice may be reduced if nondrug reinforcers are relatively less costly, supporting the use of low-cost reinforcers in behavioral therapies like contingency management.

The influence of reinforcement schedule on experience-dependent changes in motivation.

The progressive ratio procedure is used across fields to assess motivation for different reinforcers, define the effects of experimental interventions on motivation, and determine experience-dependent changes in motivation. However, less is known about how operant training schedules affect performance on this widely utilized task. Here we designed an experiment to examine the effect of variable ratio versus fixed ratio training schedules of reinforcement on progressive ratio performance while holding other performance variables constant between groups. We found a robust increase in maximum ratio completed between the pretest and posttraining test highlighting a robust training effect on progressive ratio performance. However, it did not matter if the training was under a fixed or variable ratio schedule. Additionally, we show that neither individual rates during training nor extinction responding correlated with maximum ratio achieved during the sessions. Finally, we show that rates during the training sessions do correlate with extinction performance, suggesting that these variables measure a different aspect of performance that does not predict motivation.

Unpredictability as a modulator of drug self-administration: Relevance for substance-use disorders.

Drug self-administration has been regarded as a gold-standard preclinical model of addiction and substance-use disorder (SUD). However, investigators are becoming increasingly aware, that certain aspects of addiction or SUDs experienced by humans are not accurately captured in our preclinical self-administration models. The current review will focus on two such aspects of current preclinical drug self-administration models: 1) Predictable vs. unpredictable drug access in terms of the time and effort put into obtaining drugs (i.e., response requirement) and drug quality (i.e., amount) and 2) rich vs. lean access to drugs. Some behavioral and neurobiological mechanisms that could contribute to excessive allocation of behavior toward drug-seeking and drug-taking at the expense of engaging in nondrug-related activities are discussed, and some directions for future research are identified. Based on the experiments reviewed, lean and unpredictable drug access could worsen drug-seeking and drug-taking behavior in individuals with SUDs. Once more fully explored, this area of research will help determine whether and how unpredictable and lean cost requirements affect drug self-administration in preclinical laboratory studies with nonhuman subjects and will help determine whether incorporating these conditions in current self-administration models will increase their predictive validity.

Modeling bout-pause response patterns in variable-ratio and variable-interval schedules using hierarchical Bayesian methodology.

Streams of operant responses are arranged in bouts separated by pauses and differences in performance in reinforcement schedules with identical inter-reinforcement intervals (IRIs) are primarily due to differences in within-bout response rate, not in bout-initiation rate. The present study used hierarchical Bayesian modeling as a new method to quantify the properties of the response bout. A Bernoulli distribution was utilized to express the probability to stay in bout/pause, while a Poisson distribution was utilized to quantify the within-bout response rates. We compared bout/pause patterns between variable-ratio (VR) and variable-interval (VI) schedules across IRIs. The model estimation revealed no difference in within-bout staying probability between schedules. However, response rates of within-bout responses were higher in VR than VI across IRIs. These results are consistent with previous analyses using a log-survivor plot to describe within-bout responses and bouts-initiation responses. In addition, a simulation study was performed to examine how sensitively the model estimate the parameters according to different bout initiation rates. These result showed that the within-bout staying probability was affected by changes in between-bout while within-bout response rate parameters were not. This suggests model estimation robustness of the model estimation to dissociate within-bout and between-bout parameters during different reinforcement schedules.

Cooperative responding in rats maintained by fixed- and variable-ratio schedules.

The present study investigated the effects of fixed-ratio (FR) and variable-ratio (VR) reinforcement schedules on patterns of cooperative responding in pairs of rats. Experiment 1 arranged FR 1, FR 10, and VR 10 schedules to establish cooperative responding (water delivery depended on the joint responding of two rats). Cooperative response rates and proportions were higher under intermittent schedules than under continuous reinforcement. The FR 10 schedule generated a break-and-run pattern, whereas the VR 10 schedule generated a relatively high and constant rate pattern. Experiment 2 evaluated the effects of parametric manipulations of FR and VR schedules on cooperative responding. Rates and proportions of cooperative responding generally increased between ratio sizes of 1 and 5 but showed no consistent trend as the ratio increased from 5 to 10. Experiment 3 contrasted cooperative responding between an FR6 schedule and a yoked control schedule. Coordinated behavior occurred at a higher rate under the former schedule. The present study showed that external consequences and the schedules under which the delivery of these consequences are based, select patterns of coordinated behavior.

Mirtazapine and ketanserin alter preference for gambling-like schedules of reinforcement in rats.

Drug and behavioral addictions have overlapping features, e.g., both manifest preference for larger, albeit costlier, reinforcement options in cost/benefit decision-making tasks. Our prior work revealed that the mixed-function serotonergic compound, mirtazapine, attenuates behaviors by rats motivated by abused drugs. To extend this work to behavioral addictions, here we determined if mirtazapine and/or ketanserin, another mixed-function serotonin-acting compound, can alter decision-making in rats that is independent of drug (or food)-motivated reward. Accordingly, we developed a novel variable-ratio task in rats wherein intracranial self-stimulation was used as the positive reinforcer. Using lever pressing for various levels of brain stimulation, the operant task provided choices between a small brain stimulation current delivered on a fixed-ratio schedule (i.e., a predictable reward) and a large brain stimulation delivered following an unpredictable number of responses (i.e., a variable-ratio schedule). This task allowed for demonstration of individualized preference and detection of shifts in motivational influences during a pharmacological treatment. Once baseline preference was established, we determined that pretreatment with mirtazapine or ketanserin significantly decreased preference for the large reinforcer presented after gambling-like schedules of reinforcement. When the rats were tested the next day without drug, preference for the unpredictable large reinforcer option was restored. These data demonstrate that mirtazapine and ketanserin can reduce preference for larger, costlier reinforcement options, and illustrate the potential for these drugs to alter behavior.

Response-bout analysis of interresponse times in variable-ratio and variable-interval schedules.

The purpose of this study was to analyze the response pattern difference between variable-ratio (VR) and variable-interval (VI) schedules of reinforcement by modeling interresponse time distributions of rats' lever presses. All eight rats showed higher response rates under VR 30 than under inter-reinforcement intervals yoked VI. The 30 models consisting of single Exponential (with and without the lower limit on interresponse times), Weibull, Normal, Log-Normal or Gamma distributions, all possible two component combinations of those, and 3 and 4 component models consisting of Weibull, Normal, Log-Normal, or Gamma distribution combinations were compared. The 4 component Log-Normal model was the best in terms of the Akaike information criterion and visual inspection of fitting outcome. Parameter estimates for the L4 model showed that the VR-VI response rate difference is due to a difference in short interresponse times or within bout responses. This results suggests that the VR-VI response rate difference is not an indication of a difference in the overall tendency to respond but it is rather a difference in terms of what types of response patterns are engendered between the two schedules.

Evidence for positive, but not negative, behavioral contrast with wheel-running reinforcement on multiple variable-ratio schedules.

Rats responded on a multiple variable-ratio (VR) 10 VR 10 schedule of reinforcement in which lever pressing was reinforced by the opportunity to run in a wheel for 30s in both the changed (manipulated) and unchanged components. To generate positive contrast, the schedule of reinforcement in the changed component was shifted to extinction; to generate negative contrast, the schedule was shifted to VR 3. With the shift to extinction in the changed component, wheel-running and local lever-pressing rates increased in the unchanged component, a result supporting positive contrast; however, the shift to a VR 3 schedule in the changed component showed no evidence of negative contrast in the unaltered setting, only wheel running decreased in the unchanged component. Changes in wheel-running rates across components were consistent in showing a compensation effect, depending on whether the schedule manipulation increased or decreased opportunities for wheel running in the changed component. These findings are the first to demonstrate positive behavioral contrast on a multiple schedule with wheel running as reinforcement in both components.

The role of induction in operant schedule performance.

Baum and Davison (2014b) showed that Baum's (2012) recasting of reinforcement as induction may be quantified by assuming that induction follows a power function of reinforcer rate. This power-function induction is readily integrated with theory based on the matching law. Herrnstein (1970) originally assumed background activities (BO) and their associated reinforcers ro to be constant, but ro should vary with BO. Further, power-function induction implies that BO should vary with reinforcer rate. Baum (1993) reported performance on a wide range of variable-ratio (VR) and variable-interval (VI) schedules. Pigeons' VR peck rate followed an inverted U-shaped relation, but VI peck rate separated into three ranges of food rate: low-to-moderate, moderate-to-high, and extremely high. As food rate increases, the concave downward relation in the low range reaches an inflection point and gives way to a concave upward relation in the higher range. At the extremes of food rate, VI peck rate decreases. A model based on competition between induced pecking and BO accounted for VI peck rate in the moderate to extreme range of food rates. Further research will account for all three ranges, either by integrating power-function induction with matching theory or with a model based on competition between induced activities.

The copyist model and the shaping view of reinforcement.

The strengthening view of reinforcement attributes behavior change to changes in the response strength or the value of the reinforcer. In contrast, the shaping view explains behavior change as shaping different response units through differential reinforcement. In this paper, we evaluate how well these two views explain: (1) the response-rate difference between variable-ratio and variable-interval schedules that provide the same reinforcement rate; and (2) the phenomenon of matching in choice. The copyist model (Tanno and Silberberg, 2012) - a shaping-view account - can provided accurate predictions of these phenomena without a strengthening mechanism; however, the model has limitations. It cannot explain the relation between behavior change and stimulus control, reinforcer amount, and reinforcer quality. These relations seem easily explained by a strengthening view. Future work should be directed at a model which combine the strengths of these two types of accounts.

The attraction of gambling.

If a repeated gamble is subjectively structured into units each consisting of a string of consecutive losses followed by a single win, longer strings will necessarily be less valuable. Longer, less valuable strings will be discounted by delay more than will shorter, more valuable strings. This implies that the whole gamble's expected, delay-discounted value will increase as delay discounting increases. With this restructuring, even games of (objectively) negative expected value, such as those at casinos, may be subjectively positive. The steeper the delay discounting, the greater the subjective value of the gamble (over normal ranges of discounting steepness). Frequent gamblers, who value gambles highly, would thus be expected to discount delayed rewards more steeply than would nongamblers.

Briefly delayed reinforcement effects on variable-ratio and yoked-interval schedule performance.

Most investigations of briefly delayed reinforcement have involved schedules that arrange a time-plus-response requirement. The present experiment examined whether briefly delaying reinforcement on schedules that have a ratio requirement differs from results with schedules that have a time-plus-response requirement. Four pigeons responded on a two-component multiple schedule. One component arranged a variable-ratio (VR) 50 and the other a variable-interval (VI) schedule in which the distribution of reinforcers was yoked to the preceding VR schedule. Across a series of conditions, delays were imposed in both schedules. These delays were brief (0.25- or 0.5-s) unsignaled delays and, as control conditions, a 5-s unsignaled delay and a 0.5-s delay signaled by a blackout of the chamber. In the yoked-VI component, the brief unsignaled delay increased response rates in six of nine opportunities and increased the proportion of short interresponse times (IRTs) (<0.4 s) in eight of nine opportunities. In the VR component, the brief unsignaled delay increased response rates and the proportion of short IRTs in only two of nine opportunities. For two of the three pigeons that were exposed to the 5-s unsignaled delay, response rates and the proportion of short IRTs decreased in both of the components. The 0.5-s signaled delay did not systematically change response rates nor did it change the distribution of short IRTs relative to the immediate reinforcement condition. The results replicate effects reported with time-based schedules and extend these observations by showing that changes commonly observed in VI performance with briefly delayed reinforcement are not characteristic of VR responding.

Discrimination of variable schedules is controlled by interresponse times proximal to reinforcement.

In Experiment 1, food-deprived rats responded to one of two schedules that were, with equal probability, associated with a sample lever. One schedule was always variable ratio, while the other schedule, depending on the trial within a session, was: (a) a variable-interval schedule; (b) a tandem variable-interval, differential-reinforcement-of-low-rate schedule; or (c) a tandem variable-interval, differential-reinforcement-of-high-rate schedule. Completion of a sample-lever schedule, which took approximately the same time regardless of schedule, presented two comparison levers, one associated with each sample-lever schedule. Pressing the comparison lever associated with the schedule just presented produced food, while pressing the other produced a blackout. Conditional-discrimination accuracy was related to the size of the difference in reinforced interresponse times and those that preceded it (predecessor interresponse times) between the variable-ratio and other comparison schedules. In Experiment 2, control by predecessor interresponse times was accentuated by requiring rats to discriminate between a variable-ratio schedule and a tandem schedule that required emission of a sequence of a long, then a short interresponse time in the tandem's terminal schedule. These discrimination data are compatible with the copyist model from Tanno and Silberberg (2012) in which response rates are determined by the succession of interresponse times between reinforcers weighted so that each interresponse time's role in rate determination diminishes exponentially as a function of its distance from reinforcement.

Single-sample discrimination of different schedules' reinforced interresponse times.

Food-deprived rats in Experiment 1 responded to one of two tandem schedules that were, with equal probability, associated with a sample lever. The tandem schedules' initial links were different random-interval schedules. Their values were adjusted to approximate equality in time to completing each tandem schedule's response requirements. The tandem schedules differed in their terminal links: One reinforced short interresponse times; the other reinforced long ones. Tandem-schedule completion presented two comparison levers, one of which was associated with each tandem schedule. Pressing the lever associated with the sample-lever tandem schedule produced a food pellet. Pressing the other produced a blackout. The difference between terminal-link reinforced interresponse times varied across 10-trial blocks within a session. Conditional-discrimination accuracy increased with the size of the temporal difference between terminal-link reinforced interresponse times. In Experiment 2, one tandem schedule was replaced by a random ratio, while the comparison schedule was either a tandem schedule that only reinforced long interresponse times or a random-interval schedule. Again, conditional-discrimination accuracy increased with the temporal difference between the two schedules' reinforced interresponse times. Most rats mastered the discrimination between random ratio and random interval, showing that the interresponse times reinforced by these schedules can serve to discriminate between these schedules.

Efeitos paramétricos do atraso de reforço sobre o responder de ratos submetidos a esquemas de razão e intervalos variáveis / Parametrical effects on delayed reinforcement on the responding of rats exposed to variable interval and variable ratio schedules

Foram investigados os efeitos produzidos pela introdução de diferentes valores de atraso do reforço sobre as respostas de pressão à barra emitidas por oito ratos expostos a esquemas de intervalo e razão variáveis (VI e VR, respectivamente). A partir das condições de linha de base sob reforçamento imediato, aplicou-se o atraso do reforço não sinalizado e não reiniciável pelo tempo máximo programado de 1 s, 5 s e 8 s a cada um dos esquemas, em diferentes condições, em ordem crescente e decrescente de atraso. Os resultados mostraram correlação negativa entre taxas de respostas e valores do atraso do reforço para praticamente todos os sujeitos experimentais, em todos os valores de atraso. Por meio da análise dos atrasos efetivamente obtidos na sessão, foi possível constatar que a distribuição de frequência dos atrasos, independentemente do esquema em vigor, VI ou VR, concentraram-se em 0,5 s, quando o atraso programado era de 2 s. Diferentemente, foram próximos de 5 s e 8 s quando os atrasos programados foram de 5 s e 8 s, respectivamente. Esses efeitos diferenciados do atraso sobre o responder podem ser compreendidos mediante uma análise das alterações produzidas pelo atraso nas contingências específicas de cada esquema.

This study examined the effects of introducing different delays of reinforcement on the rate of leverpressing by eight rats under variable interval and variable ratio schedules (VI and VR, respectively). Following baseline conditions under immediate reinforcement, unsignaled and non-resetting delays of reinforcement were introduced upon each of these schedules. In different conditions, delays of 2 s, 5 s and 8 s were presented in both increasing and decreasing sequences. The results showed negative correlations between rate of responding and delay of reinforcement for most subjects and delays. By analyzing the frequency distribution of the delays actually obtained in each single reinforcer delivery, it was possible to verify a higher concentration of 0.5-s delays when a 2-s delay was programmed. Incontrast, actual delays concentrated around 5 s and 8 s when the programmed delays were 5 s and 8 s,respectively. Such differential effects generated by the delays of reinforcement on responding can beunderstood by analyzing how the delay affected the ongoing contingencies in each particular schedule.