Modeling VI and VDRL feedback functions: Searching normative rules through computational simulation.

We present the mathematical description of feedback functions of variable interval and variable differential reinforcement of low rates as functions of schedule size only. These results were obtained using an R script named Beak, which was built to simulate rates of behavior interacting with simple schedules of reinforcement. Using Beak, we have simulated data that allow an assessment of different reinforcement feedback functions. This was made with unparalleled precision, as simulations provide huge samples of data and, more importantly, simulated behavior is not changed by the reinforcement it produces. Therefore, we can vary response rates systematically. We've compared different reinforcement feedback functions for random interval schedules, using the following criteria: meaning, precision, parsimony, and generality. Our results indicate that the best feedback function for the random interval schedule was published by Baum (1981). We also propose that the model used by Killeen (1975) is a viable feedback function for the random differential reinforcement of low rates schedule. We argue that Beak paves the way for greater understanding of schedules of reinforcement, addressing still open questions about quantitative features of simple schedules. Also, Beak could guide future experiments that use schedules as theoretical and methodological tools.

The effects of fatty acid amide hydrolase inhibition and monoacylglycerol lipase inhibition on habit formation in mice.

Emerging data indicate that endocannabinoid signaling is critical to the formation of habitual behavior. Previous work demonstrated that antagonism of cannabinoid receptor type 1 (CB1R) with AM251 during operant training impairs habit formation, but it is not known if this behavioral effect is specific to disrupted signaling of the endocannabinoid ligands anandamide or 2-arachidonoyl glycerol (2-AG). Here, we used selective pharmacological compounds during operant training to determine the impact of fatty acid amide hydrolase (FAAH) inhibition to increase anandamide (and other n-acylethanolamines) or monoacylglycerol lipase (MAGL) inhibition to increase 2-AG levels on the formation of habitual behaviors in mice using a food-reinforced contingency degradation procedure. We found, contrary to our hypothesis, that inhibition of FAAH and of MAGL disrupted the formation of habits. Next, AM251 was administered during training to verify that impaired habit formation could be assessed using contingency degradation. AM251-exposed mice responded at lower rates during training and at higher rates in the test. To understand the inconsistency with published data, we performed a proof-of-principle dose-response experiment to compare AM251 in our vehicle-solution to the published vehicle-suspension on response rates. We found consistent reductions in response rate with increasing doses of AM251 in solution and an inconsistent dose-response relationship with AM251 in suspension. Together, our data suggest that further characterization of the role of CB1R signaling in the formation of habitual responding is warranted and that augmenting endocannabinoids may have clinical utility for prophylactically preventing aberrant habit formation such as that hypothesized to occur in substance use disorders.

Hazard function effects on promoting self-control in variable interval time-based interventions in rats.

The present experiments investigated properties of time-based interventions used to increase self-control. Rats received impulsive-choice assessments before and after interventions that consisted of different distributions of delays to reinforcement. In Experiment 1, rats received an intervention with an increasing hazard function where delays were more evenly distributed, a decreasing hazard function where delays were mostly short, or a constant hazard function where delays were exponentially distributed. Surprisingly, rats that received the decreasing hazard function made the most self-controlled choices. Response rates during intervention trials showed that rats anticipated reinforcement based on the shape of the distributions they received. In Experiment 2, rats received an intervention with a decreasing hazard function with a steep slope or a shallow slope. Both time-based interventions increased self-control and produced similar response-rate patterns, indicating that the slope of the decreasing hazard function may not play a strong role in intervention efficacy. While this research aligns with previous literature showing that time-based interventions improved self-control, exposure to short delays produced the biggest improvements. Ultimately, exposure to short delays may increase the subjective value of the larger-later choice while occasional long delays may promote the ability to wait, which may have important implications for translational applications.

Longer operant lever-press duration requirements induce fewer but longer response bouts in rats.

Operant behavior is organized in bouts that are particularly visible under variable-interval (VI) schedules of reinforcement. Previous research showed that increasing the work required to produce a response decreases the rate at which bouts are emitted and increases the minimum interresponse time (IRT). In the current study, the minimum effective IRT was directly manipulated by changing the minimum duration of effective lever presses reinforced on a VI 40-s schedule. Contrary to assumptions of previous models, response durations were variable. Response durations were typically 0.5 s greater than the minimum duration threshold; durations that exceeded this threshold were approximately log-normally distributed. As the required duration threshold increased, rats emitted fewer but longer bouts. This effect may reflect an effort-induced reduction in motivation and a duration-induced facilitation of a response-outcome association.

Safety and cost-effectiveness of individualised screening for diabetic retinopathy: the ISDR open-label, equivalence RCT.

AIMS/HYPOTHESIS: Using variable diabetic retinopathy screening intervals, informed by personal risk levels, offers improved engagement of people with diabetes and reallocation of resources to high-risk groups, while addressing the increasing prevalence of diabetes. However, safety data on extending screening intervals are minimal. The aim of this study was to evaluate the safety and cost-effectiveness of individualised, variable-interval, risk-based population screening compared with usual care, with wide-ranging input from individuals with diabetes. METHODS: This was a two-arm, parallel-assignment, equivalence RCT (minimum 2 year follow-up) in individuals with diabetes aged 12 years or older registered with a single English screening programme. Participants were randomly allocated 1:1 at baseline to individualised screening at 6, 12 or 24 months for those at high, medium and low risk, respectively, as determined at each screening episode by a risk-calculation engine using local demographic, screening and clinical data, or to annual screening (control group). Screening staff and investigators were observer-masked to allocation and interval. Data were collected within the screening programme. The primary outcome was attendance (safety). A secondary safety outcome was the development of sight-threatening diabetic retinopathy. Cost-effectiveness was evaluated within a 2 year time horizon from National Health Service and societal perspectives. RESULTS: A total of 4534 participants were randomised. After withdrawals, there were 2097 participants in the individualised screening arm and 2224 in the control arm. Attendance rates at first follow-up were equivalent between the two arms (individualised screening 83.6%; control arm 84.7%; difference -1.0 [95% CI -3.2, 1.2]), while sight-threatening diabetic retinopathy detection rates were non-inferior in the individualised screening arm (individualised screening 1.4%, control arm 1.7%; difference -0.3 [95% CI -1.1, 0.5]). Sensitivity analyses confirmed these findings. No important adverse events were observed. Mean differences in complete case quality-adjusted life-years (EuroQol Five-Dimension Questionnaire, Health Utilities Index Mark 3) did not significantly differ from zero; multiple imputation supported the dominance of individualised screening. Incremental cost savings per person with individualised screening were £17.34 (95% CI 17.02, 17.67) from the National Health Service perspective and £23.11 (95% CI 22.73, 23.53) from the societal perspective, representing a 21% reduction in overall programme costs. Overall, 43.2% fewer screening appointments were required in the individualised arm. CONCLUSIONS/INTERPRETATION: Stakeholders involved in diabetes care can be reassured by this study, which is the largest ophthalmic RCT in diabetic retinopathy screening to date, that extended and individualised, variable-interval, risk-based screening is feasible and can be safely and cost-effectively introduced in established systematic programmes. Because of the 2 year time horizon of the trial and the long time frame of the disease, robust monitoring of attendance and retinopathy rates should be included in any future implementation. TRIAL REGISTRATION: ISRCTN 87561257 FUNDING: The study was funded by the UK National Institute for Health Research. Graphical abstract.

The effects of force requirements on microstructure: Subcriterion responses and the mismeasurement of bouts.

The effects of response force on microstructure were evaluated. A strain-gauge operandum permitted the manipulation of the force required to produce reinforcers (criterion responses) independently from the force defining response threshold. Thus, we could detect subcriterion forces that fell short of the force criterion. Eight rats earned food according to variable-interval (VI) 30- and 120-s schedules. The force requirements were set to 5.6 or 32.0 g; the response threshold was fixed at 5.6 g. Interresponse times were computed when subcriterion responses were both included and omitted from the analysis. Log-survivor functions of interresponse times showed that increasing force requirements elevated the mean between-bout interval of the VI 120-s schedule, but only if subcriterion behavior was excluded. Omitting subcriterion responses thus leads to overestimation of intervals separating response bouts. Increasing force requirements also increased the skewness of the between-bout distribution. A subsequent analysis found that subcriterion responses are most plentiful following reinforcer delivery, which helps to explain why their omission might inflate between-bout intervals, as this period is an important transition from reinforcer consumption to engagement in operant activity. The data suggest caution interpreting the effects of force on microstructure when subcriterion behavior is not or cannot be measured.

Matching theory and induction explain operant performance.

Matching theory is a general framework for understanding allocation of behavior among activities. It applies to choice in concurrent schedules and was extended to single schedules by assuming that other unrecorded behavior competes with operant behavior. Baum and Davison (2014) found that the competing activities apparently are induced by the "reinforcers" (phylogenetically important events, e.g., food) according to power functions. Combined with power-function induction, matching theory provides new equations with greater explanatory power. Four pigeons were exposed to conditions in which 7 different schedules of food delivery were presented within each experimental session. We replicated earlier results with variable-interval schedules: (a) a negatively accelerated increase of peck rate as food rate increased in the low range of food rates; (b) an upturn in pecking at higher rates; and (c) a downturn in pecking at extremely high food rates. When the contingency between pecking and food was removed, the food continued to induce pecking, even after 20 sessions with no contingency. A ratio schedule inserted in place of 1 variable-interval schedule maintained peck rates comparable to peck rates maintained by short interval schedules. We explained the results by fitting equations that combined matching theory, competition, and induction.

Repeated, within-session resurgence.

Resurgence is a reliable, transient effect that only occasionally is replicated more than once within a single experiment or subject. In the present experiments, within-session resurgence was generated repeatedly by dividing individual sessions into three phases (Training, Alternative-Reinforcement, and Resurgence-Test). In Experiments 1 and 2, resurgence reliably occurred in most of the 22-30 daily sessions when responding was reinforced on, respectively, fixed- and variable-interval schedules. Resurgence magnitude and duration did decrease across replications for some subjects, but not for others. To examine the utility of the procedure in studying the effects of an independent variable on resurgence, in Experiment 3 the effects of rich and lean baseline and alternative reinforcement rates on resurgence were compared. The target response was eliminated more rapidly, resurgence occurred more often, and usually was greater following rich alternative reinforcement rates. Resurgence was of greater magnitude when the baseline reinforcement rate was relatively lean compared to the alternative reinforcement rate. These experiments provide a reliable method for generating resurgence within individual sessions, instead of across multiple-session conditions, that can be repeated over many successive sessions.

Modeling bout-pause response patterns in variable-ratio and variable-interval schedules using hierarchical Bayesian methodology.

Streams of operant responses are arranged in bouts separated by pauses and differences in performance in reinforcement schedules with identical inter-reinforcement intervals (IRIs) are primarily due to differences in within-bout response rate, not in bout-initiation rate. The present study used hierarchical Bayesian modeling as a new method to quantify the properties of the response bout. A Bernoulli distribution was utilized to express the probability to stay in bout/pause, while a Poisson distribution was utilized to quantify the within-bout response rates. We compared bout/pause patterns between variable-ratio (VR) and variable-interval (VI) schedules across IRIs. The model estimation revealed no difference in within-bout staying probability between schedules. However, response rates of within-bout responses were higher in VR than VI across IRIs. These results are consistent with previous analyses using a log-survivor plot to describe within-bout responses and bouts-initiation responses. In addition, a simulation study was performed to examine how sensitively the model estimate the parameters according to different bout initiation rates. These result showed that the within-bout staying probability was affected by changes in between-bout while within-bout response rate parameters were not. This suggests model estimation robustness of the model estimation to dissociate within-bout and between-bout parameters during different reinforcement schedules.

A new variable interval schedule with constant hazard rate and finite time range.

We propose a new variable interval (VI) schedule that achieves constant probability of reinforcement in time while using a bounded range of intervals. By sampling each trial duration from a uniform distribution ranging from 0 to 2 T seconds, and then applying a reinforcement rule that depends linearly on trial duration, the schedule alternates reinforced and unreinforced trials, each less than 2 T seconds, while preserving a constant hazard function.

Behavioral toxicity of sodium cyanide following oral ingestion in rats: Dose-dependent onset, severity, survival, and recovery.

Sodium cyanide (NaCN) is a commonly and widely used industrial and laboratory chemical reagent that is highly toxic. Its availability and rapid harmful/lethal effects combine to make cyanide a potential foodborne/waterborne intentional-poisoning hazard. Thus, laboratory studies are needed to understand the dose-dependent progression of toxicity/lethality following ingestion of cyanide-poisoned foods/liquids. We developed an oral-dosing method in which a standard pipette was used to dispense a sodium cyanide solution into the cheek, and the rat then swallowed the solution. Following poisoning (4-128 mg/kg), overt toxic signs were recorded and survival was evaluated periodically up to 30 hours thereafter. Toxic signs for NaCN doses higher than 16 mg/kg progressed quickly from head burial and mastication, to lethargy, convulsions, gasping/respiratory distress, and death. In a follow-on study, trained operant-behavioral performance was assessed immediately following cyanide exposure (4-64 mg/kg) continuously for 5 h and again the following day. Onset of behavioral intoxication (i.e., behavioral suppression) occurred more rapidly and lasted longer as the NaCN dose increased. This oral-consumption method with concomitant operantbehavioral assessment allowed for accurate dosing and quantification of intoxication onset, severity, and recovery, and will also be valuable in characterizing similar outcomes following varying medical countermeasure drugs and doses.

Suppressive and enhancing effects of nicotine on food-seeking behavior.

The present study examined how systemic low doses of nicotine affect the microstructure of reinforced food-seeking behavior in rats. Rats were first given an acute saline or nicotine treatment (0.1-0.6mg/kg, with an inter-injection interval of at least 48h), and then a chronic saline or nicotine treatment (0.3mg/kg/day for 10 consecutive days). Immediately after each injection, rats were required to press a lever five times to obtain food that was available at unpredictable times (on average every 80s) with constant probability. Acute nicotine dose-dependently suppressed behavior prior to the delivery of the first reinforcer, but enhanced food-reinforced behavior afterwards. These effects were primarily observed in the time it took rats to initiate food-seeking behavior. Enhancing effects were also observed in the microstructure of food-seeking behavior, with lower nicotine doses (0.1-0.3mg/kg) increasing the rate at which response bouts were initiated, and higher doses (0.3-0.6mg/kg) increasing within-bout response rates. A pre-feeding control suggests that changes in appetite alone cannot explain these effects. Over the course of chronic nicotine exposure, tolerance developed to the suppressive, but not to the enhancing effects of nicotine on food-seeking behavior. These results suggest that (a) lower doses of nicotine enhance the reward value of food and/or food-associated stimuli, (b) higher doses of nicotine enhance motoric activity, and (c) ostensive sensitization effects of nicotine on behavior partially reflect a tolerance to its transient suppressive motoric effects.

Behavioral intoxication following voluntary oral ingestion of tetramethylenedisulfotetramine: Dose-dependent onset, severity, survival, and recovery.

Tetramethylenedisulfotetramine (tetramine, or TETS) is a highly toxic rodenticide that has been responsible for over 14,000 accidental and intentional poisonings worldwide. Although the vast majority of TETS poisonings involved tainted food or drink, the laboratory in vivo studies of TETS intoxication used intraperitoneal injection or gavage for TETS exposure. Seeking to develop and characterize a more realistic model of TETS intoxication in the present study, rats were trained to rapidly and voluntarily consume a poisoned food morsel. Initially, the overt toxic effects of TETS consumption across a large range of doses were characterized, then a focused range of doses was selected for more intensive behavioral evaluation (in operant test chambers providing a variable-interval schedule of food reinforcement). The onset of intoxication following voluntary oral consumption of TETS was rapid, and clear dose-dependent response-rate suppression was observed across multiple performance measures within the operant-chamber environment. At most doses, recovery of operant performance did not occur within 30h. Food consumption and body weight changes were also dose dependent and corroborated the behavioral measures of intoxication. This voluntary oral-poisoning method with concomitant operant-behavioral assessment shows promise for future studies of TETS (and other toxic chemicals of interest) and may be extremely valuable in characterizing treatment outcomes.

The effect of lever height on the microstructure of operant behavior.

The effect of lever height on the temporal organization of reinforced lever pressing was examined. Lever pressing was reinforced on a variable-interval 30-s schedule in rats, with lever height manipulated across six successive conditions. Parameters of the organization of responses in bouts (bout length distribution, bout-initiation rate, within-bout rate, and sequential dependency) were estimated. These estimates revealed (1) a qualitative change in the distribution of IRTs and their sequential dependency when the lever was too high, (2) a mixture of geometrically-distributed bout lengths at all lever heights, and (3) longer bouts at lower and intermediate lever heights. In accordance with previous data, these findings suggest that lower and intermediate lever heights favored lever pressing with longer bout lengths, faster bout initiation, faster within-bout responding, and more sequentially dependent timing. These results underscore the disociability of motoric capacity in operant performance, and may reflect the influence of the body size on the temporal organization of the operant.

Extinction of Pavlovian conditioning: The influence of trial number and reinforcement history.

Pavlovian conditioning is sensitive to the temporal relationship between the conditioned stimulus (CS) and the unconditioned stimulus (US). This has motivated models that describe learning as a process that continuously updates associative strength during the trial or specifically encodes the CS-US interval. These models predict that extinction of responding is also continuous, such that response loss is proportional to the cumulative duration of exposure to the CS without the US. We review evidence showing that this prediction is incorrect, and that extinction is trial-based rather than time-based. We also present two experiments that test the importance of trials versus time on the Partial Reinforcement Extinction Effect (PREE), in which responding extinguishes more slowly for a CS that was inconsistently reinforced with the US than for a consistently reinforced one. We show that increasing the number of extinction trials of the partially reinforced CS, relative to the consistently reinforced CS, overcomes the PREE. However, increasing the duration of extinction trials by the same amount does not overcome the PREE. We conclude that animals learn about the likelihood of the US per trial during conditioning, and learn trial-by-trial about the absence of the US during extinction. Moreover, what they learn about the likelihood of the US during conditioning affects how sensitive they are to the absence of the US during extinction.

A bout analysis of operant response disruption.

Operant behavior appears to be organized in bouts of responses, whose parameters are differentially sensitive to various manipulations. This study investigated potential differential effects of three forms of operant response disruption-extinction (EXT), non-contingent reinforcement (NCR), and prefeeding (PRE)-on response bouts. In Experiment 1, Wistar Kyoto rats (WKY) were trained on a tandem variable-time (VT) 120s fixed-ratio (FR) 5 schedule of reinforcement; after stability was established, their responding was disrupted for three sessions with one of the three disrupters (EXT, NCR, or PRE). In Experiment 2, Long Evans (LE) rats were trained on a tandem VT 240s FR 5 to stability, and their responding disrupted with EXT or NCR. In EXT and NCR, response rates declined significantly and progressively over the course of the session, primarily due to a declining bout-initiation rate in EXT, and to fewer responses per bout in NCR. In contrast, a session-wide drop in response rate was observed in PRE, primarily due to a reduction in bout-initiation rate at the start of the session. These findings suggest that each form of disruption differentially impacts dissociable aspects of behavior. Theories of behavioral persistence should account for these functional relations, which appear to be obscured in response rate measures.

Pre-asymptotic response rates as a function of the delay-of-reinforcement gradient summation for Catania's Operant Reserve: A reply to Berg & McDowell (2011).

Catania's Operant Reserve (COR; Catania, 2005) is a computational model of operant behaviour. In COR, responding depletes the reserve while reinforcement replenishes the reserve. The replenishment to the reserve depends on the location of responses within the most recent inter-reinforcement interval. The rule that maps replenishment to the responses within an inter-reinforcement interval is given by a delay-of-reinforcement gradient (DOR). Previous research (Berg and McDowell, 2011) found that non-linear DORs produce sigmoidal response rates on single variable-interval schedules while a linear DOR produces hyperbolic response rates. Berg and McDowell took these sigmoidal response rates as evidence against the viability of COR for modelling variable-interval performance. However, Berg and McDowell did not consider the effect of the slope of each DOR. The present conjecture is that the response rates from COR can be made hyperbolic by manipulating the area under the DOR. Our results show that the manipulation of the area under the DOR allows COR to produce hyperbolic response rates regardless of the parametric form of the DOR. Hence, if COR is to be used to model single variable-interval performance, the reinforcement effect over time should be taken into consideration through the manipulation of the area under the DOR.

Response-bout analysis of interresponse times in variable-ratio and variable-interval schedules.

The purpose of this study was to analyze the response pattern difference between variable-ratio (VR) and variable-interval (VI) schedules of reinforcement by modeling interresponse time distributions of rats' lever presses. All eight rats showed higher response rates under VR 30 than under inter-reinforcement intervals yoked VI. The 30 models consisting of single Exponential (with and without the lower limit on interresponse times), Weibull, Normal, Log-Normal or Gamma distributions, all possible two component combinations of those, and 3 and 4 component models consisting of Weibull, Normal, Log-Normal, or Gamma distribution combinations were compared. The 4 component Log-Normal model was the best in terms of the Akaike information criterion and visual inspection of fitting outcome. Parameter estimates for the L4 model showed that the VR-VI response rate difference is due to a difference in short interresponse times or within bout responses. This results suggests that the VR-VI response rate difference is not an indication of a difference in the overall tendency to respond but it is rather a difference in terms of what types of response patterns are engendered between the two schedules.

An alternative to the stay/switch equation assessed when using a changeover-delay.

An alternative to the generalized matching equation for understanding concurrent performances is the stay/switch model. For the stay/switch model, the important events are the contingencies and behaviors at each alternative. The current experiment compares the descriptions by two stay/switch equations, the original, empirically derived stay/switch equation and a more theoretically derived equation based on ratios of stay to switch responses matching ratios of stay to switch reinforcers. The present experiment compared descriptions by the original stay/switch equation when using and not using a changeover delay. It also compared descriptions by the more theoretical equation with and without a changeover delay. Finally, it compared descriptions of the concurrent performances by these two equations. Rats were trained in 15 conditions on identical concurrent random-interval schedules in each component of a multiple schedule. A COD operated in only one component. There were no consistent differences in the variance accounted for by each equation of concurrent performances whether or not a COD was used. The simpler equation found greater sensitivity to stay than to switch reinforcers. It also found a COD eliminated the influence of switch reinforcers. Because estimates of parameters were more meaningful when using the more theoretical stay/switch equation it is preferred.

The copyist model and the shaping view of reinforcement.

The strengthening view of reinforcement attributes behavior change to changes in the response strength or the value of the reinforcer. In contrast, the shaping view explains behavior change as shaping different response units through differential reinforcement. In this paper, we evaluate how well these two views explain: (1) the response-rate difference between variable-ratio and variable-interval schedules that provide the same reinforcement rate; and (2) the phenomenon of matching in choice. The copyist model (Tanno and Silberberg, 2012) - a shaping-view account - can provided accurate predictions of these phenomena without a strengthening mechanism; however, the model has limitations. It cannot explain the relation between behavior change and stimulus control, reinforcer amount, and reinforcer quality. These relations seem easily explained by a strengthening view. Future work should be directed at a model which combine the strengths of these two types of accounts.