The active time model of concurrent choice.

The following paper describes a steady-state model of concurrent choice, termed the active time model (ATM). ATM is derived from maximization principles and is characterized by a semi-Markov process. The model proposes that the controlling stimulus in concurrent variable-interval (VI) VI schedules of reinforcement is the time interval since the most recent response, termed here "the active interresponse time" or simply "active time." In the model after a response is generated, it is categorized by a function that relates active times to switch/stay probabilities. In the paper the output of ATM is compared with predictions made by three other models of operant conditioning: melioration, a version of scalar expectancy theory (SET), and momentary maximization. Data sets considered include preferences in multiple-concurrent VI VI schedules, molecular choice patterns, correlations between switching and perseveration, and molar choice proportions. It is shown that ATM can account for all of these data sets, while the other models produce more limited fits. However, rather than argue that ATM is the singular model for concurrent VI VI choice, a consideration of its concept space leads to the conclusion that operant choice is multiply-determined, and that an adaptive viewpoint-one that considers experimental procedures both as selecting mechanisms for animal choice as well as tests of the controlling variables of that choice-is warranted.

When Do We Not Face Our Fears? Investigating the Boundary Conditions of Costly Pain-Related Avoidance Generalization.

Excessive generalization of fear and avoidance are hallmark symptoms of chronic pain disability, yet research focusing on the mechanisms underlying generalization of avoidance specifically, is scarce. Two experiments investigated the boundary conditions of costly pain-related avoidance generalization in healthy participants who learned to avoid pain by performing increasingly effortful (in terms of deviation and force) arm-movements using a robot-arm (acquisition). During generalization, novel, but similar arm-movements, without pain, were tested. Experiment 1 (N = 64) aimed to facilitate generalization to these movements by reducing visual contextual changes between acquisition and generalization, whereas Experiment 2 (N = 70) aimed to prevent extinction by increasing pain uncertainty. Both experiments showed generalization of pain-expectancies and pain-related fear. However, Experiment 2 was the first and only to also demonstrate generalization of avoidance, ie, choosing the novel effortful arm-movements in the absence of pain. These results suggest that uncertainty about the occurrence of pain may delay recovery, due to reduced disconfirmation of threat beliefs when exploring, resulting in persistent avoidance. PERSPECTIVE: This article demonstrates generalization of instrumentally acquired costly pain-related avoidance in healthy people under conditions of uncertainty. The results suggest that targeting pain-related uncertainty may be a useful tool for clinicians adopting a psychological approach to treating excessive pain-related avoidance in chronic pain.

Economic demand analysis of within-session dose-reduction during nicotine self-administration.

BACKGROUND: This study determined if a within-session dose-reduction design sufficiently captures elasticity of demand for nicotine in male and female rats using environmental enrichment to manipulate demand elasticity. METHODS: Male and female Sprague-Dawley rats were trained to self-administer nicotine (60 µg/kg/infusion). In Experiment 1, rats began daily dose-reduction for nine sessions following acquisition. Rats then underwent a minimum of five within-session dose-reduction sessions where each dose was available for 10 min. In Experiment 2, rats were reared in isolated, social, or enriched housing followed by acquisition of nicotine self-administration. Rats then underwent within-session dose-reduction. Housing environments were then switched, followed by additional testing sessions. Consumption was calculated for each dose and exponential demand curves were fit. RESULTS: No sex differences in acquisition of nicotine self-administration were detected for either experiment. In experiment 1, demand intensity (Q0; estimated intake if nicotine were freely available), was higher with between- compared to within-session dose-reduction, although elasticity of demand (α; rate of decline in nicotine intake as a function of increasing unit price), was lower. In Experiment 2, animals reared in enrichment had fewer infusions during acquisition compared to animals in isolation. Enriched males had reduced demand intensity compared to both isolated and social males, whereas isolated females had reduced intensity compared to enriched females. CONCLUSIONS: The within-session dose-reduction procedure for nicotine self-administration replicated effects of environmental enrichment on consumption behaviors. Additionally, this procedure captured differences in nicotine demand due to sex, laying important groundwork for future translational research on mechanisms of nicotine dependence.

Subcortical Substrates of Explore-Exploit Decisions in Primates.

The explore-exploit dilemma refers to the challenge of deciding when to forego immediate rewards and explore new opportunities that could lead to greater rewards in the future. While motivational neural circuits facilitate learning based on past choices and outcomes, it is unclear whether they also support computations relevant for deciding when to explore. We recorded neural activity in the amygdala and ventral striatum of rhesus macaques as they solved a task that required them to balance novelty-driven exploration with exploitation of what they had already learned. Using a partially observable Markov decision process (POMDP) model to quantify explore-exploit trade-offs, we identified that the ventral striatum and amygdala differ in how they represent the immediate value of exploitative choices and the future value of exploratory choices. These findings show that subcortical motivational circuits are important in guiding explore-exploit decisions.

An automated home-cage-based 5-choice serial reaction time task for rapid assessment of attention and impulsivity in rats.

RATIONALE: The 5-choice serial reaction time task (5-CSRTT) is a widely used operant task for measuring attention and motor impulsivity in rodents. Training animals in this task requires an extensive period of daily operant sessions. Recently, a self-paced, automated version of this task has been developed for mice, which substantially reduces training time. Whether a similar approach is effective for rats is currently unknown. OBJECTIVE: Here, we tested whether attention and impulsivity can be assessed in rats with a self-paced version of the 5-CSRTT. METHODS: Operant boxes were connected to home-cages with tunnels. Two groups of rats self-paced their training by means of an automated script. The first group of animals was allowed unlimited access (UA) to start trials in the task; for the second group, trial availability was restricted to the first 2.5 h of the dark cycle (TR). Task parameter manipulations, such as variable inter-trial intervals and stimulus durations as well as pharmacological challenges with scopolamine, were tested to validate the task. RESULTS: Self-paced training took less than 1 week. Animals in the UA group showed higher levels of omissions compared with the TR group. In both protocols, variable inter-trial intervals increased impulsivity, and variable stimulus durations decreased attentional performance. Scopolamine affected cognitive performance in the TR group only. CONCLUSIONS: Home-cage-based training of the 5-CSRTT in rats, especially the TR protocol, presents a valid and fast alternative for measuring attention and impulsivity.

Cues play a critical role in estrous cycle-dependent enhancement of cocaine reinforcement.

While preclinical work has aimed to outline the neural mechanisms of drug addiction, it has overwhelmingly focused on male subjects. There has been a push in recent years to incorporate females into existing addiction models; however, males and females often have different behavioral strategies, making it important to not only include females, but to develop models that assess the factors that comprise female drug addiction. Traditional self-administration models often include light or tone cues that serve as discriminative stimuli and/or consequent stimuli, making it nearly impossible to disentangle the effects of cue learning, the cues themselves, and acute effects of psychostimulant drugs. To disentangle the interaction between drug-associated cues and the consummatory and appetitive responding driven by cocaine, we have developed a new behavioral procedure that combines Pavlovian-instrumental transfer with behavioral economic analysis. This task can be completed within a single session, allowing for studies looking at estrous cycle stage-dependent effects in intact cycling females, something that has been difficult in the past. In this study, we found no differences in self-administration across the estrous cycle in the absence of cues; however, when cues were introduced, the cues that acquired value during estrus-but not during diestrus or in males-increased motivation. Cues paired during estrus also increased c-fos expression to a greater extent in striatal regions, an effect that may underlie the observed increases in seeking induced by these cues, even weeks later. Together, these data suggest that fundamental differences in the motivational properties of psychostimulant drugs between males and females are complex and are driven primarily by the interaction between drug-associated stimuli and drug effects.

Striatal dopamine D2 receptors regulate effort but not value-based decision making and alter the dopaminergic encoding of cost.

Deficits in goal-directed motivation represent a debilitating symptom for many patients with schizophrenia. Impairments in motivation can arise from deficits in processing information about effort and or value, disrupting effective cost-benefit decision making. We have previously shown that upregulated dopamine D2 receptor expression within the striatum (D2R-OE mice) decreases goal-directed motivation. Here, we determine the behavioral and neurochemical mechanisms behind this deficit. Female D2R-OE mice were tested in several behavioral paradigms including recently developed tasks that independently assess the impact of Value or Effort manipulations on cost-benefit decision making. In vivo microdialysis was used to measure extracellular dopamine in the striatum during behavior. In a value-based choice task, D2R-OE mice show normal sensitivity to changes in reward value and used reward value to guide their actions. In an effort-based choice task, D2R-OE mice evaluate the cost of increasing the number of responses greater relative to the effort cost of longer duration responses compared to controls. This shift away from choosing to repeatedly execute a response is accompanied by a dampening of extracellular dopamine in the striatum during goal-directed behavior. In the ventral striatum, extracellular dopamine level negatively correlates with response cost in controls, but this relationship is lost in D2R-OE mice. These results show that D2R signaling in the striatum, as observed in some patients with schizophrenia, alters the relationship between effort expenditure and extracellular dopamine. This dysregulation produces motivation deficits that are specific to effort but not value-based decision making, paralleling the effort-based motivational deficits observed in schizophrenia.

A Robotic System for Adaptive Training and Function Assessment of Forelimb Retraction in Mice.

Rodent models are decisive for translational research in healthy and pathological conditions of motor function thanks to specific similarities with humans. Here, we present an upgraded version of the M-Platform, a robotic device previously designed to train mice during forelimb retraction tasks. This new version significantly extends its possibilities for murine experiments during motor tasks: 1) an actuation system for friction adjustment allows to automatically adapt pulling difficulty; 2) the device can be used both for training, with a retraction task, and for assessment, with an isometric task; and 3) the platform can be integrated with a neurophysiology systems to record simultaneous cortical neural activity. Results of the validation experiments with healthy mice confirmed that the M-Platform permits precise adjustments of friction during the task, thus allowing to change its difficulty and that these variations induce a different improvement in motor performance, after specific training sessions. Moreover, simultaneous and high quality (high signal-to-noise ratio) neural signals can be recorded from the rostral forelimb area (RFA) during task execution. With the novel features presented herein, the M-Platform may allow to investigate the outcome of a customized motor rehabilitation protocol after neural injury, to analyze task-related signals from brain regions interested by neuroplastic events and to perform optogenetic silencing or stimulation during experiments in transgenic mice.

Delta/mu opioid receptor interactions in operant conditioning assays of pain-depressed responding and drug-induced rate suppression: assessment of therapeutic index in male Sprague Dawley rats.

RATIONALE AND OBJECTIVES: Although delta/mu receptor interactions vary as a function of behavioral endpoint, there have been no assessments of these interactions using assays of pain-depressed responding. This is the first report of delta/mu interactions using an assay of pain-depressed behavior. METHODS: A mult-cycle FR10 operant schedule was utilized in the presence of (nociception) and in the absence of (rate suppression) a lactic acid inflammatory pain-like manipulation. SNC80 and methadone were used as selective/high efficacy delta and mu agonists, respectively. Both SNC80 and methadone alone produced a dose-dependent restoration of pain-depressed responding and dose-dependent response rate suppression. Three fixed ratio mixtures, based on the relative potencies of the drugs in the nociception assay, also produced dose-dependent antinociception and sedation. Isobolographic analysis indicated that all three mixtures produced supra-additive antinociceptive effects and simply additive sedation effects. CONCLUSIONS: The therapeutic index (TI) inversely varied as a function of amount of SNC80 in the mixture, such that lower amounts of SNC80 produced a higher TI, and larger amounts produced a lower TI. Compared to literature using standard pain-elicited assays, the orderly relationship between SNC80 and TI reported here may be a unique function of assessing pain-depressed behavior.

To work or not to work: Neural representation of cost and benefit of instrumental action.

By definition, instrumental actions are performed in order to obtain certain goals. Nevertheless, the attainment of goals typically implies obstacles, and response vigor is known to reflect an integration of subjective benefit and cost. Whereas several brain regions have been associated with cost/benefit ratio decision-making, trial-by-trial fluctuations in motivation are not well understood. We review recent evidence supporting the motivational implications of signal fluctuations in the mesocorticolimbic system. As an extension of "set-point" theories of instrumental action, we propose that response vigor is determined by a rapid integration of brain signals that reflect value and cost on a trial-by-trial basis giving rise to an online estimate of utility. Critically, we posit that fluctuations in key nodes of the network can predict deviations in response vigor and that variability in instrumental behavior can be accounted for by models devised from optimal control theory, which incorporate the effortful control of noise. Notwithstanding, the post hoc analysis of signaling dynamics has caveats that can effectively be addressed in future research with the help of two novel fMRI imaging techniques. First, adaptive fMRI paradigms can be used to establish a time-order relationship, which is a prerequisite for causality, by using observed signal fluctuations as triggers for stimulus presentation. Second, real-time fMRI neurofeedback can be employed to induce predefined brain states that may facilitate benefit or cost aspects of instrumental actions. Ultimately, understanding temporal dynamics in brain networks subserving response vigor holds the promise for targeted interventions that could help to readjust the motivational balance of behavior.

A hidden Markov model for decoding and the analysis of replay in spike trains.

We present a hidden Markov model that describes variation in an animal's position associated with varying levels of activity in action potential spike trains of individual place cell neurons. The model incorporates a coarse-graining of position, which we find to be a more parsimonious description of the system than other models. We use a sequential Monte Carlo algorithm for Bayesian inference of model parameters, including the state space dimension, and we explain how to estimate position from spike train observations (decoding). We obtain greater accuracy over other methods in the conditions of high temporal resolution and small neuronal sample size. We also present a novel, model-based approach to the study of replay: the expression of spike train activity related to behaviour during times of motionlessness or sleep, thought to be integral to the consolidation of long-term memories. We demonstrate how we can detect the time, information content and compression rate of replay events in simulated and real hippocampal data recorded from rats in two different environments, and verify the correlation between the times of detected replay events and of sharp wave/ripples in the local field potential.

A comparison of preference-assessment methods.

In Study 1, we evaluated preference stability across 4 preference-assessment methods for 6 individuals, 5 of whom had autism spectrum disorder and 1 of whom had traumatic brain injury. We also measured participants' problem behavior as a corollary measure during all assessment methods. The highest mean correlation coefficients and Kendall rank coefficients of concordance across administrations were observed for the paired-stimulus and multiple-stimulus-without-replacement methods. Lower correspondence across administrations was observed for the free-operant and response-restriction methods. Although differentially higher levels of problem behavior did not occur with a single method, lower levels were consistently observed with the free-operant method. During Study 2, we evaluated the implications of lower coefficients on reinforcer efficacy by comparing an initially identified and an immediately identified high-preference stimulus in a reinforcer assessment. Initially identified and immediately identified high-preference stimuli were equally effective reinforcers, suggesting that fluctuations in preference do not necessarily affect reinforcer efficacy in practice.

Mechanical Conflict System: A Novel Operant Method for the Assessment of Nociceptive Behavior.

A new operant test for preclinical pain research, termed the Mechanical Conflict System (MCS), is presented. Rats were given a choice either to remain in a brightly lit compartment or to escape to a dark compartment by crossing an array of height-adjustable nociceptive probes. Latency to escape the light compartment was evaluated with varying probe heights (0, .5, 1, 2, 3, and 4 mm above compartment floor) in rats with neuropathic pain induced by constriction nerve injury (CCI) and in naive control rats. Escape responses in CCI rats were assessed following intraperitoneal administration of pregabalin (10 and 30 mg/kg), morphine (2.5 and 5 mg/kg), and the tachykinin NK1 receptor antagonist, RP 67580 (1 and 10 mg/kg). Results indicate that escape latency increased as a function of probe height in both naive and CCI rats. Pregabalin (10 and 30 mg/kg) and morphine (5 mg/kg), but not RP 67580, decreased latency to escape in CCI rats suggesting an antinociceptive effect. In contrast, morphine (10 mg/kg) but not pregabalin (30 mg/kg) increased escape latency in naive rats suggesting a possible anxiolytic action of morphine in response to light-induced fear. No order effects following multiple test sessions were observed. We conclude that the MCS is a valid method to assess behavioral signs of affective pain in rodents.

Assessment of individual differences in the rat nucleus accumbens transcriptome following taste-heroin extended access.

Heroin addiction is a disease of chronic relapse that harms the individual through devaluation of personal responsibilities in favor of finding and using drugs. Only some recreational heroin users devolve into addiction but the basis of these individual differences is not known. We have shown in rats that avoidance of a heroin-paired taste cue reliably identifies individual animals with greater addiction-like behavior for heroin. Here rats received 5min access to a 0.15% saccharin solution followed by the opportunity to self-administer either saline or heroin for 6h. Large Suppressors of the heroin-paired taste cue displayed increased drug escalation, motivation for drug, and drug loading behavior compared with Small Suppressors. Little is known about the molecular mechanisms of these individual differences in addiction-like behavior. We examined the individual differences in mRNA expression in the nucleus accumbens (NAc) of rats that were behaviorally stratified by addiction-like behavior using next-generation sequencing. We hypothesized that based on the avoidance of the drug-paired cue there will be a unique mRNA profile in the NAc. Analysis of strand-specific whole genome RNA-Seq data revealed a number of genes differentially regulated in NAc based on the suppression of the natural saccharine reward. Large Suppressors exhibited a unique mRNA prolife compared to Saline controls and Small Suppressors. Genes related to immunity, neuronal activity, and behavior were differentially expressed among the 3 groups. In total, individual differences in avoidance of a heroin-paired taste cue are associated with addiction-like behavior along with differential NAc gene expression.

Role of estrogen receptor-α on food demand elasticity.

Estrogens have been shown to have an inhibitory effect on food intake under free-feeding conditions, yet the effects of estrogens on food-maintained operant responding have been studied to a much lesser extent and, thus, are not well understood. Therefore, the purpose of the present experiment was to use a behavioral economics paradigm to assess differences in demand elasticity between mice with knockout of the estrogen receptor subtype α, knockout of subtype ß, and their wild type controls. The mice responded in a closed economy, and the price of food was increased by increasing the fixed-ratio response requirement every four sessions. Overall, we found that mice with the knockout of receptor subtype α had the most elastic demand functions. Therefore, under these conditions, estrogens increased food seeking via activation of the receptor subtype α. The results were inconsistent with those reported by previous studies that employed free-feeding conditions.

Avoidance behavior: a free-operant lever-press avoidance task for the assessment of the effects of safety signals.

This protocol details a free-operant avoidance paradigm that has been developed to evaluate the relative contribution of different sources of reinforcement of avoidance behavior that may play an important role in the development and maintenance of human anxiety disorders. The task enables the assessment of the effects of safety cues that signal a period free from danger on lever-press avoidance behavior. Avoidance behavior trained using this protocol has been shown to be sensitive to both behavioral and pharmacological manipulations and has been optimized so that it takes approximately 1 month for rats to perform at high levels of stable avoidance responding.

Economics of food intake in mice: energy yield of the reinforcer.

One of the Zeitgeists of the field for the study of ingestive behavior is that organisms are endowed with internal self-regulatory mechanisms that ensure optimal nutrition. However, the alarming increase in the prevalence of obesity challenges us to reconsider the extent to which internal regulatory mechanisms affect food intake, especially in a free market economy. Cued by the pioneering work of George Collier and his students, we have been examining food intake (demand) in mice when the effort or price of food is manipulated. We present two new experiments in mice that investigate the effect of energy yield per unit of food earned on working for food. The first experiment shows that when the nominal energy yield of each food pellet is halved by cellulose dilution, mice show relatively inelastic calorie-related demand despite the fact the cellulose diluted diet is unpalatable. The second experiment shows that the size of the pellet reinforcer does not have a major effect on food demand except in the extreme condition of small reward and high unit price. New analyses of distributions of responding are presented which suggest that mice work for "target" numbers of food rewards with only a small influence of price or energy gain.

Dissociable contributions of anterior cingulate cortex and basolateral amygdala on a rodent cost/benefit decision-making task of cognitive effort.

Personal success often requires the choice to expend greater effort for larger rewards, and deficits in such effortful decision making accompany a number of illnesses including depression, schizophrenia, and attention-deficit/hyperactivity disorder. Animal models have implicated brain regions such as the basolateral amygdala (BLA) and anterior cingulate cortex (ACC) in physical effort-based choice, but disentangling the unique contributions of these two regions has proven difficult, and effort demands in industrialized society are predominantly cognitive in nature. Here we utilize the rodent cognitive effort task (rCET), a modification of the five-choice serial reaction-time task, wherein animals can choose to expend greater visuospatial attention to obtain larger sucrose rewards. Temporary inactivation (via baclofen-muscimol) of BLA and ACC showed dissociable effects: BLA inactivation caused hard-working rats to 'slack off' and 'slacker' rats to work harder, whereas ACC inactivation caused all animals to reduce willingness to expend mental effort. Furthermore, BLA inactivation increased the time needed to make choices, whereas ACC inactivation increased motor impulsivity. These data illuminate unique contributions of BLA and ACC to effort-based decision making, and imply overlapping yet distinct circuitry for cognitive vs physical effort. Our understanding of effortful decision making may therefore require expanding our models beyond purely physical costs.

Effects of price and pellet type on food waste in mice.

When laboratory mice are provided with free access to food, they often fragment their food such that it collects on the cage floor - wasted. An operant analysis of food waste, however, has not yet been conducted. The purpose of the present study was to evaluate the effect of response requirement and pellet type on food waste using a behavioral economic paradigm. Sixteen mice responded under a series of escalating fixed ratio schedules. Nose pokes were reinforced with either a grain-based pellet or a fiber-based pellet (diluted with non-digestible cellulose) across conditions. We found that mice spilled a greater percent of the total earned pellets at low response requirements. Additionally, mice spilled more fiber-based pellets relative to grain-based pellets. This difference was most pronounced when the fixed ratio requirement was low and was attenuated as the fixed ratio was increased, and this decrease in food waste across prices was well accounted for by an exponential model. Mice may have been extracting the calorically dense components of the fiber-based pellets only when the schedule of reinforcement was rich. When the schedule of reinforcement was lean, responding for a new pellet likely was a more functional behavior than fragmenting a pellet and discarding portions.

Assessment of behavioral flexibility after middle cerebral artery occlusion in mice.

Middle cerebral artery occlusion (MCAO) is the most common animal model of cerebral ischemia and induces various functional impairments. Long-lasting deficits resulting from MCAO however, remain insufficiently characterized, especially regarding cognition. Yet, behavioral flexibility, a prominent cognitive process is found impaired after stroke in humans. We thus used an operant-based task to assess behavioral flexibility in mice after MCAO. Three weeks after 30 min MCAO surgery, mice were subjected to a battery of sensorimotor tests (rotarod, vertical pole test, spontaneous locomotion and grip-strength test). Behavioral flexibility was then assessed in an operant task, in which mice, rewarded according to a FR5 schedule of reinforcement, had to alternate their operant responses between two levers from trial to trial. Regarding sensory and motor functioning, only the pole test yielded a significant difference between MCAO and sham mice. In the operant flexibility task, results showed a behavioral flexibility deficit in MCAO mice; neither the operant response acquisition nor the appeal for food rewards was altered. In conclusion, our operant-based task revealed a long-lasting behavioral flexibility deficit after MCAO in mice.