Robert A. Rescorla (1940-2020).

Memorializes Robert Arthur Rescorla (1940-2020), emeritus professor of psychology at the University of Pennsylvania. Rescorla was the world's most distinguished scholar in animal learning and a great teacher. Rescorla thought of himself as primarily an experimen talist, and his experiments on Pavlovian conditioning and instrumental learning would win any prize for the aesthetics of experimental design. Most of his nearly 200 empirical articles included multiple experiments with replications. Among Re scorla's most beautiful experiments were the studies of extinc tion performed in the last decade of his career. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

The physical basis of memory.

Neuroscientists are searching for the engram within the conceptual framework established by John Locke's theory of mind. This framework was elaborated before the development of information theory, before the development of information processing machines and the science of computation, before the discovery that molecules carry hereditary information, before the discovery of the codon code and the molecular machinery for editing the messages written in this code and translating it into transcription factors that mark abstract features of organic structure such as anterior and distal. The search for the engram needs to abandon Locke's conceptual framework and work within a framework informed by these developments. The engram is the medium by which information extracted from past experience is transmitted to the computations that inform future behavior. The information-conveying symbols in the engram are rapidly generated in the course of computations, which implies that they are molecules.

Number and time in acquisition, extinction and recovery.

We measured rate of acquisition, trials to extinction, cumulative responses in extinction, and the spontaneous recovery of anticipatory hopper poking in a Pavlovian protocol with mouse subjects. We varied by factors of 4 number of sessions, trials per session, intersession interval, and span of training (number of days over which training extended). We find that different variables affect each measure: Rate of acquisition [1/(trials to acquisition)] is faster when there are fewer trials per session. Terminal rate of responding is faster when there are more total training trials. Trials to extinction and amount of responding during extinction are unaffected by these variables. The number of training trials has no effect on recovery in a 4-trial probe session 21 days after extinction. However, recovery is greater when the span of training is greater, regardless of how many sessions there are within that span. Our results and those of others suggest that the numbers and durations and spacings of longer-duration "episodes" in a conditioning protocol (sessions and the spans in days of training and extinction) are important variables and that different variables affect different aspects of subjects' behavior. We discuss the theoretical and clinical implications of these and related findings and conclusions-for theories of conditioning and for neuroscience.

Contingency, contiguity, and causality in conditioning: Applying information theory and Weber's Law to the assignment of credit problem.

Contingency is a critical concept for theories of associative learning and the assignment of credit problem in reinforcement learning. Measuring and manipulating it has, however, been problematic. The information-theoretic definition of contingency-normalized mutual information-makes it a readily computed property of the relation between reinforcing events, the stimuli that predict them and the responses that produce them. When necessary, the dynamic range of the required temporal representation divided by the Weber fraction gives a psychologically realistic plug-in estimates of the entropies. There is no measurable prospective contingency between a peck and reinforcement when pigeons peck on a variable interval schedule of reinforcement. There is, however, a perfect retrospective contingency between reinforcement and the immediately preceding peck. Degrading the retrospective contingency by gratis reinforcement reveals a critical value (.25), below which performance declines rapidly. Contingency is time scale invariant, whereas the perception of proximate causality depends-we assume-on there being a short, fixed psychologically negligible critical interval between cause and effect. Increasing the interval between a response and reinforcement that it triggers degrades the retrograde contingency, leading to a decline in performance that restores it to at or above its critical value. Thus, there is no critical interval in the retrospective effect of reinforcement. We conclude with a short review of the broad explanatory scope of information-theoretic contingencies when regarded as causal variables in conditioning. We suggest that the computation of contingencies may supplant the computation of the sum of all future rewards in models of reinforcement learning. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Theoretical implications of quantitative properties of interval timing and probability estimation in mouse and rat.

In three experiments with mice ( Mus musculus ) and rats (Rattus norvigicus), we used a switch paradigm to measure quantitative properties of the interval-timing mechanism. We found that: 1) Rodents adjusted the precision of their timed switches in response to changes in the interval between the short and long feed latencies (the temporal goalposts). 2) The variability in the timing of the switch response was reduced or unchanged in the face of large trial-to-trial random variability in the short and long feed latencies. 3) The adjustment in the distribution of switch latencies in response to changes in the relative frequency of short and long trials was sensitive to the asymmetry in the Kullback-Leibler divergence. The three results suggest that durations are represented with adjustable precision, that they are timed by multiple timers, and that there is a trial-by-trial (episodic) record of feed latencies in memory.

The Coding Question.

Recent electrophysiological results imply that the duration of the stimulus onset asynchrony in eyeblink conditioning is encoded by a mechanism intrinsic to the cerebellar Purkinje cell. This raises the general question - how is quantitative information (durations, distances, rates, probabilities, amounts, etc.) transmitted by spike trains and encoded into engrams? The usual assumption is that information is transmitted by firing rates. However, rate codes are energetically inefficient and computationally awkward. A combinatorial code is more plausible. If the engram consists of altered synaptic conductances (the usual assumption), then we must ask how numbers may be written to synapses. It is much easier to formulate a coding hypothesis if the engram is realized by a cell-intrinsic molecular mechanism.

Numbers and brains.

The representation of discrete and continuous quantities appears to be ancient and pervasive in animal brains. Because numbers are the natural carriers of these representations, we may discover that in brains, it's numbers all the way down.

Navigation: Whence Our Sense of Direction?

Behavioral data have long implied our sense of direction derives from global environmental shape; electrophysiological evidence, however, has seemed to imply it derives from salient non-geometric landmarks. Experiments on the re-establishment of place fields in disoriented mice now align the electrophysiological data with the behavioral data.

Finding numbers in the brain.

After listing functional constraints on what numbers in the brain must do, I sketch the two's complement fixed-point representation of numbers because it has stood the test of time and because it illustrates the non-obvious ways in which an effective coding scheme may operate. I briefly consider its neurobiological implementation. It is easier to imagine its implementation at the cell-intrinsic molecular level, with thermodynamically stable, volumetrically minimal polynucleotides encoding the remembered numbers, than at the circuit level, with plastic synapses encoding them.This article is part of a discussion meeting issue 'The origins of numerical abilities'.

Bayesian change-point analysis reveals developmental change in a classic theory of mind task.

Although learning and development reflect changes situated in an individual brain, most discussions of behavioral change are based on the evidence of group averages. Our reliance on group-averaged data creates a dilemma. On the one hand, we need to use traditional inferential statistics. On the other hand, group averages are highly ambiguous when we need to understand change in the individual; the average pattern of change may characterize all, some, or none of the individuals in the group. Here we present a new method for statistically characterizing developmental change in each individual child we study. Using false-belief tasks, fifty-two children in two cohorts were repeatedly tested for varying lengths of time between 3 and 5 years of age. Using a novel Bayesian change point analysis, we determined both the presence and-just as importantly-the absence of change in individual longitudinal cumulative records. Whenever the analysis supports a change conclusion, it identifies in that child's record the most likely point at which change occurred. Results show striking variability in patterns of change and stability across individual children. We then group the individuals by their various patterns of change or no change. The resulting patterns provide scarce support for sudden changes in competence and shed new light on the concepts of "passing" and "failing" in developmental studies.

Automated, quantitative cognitive/behavioral screening of mice: for genetics, pharmacology, animal cognition and undergraduate instruction.

We describe a high-throughput, high-volume, fully automated, live-in 24/7 behavioral testing system for assessing the effects of genetic and pharmacological manipulations on basic mechanisms of cognition and learning in mice. A standard polypropylene mouse housing tub is connected through an acrylic tube to a standard commercial mouse test box. The test box has 3 hoppers, 2 of which are connected to pellet feeders. All are internally illuminable with an LED and monitored for head entries by infrared (IR) beams. Mice live in the environment, which eliminates handling during screening. They obtain their food during two or more daily feeding periods by performing in operant (instrumental) and Pavlovian (classical) protocols, for which we have written protocol-control software and quasi-real-time data analysis and graphing software. The data analysis and graphing routines are written in a MATLAB-based language created to simplify greatly the analysis of large time-stamped behavioral and physiological event records and to preserve a full data trail from raw data through all intermediate analyses to the published graphs and statistics within a single data structure. The data-analysis code harvests the data several times a day and subjects it to statistical and graphical analyses, which are automatically stored in the "cloud" and on in-lab computers. Thus, the progress of individual mice is visualized and quantified daily. The data-analysis code talks to the protocol-control code, permitting the automated advance from protocol to protocol of individual subjects. The behavioral protocols implemented are matching, autoshaping, timed hopper-switching, risk assessment in timed hopper-switching, impulsivity measurement, and the circadian anticipation of food availability. Open-source protocol-control and data-analysis code makes the addition of new protocols simple. Eight test environments fit in a 48 in x 24 in x 78 in cabinet; two such cabinets (16 environments) may be controlled by one computer.

The perception of probability.

We present a computational model to explain the results from experiments in which subjects estimate the hidden probability parameter of a stepwise nonstationary Bernoulli process outcome by outcome. The model captures the following results qualitatively and quantitatively, with only 2 free parameters: (a) Subjects do not update their estimate after each outcome; they step from one estimate to another at irregular intervals. (b) The joint distribution of step widths and heights cannot be explained on the assumption that a threshold amount of change must be exceeded in order for them to indicate a change in their perception. (c) The mapping of observed probability to the median perceived probability is the identity function over the full range of probabilities. (d) Precision (how close estimates are to the best possible estimate) is good and constant over the full range. (e) Subjects quickly detect substantial changes in the hidden probability parameter. (f) The perceived probability sometimes changes dramatically from one observation to the next. (g) Subjects sometimes have second thoughts about a previous change perception, after observing further outcomes. (h) The frequency with which they perceive changes moves in the direction of the true frequency over sessions. (Explaining this finding requires 2 additional parametric assumptions.) The model treats the perception of the current probability as a by-product of the construction of a compact encoding of the experienced sequence in terms of its change points. It illustrates the why and the how of intermittent Bayesian belief updating and retrospective revision in simple perception. It suggests a reinterpretation of findings in the recent literature on the neurobiology of decision making.

Cognitive assessment of mice strains heterozygous for cell-adhesion genes reveals strain-specific alterations in timing.

We used a fully automated system for the behavioural measurement of physiologically meaningful properties of basic mechanisms of cognition to test two strains of heterozygous mutant mice, Bfc (batface) and L1, and their wild-type littermate controls. Both of the target genes are involved in the establishment and maintenance of synapses. We find that the Bfc heterozygotes show reduced precision in their representation of interval duration, whereas the L1 heterozygotes show increased precision. These effects are functionally specific, because many other measures made on the same mice are unaffected, namely: the accuracy of matching temporal investment ratios to income ratios in a matching protocol, the rate of instrumental and classical conditioning, the latency to initiate a cued instrumental response, the trials on task and the impulsivity in a switch paradigm, the accuracy with which mice adjust timed switches to changes in the temporal constraints, the days to acquisition, and mean onset time and onset variability in the circadian anticipation of food availability.

Temporal contingency.

Contingency, and more particularly temporal contingency, has often figured in thinking about the nature of learning. However, it has never been formally defined in such a way as to make it a measure that can be applied to most animal learning protocols. We use elementary information theory to define contingency in such a way as to make it a measurable property of almost any conditioning protocol. We discuss how making it a measurable construct enables the exploration of the role of different contingencies in the acquisition and performance of classically and operantly conditioned behavior.

It's the information!

Learning in conditioning protocols has long been thought to depend on temporal contiguity between the conditioned stimulus and the unconditioned stimulus. This conceptualization has led to a preponderance of associative models of conditioning. We suggest that trial-based associative models that posit contiguity as the primary principle underlying learning are flawed, and provide a brief review of an alternative, information theoretic approach to conditioning. The information that a CS conveys about the timing of the next US can be derived from the temporal parameters of a conditioning protocol. According to this view, a CS will support conditioned responding if, and only if, it reduces uncertainty about the timing of the next US.

The neuroscience of learning: beyond the Hebbian synapse.

From the traditional perspective of associative learning theory, the hypothesis linking modifications of synaptic transmission to learning and memory is plausible. It is less so from an information-processing perspective, in which learning is mediated by computations that make implicit commitments to physical and mathematical principles governing the domains where domain-specific cognitive mechanisms operate. We compare the properties of associative learning and memory to the properties of long-term potentiation, concluding that the properties of the latter do not explain the fundamental properties of the former. We briefly review the neuroscience of reinforcement learning, emphasizing the representational implications of the neuroscientific findings. We then review more extensively findings that confirm the existence of complex computations in three information-processing domains: probabilistic inference, the representation of uncertainty, and the representation of space. We argue for a change in the conceptual framework within which neuroscientists approach the study of learning mechanisms in the brain.

Mice take calculated risks.

Animals successfully navigate the world despite having only incomplete information about behaviorally important contingencies. It is an open question to what degree this behavior is driven by estimates of stochastic parameters (brain-constructed models of the experienced world) and to what degree it is directed by reinforcement-driven processes that optimize behavior in the limit without estimating stochastic parameters (model-free adaptation processes, such as associative learning). We find that mice adjust their behavior in response to a change in probability more quickly and abruptly than can be explained by differential reinforcement. Our results imply that mice represent probabilities and perform calculations over them to optimize their behavior, even when the optimization produces negligible material gain.

Conditioned [corrected] stimulus informativeness governs conditioned stimulus-unconditioned stimulus associability.

In a conditioning protocol, the onset of the conditioned stimulus ([CS]) provides information about when to expect reinforcement (unconditioned stimulus [US]). There are two sources of information from the CS in a delay conditioning paradigm in which the CS-US interval is fixed. The first depends on the informativeness, the degree to which CS onset reduces the average expected time to onset of the next US. The second depends only on how precisely a subject can represent a fixed-duration interval (the temporal Weber fraction). In three experiments with mice, we tested the differential impact of these two sources of information on rate of acquisition of conditioned responding (CS-US associability). In Experiment 1, we showed that associability (the inverse of trials to acquisition) increased in proportion to informativeness. In Experiment 2, we showed that fixing the duration of the US-US interval or the CS-US interval or both had no effect on associability. In Experiment 3, we equated the increase in information produced by varying the C/T ratio with the increase produced by fixing the duration of the CS-US interval. Associability increased with increased informativeness, but, as in Experiment 2, fixing the CS-US duration had no effect on associability. These results are consistent with the view that CS-US associability depends on the increased rate of reward signaled by CS onset. The results also provide further evidence that conditioned responding is temporally controlled when it emerges.