RESUMEN
Reinforcement learning inspires much theorizing in neuroscience, cognitive science, machine learning, and AI. A central question concerns the conditions that produce the perception of a contingency between an action and reinforcement-the assignment-of-credit problem. Contemporary models of associative and reinforcement learning do not leverage the temporal metrics (measured intervals). Our information-theoretic approach formalizes contingency by time-scale invariant temporal mutual information. It predicts that learning may proceed rapidly even with extremely long action-reinforcer delays. We show that rats can learn an action after a single reinforcement, even with a 16-min delay between the action and reinforcement (15-fold longer than any delay previously shown to support such learning). By leveraging metric temporal information, our solution obviates the need for windows of associability, exponentially decaying eligibility traces, microstimuli, or distributions over Bayesian belief states. Its three equations have no free parameters; they predict one-shot learning without iterative simulation.
Asunto(s)
Refuerzo en Psicología , Animales , Ratas , Aprendizaje/fisiología , Factores de Tiempo , Teorema de BayesRESUMEN
We develop a mathematical approach to formally proving that certain neural computations and representations exist based on patterns observed in an organism's behaviour. To illustrate, we provide a simple set of conditions under which an ant's ability to determine how far it is from its nest would logically imply neural structures isomorphic to the natural numbers â . We generalise these results to arbitrary behaviours and representations and show what mathematical characterisation of neural computation and representation is simplest while being maximally predictive of behaviour. We develop this framework in detail using a path integration example, where an organism's ability to search for its nest in the correct location implies representational structures isomorphic to two-dimensional coordinates under addition. We also study a system for processing a n b n strings common in comparative work. Our approach provides an objective way to determine what theory of a physical system is best, addressing a fundamental challenge in neuroscientific inference. These results motivate considering which neurobiological structures have the requisite formal structure and are otherwise physically plausible given relevant physical considerations such as generalisability, information density, thermodynamic stability and energetic cost.
Asunto(s)
Neurociencias , Animales , Neurociencias/métodos , Modelos Neurológicos , Conducta Animal/fisiología , HumanosRESUMEN
Honeybees (Apis mellifera carnica) communicate the direction and distance to a food source by means of a waggle dance. We ask whether bees recruited by the dance use it only as a flying instruction, with the technical form of a polar vector, or also translate it into a location vector that enables them to set courses directed toward the food source from arbitrary locations within their familiar territory. The flights of recruits captured on exiting the hive and released at distant sites were tracked by radar. The recruits performed first a straight flight in approximately the compass direction indicated by the dance. However, this "vector" portion of their flights and the ensuing tortuous "search" portion were strongly and differentially affected by the release site. Searches were biased toward the true location of the food and away from the location specified by translating the origin for the danced polar vector to the release site. We conclude that by following the dance recruits get two messages, a polar flying instruction (bearing and range from the hive) and a location vector that enables them to approach the source from anywhere in their familiar territory. The dance communication is much richer than thought so far.
Asunto(s)
Comunicación Animal , Deportes , Abejas , Animales , Alimentos , ComunicaciónRESUMEN
Temporal information-processing is critical for adaptive behavior and goal-directed action. It is thus crucial to understand how the temporal distance between behaviorally relevant events is encoded to guide behavior. However, research on temporal representations has yielded mixed findings as to whether organisms utilize relative versus absolute judgments of time intervals. To address this fundamental question about the timing mechanism, we tested mice in a duration discrimination procedure in which they learned to correctly categorize tones of different durations as short or long. After being trained on a pair of target intervals, the mice were transferred to conditions in which cue durations and corresponding response locations were systematically manipulated so that either the relative or absolute mapping remained constant. The findings indicate that transfer occurred most readily when relative relationships of durations and response locations were preserved. In contrast, when subjects had to re-map these relative relations, even when positive transfer initially occurred based on absolute mappings, their temporal discrimination performance was impaired, and they required extensive training to re-establish temporal control. These results demonstrate that mice can represent experienced durations both as having a certain magnitude (absolute representation) and as being shorter or longer of the two durations (an ordinal relation to other cue durations), with relational control having a more enduring influence in temporal discriminations. (PsycInfo Database Record (c) 2023 APA, all rights reserved).
Asunto(s)
Aprendizaje , Percepción del Tiempo , Ratones , Animales , Percepción del Tiempo/fisiología , Motivación , ColumbidaeRESUMEN
Optimal behavior requires interpreting environmental cues that indicate when to perform actions. Dopamine is important for learning about reward-predicting events, but its role in adapting to inhibitory cues is unclear. Here we show that when mice can earn rewards in the absence but not presence of an auditory cue, dopamine level in the ventral striatum accurately reflects reward availability in real-time over a sustained period (80 s). In addition, unpredictable transitions between different states of reward availability are accompanied by rapid (~1-2 s) dopamine transients that deflect negatively at the onset and positively at the offset of the cue. This Dopamine encoding of reward availability and transitions between reward availability states is not dependent on reward or activity evoked dopamine release, appears before mice learn the task and is sensitive to motivational state. Our findings are consistent across different techniques including electrochemical recordings and fiber photometry with genetically encoded optical sensors for calcium and dopamine.
Asunto(s)
Dopamina , Estriado Ventral , Animales , Señales (Psicología) , Dopamina/fisiología , Ratones , Núcleo Accumbens , RecompensaRESUMEN
Numbers are symbols manipulated in accord with the axioms of arithmetic. They sometimes represent discrete and continuous quantities (e.g., numerosities, durations, rates, distances, directions, and probabilities), but they are often simply names. Brains, including insect brains, represent the rational numbers with a fixed-point data type, consisting of a significand and an exponent, thereby conveying both magnitude and precision.
Asunto(s)
Encéfalo , Lenguaje , Humanos , MatemáticaRESUMEN
Karl Lashley began the search for the engram nearly seventy years ago. In the time since, much has been learned but divisions remain. In the contemporary neurobiology of learning and memory, two profoundly different conceptions contend: the associative/connectionist (A/C) conception and the computational/representational (C/R) conception. Both theories ground themselves in the belief that the mind is emergent from the properties and processes of a material brain. Where these theories differ is in their description of what the neurobiological substrate of memory is and where it resides in the brain. The A/C theory of memory emphasizes the need to distinguish memory cognition from the memory engram and postulates that memory cognition is an emergent property of patterned neural activity routed through engram circuits. In this model, learning re-organizes synapse association strengths to guide future neural activity. Importantly, the version of the A/C theory advocated for here contends that synaptic change is not symbolic and, despite normally being necessary, is not sufficient for memory cognition. Instead, synaptic change provides the capacity and a blueprint for reinstating symbolic patterns of neural activity. Unlike the A/C theory, which posits that memory emerges at the circuit level, the C/R conception suggests that memory manifests at the level of intracellular molecular structures. In C/R theory, these intracellular structures are information-conveying and have properties compatible with the view that brain computation utilizes a read/write memory, functionally similar to that in a computer. New research has energized both sides and highlighted the need for new discussion. Both theories, the key questions each theory has yet to resolve and several potential paths forward are presented here.
Asunto(s)
Encéfalo/fisiología , Memoria/fisiología , Modelos Neurológicos , Plasticidad Neuronal , Neuronas/fisiología , Animales , Humanos , Aprendizaje/fisiología , Vías Nerviosas/fisiologíaRESUMEN
Shannon's theory lays the foundation for understanding the flow of information from world into brain: There must be a set of possible messages. Brain structure determines what they are. Many messages convey quantitative facts (distances, directions, durations, etc.). It is impossible to consider how neural tissue processes these numbers without first considering how it encodes them.
Asunto(s)
Metáfora , Descanso , EncéfaloRESUMEN
Numerosity, or the ability to understand and distinguish between discrete quantities, was first formalized for study in animals by Mechner (1958a). Rats had to press one lever (the counting lever) n times to arm food release from pressing a second lever (the reward lever). The only cue that n presses had been made to the counting lever was the animal's representation of how many times it had pressed it. In the years that have passed since, many researchers have modified the task in meaningful ways to attempt to tease apart timing-based and count-based strategies. Strong evidence has amassed that the two are fundamentally different and separable skills but, to date, no study has effectively examined the differential contributions of the two strategies in Mechner's original task. By examining performance mid-trial and correlating it with whole-trial performance, we were able to identify patterns of correlation consistent with counting and timing strategies. Due to the independent nature of these correlation patterns, this technique was uniquely able to provide evidence for strategies that combined both timing and counting components. The results show that most mice demonstrated this combined strategy. This provides direct evidence that mice can and do use numerosity to complete Mechner's original task. A rational agent with fallible estimates of both counts made and time elapsed in making them should use both estimates when deciding when to switch to the second lever.
RESUMEN
Animals optimize their behavior to maximize rewards by utilizing cues from the environment. In discrimination learning, cues signal when rewards can and cannot be earned by making a particular response. In our experiment, we trained male mice to press a lever to receive a reward on a random interval schedule. We then introduced a prolonged tone (20, 40, or 80 sec), during which no rewards could be earned. We sought to test our hypothesis that the duration of the tone and frequency of reward during the inter-tone-intervals affect the informativeness of cues and led to differences in discriminative behavior. Learning was expressed as an increase in lever pressing during the intertrial interval (ITI) and, when the informativeness of the cue was high, animals also reduced their lever pressing during the tone. Additionally, we found that the depth of discriminative learning was linearly related to the informativeness of the cues. Our results show that the time-scale invariant information-theoretic definition of contingency applied to excitatory conditioning can also be applied to inhibitory conditioning. (PsycINFO Database Record (c) 2019 APA, all rights reserved).
Asunto(s)
Condicionamiento Operante/fisiología , Aprendizaje Discriminativo/fisiología , Estimulación Acústica , Animales , Conducta Animal/fisiología , Señales (Psicología) , Masculino , Ratones , Recompensa , Factores de TiempoRESUMEN
The search for memory is one of the oldest quests in written human history. For at least two millennia, we have tried to understand how we learn and remember. We have gradually converged on the brain and looked inside it to find the basis of knowledge, the trace of memory. The search for memory has been conducted on multiple levels, from the organ to the cell to the synapse, and has been distributed across disciplines with less chronological or intellectual overlap than one might hope. Frequently, the study of the mind and its memories has been severely restricted by technological or philosophical limitations. However, in the last few years, certain technologies have emerged, offering new routes of inquiry into the basis of memory. The 2016 Kavli Futures Symposium was devoted to the past and future of memory studies. At the workshop, participants evaluated the logic and data underlying the existing and emerging theories of memory. In this paper, written in the spirit of the workshop, we briefly review the history of the hunt for memory, summarizing some of the key debates at each level of spatial resolution. We then discuss the exciting new opportunities to unravel the mystery of memory.
Asunto(s)
Encéfalo/fisiología , Aprendizaje/fisiología , Memoria/fisiología , Plasticidad Neuronal/fisiología , Humanos , Recuerdo Mental/fisiología , Sinapsis/fisiologíaRESUMEN
Mammals navigate by means of a metric cognitive map. Insects, most notably bees and ants, are also impressive navigators. The question whether they, too, have a metric cognitive map is important to cognitive science and neuroscience. Experimentally captured and displaced bees often depart from the release site in the compass direction they were bent on before their capture, even though this no longer heads them toward their goal. When they discover their error, however, the bees set off more or less directly toward their goal. This ability to orient toward a goal from an arbitrary point in the familiar environment is evidence that they have an integrated metric map of the experienced environment. We report a test of an alternative hypothesis, which is that all the bees have in memory is a collection of snapshots that enable them to recognize different landmarks and, associated with each such snapshot, a sun-compass-referenced home vector derived from dead reckoning done before and after previous visits to the landmark. We show that a large shift in the sun-compass rapidly induced by general anesthesia does not alter the accuracy or speed of the homeward-oriented flight made after the bees discover the error in their initial postrelease flight. This result rules out the sun-referenced home-vector hypothesis, further strengthening the now extensive evidence for a metric cognitive map in bees.
Asunto(s)
Abejas/fisiología , Cognición , Fenómenos de Retorno al Lugar Habitual/fisiología , Anestésicos/química , Animales , Teorema de Bayes , Encéfalo/fisiología , Ritmo Circadiano , Señales (Psicología) , Vuelo Animal , Isoflurano/química , Memoria , Oportunidad Relativa , Orientación , Conducta Espacial , Luz SolarRESUMEN
Most studies in the neurobiology of learning assume that the underlying learning process is a pairing - dependent change in synaptic strength that requires repeated experience of events presented in close temporal contiguity. However, much learning is rapid and does not depend on temporal contiguity, which has never been precisely defined. These points are well illustrated by studies showing that the temporal relations between events are rapidly learned- even over long delays- and that this knowledge governs the form and timing of behavior. The speed with which anticipatory responses emerge in conditioning paradigms is determined by the information that cues provide about the timing of rewards. The challenge for understanding the neurobiology of learning is to understand the mechanisms in the nervous system that encode information from even a single experience, the nature of the memory mechanisms that can encode quantities such as time, and how the brain can flexibly perform computations based on this information.
Asunto(s)
Encéfalo/fisiología , Aprendizaje/fisiología , Memoria/fisiología , Percepción del Tiempo/fisiología , Animales , Anticipación Psicológica/fisiología , Condicionamiento Psicológico/fisiología , Humanos , Factores de TiempoRESUMEN
Human and mouse subjects tried to anticipate at which of 2 locations a reward would appear. On a randomly scheduled fraction of the trials, it appeared with a short latency at one location; on the complementary fraction, it appeared after a longer latency at the other location. Subjects of both species accurately assessed the exogenous uncertainty (the probability of a short versus a long trial) and the endogenous uncertainty (from the scalar variability in their estimates of an elapsed duration) to compute the optimal target latency for a switch from the short- to the long-latency location. The optimal latency was arrived at so rapidly that there was no reliably discernible improvement over trials. Under these nonverbal conditions, humans and mice accurately assess risks and behave nearly optimally. That this capacity is well-developed in the mouse opens up the possibility of a genetic approach to the neurobiological mechanisms underlying risk assessment.
Asunto(s)
Medición de Riesgo , Adulto , Alimentación Animal , Animales , Femenino , Humanos , Masculino , Ratones , Ratones Endogámicos C57BL , Modelos Biológicos , Modelos Genéticos , Modelos Estadísticos , Probabilidad , Proyectos de Investigación , Factores de TiempoRESUMEN
We investigated how the common measures of timing performance behaved in the course of training on the peak procedure in C3H mice. Following fixed interval (FI) pre-training, mice received 16 days of training in the peak procedure. The peak time and spread were derived from the average response rates while the start and stop times and their relative variability were derived from a single-trial analysis. Temporal precision (response spread) appeared to improve in the course of training. This apparent improvement in precision was, however, an averaging artifact; it was mediated by the staggered appearance of timed stops, rather than by the delayed occurrence of start times. Trial-by-trial analysis of the stop times for individual subjects revealed that stops appeared abruptly after three to five sessions and their timing did not change as training was prolonged. Start times and the precision of start and stop times were generally stable throughout training. Our results show that subjects do not gradually learn to time their start or stop of responding. Instead, they learn the duration of the FI, with robust temporal control over the start of the response; the control over the stop of response appears abruptly later.
Asunto(s)
Aprendizaje/fisiología , Memoria/fisiología , Esquema de Refuerzo , Percepción del Tiempo/fisiología , Animales , Condicionamiento Operante/fisiología , Masculino , Ratones , Ratones Endogámicos C3H , Refuerzo en PsicologíaRESUMEN
Two experiments measured the human ability to reproduce locomotor distances of 4.6-100 m without visual feedback and compared distance production with time production. Subjects were not permitted to count steps. It was found that the precision of human odometry follows Weber's law that variability is proportional to distance. The coefficients of variation for distance production were much lower than those measured for time production for similar durations. Gait parameters recorded during the task (average step length and step frequency) were found to be even less variable suggesting that step integration could be the basis for non-visual human odometry.
Asunto(s)
Percepción de Distancia , Caminata , Femenino , Marcha , Humanos , Masculino , Desempeño Psicomotor , Privación Sensorial , Percepción del TiempoRESUMEN
The effects of altering the contingency between the conditioned stimulus (CS) and the unconditioned stimulus (US) on the acquisition of autoshaped responding was investigated by changing the frequency of unsignaled USs during the intertrial interval. The addition of the unsignaled USs had an effect on acquisition speed comparable with that of massing trials. The effects of these manipulations can be understood in terms of their effect on the amount of information (number of bits) that the average CS conveys to the subject about the timing of the next US. The number of reinforced CSs prior to acquisition is inversely related to the information content of the CS.
Asunto(s)
Aprendizaje por Asociación/fisiología , Condicionamiento Clásico/fisiología , Aprendizaje Discriminativo/fisiología , Inhibición Psicológica , Animales , Conducta Animal , Columbidae , Esquema de Refuerzo , Refuerzo en Psicología , Factores de TiempoRESUMEN
Meck and Church (1983) estimated a 5:1 scale factor relating the mental magnitudes representing number to the mental magnitudes representing duration. We repeated their experiment with human subjects. We obtained transfer regardless of the objective scaling between the ranges; a 5:1 scaling for number versus duration (measured in seconds) was not necessary. We obtained transfer even when the proportions between the endpoints of the number range were different. We conclude that, at least in human subjects, transfer from a discrimination based on continuous quantity (duration) to a discrimination based on discrete quantity (number) is mediated by the cross-domain comparability of within-domain proportions. The results of our second and third experiments also suggest that the subjects compare a probe with a criterion determined by the range of stimuli tested rather than by trial-specific referents, in accordance with the pseudologistic model of Killeen, Fetterman, and Bizo (1997).
Asunto(s)
Cognición , Discriminación en Psicología , Modelos Estadísticos , Transferencia de Experiencia en Psicología , HumanosRESUMEN
The negatively accelerated, gradually increasing learning curve is an artifact of group averaging in several commonly used basic learning paradigms (pigeon autoshaping, delay- and trace-eye-blink conditioning in the rabbit and rat, autoshaped hopper entry in the rat, plus maze performance in the rat, and water maze performance in the mouse). The learning curves for individual subjects show an abrupt, often step-like increase from the untrained level of responding to the level seen in the well trained subject. The rise is at least as abrupt as that commonly seen in psychometric functions in stimulus detection experiments. It may indicate that the appearance of conditioned behavior is mediated by an evidence-based decision process, as in stimulus detection experiments. If the appearance of conditioned behavior is taken instead to reflect the increase in an underlying associative strength, then a negligible portion of the function relating associative strength to amount of experience is behaviorally visible. Consequently, rate of learning cannot be estimated from the group-average curve; the best measure is latency to the onset of responding, determined for each subject individually.