Structured Event Memory: A neuro-symbolic model of event cognition.

Humans spontaneously organize a continuous experience into discrete events and use the learned structure of these events to generalize and organize memory. We introduce the Structured Event Memory (SEM) model of event cognition, which accounts for human abilities in event segmentation, memory, and generalization. SEM is derived from a probabilistic generative model of event dynamics defined over structured symbolic scenes. By embedding symbolic scene representations in a vector space and parametrizing the scene dynamics in this continuous space, SEM combines the advantages of structured and neural network approaches to high-level cognition. Using probabilistic reasoning over this generative model, SEM can infer event boundaries, learn event schemata, and use event knowledge to reconstruct past experience. We show that SEM can scale up to high-dimensional input spaces, producing human-like event segmentation for naturalistic video data, and accounts for a wide array of memory phenomena. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Generalizing to generalize: Humans flexibly switch between compositional and conjunctive structures during reinforcement learning.

Humans routinely face novel environments in which they have to generalize in order to act adaptively. However, doing so involves the non-trivial challenge of deciding which aspects of a task domain to generalize. While it is sometimes appropriate to simply re-use a learned behavior, often adaptive generalization entails recombining distinct components of knowledge acquired across multiple contexts. Theoretical work has suggested a computational trade-off in which it can be more or less useful to learn and generalize aspects of task structure jointly or compositionally, depending on previous task statistics, but it is unknown whether humans modulate their generalization strategy accordingly. Here we develop a series of navigation tasks that separately manipulate the statistics of goal values ("what to do") and state transitions ("how to do it") across contexts and assess whether human subjects generalize these task components separately or conjunctively. We find that human generalization is sensitive to the statistics of the previously experienced task domain, favoring compositional or conjunctive generalization when the task statistics are indicative of such structures, and a mixture of the two when they are more ambiguous. These results support a normative "meta-generalization" account and suggests that people not only generalize previous task components but also generalize the statistical structure most likely to support generalization.

Finding structure in multi-armed bandits.

How do humans search for rewards? This question is commonly studied using multi-armed bandit tasks, which require participants to trade off exploration and exploitation. Standard multi-armed bandits assume that each option has an independent reward distribution. However, learning about options independently is unrealistic, since in the real world options often share an underlying structure. We study a class of structured bandit tasks, which we use to probe how generalization guides exploration. In a structured multi-armed bandit, options have a correlation structure dictated by a latent function. We focus on bandits in which rewards are linear functions of an option's spatial position. Across 5 experiments, we find evidence that participants utilize functional structure to guide their exploration, and also exhibit a learning-to-learn effect across rounds, becoming progressively faster at identifying the latent function. Our experiments rule out several heuristic explanations and show that the same findings obtain with non-linear functions. Comparing several models of learning and decision making, we find that the best model of human behavior in our tasks combines three computational mechanisms: (1) function learning, (2) clustering of reward distributions across rounds, and (3) uncertainty-guided exploration. Our results suggest that human reinforcement learning can utilize latent structure in sophisticated ways to improve efficiency.

Compositional clustering in task structure learning.

Humans are remarkably adept at generalizing knowledge between experiences in a way that can be difficult for computers. Often, this entails generalizing constituent pieces of experiences that do not fully overlap, but nonetheless share useful similarities with, previously acquired knowledge. However, it is often unclear how knowledge gained in one context should generalize to another. Previous computational models and data suggest that rather than learning about each individual context, humans build latent abstract structures and learn to link these structures to arbitrary contexts, facilitating generalization. In these models, task structures that are more popular across contexts are more likely to be revisited in new contexts. However, these models can only re-use policies as a whole and are unable to transfer knowledge about the transition structure of the environment even if only the goal has changed (or vice-versa). This contrasts with ecological settings, where some aspects of task structure, such as the transition function, will be shared between context separately from other aspects, such as the reward function. Here, we develop a novel non-parametric Bayesian agent that forms independent latent clusters for transition and reward functions, affording separable transfer of their constituent parts across contexts. We show that the relative performance of this agent compared to an agent that jointly clusters reward and transition functions depends environmental task statistics: the mutual information between transition and reward functions and the stochasticity of the observations. We formalize our analysis through an information theoretic account of the priors, and propose a meta learning agent that dynamically arbitrates between strategies across task domains to optimize a statistical tradeoff.

A cholinergic feedback circuit to regulate striatal population uncertainty and optimize reinforcement learning.

Convergent evidence suggests that the basal ganglia support reinforcement learning by adjusting action values according to reward prediction errors. However, adaptive behavior in stochastic environments requires the consideration of uncertainty to dynamically adjust the learning rate. We consider how cholinergic tonically active interneurons (TANs) may endow the striatum with such a mechanism in computational models spanning three Marr's levels of analysis. In the neural model, TANs modulate the excitability of spiny neurons, their population response to reinforcement, and hence the effective learning rate. Long TAN pauses facilitated robustness to spurious outcomes by increasing divergence in synaptic weights between neurons coding for alternative action values, whereas short TAN pauses facilitated stochastic behavior but increased responsiveness to change-points in outcome contingencies. A feedback control system allowed TAN pauses to be dynamically modulated by uncertainty across the spiny neuron population, allowing the system to self-tune and optimize performance across stochastic environments.

Adolescents let sufficient evidence accumulate before making a decision when large incentives are at stake.

Adolescent decision-making has been described as impulsive and suboptimal in the presence of incentives. In this study we examined the neural substrates of adolescent decision-making using a perceptual discrimination task for which small and large rewards were associated with correctly detecting the direction of motion of a cloud of moving dots. Adults showed a reward bias of faster reaction times on trials for which the direction of motion was associated with a large reward. Adolescents, in contrast, were slower to make decisions on trials associated with large rewards. This behavioral pattern in adolescents was paralleled by greater recruitment of fronto-parietal regions important in representing the accumulation of evidence sufficient for selecting one choice over its alternative and the certainty of that choice. The findings suggest that when large incentives are dependent on performance, adolescents may require more evidence to accumulate prior to responding, to be certain to maximize their gains. Adults, in contrast, appear to be quicker in evaluating the evidence for a decision when primed by rewards. Overall these findings suggest that rather than reacting hastily, adolescents can be incentivized to take more time to make decisions when large rewards are at stake. A video abstract of this article can be viewed at http://youtu.be/1g4F5vzFDl0.

Behavioral and neural correlates of delay of gratification 40 years later.

We examined the neural basis of self-regulation in individuals from a cohort of preschoolers who performed the delay-of-gratification task 4 decades ago. Nearly 60 individuals, now in their mid-forties, were tested on "hot" and "cool" versions of a go/nogo task to assess whether delay of gratification in childhood predicts impulse control abilities and sensitivity to alluring cues (happy faces). Individuals who were less able to delay gratification in preschool and consistently showed low self-control abilities in their twenties and thirties performed more poorly than did high delayers when having to suppress a response to a happy face but not to a neutral or fearful face. This finding suggests that sensitivity to environmental hot cues plays a significant role in individuals' ability to suppress actions toward such stimuli. A subset of these participants (n = 26) underwent functional imaging for the first time to test for biased recruitment of frontostriatal circuitry when required to suppress responses to alluring cues. Whereas the prefrontal cortex differentiated between nogo and go trials to a greater extent in high delayers, the ventral striatum showed exaggerated recruitment in low delayers. Thus, resistance to temptation as measured originally by the delay-of-gratification task is a relatively stable individual difference that predicts reliable biases in frontostriatal circuitries that integrate motivational and control processes.