Distinct replay signatures for prospective decision-making and memory preservation.

Theories of neural replay propose that it supports a range of functions, most prominently planning and memory consolidation. Here, we test the hypothesis that distinct signatures of replay in the same task are related to model-based decision-making ("planning") and memory preservation. We designed a reward learning task wherein participants utilized structure knowledge for model-based evaluation, while at the same time had to maintain knowledge of two independent and randomly alternating task environments. Using magnetoencephalography and multivariate analysis, we first identified temporally compressed sequential reactivation, or replay, both prior to choice and following reward feedback. Before choice, prospective replay strength was enhanced for the current task-relevant environment when a model-based planning strategy was beneficial. Following reward receipt, and consistent with a memory preservation role, replay for the alternative distal task environment was enhanced as a function of decreasing recency of experience with that environment. Critically, these planning and memory preservation relationships were selective to pre-choice and post-feedback periods, respectively. Our results provide support for key theoretical proposals regarding the functional role of replay and demonstrate that the relative strength of planning and memory-related signals are modulated by ongoing computational and task demands.

Trajectories through semantic spaces in schizophrenia and the relationship to ripple bursts.

Human cognition is underpinned by structured internal representations that encode relationships between entities in the world (cognitive maps). Clinical features of schizophrenia-from thought disorder to delusions-are proposed to reflect disorganization in such conceptual representations. Schizophrenia is also linked to abnormalities in neural processes that support cognitive map representations, including hippocampal replay and high-frequency ripple oscillations. Here, we report a computational assay of semantically guided conceptual sampling and exploit this to test a hypothesis that people with schizophrenia (PScz) exhibit abnormalities in semantically guided cognition that relate to hippocampal replay and ripples. Fifty-two participants [26 PScz (13 unmedicated) and 26 age-, gender-, and intelligence quotient (IQ)-matched nonclinical controls] completed a category- and letter-verbal fluency task, followed by a magnetoencephalography (MEG) scan involving a separate sequence-learning task. We used a pretrained word embedding model of semantic similarity, coupled to a computational model of word selection, to quantify the degree to which each participant's verbal behavior was guided by semantic similarity. Using MEG, we indexed neural replay and ripple power in a post-task rest session. Across all participants, word selection was strongly influenced by semantic similarity. The strength of this influence showed sensitivity to task demands (category > letter fluency) and predicted performance. In line with our hypothesis, the influence of semantic similarity on behavior was reduced in schizophrenia relative to controls, predicted negative psychotic symptoms, and correlated with an MEG signature of hippocampal ripple power (but not replay). The findings bridge a gap between phenomenological and neurocomputational accounts of schizophrenia.

Credit assignment to state-independent task representations and its relationship with model-based decision making.

Model-free learning enables an agent to make better decisions based on prior experience while representing only minimal knowledge about an environment's structure. It is generally assumed that model-free state representations are based on outcome-relevant features of the environment. Here, we challenge this assumption by providing evidence that a putative model-free system assigns credit to task representations that are irrelevant to an outcome. We examined data from 769 individuals performing a well-described 2-step reward decision task where stimulus identity but not spatial-motor aspects of the task predicted reward. We show that participants assigned value to spatial-motor representations despite it being outcome irrelevant. Strikingly, spatial-motor value associations affected behavior across all outcome-relevant features and stages of the task, consistent with credit assignment to low-level state-independent task representations. Individual difference analyses suggested that the impact of spatial-motor value formation was attenuated for individuals who showed greater deployment of goal-directed (model-based) strategies. Our findings highlight a need for a reconsideration of how model-free representations are formed and regulated according to the structure of the environment.

Compositional Sequence Generation in the Entorhinal-Hippocampal System.

Neurons in the medial entorhinal cortex exhibit multiple, periodically organized, firing fields which collectively appear to form an internal representation of space. Neuroimaging data suggest that this grid coding is also present in other cortical areas such as the prefrontal cortex, indicating that it may be a general principle of neural functionality in the brain. In a recent analysis through the lens of dynamical systems theory, we showed how grid coding can lead to the generation of a diversity of empirically observed sequential reactivations of hippocampal place cells corresponding to traversals of cognitive maps. Here, we extend this sequence generation model by describing how the synthesis of multiple dynamical systems can support compositional cognitive computations. To empirically validate the model, we simulate two experiments demonstrating compositionality in space or in time during sequence generation. Finally, we describe several neural network architectures supporting various types of compositionality based on grid coding and highlight connections to recent work in machine learning leveraging analogous techniques.

Rapid Visuomotor Responses Reflect Value-Based Decisions.

Cognitive decision-making is known to be sensitive to the values of potential options, which are the probability and size of rewards associated with different choices. Here, we examine whether rapid motor responses to perturbations of visual feedback about movement, which mediate low-level and involuntary feedback control loops, reflect computations associated with high-level value-based decision-making. In three experiments involving human participants, we varied the value associated with different potential targets for reaching movements by controlling the distributions of rewards across the targets (Experiment 1), the probability with which each target could be specified (Experiment 2), or both (Experiment 3). We found that the size of rapid and involuntary feedback responses to movement perturbations was strongly influenced by the relative value between targets. A statistical model of relative value that includes a term for risk sensitivity provided the best fit to the visuomotor response data, illustrating that feedback control policies are biased to favor more frequent task success at the expense of the overall extrinsic reward accumulated through movement. Importantly however, the regulation of rapid feedback responses was associated with successful pursuit of high-value task outcomes. This implies that when we move, the brain specifies a set of feedback control gains that enable low-level motor areas not only to generate efficient and accurate movement, but also to rapidly and adaptively respond to evolving sensory information in a manner consistent with value-based decision-making.SIGNIFICANCE STATEMENT Current theories of sensorimotor control suggest that, rather than selecting and planning the details of movements in advance, the role of the brain is to set time-varying feedback gains that continuously transform sensory information into motor commands by feedback control. Here, we examine whether the fastest motor responses to perturbations of movement, which mediate low-level and involuntary feedback control loops (i.e., reflexes), reflect computations associated with high-level, value-based decision-making. We find that rapid feedback responses during reaching reflect the relative probabilities and rewards associated with target options. This suggests that low-order components of the sensorimotor control hierarchy, which generate rapid and automatic responses, can continuously evaluate evolving sensory evidence and initiate responses according to the prospect of reward.

Opportunities for emotion and mental health research in the resource-rationality framework.

We discuss opportunities in applying the resource-rationality framework toward answering questions in emotion and mental health research. These opportunities rely on characterization of individual differences in cognitive strategies; an endeavor that may be at odds with the normative approach outlined in the target article. We consider ways individual differences might enter the framework and the translational opportunities offered by each.

Characterizing the associative content of brain structures involved in habitual and goal-directed actions in humans: a multivariate FMRI study.

While there is accumulating evidence for the existence of distinct neural systems supporting goal-directed and habitual action selection in the mammalian brain, much less is known about the nature of the information being processed in these different brain regions. Associative learning theory predicts that brain systems involved in habitual control, such as the dorsolateral striatum, should contain stimulus and response information only, but not outcome information, while regions involved in goal-directed action, such as ventromedial and dorsolateral prefrontal cortex and dorsomedial striatum, should be involved in processing information about outcomes as well as stimuli and responses. To test this prediction, human participants underwent fMRI while engaging in a binary choice task designed to enable the separate identification of these different representations with a multivariate classification analysis approach. Consistent with our predictions, the dorsolateral striatum contained information about responses but not outcomes at the time of an initial stimulus, while the regions implicated in goal-directed action selection contained information about both responses and outcomes. These findings suggest that differential contributions of these regions to habitual and goal-directed behavioral control may depend in part on basic differences in the type of information that these regions have access to at the time of decision making.

Evidence for model-based computations in the human amygdala during Pavlovian conditioning.

Contemporary computational accounts of instrumental conditioning have emphasized a role for a model-based system in which values are computed with reference to a rich model of the structure of the world, and a model-free system in which values are updated without encoding such structure. Much less studied is the possibility of a similar distinction operating at the level of Pavlovian conditioning. In the present study, we scanned human participants while they participated in a Pavlovian conditioning task with a simple structure while measuring activity in the human amygdala using a high-resolution fMRI protocol. After fitting a model-based algorithm and a variety of model-free algorithms to the fMRI data, we found evidence for the superiority of a model-based algorithm in accounting for activity in the amygdala compared to the model-free counterparts. These findings support an important role for model-based algorithms in describing the processes underpinning Pavlovian conditioning, as well as providing evidence of a role for the human amygdala in model-based inference.

The generative neural microdynamics of cognitive processing.

The entorhinal cortex and hippocampus form a recurrent network that informs many cognitive processes, including memory, planning, navigation, and imagination. Neural recordings from these regions reveal spatially organized population codes corresponding to external environments and abstract spaces. Aligning the former cognitive functionalities with the latter neural phenomena is a central challenge in understanding the entorhinal-hippocampal circuit (EHC). Disparate experiments demonstrate a surprising level of complexity and apparent disorder in the intricate spatiotemporal dynamics of sequential non-local hippocampal reactivations, which occur particularly, though not exclusively, during immobile pauses and rest. We review these phenomena with a particular focus on their apparent lack of physical simulative realism. These observations are then integrated within a theoretical framework and proposed neural circuit mechanisms that normatively characterize this neural complexity by conceiving different regimes of hippocampal microdynamics as neuromarkers of diverse cognitive computations.

Flexible modulation of sequence generation in the entorhinal-hippocampal system.

Exploration, consolidation and planning depend on the generation of sequential state representations. However, these algorithms require disparate forms of sampling dynamics for optimal performance. We theorize how the brain should adapt internally generated sequences for particular cognitive functions and propose a neural mechanism by which this may be accomplished within the entorhinal-hippocampal circuit. Specifically, we demonstrate that the systematic modulation along the medial entorhinal cortex dorsoventral axis of grid population input into the hippocampus facilitates a flexible generative process that can interpolate between qualitatively distinct regimes of sequential hippocampal reactivations. By relating the emergent hippocampal activity patterns drawn from our model to empirical data, we explain and reconcile a diversity of recently observed, but apparently unrelated, phenomena such as generative cycling, diffusive hippocampal reactivations and jumping trajectory events.

Internal Models in Biological Control.

Rationality principles such as optimal feedback control and Bayesian inference underpin a probabilistic framework that has accounted for a range of empirical phenomena in biological sensorimotor control. To facilitate the optimization of flexible and robust behaviors consistent with these theories, the ability to construct internal models of the motor system and environmental dynamics can be crucial. In the context of this theoretic formalism, we review the computational roles played by such internal models and the neural and behavioral evidence for their implementation in the brain.

A Mobile Phone App for the Generation and Characterization of Motor Habits.

Habits are a powerful route to efficiency; the ability to constantly shift between goal-directed and habitual strategies, as well as integrate them into behavioral output, is key to optimal performance in everyday life. When such ability is impaired, it may lead to loss of control and to compulsive behavior. Habits have successfully been induced and investigated in rats using methods such as overtraining stimulus-response associations and outcome devaluation, respectively. However, such methods have ineffectively measured habits in humans because (1) human habits usually involve more complex sequences of actions than in rats and (2) of pragmatic impediments posed by the extensive time (weeks or even months), it may take for routine habits to develop. We present here a novel behavioral paradigm-a mobile-phone app methodology-for inducing and measuring habits in humans during their everyday schedule and environment. It assumes that practice is key to achieve automaticity and proficiency and that the use of a hierarchical sequence of actions is the best strategy for capturing the cognitive mechanisms involved in habit formation (including "chunking") and consolidation. The task is a gamified self-instructed and self-paced app on a mobile phone that enables subjects to learn and practice two sequences of finger movements, composed of chords and single presses. It involves a step-wise learning procedure in which subjects begin responding to a visual and auditory cued sequence by generating responses on the screen using four fingers. Such cues progressively disappear throughout 1 month of training, enabling the subject ultimately to master the motor skill involved. We present preliminary data for the acquisition of motor sequence learning in 29 healthy individuals, each trained over a month period. We demonstrate an asymptotic improvement in performance, as well as its automatic nature. We also report how people integrate the task into their daily routine, the development of motor precision throughout training, and the effect of intermittent reinforcement and reward extinction in habit preservation. The findings help to validate this "real world" app for measuring human habits.

Category-dependent and category-independent goal-value codes in human ventromedial prefrontal cortex.

To choose between manifestly distinct options, it is suggested that the brain assigns values to goals using a common currency. Although previous studies have reported activity in ventromedial prefrontal cortex (vmPFC) correlating with the value of different goal stimuli, it remains unclear whether such goal-value representations are independent of the associated stimulus categorization, as required by a common currency. Using multivoxel pattern analyses on functional magnetic resonance imaging (fMRI) data, we found a region of medial prefrontal cortex to contain a distributed goal-value code that is independent of stimulus category. More ventrally in the vmPFC, we found spatially distinct areas of the medial orbitofrontal cortex to contain unique category-dependent distributed value codes for food and consumer items. These results implicate the medial prefrontal cortex in the implementation of a common currency and suggest a ventral versus dorsal topographical organization of value signals in the vmPFC.