Correcting the hebbian mistake: Toward a fully error-driven hippocampus.

The hippocampus plays a critical role in the rapid learning of new episodic memories. Many computational models propose that the hippocampus is an autoassociator that relies on Hebbian learning (i.e., "cells that fire together, wire together"). However, Hebbian learning is computationally suboptimal as it does not learn in a way that is driven toward, and limited by, the objective of achieving effective retrieval. Thus, Hebbian learning results in more interference and a lower overall capacity. Our previous computational models have utilized a powerful, biologically plausible form of error-driven learning in hippocampal CA1 and entorhinal cortex (EC) (functioning as a sparse autoencoder) by contrasting local activity states at different phases in the theta cycle. Based on specific neural data and a recent abstract computational model, we propose a new model called Theremin (Total Hippocampal ERror MINimization) that extends error-driven learning to area CA3-the mnemonic heart of the hippocampal system. In the model, CA3 responds to the EC monosynaptic input prior to the EC disynaptic input through dentate gyrus (DG), giving rise to a temporal difference between these two activation states, which drives error-driven learning in the EC→CA3 and CA3↔CA3 projections. In effect, DG serves as a teacher to CA3, correcting its patterns into more pattern-separated ones, thereby reducing interference. Results showed that Theremin, compared with our original Hebbian-based model, has significantly increased capacity and learning speed. The model makes several novel predictions that can be tested in future studies.

Deep Predictive Learning in Neocortex and Pulvinar.

How do humans learn from raw sensory experience? Throughout life, but most obviously in infancy, we learn without explicit instruction. We propose a detailed biological mechanism for the widely embraced idea that learning is driven by the differences between predictions and actual outcomes (i.e., predictive error-driven learning). Specifically, numerous weak projections into the pulvinar nucleus of the thalamus generate top-down predictions, and sparse driver inputs from lower areas supply the actual outcome, originating in Layer 5 intrinsic bursting neurons. Thus, the outcome representation is only briefly activated, roughly every 100 msec (i.e., 10 Hz, alpha), resulting in a temporal difference error signal, which drives local synaptic changes throughout the neocortex. This results in a biologically plausible form of error backpropagation learning. We implemented these mechanisms in a large-scale model of the visual system and found that the simulated inferotemporal pathway learns to systematically categorize 3-D objects according to invariant shape properties, based solely on predictive learning from raw visual inputs. These categories match human judgments on the same stimuli and are consistent with neural representations in inferotemporal cortex in primates.

Regional specialization within the human striatum for diverse psychological functions.

Decades of animal and human neuroimaging research have identified distinct, but overlapping, striatal zones, which are interconnected with separable corticostriatal circuits, and are crucial for the organization of functional systems. Despite continuous efforts to subdivide the human striatum based on anatomical and resting-state functional connectivity, characterizing the different psychological processes related to each zone remains a work in progress. Using an unbiased, data-driven approach, we analyzed large-scale coactivation data from 5,809 human imaging studies. We (i) identified five distinct striatal zones that exhibited discrete patterns of coactivation with cortical brain regions across distinct psychological processes and (ii) identified the different psychological processes associated with each zone. We found that the reported pattern of cortical activation reliably predicted which striatal zone was most strongly activated. Critically, activation in each functional zone could be associated with distinct psychological processes directly, rather than inferred indirectly from psychological functions attributed to associated cortices. Consistent with well-established findings, we found an association of the ventral striatum (VS) with reward processing. Confirming less well-established findings, the VS and adjacent anterior caudate were associated with evaluating the value of rewards and actions, respectively. Furthermore, our results confirmed a sometimes overlooked specialization of the posterior caudate nucleus for executive functions, often considered the exclusive domain of frontoparietal cortical circuits. Our findings provide a precise functional map of regional specialization within the human striatum, both in terms of the differential cortical regions and psychological functions associated with each striatal zone.

Latent structure in random sequences drives neural learning toward a rational bias.

People generally fail to produce random sequences by overusing alternating patterns and avoiding repeating ones-the gambler's fallacy bias. We can explain the neural basis of this bias in terms of a biologically motivated neural model that learns from errors in predicting what will happen next. Through mere exposure to random sequences over time, the model naturally develops a representation that is biased toward alternation, because of its sensitivity to some surprisingly rich statistical structure that emerges in these random sequences. Furthermore, the model directly produces the best-fitting bias-gain parameter for an existing Bayesian model, by which we obtain an accurate fit to the human data in random sequence production. These results show that our seemingly irrational, biased view of randomness can be understood instead as the perfectly reasonable response of an effective learning mechanism to subtle statistical structure embedded in random sequences.

Prediction error and somatosensory insula activation in women recovered from anorexia nervosa.

BACKGROUND: Previous research in patients with anorexia nervosa showed heightened brain response during a taste reward conditioning task and heightened sensitivity to rewarding and punishing stimuli. Here we tested the hypothesis that individuals recovered from anorexia nervosa would also experience greater brain activation during this task as well as higher sensitivity to salient stimuli than controls. METHODS: Women recovered from restricting-type anorexia nervosa and healthy control women underwent fMRI during application of a prediction error taste reward learning paradigm. RESULTS: Twenty-four women recovered from anorexia nervosa (mean age 30.3 ± 8.1 yr) and 24 control women (mean age 27.4 ± 6.3 yr) took part in this study. The recovered anorexia nervosa group showed greater left posterior insula activation for the prediction error model analysis than the control group (family-wise error- and small volume-corrected p < 0.05). A group × condition analysis found greater posterior insula response in women recovered from anorexia nervosa than controls for unexpected stimulus omission, but not for unexpected receipt. Sensitivity to punishment was elevated in women recovered from anorexia nervosa. LIMITATIONS: This was a cross-sectional study, and the sample size was modest. CONCLUSION: Anorexia nervosa after recovery is associated with heightened prediction error-related brain response in the posterior insula as well as greater response to unexpected reward stimulus omission. This finding, together with behaviourally increased sensitivity to punishment, could indicate that individuals recovered from anorexia nervosa are particularly responsive to punishment. The posterior insula processes somatosensory stimuli, including unexpected bodily states, and greater response could indicate altered perception or integration of unexpected or maybe unwanted bodily feelings. Whether those findings develop during the ill state or whether they are biological traits requires further study.

Indirection and symbol-like processing in the prefrontal cortex and basal ganglia.

The ability to flexibly, rapidly, and accurately perform novel tasks is a hallmark of human behavior. In our everyday lives we are often faced with arbitrary instructions that we must understand and follow, and we are able to do so with remarkable ease. It has frequently been argued that this ability relies on symbol processing, which depends critically on the ability to represent variables and bind them to arbitrary values. Whereas symbol processing is a fundamental feature of all computer systems, it remains a mystery whether and how this ability is carried out by the brain. Here, we provide an example of how the structure and functioning of the prefrontal cortex/basal ganglia working memory system can support variable binding, through a form of indirection (akin to a pointer in computer science). We show how indirection enables the system to flexibly generalize its behavior substantially beyond its direct experience (i.e., systematicity). We argue that this provides a biologically plausible mechanism that approximates a key component of symbol processing, exhibiting both the flexibility, but also some of the limitations, that are associated with this ability in humans.

Theta coordinated error-driven learning in the hippocampus.

The learning mechanism in the hippocampus has almost universally been assumed to be Hebbian in nature, where individual neurons in an engram join together with synaptic weight increases to support facilitated recall of memories later. However, it is also widely known that Hebbian learning mechanisms impose significant capacity constraints, and are generally less computationally powerful than learning mechanisms that take advantage of error signals. We show that the differential phase relationships of hippocampal subfields within the overall theta rhythm enable a powerful form of error-driven learning, which results in significantly greater capacity, as shown in computer simulations. In one phase of the theta cycle, the bidirectional connectivity between CA1 and entorhinal cortex can be trained in an error-driven fashion to learn to effectively encode the cortical inputs in a compact and sparse form over CA1. In a subsequent portion of the theta cycle, the system attempts to recall an existing memory, via the pathway from entorhinal cortex to CA3 and CA1. Finally the full theta cycle completes when a strong target encoding representation of the current input is imposed onto the CA1 via direct projections from entorhinal cortex. The difference between this target encoding and the attempted recall of the same representation on CA1 constitutes an error signal that can drive the learning of CA3 to CA1 synapses. This CA3 to CA1 pathway is critical for enabling full reinstatement of recalled hippocampal memories out in cortex. Taken together, these new learning dynamics enable a much more robust, high-capacity model of hippocampal learning than was available previously under the classical Hebbian model.

A complementary learning systems model of how sleep moderates retrieval practice effects.

While many theories assume that sleep is critical in stabilizing and strengthening memories, our recent behavioral study (Liu & Ranganath, 2021, Psychonomic Bulletin & Review, 28[6], 2035-2044) suggests that sleep does not simply stabilize memories. Instead, it plays a more complex role, integrating information across two temporally distinct learning episodes. In the current study, we simulated the results of Liu and Ranganath (2021) using our biologically plausible computational model, TEACH, developed based on the complementary learning systems (CLS) framework. Our model suggests that when memories are activated during sleep, the reduced influence of temporal context establishes connections across temporally separated events through mutual training between the hippocampus and neocortex. In addition to providing a compelling mechanistic explanation for the selective effect of sleep, this model offers new examples of the diverse ways in which the cortex and hippocampus can interact during learning.

Assembling old tricks for new tasks: a neural model of instructional learning and control.

We can learn from the wisdom of others to maximize success. However, it is unclear how humans take advice to flexibly adapt behavior. On the basis of data from neuroanatomy, neurophysiology, and neuroimaging, a biologically plausible model is developed to illustrate the neural mechanisms of learning from instructions. The model consists of two complementary learning pathways. The slow-learning parietal pathway carries out simple or habitual stimulus-response (S-R) mappings, whereas the fast-learning hippocampal pathway implements novel S-R rules. Specifically, the hippocampus can rapidly encode arbitrary S-R associations, and stimulus-cued responses are later recalled into the basal ganglia-gated pFC to bias response selection in the premotor and motor cortices. The interactions between the two model learning pathways explain how instructions can override habits and how automaticity can be achieved through motor consolidation.

Neural inhibition enables selection during language processing.

Whether grocery shopping or choosing words to express a thought, selecting between options can be challenging, especially for people with anxiety. We investigate the neural mechanisms supporting selection during language processing and its breakdown in anxiety. Our neural network simulations demonstrate a critical role for competitive, inhibitory dynamics supported by GABAergic interneurons. As predicted by our model, we find that anxiety (associated with reduced neural inhibition) impairs selection among options and associated prefrontal cortical activity, even in a simple, nonaffective verb-generation task, and the GABA agonist midazolam (which increases neural inhibition) improves selection, whereas retrieval from semantic memory is unaffected when selection demands are low. Neural inhibition is key to choosing our words.

Inhibiting PKMζ reveals dorsal lateral and dorsal medial striatum store the different memories needed to support adaptive behavior.

Evidence suggests that two regions of the striatum contribute differential support to instrumental response selection. The dorsomedial striatum (DMS) is thought to support expectancy-mediated actions, and the dorsolateral striatum (DLS) is thought to support habits. Currently it is unclear whether these regions store task-relevant information or just coordinate the learning and retention of these solutions by other brain regions. To address this issue, we developed a two-lever concurrent variable-interval reinforcement operant conditioning task and used it to assess the trained rat's sensitivity to contingency shifts. Consistent with the view that these two regions make different contributions to actions and habits, injecting the NMDA antagonist DL-AP5 into the DMS just prior to the shift impaired the rat's performance but enhanced performance when injected into the DLS. To determine if these regions support memory content, we first trained rats on a biased concurrent schedule (Lever 1: VI 40" and Lever 2: VI 10"). With the intent of "erasing" the memory content stored in striatum, after this training we inhibited the putative memory-maintenance protein kinase C isozyme protein kinase Mζ (PKMζ). Infusing zeta inhibitory peptide (ZIP) into the DLS enhanced the rat's ability to adapt to the contingency shift 2 d later, whereas injecting it into the DMS had the opposite effect. Infusing GluR2(3Y) into the DMS 1 h before ZIP infusions prevented ZIP from impairing the rat's sensitivity to the contingency shift. These results support the hypothesis that the DMS stores information needed to support actions and the DLS stores information needed to support habits.

Altered Associations Between Motivated Performance and Frontostriatal Functional Connectivity During Reward Anticipation in Schizophrenia.

BACKGROUND AND HYPOTHESIS: The neuronal mechanisms that underlie deficits in effort cost computation in schizophrenia (SZ) are poorly understood. Given the role of frontostriatal circuits in valence-oriented motivation, we hypothesized that these circuits are either dysfunctional in SZ or do not appropriately predict behavior in SZ when task conditions are difficult and good performance is rewarded. STUDY DESIGN: A total of 52 people with recent onset SZ-spectrum disorders and 48 healthy controls (HCs) performed a 3T fMRI task with 2 valence conditions (rewarded vs neutral) and 2 difficulty conditions. Frontostriatal connectivity was extracted during the cue (anticipatory) phase. Individual behavior was fit using a drift-diffusion model, allowing the performance parameter, drift rate (DR), to vary between task conditions. Three models were examined: A group × condition model of DR, a group × condition model of connectivity, and a regression model of connectivity predicting DR depending on group and condition. STUDY RESULTS: DRs showed the expected positive correlation with accuracy and a negative association with reaction time. The SZ group showed a deficit in DR but did not differ in overall connectivity or show a valence-specific deficit in connectivity. Significant group × valence × difficulty interactions, however, were observed on the relationship between right dorsolateral prefrontal (DLPFC)-striatal connectivity and DR (DLPFC-Caudate: F = 10.92, PFDR = .004; DLPFC-Putamen: F = 5.14, PFDR = .048) driven by more positive relationships between DR and connectivity during cues for the difficult-rewarded condition in HCs compared to SZ. CONCLUSIONS: These findings suggest that frontostriatal connectivity is less predictive of performance in SZ when task difficulty is increased and a reward incentive is applied.

Expectancy, ambiguity, and behavioral flexibility: separable and complementary roles of the orbital frontal cortex and amygdala in processing reward expectancies.

Appetitive goal-directed behavior can be associated with a cue-triggered expectancy that it will lead to a particular reward, a process thought to depend on the OFC and basolateral amygdala complex. We developed a biologically informed neural network model of this system to investigate the separable and complementary roles of these areas as the main components of a flexible expectancy system. These areas of interest are part of a neural network with additional subcortical areas, including the central nucleus of amygdala, ventral (limbic) and dorsomedial (associative) striatum. Our simulations are consistent with the view that the amygdala maintains Pavlovian associations through incremental updating of synaptic strength and that the OFC supports flexibility by maintaining an activation-based working memory of the recent reward history. Our model provides a mechanistic explanation for electrophysiological evidence that cue-related firing in OFC neurons is nonselectively early after a contingency change and why this nonselective firing is critical for promoting plasticity in the amygdala. This ambiguous activation results from the simultaneous maintenance of recent outcomes and obsolete Pavlovian contingencies in working memory. Furthermore, at the beginning of reversal, the OFC is critical for supporting responses that are no longer inappropriate. This result is inconsistent with an exclusive inhibitory account of OFC function.

Distinct contributions of the caudate nucleus, rostral prefrontal cortex, and parietal cortex to the execution of instructed tasks.

When we behave according to rules and instructions, our brains interpret abstract representations of what to do and transform them into actual behavior. In order to investigate the neural mechanisms behind this process, we devised an fMRI experiment that explicitly isolated rule interpretation from rule encoding and execution. Our results showed that a specific network of regions (including the left rostral prefrontal cortex, the caudate nucleus, and the bilateral posterior parietal cortices) is responsible for translating rules into executable form. An analysis of activation patterns across conditions revealed that the posterior parietal cortices represent a mental template for the task to perform, that the inferior parietal gyrus and the caudate nucleus are responsible for instantiating the template in the proper context, and that the left rostral prefrontal cortex integrates information across complex relationships.

The Structure of Systematicity in the Brain.

A hallmark of human intelligence is the ability to adapt to new situations, by applying learned rules to new content (systematicity) and thereby enabling an open-ended number of inferences and actions (generativity). Here, we propose that the human brain accomplishes these feats through pathways in the parietal cortex that encode the abstract structure of space, events, and tasks, and pathways in the temporal cortex that encode information about specific people, places, and things (content). Recent neural network models show how the separation of structure and content might emerge through a combination of architectural biases and learning, and these networks show dramatic improvements in the ability to capture systematic, generative behavior. We close by considering how the hippocampal formation may form integrative memories that enable rapid learning of new structure and content representations.

A Neural Network Model of Continual Learning with Cognitive Control.

Neural networks struggle in continual learning settings from catastrophic forgetting: when trials are blocked, new learning can overwrite the learning from previous blocks. Humans learn effectively in these settings, in some cases even showing an advantage of blocking, suggesting the brain contains mechanisms to overcome this problem. Here, we build on previous work and show that neural networks equipped with a mechanism for cognitive control do not exhibit catastrophic forgetting when trials are blocked. We further show an advantage of blocking over interleaving when there is a bias for active maintenance in the control signal, implying a tradeoff between maintenance and the strength of control. Analyses of map-like representations learned by the networks provided additional insights into these mechanisms. Our work highlights the potential of cognitive control to aid continual learning in neural networks, and offers an explanation for the advantage of blocking that has been observed in humans.

Separate neural substrates for skill learning and performance in the ventral and dorsal striatum.

It is widely accepted that the striatum of the basal ganglia is a primary substrate for the learning and performance of skills. We provide evidence that two regions of the rat striatum, ventral and dorsal, play distinct roles in instrumental conditioning (skill learning), with the ventral striatum being critical for learning and the dorsal striatum being important for performance but, notably, not for learning. This implies an actor (dorsal) versus director (ventral) division of labor, which is a new variant of the widely discussed actor-critic architecture. Our results also imply that the successful performance of a skill can ultimately result in its establishment as a habit outside the basal ganglia.

Complementary Structure-Learning Neural Networks for Relational Reasoning.

The neural mechanisms supporting flexible relational inferences, especially in novel situations, are a major focus of current research. In the complementary learning systems framework, pattern separation in the hippocampus allows rapid learning in novel environments, while slower learning in neocortex accumulates small weight changes to extract systematic structure from well-learned environments. In this work, we adapt this framework to a task from a recent fMRI experiment where novel transitive inferences must be made according to implicit relational structure. We show that computational models capturing the basic cognitive properties of these two systems can explain relational transitive inferences in both familiar and novel environments, and reproduce key phenomena observed in the fMRI experiment.

The Neural Correlates of Cued Reward Omission.

Compared to our understanding of positive prediction error signals occurring due to unexpected reward outcomes, less is known about the neural circuitry in humans that drives negative prediction errors during omission of expected rewards. While classical learning theories such as Rescorla-Wagner or temporal difference learning suggest that both types of prediction errors result from a simple subtraction, there has been recent evidence suggesting that different brain regions provide input to dopamine neurons which contributes to specific components of this prediction error computation. Here, we focus on the brain regions responding to negative prediction error signals, which has been well-established in animal studies to involve a distinct pathway through the lateral habenula. We examine the activity of this pathway in humans, using a conditioned inhibition paradigm with high-resolution functional MRI. First, participants learned to associate a sensory stimulus with reward delivery. Then, reward delivery was omitted whenever this stimulus was presented simultaneously with a different sensory stimulus, the conditioned inhibitor (CI). Both reward presentation and the reward-predictive cue activated midbrain dopamine regions, insula and orbitofrontal cortex. While we found significant activity at an uncorrected threshold for the CI in the habenula, consistent with our predictions, it did not survive correction for multiple comparisons and awaits further replication. Additionally, the pallidum and putamen regions of the basal ganglia showed modulations of activity for the inhibitor that did not survive the corrected threshold.