A complementary learning systems model of how sleep moderates retrieval practice effects.

While many theories assume that sleep is critical in stabilizing and strengthening memories, our recent behavioral study (Liu & Ranganath, 2021, Psychonomic Bulletin & Review, 28[6], 2035-2044) suggests that sleep does not simply stabilize memories. Instead, it plays a more complex role, integrating information across two temporally distinct learning episodes. In the current study, we simulated the results of Liu and Ranganath (2021) using our biologically plausible computational model, TEACH, developed based on the complementary learning systems (CLS) framework. Our model suggests that when memories are activated during sleep, the reduced influence of temporal context establishes connections across temporally separated events through mutual training between the hippocampus and neocortex. In addition to providing a compelling mechanistic explanation for the selective effect of sleep, this model offers new examples of the diverse ways in which the cortex and hippocampus can interact during learning.

Altered Associations Between Motivated Performance and Frontostriatal Functional Connectivity During Reward Anticipation in Schizophrenia.

BACKGROUND AND HYPOTHESIS: The neuronal mechanisms that underlie deficits in effort cost computation in schizophrenia (SZ) are poorly understood. Given the role of frontostriatal circuits in valence-oriented motivation, we hypothesized that these circuits are either dysfunctional in SZ or do not appropriately predict behavior in SZ when task conditions are difficult and good performance is rewarded. STUDY DESIGN: A total of 52 people with recent onset SZ-spectrum disorders and 48 healthy controls (HCs) performed a 3T fMRI task with 2 valence conditions (rewarded vs neutral) and 2 difficulty conditions. Frontostriatal connectivity was extracted during the cue (anticipatory) phase. Individual behavior was fit using a drift-diffusion model, allowing the performance parameter, drift rate (DR), to vary between task conditions. Three models were examined: A group × condition model of DR, a group × condition model of connectivity, and a regression model of connectivity predicting DR depending on group and condition. STUDY RESULTS: DRs showed the expected positive correlation with accuracy and a negative association with reaction time. The SZ group showed a deficit in DR but did not differ in overall connectivity or show a valence-specific deficit in connectivity. Significant group × valence × difficulty interactions, however, were observed on the relationship between right dorsolateral prefrontal (DLPFC)-striatal connectivity and DR (DLPFC-Caudate: F = 10.92, PFDR = .004; DLPFC-Putamen: F = 5.14, PFDR = .048) driven by more positive relationships between DR and connectivity during cues for the difficult-rewarded condition in HCs compared to SZ. CONCLUSIONS: These findings suggest that frontostriatal connectivity is less predictive of performance in SZ when task difficulty is increased and a reward incentive is applied.

Correcting the hebbian mistake: Toward a fully error-driven hippocampus.

The hippocampus plays a critical role in the rapid learning of new episodic memories. Many computational models propose that the hippocampus is an autoassociator that relies on Hebbian learning (i.e., "cells that fire together, wire together"). However, Hebbian learning is computationally suboptimal as it does not learn in a way that is driven toward, and limited by, the objective of achieving effective retrieval. Thus, Hebbian learning results in more interference and a lower overall capacity. Our previous computational models have utilized a powerful, biologically plausible form of error-driven learning in hippocampal CA1 and entorhinal cortex (EC) (functioning as a sparse autoencoder) by contrasting local activity states at different phases in the theta cycle. Based on specific neural data and a recent abstract computational model, we propose a new model called Theremin (Total Hippocampal ERror MINimization) that extends error-driven learning to area CA3-the mnemonic heart of the hippocampal system. In the model, CA3 responds to the EC monosynaptic input prior to the EC disynaptic input through dentate gyrus (DG), giving rise to a temporal difference between these two activation states, which drives error-driven learning in the EC→CA3 and CA3↔CA3 projections. In effect, DG serves as a teacher to CA3, correcting its patterns into more pattern-separated ones, thereby reducing interference. Results showed that Theremin, compared with our original Hebbian-based model, has significantly increased capacity and learning speed. The model makes several novel predictions that can be tested in future studies.

The Structure of Systematicity in the Brain.

A hallmark of human intelligence is the ability to adapt to new situations, by applying learned rules to new content (systematicity) and thereby enabling an open-ended number of inferences and actions (generativity). Here, we propose that the human brain accomplishes these feats through pathways in the parietal cortex that encode the abstract structure of space, events, and tasks, and pathways in the temporal cortex that encode information about specific people, places, and things (content). Recent neural network models show how the separation of structure and content might emerge through a combination of architectural biases and learning, and these networks show dramatic improvements in the ability to capture systematic, generative behavior. We close by considering how the hippocampal formation may form integrative memories that enable rapid learning of new structure and content representations.

A Neural Network Model of Continual Learning with Cognitive Control.

Neural networks struggle in continual learning settings from catastrophic forgetting: when trials are blocked, new learning can overwrite the learning from previous blocks. Humans learn effectively in these settings, in some cases even showing an advantage of blocking, suggesting the brain contains mechanisms to overcome this problem. Here, we build on previous work and show that neural networks equipped with a mechanism for cognitive control do not exhibit catastrophic forgetting when trials are blocked. We further show an advantage of blocking over interleaving when there is a bias for active maintenance in the control signal, implying a tradeoff between maintenance and the strength of control. Analyses of map-like representations learned by the networks provided additional insights into these mechanisms. Our work highlights the potential of cognitive control to aid continual learning in neural networks, and offers an explanation for the advantage of blocking that has been observed in humans.

Complementary Structure-Learning Neural Networks for Relational Reasoning.

The neural mechanisms supporting flexible relational inferences, especially in novel situations, are a major focus of current research. In the complementary learning systems framework, pattern separation in the hippocampus allows rapid learning in novel environments, while slower learning in neocortex accumulates small weight changes to extract systematic structure from well-learned environments. In this work, we adapt this framework to a task from a recent fMRI experiment where novel transitive inferences must be made according to implicit relational structure. We show that computational models capturing the basic cognitive properties of these two systems can explain relational transitive inferences in both familiar and novel environments, and reproduce key phenomena observed in the fMRI experiment.

Deep Predictive Learning in Neocortex and Pulvinar.

How do humans learn from raw sensory experience? Throughout life, but most obviously in infancy, we learn without explicit instruction. We propose a detailed biological mechanism for the widely embraced idea that learning is driven by the differences between predictions and actual outcomes (i.e., predictive error-driven learning). Specifically, numerous weak projections into the pulvinar nucleus of the thalamus generate top-down predictions, and sparse driver inputs from lower areas supply the actual outcome, originating in Layer 5 intrinsic bursting neurons. Thus, the outcome representation is only briefly activated, roughly every 100 msec (i.e., 10 Hz, alpha), resulting in a temporal difference error signal, which drives local synaptic changes throughout the neocortex. This results in a biologically plausible form of error backpropagation learning. We implemented these mechanisms in a large-scale model of the visual system and found that the simulated inferotemporal pathway learns to systematically categorize 3-D objects according to invariant shape properties, based solely on predictive learning from raw visual inputs. These categories match human judgments on the same stimuli and are consistent with neural representations in inferotemporal cortex in primates.

The Neural Correlates of Cued Reward Omission.

Compared to our understanding of positive prediction error signals occurring due to unexpected reward outcomes, less is known about the neural circuitry in humans that drives negative prediction errors during omission of expected rewards. While classical learning theories such as Rescorla-Wagner or temporal difference learning suggest that both types of prediction errors result from a simple subtraction, there has been recent evidence suggesting that different brain regions provide input to dopamine neurons which contributes to specific components of this prediction error computation. Here, we focus on the brain regions responding to negative prediction error signals, which has been well-established in animal studies to involve a distinct pathway through the lateral habenula. We examine the activity of this pathway in humans, using a conditioned inhibition paradigm with high-resolution functional MRI. First, participants learned to associate a sensory stimulus with reward delivery. Then, reward delivery was omitted whenever this stimulus was presented simultaneously with a different sensory stimulus, the conditioned inhibitor (CI). Both reward presentation and the reward-predictive cue activated midbrain dopamine regions, insula and orbitofrontal cortex. While we found significant activity at an uncorrected threshold for the CI in the habenula, consistent with our predictions, it did not survive correction for multiple comparisons and awaits further replication. Additionally, the pallidum and putamen regions of the basal ganglia showed modulations of activity for the inhibitor that did not survive the corrected threshold.

A systems-neuroscience model of phasic dopamine.

We describe a neurobiologically informed computational model of phasic dopamine signaling to account for a wide range of findings, including many considered inconsistent with the simple reward prediction error (RPE) formalism. The central feature of this PVLV framework is a distinction between a primary value (PV) system for anticipating primary rewards (Unconditioned Stimuli [USs]), and a learned value (LV) system for learning about stimuli associated with such rewards (CSs). The LV system represents the amygdala, which drives phasic bursting in midbrain dopamine areas, while the PV system represents the ventral striatum, which drives shunting inhibition of dopamine for expected USs (via direct inhibitory projections) and phasic pausing for expected USs (via the lateral habenula). Our model accounts for data supporting the separability of these systems, including individual differences in CS-based (sign-tracking) versus US-based learning (goal-tracking). Both systems use competing opponent-processing pathways representing evidence for and against specific USs, which can explain data dissociating the processes involved in acquisition versus extinction conditioning. Further, opponent processing proved critical in accounting for the full range of conditioned inhibition phenomena, and the closely related paradigm of second-order conditioning. Finally, we show how additional separable pathways representing aversive USs, largely mirroring those for appetitive USs, also have important differences from the positive valence case, allowing the model to account for several important phenomena in aversive conditioning. Overall, accounting for all of these phenomena strongly constrains the model, thus providing a well-validated framework for understanding phasic dopamine signaling. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Unraveling the Mysteries of Motivation.

Motivation plays a central role in human behavior and cognition but is not well captured by widely used artificial intelligence (AI) and computational modeling frameworks. This Opinion article addresses two central questions regarding the nature of motivation: what are the nature and dynamics of the internal goals that drive our motivational system and how can this system be sufficiently flexible to support our ability to rapidly adapt to novel situations, tasks, etc.? In reviewing existing systems and neuroscience research and theorizing on these questions, a wealth of insights to constrain the development of computational models of motivation can be found.

How Sequential Interactive Processing Within Frontostriatal Loops Supports a Continuum of Habitual to Controlled Processing.

We address the distinction between habitual/automatic vs. goal-directed/controlled behavior, from the perspective of a computational model of the frontostriatal loops. The model exhibits a continuum of behavior between these poles, as a function of the interactive dynamics among different functionally-specialized brain areas, operating iteratively over multiple sequential steps, and having multiple nested loops of similar decision making circuits. This framework blurs the lines between these traditional distinctions in many ways. For example, although habitual actions have traditionally been considered purely automatic, the outer loop must first decide to allow such habitual actions to proceed. Furthermore, because the part of the brain that generates proposed action plans is common across habitual and controlled/goal-directed behavior, the key differences are instead in how many iterations of sequential decision-making are taken, and to what extent various forms of predictive (model-based) processes are engaged. At the core of every iterative step in our model, the basal ganglia provides a "model-free" dopamine-trained Go/NoGo evaluation of the entire distributed plan/goal/evaluation/prediction state. This evaluation serves as the fulcrum of serializing otherwise parallel neural processing. Goal-based inputs to the nominally model-free basal ganglia system are among several ways in which the popular model-based vs. model-free framework may not capture the most behaviorally and neurally relevant distinctions in this area.

Computational models of motivated frontal function.

Computational models of frontal function have made important contributions to understanding how the frontal lobes support a wide range of important functions, in their interactions with other brain areas including, critically, the basal ganglia (BG). We focus here on the specific case of how different frontal areas support goal-directed, motivated decision-making, by representing three essential types of information: possible plans of action (in more dorsal and lateral frontal areas), affectively significant outcomes of those action plans (in ventral, medial frontal areas including the orbital frontal cortex), and the overall utility of a given plan compared to other possible courses of action (in anterior cingulate cortex). Computational models of goal-directed action selection at multiple different levels of analysis provide insight into the nature of learning and processing in these areas and the relative contributions of the frontal cortex versus the BG. The most common neurologic disorders implicate these areas, and understanding their precise function and modes of dysfunction can contribute to the new field of computational psychiatry, within the broader field of computational neuroscience.

Regional specialization within the human striatum for diverse psychological functions.

Decades of animal and human neuroimaging research have identified distinct, but overlapping, striatal zones, which are interconnected with separable corticostriatal circuits, and are crucial for the organization of functional systems. Despite continuous efforts to subdivide the human striatum based on anatomical and resting-state functional connectivity, characterizing the different psychological processes related to each zone remains a work in progress. Using an unbiased, data-driven approach, we analyzed large-scale coactivation data from 5,809 human imaging studies. We (i) identified five distinct striatal zones that exhibited discrete patterns of coactivation with cortical brain regions across distinct psychological processes and (ii) identified the different psychological processes associated with each zone. We found that the reported pattern of cortical activation reliably predicted which striatal zone was most strongly activated. Critically, activation in each functional zone could be associated with distinct psychological processes directly, rather than inferred indirectly from psychological functions attributed to associated cortices. Consistent with well-established findings, we found an association of the ventral striatum (VS) with reward processing. Confirming less well-established findings, the VS and adjacent anterior caudate were associated with evaluating the value of rewards and actions, respectively. Furthermore, our results confirmed a sometimes overlooked specialization of the posterior caudate nucleus for executive functions, often considered the exclusive domain of frontoparietal cortical circuits. Our findings provide a precise functional map of regional specialization within the human striatum, both in terms of the differential cortical regions and psychological functions associated with each striatal zone.

Prediction error and somatosensory insula activation in women recovered from anorexia nervosa.

BACKGROUND: Previous research in patients with anorexia nervosa showed heightened brain response during a taste reward conditioning task and heightened sensitivity to rewarding and punishing stimuli. Here we tested the hypothesis that individuals recovered from anorexia nervosa would also experience greater brain activation during this task as well as higher sensitivity to salient stimuli than controls. METHODS: Women recovered from restricting-type anorexia nervosa and healthy control women underwent fMRI during application of a prediction error taste reward learning paradigm. RESULTS: Twenty-four women recovered from anorexia nervosa (mean age 30.3 ± 8.1 yr) and 24 control women (mean age 27.4 ± 6.3 yr) took part in this study. The recovered anorexia nervosa group showed greater left posterior insula activation for the prediction error model analysis than the control group (family-wise error- and small volume-corrected p < 0.05). A group × condition analysis found greater posterior insula response in women recovered from anorexia nervosa than controls for unexpected stimulus omission, but not for unexpected receipt. Sensitivity to punishment was elevated in women recovered from anorexia nervosa. LIMITATIONS: This was a cross-sectional study, and the sample size was modest. CONCLUSION: Anorexia nervosa after recovery is associated with heightened prediction error-related brain response in the posterior insula as well as greater response to unexpected reward stimulus omission. This finding, together with behaviourally increased sensitivity to punishment, could indicate that individuals recovered from anorexia nervosa are particularly responsive to punishment. The posterior insula processes somatosensory stimuli, including unexpected bodily states, and greater response could indicate altered perception or integration of unexpected or maybe unwanted bodily feelings. Whether those findings develop during the ill state or whether they are biological traits requires further study.

How the credit assignment problems in motor control could be solved after the cerebellum predicts increases in error.

We present a cerebellar architecture with two main characteristics. The first one is that complex spikes respond to increases in sensory errors. The second one is that cerebellar modules associate particular contexts where errors have increased in the past with corrective commands that stop the increase in error. We analyze our architecture formally and computationally for the case of reaching in a 3D environment. In the case of motor control, we show that there are synergies of this architecture with the Equilibrium-Point hypothesis, leading to novel ways to solve the motor error and distal learning problems. In particular, the presence of desired equilibrium lengths for muscles provides a way to know when the error is increasing, and which corrections to apply. In the context of Threshold Control Theory and Perceptual Control Theory we show how to extend our model so it implements anticipative corrections in cascade control systems that span from muscle contractions to cognitive operations.

Latent structure in random sequences drives neural learning toward a rational bias.

People generally fail to produce random sequences by overusing alternating patterns and avoiding repeating ones-the gambler's fallacy bias. We can explain the neural basis of this bias in terms of a biologically motivated neural model that learns from errors in predicting what will happen next. Through mere exposure to random sequences over time, the model naturally develops a representation that is biased toward alternation, because of its sensitivity to some surprisingly rich statistical structure that emerges in these random sequences. Furthermore, the model directly produces the best-fitting bias-gain parameter for an existing Bayesian model, by which we obtain an accurate fit to the human data in random sequence production. These results show that our seemingly irrational, biased view of randomness can be understood instead as the perfectly reasonable response of an effective learning mechanism to subtle statistical structure embedded in random sequences.

Thalamic pathways underlying prefrontal cortex-medial temporal lobe oscillatory interactions.

As focus shifts to large-scale network interactions involved in memory, it is becoming increasingly clear that oscillatory dynamics are critically involved. A number of studies have shown a negative correlation between memory retrieval in alpha and beta power, and a positive correlation between retrieval and theta power. In this opinion article, we suggest three thalamic sub-regions responsible for the coordination of oscillatory activity and the facilitation of memory processes. Specifically, the medial dorsal nucleus is related to changes in beta synchrony, the pulvinar is responsible for alpha synchrony, and the anterior thalamus is related to theta synchrony. These pathways may be modulated via frontal control, and changes in oscillations could be used to track the engagement of underlying memory systems.

Proceedings From FDA/A.S.P.E.N. Public Workshop: Clinical Trial Design for Intravenous Fat Emulsion Products, October 29, 2013.

The development of intravenous fat emulsion (IVFE) is the culmination of physiological, biochemical, nutritional, and medical scientific advancements. IVFEs have the ability to deliver critical nutritional substrates to the patient. Recent literature purports that they may also play roles in modulation of immune functionality and pulmonary physiology, but data supporting these potential benefits are limited. While soybean-based IVFEs have comprised the dominant fat in U.S. markets, a number of other novel IVFEs may prove to optimize the care of children and adults in both hospitalized and home settings. The October 2013 U.S. Food and Drug Administration (FDA)/American Society for Parenteral and Enteral Nutrition (A.S.P.E.N.) Public Workshop brought together scientists, researchers, and clinical experts to present updated clinical perspectives of IVFEs, including historical development, current state of usage throughout the world, and considerations for the regulatory approval of new IVFEs in the United States.

A continuous-time neural model for sequential action.

Action selection, planning and execution are continuous processes that evolve over time, responding to perceptual feedback as well as evolving top-down constraints. Existing models of routine sequential action (e.g. coffee- or pancake-making) generally fall into one of two classes: hierarchical models that include hand-built task representations, or heterarchical models that must learn to represent hierarchy via temporal context, but thus far lack goal-orientedness. We present a biologically motivated model of the latter class that, because it is situated in the Leabra neural architecture, affords an opportunity to include both unsupervised and goal-directed learning mechanisms. Moreover, we embed this neurocomputational model in the theoretical framework of the theory of event coding (TEC), which posits that actions and perceptions share a common representation with bidirectional associations between the two. Thus, in this view, not only does perception select actions (along with task context), but actions are also used to generate perceptions (i.e. intended effects). We propose a neural model that implements TEC to carry out sequential action control in hierarchically structured tasks such as coffee-making. Unlike traditional feedforward discrete-time neural network models, which use static percepts to generate static outputs, our biological model accepts continuous-time inputs and likewise generates non-stationary outputs, making short-timescale dynamic predictions.