Learning environment-specific learning rates.

People often have to switch back and forth between different environments that come with different problems and volatilities. While volatile environments require fast learning (i.e., high learning rates), stable environments call for lower learning rates. Previous studies have shown that people adapt their learning rates, but it remains unclear whether they can also learn about environment-specific learning rates, and instantaneously retrieve them when revisiting environments. Here, using optimality simulations and hierarchical Bayesian analyses across three experiments, we show that people can learn to use different learning rates when switching back and forth between two different environments. We even observe a signature of these environment-specific learning rates when the volatility of both environments is suddenly the same. We conclude that humans can flexibly adapt and learn to associate different learning rates to different environments, offering important insights for developing theories of meta-learning and context-specific control.

Manipulating Prior Beliefs Causally Induces Under- and Overconfidence.

Humans differ vastly in the confidence they assign to decisions. Although such under- and overconfidence relate to fundamental life outcomes, a computational account specifying the underlying mechanisms is currently lacking. We propose that prior beliefs in the ability to perform a task explain confidence differences across participants and tasks, despite similar performance. In two perceptual decision-making experiments, we show that manipulating prior beliefs about performance during training causally influences confidence in healthy adults (N = 50 each; Experiment 1: 8 men, one nonbinary; Experiment 2: 5 men) during a test phase, despite unaffected objective performance. This is true when prior beliefs are induced via manipulated comparative feedback and via manipulated training-phase difficulty. Our results were accounted for within an accumulation-to-bound model, explicitly modeling prior beliefs on the basis of earlier task exposure. Decision confidence is quantified as the probability of being correct conditional on prior beliefs, causing under- or overconfidence. We provide a fundamental mechanistic insight into the computations underlying under- and overconfidence.

Modulating hierarchical learning by high-definition transcranial alternating current stimulation at theta frequency.

Considerable evidence highlights the dorsolateral prefrontal cortex (DLPFC) as a key region for hierarchical (i.e. multilevel) learning. In a previous electroencephalography (EEG) study, we found that the low-level prediction errors were encoded by frontal theta oscillations (4-7 Hz), centered on right DLPFC (rDLPFC). However, the causal relationship between frontal theta oscillations and hierarchical learning remains poorly understood. To investigate this question, in the current study, participants received theta (6 Hz) and sham high-definition transcranial alternating current stimulation (HD-tACS) over the rDLPFC while performing the probabilistic reversal learning task. Behaviorally, theta tACS induced a significant reduction in accuracy for the stable environment, but not for the volatile environment, relative to the sham condition. Computationally, we implemented a combination of a hierarchical Bayesian learning and a decision model. Theta tACS induced a significant increase in low-level (i.e. probability-level) learning rate and uncertainty of low-level estimation relative to sham condition. Instead, the temperature parameter of the decision model, which represents (inverse) decision noise, was not significantly altered due to theta stimulation. These results indicate that theta frequency may modulate the (low-level) learning rate. Furthermore, environmental features (e.g. its stability) may determine whether learning is optimized as a result.

How much data do we need to estimate computational models of decision-making? The COMPASS toolbox.

How much data are needed to obtain useful parameter estimations from a computational model? The standard approach to address this question is to carry out a goodness-of-recovery study. Here, the correlation between individual-participant true and estimated parameter values determines when a sample size is large enough. However, depending on one's research question, this approach may be suboptimal, potentially leading to sample sizes that are either too small (underpowered) or too large (overcostly or unfeasible). In this paper, we formulate a generalized concept of statistical power and use this to propose a novel approach toward determining how much data is needed to obtain useful parameter estimates from a computational model. We describe a Python-based toolbox (COMPASS) that allows one to determine how many participants are needed to fit one specific computational model, namely the Rescorla-Wagner model of learning and decision-making. Simulations revealed that a high number of trials per person (more than the number of persons) are a prerequisite for high-powered studies in this particular setting.

Thunderstruck: The ACDC model of flexible sequences and rhythms in recurrent neural circuits.

Adaptive sequential behavior is a hallmark of human cognition. In particular, humans can learn to produce precise spatiotemporal sequences given a certain context. For instance, musicians can not only reproduce learned action sequences in a context-dependent manner, they can also quickly and flexibly reapply them in any desired tempo or rhythm without overwriting previous learning. Existing neural network models fail to account for these properties. We argue that this limitation emerges from the fact that sequence information (i.e., the position of the action) and timing (i.e., the moment of response execution) are typically stored in the same neural network weights. Here, we augment a biologically plausible recurrent neural network of cortical dynamics to include a basal ganglia-thalamic module which uses reinforcement learning to dynamically modulate action. This "associative cluster-dependent chain" (ACDC) model modularly stores sequence and timing information in distinct loci of the network. This feature increases computational power and allows ACDC to display a wide range of temporal properties (e.g., multiple sequences, temporal shifting, rescaling, and compositionality), while still accounting for several behavioral and neurophysiological empirical observations. Finally, we apply this ACDC network to show how it can learn the famous "Thunderstruck" song intro and then flexibly play it in a "bossa nova" rhythm without further training.

Obsessive-compulsive disorder is characterized by decreased Pavlovian influence on instrumental behavior.

Obsessive-compulsive disorder (OCD) is characterized by uncontrollable repetitive actions thought to rely on abnormalities within fundamental instrumental learning systems. We investigated cognitive and computational mechanisms underlying Pavlovian biases on instrumental behavior in both clinical OCD patients and healthy controls using a Pavlovian-Instrumental Transfer (PIT) task. PIT is typically evidenced by increased responding in the presence of a positive (previously rewarded) Pavlovian cue, and reduced responding in the presence of a negative cue. Thirty OCD patients and thirty-one healthy controls completed the Pavlovian Instrumental Transfer test, which included instrumental training, Pavlovian training for positive, negative and neutral cues, and a PIT phase in which participants performed the instrumental task in the presence of the Pavlovian cues. Modified Rescorla-Wagner models were fitted to trial-by-trial data of participants to estimate underlying computational mechanism and quantify individual differences during training and transfer stages. Bayesian hierarchical methods were used to estimate free parameters and compare the models. Behavioral and computational results indicated a weaker Pavlovian influence on instrumental behavior in OCD patients than in HC, especially for negative Pavlovian cues. Our results contrast with the increased PIT effects reported for another set of disorders characterized by compulsivity, substance use disorders, in which PIT is enhanced. A possible reason for the reduced PIT in OCD may be impairment in using the contextual information provided by the cues to appropriately adjust behavior, especially when inhibiting responding when a negative cue is present. This study provides deeper insight into our understanding of deficits in OCD from the perspective of Pavlovian influences on instrumental behavior and may have implications for OCD treatment modalities focused on reducing compulsive behaviors.

The neural substrates of how model-based learning affects risk taking: Functional coupling between right cerebellum and left caudate.

Higher executive control capacity allows people to appropriately evaluate risk and avoid both excessive risk aversion and excessive risk-taking. The neural mechanisms underlying this relationship between executive function and risk taking are still unknown. We used voxel-based morphometry (VBM) analysis combined with resting-state functional connectivity (rs-FC) to evaluate how one component of executive function, model-based learning, relates to risk taking. We measured individuals' use of the model-based learning system with the two-step task, and risk taking with the Balloon Analogue Risk Task. Behavioral results indicated that risk taking was positively correlated with the model-based weighting parameter ω. The VBM results showed a positive association between model-based learning and gray matter volume in the right cerebellum (RCere) and left inferior parietal lobule (LIPL). Functional connectivity results suggested that the coupling between RCere and the left caudate (LCAU) was correlated with both model-based learning and risk taking. Mediation analysis indicated that RCere-LCAU functional connectivity completely mediated the effect of model-based learning on risk taking. These results indicate that learners who favor model-based strategies also engage in more appropriate risky behaviors through interactions between reward-based learning, error-based learning and executive control subserved by a caudate, cerebellar and parietal network.

Electrophysiological Signatures of Hierarchical Learning.

Human perception and learning is thought to rely on a hierarchical generative model that is continuously updated via precision-weighted prediction errors (pwPEs). However, the neural basis of such cognitive process and how it unfolds during decision-making remain poorly understood. To investigate this question, we combined a hierarchical Bayesian model (i.e., Hierarchical Gaussian Filter [HGF]) with electroencephalography (EEG), while participants performed a probabilistic reversal learning task in alternatingly stable and volatile environments. Behaviorally, the HGF fitted significantly better than two control, nonhierarchical, models. Neurally, low-level and high-level pwPEs were independently encoded by the P300 component. Low-level pwPEs were reflected in the theta (4-8 Hz) frequency band, but high-level pwPEs were not. Furthermore, the expressions of high-level pwPEs were stronger for participants with better HGF fit. These results indicate that the brain employs hierarchical learning and encodes both low- and high-level learning signals separately and adaptively.

Learning to Synchronize: Midfrontal Theta Dynamics during Rule Switching.

In recent years, several hierarchical extensions of well-known learning algorithms have been proposed. For example, when stimulus-action mappings vary across time or context, the brain may learn two or more stimulus-action mappings in separate modules, and additionally (at a hierarchically higher level) learn to appropriately switch between those modules. However, how the brain mechanistically coordinates neural communication to implement such hierarchical learning remains unknown. Therefore, the current study tests a recent computational model that proposed how midfrontal theta oscillations implement such hierarchical learning via the principle of binding by synchrony (Sync model). More specifically, the Sync model uses bursts at theta frequency to flexibly bind appropriate task modules by synchrony. The 64-channel EEG signal was recorded while 27 human subjects (female: 21, male: 6) performed a probabilistic reversal learning task. In line with the Sync model, postfeedback theta power showed a linear relationship with negative prediction errors, but not with positive prediction errors. This relationship was especially pronounced for subjects with better behavioral fit (measured via Akaike information criterion) of the Sync model. Also consistent with Sync model simulations, theta phase-coupling between midfrontal electrodes and temporoparietal electrodes was stronger after negative feedback. Our data suggest that the brain uses theta power and synchronization for flexibly switching between task rule modules, as is useful, for example, when multiple stimulus-action mappings must be retained and used.SIGNIFICANCE STATEMENT Everyday life requires flexibility in switching between several rules. A key question in understanding this ability is how the brain mechanistically coordinates such switches. The current study tests a recent computational framework (Sync model) that proposed how midfrontal theta oscillations coordinate activity in hierarchically lower task-related areas. In line with predictions of this Sync model, midfrontal theta power was stronger when rule switches were most likely (strong negative prediction error), especially in subjects who obtained a better model fit. Additionally, also theta phase connectivity between midfrontal and task-related areas was increased after negative feedback. Thus, the data provided support for the hypothesis that the brain uses theta power and synchronization for flexibly switching between rules.

Signed Reward Prediction Errors in the Ventral Striatum Drive Episodic Memory.

Recent behavioral evidence implicates reward prediction errors (RPEs) as a key factor in the acquisition of episodic memory. Yet, important neural predictions related to the role of RPEs in episodic memory acquisition remain to be tested. Humans (both sexes) performed a novel variable-choice task where we experimentally manipulated RPEs and found support for key neural predictions with fMRI. Our results show that in line with previous behavioral observations, episodic memory accuracy increases with the magnitude of signed (i.e., better/worse-than-expected) RPEs (SRPEs). Neurally, we observe that SRPEs are encoded in the ventral striatum (VS). Crucially, we demonstrate through mediation analysis that activation in the VS mediates the experimental manipulation of SRPEs on episodic memory accuracy. In particular, SRPE-based responses in the VS (during learning) predict the strength of subsequent episodic memory (during recollection). Furthermore, functional connectivity between task-relevant processing areas (i.e., face-selective areas) and hippocampus and ventral striatum increased as a function of RPE value (during learning), suggesting a central role of these areas in episodic memory formation. Our results consolidate reinforcement learning theory and striatal RPEs as key factors subtending the formation of episodic memory.SIGNIFICANCE STATEMENT Recent behavioral research has shown that reward prediction errors (RPEs), a key concept of reinforcement learning theory, are crucial to the formation of episodic memories. In this study, we reveal the neural underpinnings of this process. Using fMRI, we show that signed RPEs (SRPEs) are encoded in the ventral striatum (VS), and crucially, that SRPE VS activity is responsible for the subsequent recollection accuracy of one-shot learned episodic memory associations.

Outcome Value and Task Aversiveness Impact Task Procrastination through Separate Neural Pathways.

The temporal decision model of procrastination has proposed that outcome value and task aversiveness are two separate aspects accounting for procrastination. If true, the human brain is likely to implicate separate neural pathways to mediate the effect of outcome value and task aversiveness on procrastination. Outcome value is plausibly constructed via a hippocampus-based pathway because of the hippocampus's unique role in episodic prospection. In contrast, task aversiveness might be represented through an amygdala-involved pathway. In the current study, participants underwent fMRI scanning when viewing both tasks and future outcomes, without any experimental instruction imposed. The results revealed that outcome value increased activations in the caudate, and suppressed procrastination through a hippocampus-caudate pathway. In contrast, task aversiveness increased activations in the anterior insula, and increased procrastination via an amygdala-insula pathway. In sum, this study demonstrates that people can incorporate both outcome value and task aversiveness into task valuation to decide whether to procrastinate or not; and it elucidates the separate neural pathways via which this occurs.

Spatial Attention in Serial Order Working Memory: An EEG Study.

Theoretical models explaining serial order processing link order information to specified position markers. However, the precise characteristics of position marking have remained largely elusive. Recent studies have shown that space is involved in marking serial position of items in verbal working memory (WM). Furthermore, it has been suggested, but not proven, that accessing these items involves horizontal shifts of spatial attention. We used continuous electroencephalography recordings to show that memory search in serial order verbal WM involves spatial attention processes that share the same electrophysiological signatures as those operating on the visuospatial WM and external space. Accessing an item from a sequence in verbal WM induced posterior "early directing attention negativity" and "anterior directing attention negativity" contralateral to the position of the item in mental space (i.e., begin items on the left; end items on the right). In the frequency domain, we observed posterior alpha suppression contralateral to the position of the item. Our results provide clear evidence for the involvement of spatial attention in retrieving serial information from verbal WM. Implications for WM models are discussed.

Cognitive Effort Modulates Connectivity between Dorsal Anterior Cingulate Cortex and Task-Relevant Cortical Areas.

Investment of cognitive effort is required in everyday life and has received ample attention in recent neurocognitive frameworks. The neural mechanism of effort investment is thought to be structured hierarchically, with dorsal anterior cingulate cortex (dACC) at the highest level, recruiting task-specific upstream areas. In the current fMRI study, we tested whether dACC is generally active when effort demand is high across tasks with different stimuli, and whether connectivity between dACC and task-specific areas is increased depending on the task requirements and effort level at hand. For that purpose, a perceptual detection task was administered that required male and female human participants to detect either a face or a house in a noisy image. Effort demand was manipulated by adding little (low effort) or much (high effort) noise to the images. Results showed a network of dACC, anterior insula (AI), and intraparietal sulcus (IPS) to be more active when effort demand was high, independent of the performed task (face or house detection). Importantly, effort demand modulated functional connectivity between dACC and face-responsive or house-responsive perceptual areas, depending on the task at hand. This shows that dACC, AI, and IPS constitute a general effort-responsive network and suggests that the neural implementation of cognitive effort involves dACC-initiated sensitization of task-relevant areas.SIGNIFICANCE STATEMENT Although cognitive effort is generally perceived as aversive, its investment is inevitable when navigating an increasingly complex society. In this study, we demonstrate how the human brain tailors the implementation of effort to the requirements of the task at hand. We show increased effort-related activity in a network of brain areas consisting of dorsal anterior cingulate cortex (dACC), anterior insula, and intraparietal sulcus, independent of task specifics. Crucially, we also show that effort-induced functional connectivity between dACC and task-relevant areas tracks specific task demands. These results demonstrate how brain regions specialized to solve a task may be energized by dACC when effort demand is high.

Neural Synchrony for Adaptive Control.

Cognitive control can be adaptive along several dimensions, including intensity (how intensely do control signals influence bottom-up processing) and selectivity (what information is selected for further processing). Furthermore, control can be exerted along slow or fast time scales. Whereas control on a slow time scale is used to proactively prepare for upcoming challenges, control can also be used on a faster time scale to react to unexpected events that require control. Importantly, a systematic comparison of these dimensions and time scales remains lacking. Moreover, most current models of adaptive control allow predictions only at a behavioral, not neurophysiological, level, thus seriously reducing the range of available empirical restrictions for informing model formulation. The current article addresses this issue by implementing a control loop in an earlier model of neural synchrony. The resulting model is tested on a Stroop task. We observe that only the model that exerts cognitive control on intensity and selectivity dimensions, as well as on two time scales, can account for relevant behavioral and neurophysiological data. Our findings hold important implications for both cognitive control and how computational models can be empirically constrained.

Theta and alpha power across fast and slow timescales in cognitive control.

Theta and alpha frequency neural oscillations are important for learning and cognitive control, but their exact role has remained obscure. In particular, it is unknown whether they operate at similar timescales, and whether they support different cognitive processes. We recorded EEG in 30 healthy human participants while they performed a learning task containing both novel (block-unique) and repeating stimuli. We investigated behavior and electrophysiology at both fast (i.e., within blocks) and slow (i.e., between blocks) timescales. Behaviorally, both response time and accuracy improved (respectively decrease and increase) over both fast and slow timescales. However, on the spectral level, theta power significantly decreased along the slow timescale, whereas alpha power significantly increased along the fast timescale. We thus demonstrate that theta and alpha both play a role during learning, but operate at different timescales. This result poses important empirical constraints for theories on learning, cognitive control, and neural oscillations.

Human midcingulate cortex encodes distributed representations of task progress.

The function of midcingulate cortex (MCC) remains elusive despite decades of investigation and debate. Complicating matters, individual MCC neurons respond to highly diverse task-related events, and MCC activation is reported in most human neuroimaging studies employing a wide variety of task manipulations. Here we investigate this issue by applying a model-based cognitive neuroscience approach involving neural network simulations, functional magnetic resonance imaging, and representational similarity analysis. We demonstrate that human MCC encodes distributed, dynamically evolving representations of extended, goal-directed action sequences. These representations are uniquely sensitive to the stage and identity of each sequence, indicating that MCC sustains contextual information necessary for discriminating between task states. These results suggest that standard univariate approaches for analyzing MCC function overlook the major portion of task-related information encoded by this brain area and point to promising new avenues for investigation.

A Postdecisional Neural Marker of Confidence Predicts Information-Seeking in Decision-Making.

Theoretical work predicts that decisions made with low confidence should lead to increased information-seeking. This is an adaptive strategy because it can increase the quality of a decision, and previous behavioral work has shown that decision-makers engage in such confidence-driven information-seeking. The present study aimed to characterize the neural markers that mediate the relationship between confidence and information-seeking. A paradigm was used in which 17 human participants (9 male) made an initial perceptual decision, and then decided whether or not they wanted to sample more evidence before committing to a final decision and confidence judgment. Predecisional and postdecisional event-related potential components were similarly modulated by the level of confidence and by information-seeking choices. Time-resolved multivariate decoding of scalp EEG signals first revealed that both information-seeking choices and decision confidence could be decoded from the time of the initial decision to the time of the subsequent information-seeking choice (within-condition decoding). No above-chance decoding was visible in the preresponse time window. Crucially, a classifier trained to decode high versus low confidence predicted information-seeking choices after the initial perceptual decision (across-condition decoding). This time window corresponds to that of a postdecisional neural marker of confidence. Collectively, our findings demonstrate, for the first time, that neural indices of confidence are functionally involved in information-seeking decisions.SIGNIFICANCE STATEMENT Despite substantial current interest in neural signatures of our sense of confidence, it remains largely unknown how confidence is used to regulate behavior. Here, we devised a task in which human participants could decide whether or not to sample additional decision-relevant information at a small monetary cost. Using neural recordings, we could predict such information-seeking choices based on a neural signature of decision confidence. Our study illuminates a neural link between decision confidence and adaptive behavioral control.

Learning to synchronize: How biological agents can couple neural task modules for dealing with the stability-plasticity dilemma.

We provide a novel computational framework on how biological and artificial agents can learn to flexibly couple and decouple neural task modules for cognitive processing. In this way, they can address the stability-plasticity dilemma. For this purpose, we combine two prominent computational neuroscience principles, namely Binding by Synchrony and Reinforcement Learning. The model learns to synchronize task-relevant modules, while also learning to desynchronize currently task-irrelevant modules. As a result, old (but currently task-irrelevant) information is protected from overwriting (stability) while new information can be learned quickly in currently task-relevant modules (plasticity). We combine learning to synchronize with task modules that learn via one of several classical learning algorithms (Rescorla-Wagner, backpropagation, Boltzmann machines). The resulting combined model is tested on a reversal learning paradigm where it must learn to switch between three different task rules. We demonstrate that our combined model has significant computational advantages over the original network without synchrony, in terms of both stability and plasticity. Importantly, the resulting models' processing dynamics are also consistent with empirical data and provide empirically testable hypotheses for future MEG/EEG studies.

Continuous track paths reveal additive evidence integration in multistep decision making.

Multistep decision making pervades daily life, but its underlying mechanisms remain obscure. We distinguish four prominent models of multistep decision making, namely serial stage, hierarchical evidence integration, hierarchical leaky competing accumulation (HLCA), and probabilistic evidence integration (PEI). To empirically disentangle these models, we design a two-step reward-based decision paradigm and implement it in a reaching task experiment. In a first step, participants choose between two potential upcoming choices, each associated with two rewards. In a second step, participants choose between the two rewards selected in the first step. Strikingly, as predicted by the HLCA and PEI models, the first-step decision dynamics were initially biased toward the choice representing the highest sum/mean before being redirected toward the choice representing the maximal reward (i.e., initial dip). Only HLCA and PEI predicted this initial dip, suggesting that first-step decision dynamics depend on additive integration of competing second-step choices. Our data suggest that potential future outcomes are progressively unraveled during multistep decision making.

Oscillatory signatures of reward prediction errors in declarative learning.

Reward prediction errors (RPEs) are crucial to learning. Whereas these mismatches between reward expectation and reward outcome are known to drive procedural learning, their role in declarative learning remains underexplored. Earlier work from our lab addressed this, and consistently found that signed reward prediction errors (SRPEs; "better-than-expected" signals) boost declarative learning. In the current EEG study, we sought to explore the neural signatures of SRPEs. Participants studied 60 Dutch-Swahili word pairs while RPE magnitudes were parametrically manipulated. Behaviorally, we replicated our previous findings that SRPEs drive declarative learning, with increased recognition for word pairs accompanied by large, positive RPEs. In the EEG data, at the start of reward feedback processing, we found an oscillatory (theta) signature consistent with unsigned reward prediction errors (URPEs; "different-than-expected" signals). Slightly later during reward feedback processing, we observed oscillatory (high-beta and high-alpha) signatures for SRPEs during reward feedback, similar to SRPE signatures during procedural learning. These findings illuminate the time course of neural oscillations in processing reward during declarative learning, providing important constraints for future theoretical work.