Humans adaptively select different computational strategies in different learning environments.

The Rescorla-Wagner rule remains the most popular tool to describe human behavior in reinforcement learning tasks. Nevertheless, it cannot fit human learning in complex environments. Previous work proposed several hierarchical extensions of this learning rule. However, it remains unclear when a flat (nonhierarchical) versus a hierarchical strategy is adaptive, or when it is implemented by humans. To address this question, current work applies a nested modeling approach to evaluate multiple models in multiple reinforcement learning environments both computationally (which approach performs best) and empirically (which approach fits human data best). We consider 10 empirical data sets (N = 407) divided over three reinforcement learning environments. Our results demonstrate that different environments are best solved with different learning strategies; and that humans adaptively select the learning strategy that allows best performance. Specifically, while flat learning fitted best in less complex stable learning environments, humans employed more hierarchically complex models in more complex environments. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Learning environment-specific learning rates.

People often have to switch back and forth between different environments that come with different problems and volatilities. While volatile environments require fast learning (i.e., high learning rates), stable environments call for lower learning rates. Previous studies have shown that people adapt their learning rates, but it remains unclear whether they can also learn about environment-specific learning rates, and instantaneously retrieve them when revisiting environments. Here, using optimality simulations and hierarchical Bayesian analyses across three experiments, we show that people can learn to use different learning rates when switching back and forth between two different environments. We even observe a signature of these environment-specific learning rates when the volatility of both environments is suddenly the same. We conclude that humans can flexibly adapt and learn to associate different learning rates to different environments, offering important insights for developing theories of meta-learning and context-specific control.

Manipulating Prior Beliefs Causally Induces Under- and Overconfidence.

Humans differ vastly in the confidence they assign to decisions. Although such under- and overconfidence relate to fundamental life outcomes, a computational account specifying the underlying mechanisms is currently lacking. We propose that prior beliefs in the ability to perform a task explain confidence differences across participants and tasks, despite similar performance. In two perceptual decision-making experiments, we show that manipulating prior beliefs about performance during training causally influences confidence in healthy adults (N = 50 each; Experiment 1: 8 men, one nonbinary; Experiment 2: 5 men) during a test phase, despite unaffected objective performance. This is true when prior beliefs are induced via manipulated comparative feedback and via manipulated training-phase difficulty. Our results were accounted for within an accumulation-to-bound model, explicitly modeling prior beliefs on the basis of earlier task exposure. Decision confidence is quantified as the probability of being correct conditional on prior beliefs, causing under- or overconfidence. We provide a fundamental mechanistic insight into the computations underlying under- and overconfidence.

How much data do we need to estimate computational models of decision-making? The COMPASS toolbox.

How much data are needed to obtain useful parameter estimations from a computational model? The standard approach to address this question is to carry out a goodness-of-recovery study. Here, the correlation between individual-participant true and estimated parameter values determines when a sample size is large enough. However, depending on one's research question, this approach may be suboptimal, potentially leading to sample sizes that are either too small (underpowered) or too large (overcostly or unfeasible). In this paper, we formulate a generalized concept of statistical power and use this to propose a novel approach toward determining how much data is needed to obtain useful parameter estimates from a computational model. We describe a Python-based toolbox (COMPASS) that allows one to determine how many participants are needed to fit one specific computational model, namely the Rescorla-Wagner model of learning and decision-making. Simulations revealed that a high number of trials per person (more than the number of persons) are a prerequisite for high-powered studies in this particular setting.

Learning where to be flexible: Using environmental cues to regulate cognitive control.

Cognitive flexibility refers to a mental state that allows efficient switching between tasks. While deciding to be flexible is often ascribed to a strategic resource-intensive executive process, people may also simply use their environment to trigger different states of cognitive flexibility. We developed a paradigm where participants were exposed to two environments with different task-switching probabilities, followed by a probe phase to test the impact of environmental cues. Our results show that people were more efficient at switching in a high-switch environment. Critically, we observe environment-specific triggering of cognitive flexibility after a 4-day training period (Experiment 2, N = 51), but not after a 1-day training period (Experiment 1, N = 52). Together, these findings suggest that people can associate the need for cognitive flexibility with their environment, providing an environmental triggering mechanism for cognitive control. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

The neural substrates of how model-based learning affects risk taking: Functional coupling between right cerebellum and left caudate.

Higher executive control capacity allows people to appropriately evaluate risk and avoid both excessive risk aversion and excessive risk-taking. The neural mechanisms underlying this relationship between executive function and risk taking are still unknown. We used voxel-based morphometry (VBM) analysis combined with resting-state functional connectivity (rs-FC) to evaluate how one component of executive function, model-based learning, relates to risk taking. We measured individuals' use of the model-based learning system with the two-step task, and risk taking with the Balloon Analogue Risk Task. Behavioral results indicated that risk taking was positively correlated with the model-based weighting parameter ω. The VBM results showed a positive association between model-based learning and gray matter volume in the right cerebellum (RCere) and left inferior parietal lobule (LIPL). Functional connectivity results suggested that the coupling between RCere and the left caudate (LCAU) was correlated with both model-based learning and risk taking. Mediation analysis indicated that RCere-LCAU functional connectivity completely mediated the effect of model-based learning on risk taking. These results indicate that learners who favor model-based strategies also engage in more appropriate risky behaviors through interactions between reward-based learning, error-based learning and executive control subserved by a caudate, cerebellar and parietal network.

Modulating hierarchical learning by high-definition transcranial alternating current stimulation at theta frequency.

Considerable evidence highlights the dorsolateral prefrontal cortex (DLPFC) as a key region for hierarchical (i.e. multilevel) learning. In a previous electroencephalography (EEG) study, we found that the low-level prediction errors were encoded by frontal theta oscillations (4-7 Hz), centered on right DLPFC (rDLPFC). However, the causal relationship between frontal theta oscillations and hierarchical learning remains poorly understood. To investigate this question, in the current study, participants received theta (6 Hz) and sham high-definition transcranial alternating current stimulation (HD-tACS) over the rDLPFC while performing the probabilistic reversal learning task. Behaviorally, theta tACS induced a significant reduction in accuracy for the stable environment, but not for the volatile environment, relative to the sham condition. Computationally, we implemented a combination of a hierarchical Bayesian learning and a decision model. Theta tACS induced a significant increase in low-level (i.e. probability-level) learning rate and uncertainty of low-level estimation relative to sham condition. Instead, the temperature parameter of the decision model, which represents (inverse) decision noise, was not significantly altered due to theta stimulation. These results indicate that theta frequency may modulate the (low-level) learning rate. Furthermore, environmental features (e.g. its stability) may determine whether learning is optimized as a result.

Obsessive-compulsive disorder is characterized by decreased Pavlovian influence on instrumental behavior.

Obsessive-compulsive disorder (OCD) is characterized by uncontrollable repetitive actions thought to rely on abnormalities within fundamental instrumental learning systems. We investigated cognitive and computational mechanisms underlying Pavlovian biases on instrumental behavior in both clinical OCD patients and healthy controls using a Pavlovian-Instrumental Transfer (PIT) task. PIT is typically evidenced by increased responding in the presence of a positive (previously rewarded) Pavlovian cue, and reduced responding in the presence of a negative cue. Thirty OCD patients and thirty-one healthy controls completed the Pavlovian Instrumental Transfer test, which included instrumental training, Pavlovian training for positive, negative and neutral cues, and a PIT phase in which participants performed the instrumental task in the presence of the Pavlovian cues. Modified Rescorla-Wagner models were fitted to trial-by-trial data of participants to estimate underlying computational mechanism and quantify individual differences during training and transfer stages. Bayesian hierarchical methods were used to estimate free parameters and compare the models. Behavioral and computational results indicated a weaker Pavlovian influence on instrumental behavior in OCD patients than in HC, especially for negative Pavlovian cues. Our results contrast with the increased PIT effects reported for another set of disorders characterized by compulsivity, substance use disorders, in which PIT is enhanced. A possible reason for the reduced PIT in OCD may be impairment in using the contextual information provided by the cues to appropriately adjust behavior, especially when inhibiting responding when a negative cue is present. This study provides deeper insight into our understanding of deficits in OCD from the perspective of Pavlovian influences on instrumental behavior and may have implications for OCD treatment modalities focused on reducing compulsive behaviors.

Computational Investigations of Learning and Synchronization in Cognitive Control.

Complex cognition requires binding together of stimulus, action, and other features, across different time scales. Several implementations of such binding have been proposed in the literature, most prominently synaptic binding (learning) and synchronization. Biologically plausible accounts of how these different types of binding interact in the human brain are still lacking. To this end, we adopt a computational approach to investigate the impact of learning and synchronization on both behavioral (reaction time, error rate) and neural (θ power) measures. We train four models varying in their ability to learn and synchronize for an extended period of time on three seminal action control paradigms varying in difficulty. Learning, but not synchronization, proved essential for behavioral improvement. Synchronization however boosts performance of difficult tasks, avoiding the computational pitfalls of catastrophic interference. At the neural level, θ power decreases with practice but increases with task difficulty. Our simulation results bring new insights in how different types of binding interact in different types of tasks, and how this is translated in both behavioral and neural metrics.

Dynamic influences on static measures of metacognition.

Humans differ in their capability to judge choice accuracy via confidence judgments. Popular signal detection theoretic measures of metacognition, such as M-ratio, do not consider the dynamics of decision making. This can be problematic if response caution is shifted to alter the tradeoff between speed and accuracy. Such shifts could induce unaccounted-for sources of variation in the assessment of metacognition. Instead, evidence accumulation frameworks consider decision making, including the computation of confidence, as a dynamic process unfolding over time. Using simulations, we show a relation between response caution and M-ratio. We then show the same pattern in human participants explicitly instructed to focus on speed or accuracy. Finally, this association between M-ratio and response caution is also present across four datasets without any reference towards speed. In contrast, when data are analyzed with a dynamic measure of metacognition, v-ratio, there is no effect of speed-accuracy tradeoff.

Imbalance between the caudate and putamen connectivity in obsessive-compulsive disorder.

BACKGROUND: Compulsive behaviors in obsessive-compulsive disorder (OCD) have been suggested to result from an imbalance in cortico-striatal connectivity. However, the nature of this impairment, the relative involvement of different striatal areas, their imbalance in genetically related but unimpaired individuals, and their relationship with cognitive dysfunction in OCD patients, remain unknown. METHODS: In the current study, striatal (i.e., caudate and putamen) whole-brain connectivity was computed in a sample of OCD patients (OCD, n = 62), unaffected first-degree relatives (UFDR, n = 53) and healthy controls (HC, n = 73) by ROI-based resting-state functional magnetic resonance imaging (rs-fMRI). A behavioral task switch paradigm outside of the scanner was also performed to measure cognitive flexibility in OCD patients. RESULTS: There were significantly increased strengths (Z-transformed Pearson correlation coefficient) in caudate connectivity in OCD patients. A significant correlation between the two types of connectivity strengths in the relevant regions was observed only in the OCD patient group. Furthermore, the caudate connectivity of patients was negatively associated with their task-switch performance. CONCLUSIONS: The imbalance between the caudate and putamen connectivity, arising from the abnormal increase of caudate activity, may serve as a clinical characteristic for obsessive-compulsive disorder.

Theta oscillations shift towards optimal frequency for cognitive control.

Cognitive control allows to flexibly guide behaviour in a complex and ever-changing environment. It is supported by theta band (4-7 Hz) neural oscillations that coordinate distant neural populations. However, little is known about the precise neural mechanisms permitting such flexible control. Most research has focused on theta amplitude, showing that it increases when control is needed, but a second essential aspect of theta oscillations, their peak frequency, has mostly been overlooked. Here, using computational modelling and behavioural and electrophysiological recordings, in three independent datasets, we show that theta oscillations adaptively shift towards optimal frequency depending on task demands. We provide evidence that theta frequency balances reliable set-up of task representation and gating of task-relevant sensory and motor information and that this frequency shift predicts behavioural performance. Our study presents a mechanism supporting flexible control and calls for a reevaluation of the mechanistic role of theta oscillations in adaptive behaviour.

Thunderstruck: The ACDC model of flexible sequences and rhythms in recurrent neural circuits.

Adaptive sequential behavior is a hallmark of human cognition. In particular, humans can learn to produce precise spatiotemporal sequences given a certain context. For instance, musicians can not only reproduce learned action sequences in a context-dependent manner, they can also quickly and flexibly reapply them in any desired tempo or rhythm without overwriting previous learning. Existing neural network models fail to account for these properties. We argue that this limitation emerges from the fact that sequence information (i.e., the position of the action) and timing (i.e., the moment of response execution) are typically stored in the same neural network weights. Here, we augment a biologically plausible recurrent neural network of cortical dynamics to include a basal ganglia-thalamic module which uses reinforcement learning to dynamically modulate action. This "associative cluster-dependent chain" (ACDC) model modularly stores sequence and timing information in distinct loci of the network. This feature increases computational power and allows ACDC to display a wide range of temporal properties (e.g., multiple sequences, temporal shifting, rescaling, and compositionality), while still accounting for several behavioral and neurophysiological empirical observations. Finally, we apply this ACDC network to show how it can learn the famous "Thunderstruck" song intro and then flexibly play it in a "bossa nova" rhythm without further training.

Electrophysiological Signatures of Hierarchical Learning.

Human perception and learning is thought to rely on a hierarchical generative model that is continuously updated via precision-weighted prediction errors (pwPEs). However, the neural basis of such cognitive process and how it unfolds during decision-making remain poorly understood. To investigate this question, we combined a hierarchical Bayesian model (i.e., Hierarchical Gaussian Filter [HGF]) with electroencephalography (EEG), while participants performed a probabilistic reversal learning task in alternatingly stable and volatile environments. Behaviorally, the HGF fitted significantly better than two control, nonhierarchical, models. Neurally, low-level and high-level pwPEs were independently encoded by the P300 component. Low-level pwPEs were reflected in the theta (4-8 Hz) frequency band, but high-level pwPEs were not. Furthermore, the expressions of high-level pwPEs were stronger for participants with better HGF fit. These results indicate that the brain employs hierarchical learning and encodes both low- and high-level learning signals separately and adaptively.

Neural Representations of Task Context and Temporal Order During Action Sequence Execution.

Routine action sequences can share a great deal of similarity in terms of their stimulus response mappings. As a consequence, their correct execution relies crucially on the ability to preserve contextual and temporal information. However, there are few empirical studies on the neural mechanism and the brain areas maintaining such information. To address this gap in the literature, we recently recorded the blood-oxygen level dependent (BOLD) response in a newly developed coffee-tea making task. The task involves the execution of four action sequences that each comprise six consecutive decision states, which allows for examining the maintenance of contextual and temporal information. Here, we report a reanalysis of this dataset using a data-driven approach, namely multivariate pattern analysis, that examines context-dependent neural activity across several predefined regions of interest. Results highlight involvement of the inferior-temporal gyrus and lateral prefrontal cortex in maintaining temporal and contextual information for the execution of hierarchically organized action sequences. Furthermore, temporal information seems to be more strongly encoded in areas over the left hemisphere.

Using top-down modulation to optimally balance shared versus separated task representations.

Human adaptive behavior requires continually learning and performing a wide variety of tasks, often with very little practice. To accomplish this, it is crucial to separate neural representations of different tasks in order to avoid interference. At the same time, sharing neural representations supports generalization and allows faster learning. Therefore, a crucial challenge is to find an optimal balance between shared versus separated representations. Typically, models of human cognition employ top-down modulatory signals to separate task representations, but there exist surprisingly little systematic computational investigations of how such modulation is best implemented. We identify and systematically evaluate two crucial features of modulatory signals. First, top-down input can be processed in an additive or multiplicative manner. Second, the modulatory signals can be adaptive (learned) or non-adaptive (random). We cross these two features, resulting in four modulation networks which are tested on a variety of input datasets and tasks with different degrees of stimulus-action mapping overlap. The multiplicative adaptive modulation network outperforms all other networks in terms of accuracy. Moreover, this network develops hidden units that optimally share representations between tasks. Specifically, different than the binary approach of currently popular latent state models, it exploits partial overlap between tasks.

Aberrant rich club organization in patients with obsessive-compulsive disorder and their unaffected first-degree relatives.

Recent studies suggested that the rich club organization promoting global brain communication and integration of information, may be abnormally increased in obsessive-compulsive disorder (OCD). However, the structural and functional basis of this organization is still not very clear. Given the heritability of OCD, as suggested by previous family-based studies, we hypothesize that aberrant rich club organization may be a trait marker for OCD. In the present study, 32 patients with OCD, 30 unaffected first-degree relatives (FDR) and 32 healthy controls (HC) underwent diffusion tensor imaging (DTI) and functional magnetic resonance imaging (fMRI). We examined the structural rich club organization and its interrelationship with functional coupling. Our results showed that rich club and peripheral connection strength in patients with OCD was lower than in HC, while it was intermediate in FDR. Finally, the coupling between structural and functional connections of the rich club, was decreased in FDR but not in OCD relative to HC, which suggests a buffering mechanism of brain functions in FDR. Overall, our findings suggest that alteration of the rich club organization may reflect a vulnerability biomarker for OCD, possibly buffered by structural and functional coupling of the rich club.

Neural Synchrony for Adaptive Control.

Cognitive control can be adaptive along several dimensions, including intensity (how intensely do control signals influence bottom-up processing) and selectivity (what information is selected for further processing). Furthermore, control can be exerted along slow or fast time scales. Whereas control on a slow time scale is used to proactively prepare for upcoming challenges, control can also be used on a faster time scale to react to unexpected events that require control. Importantly, a systematic comparison of these dimensions and time scales remains lacking. Moreover, most current models of adaptive control allow predictions only at a behavioral, not neurophysiological, level, thus seriously reducing the range of available empirical restrictions for informing model formulation. The current article addresses this issue by implementing a control loop in an earlier model of neural synchrony. The resulting model is tested on a Stroop task. We observe that only the model that exerts cognitive control on intensity and selectivity dimensions, as well as on two time scales, can account for relevant behavioral and neurophysiological data. Our findings hold important implications for both cognitive control and how computational models can be empirically constrained.

Reward prediction errors drive declarative learning irrespective of agency.

Recent years have witnessed a steady increase in the number of studies investigating the role of reward prediction errors (RPEs) in declarative learning. Specifically, in several experimental paradigms, RPEs drive declarative learning, with larger and more positive RPEs enhancing declarative learning. However, it is unknown whether this RPE must derive from the participant's own response, or whether instead, any RPE is sufficient to obtain the learning effect. To test this, we generated RPEs in the same experimental paradigm where we combined an agency and a nonagency condition. We observed no interaction between RPE and agency, suggesting that any RPE (irrespective of its source) can drive declarative learning. This result holds implications for declarative learning theory.

Theta and alpha power across fast and slow timescales in cognitive control.

Theta and alpha frequency neural oscillations are important for learning and cognitive control, but their exact role has remained obscure. In particular, it is unknown whether they operate at similar timescales, and whether they support different cognitive processes. We recorded EEG in 30 healthy human participants while they performed a learning task containing both novel (block-unique) and repeating stimuli. We investigated behavior and electrophysiology at both fast (i.e., within blocks) and slow (i.e., between blocks) timescales. Behaviorally, both response time and accuracy improved (respectively decrease and increase) over both fast and slow timescales. However, on the spectral level, theta power significantly decreased along the slow timescale, whereas alpha power significantly increased along the fast timescale. We thus demonstrate that theta and alpha both play a role during learning, but operate at different timescales. This result poses important empirical constraints for theories on learning, cognitive control, and neural oscillations.