Surprise-minimization as a solution to the structural credit assignment problem.

The structural credit assignment problem arises when the causal structure between actions and subsequent outcomes is hidden from direct observation. To solve this problem and enable goal-directed behavior, an agent has to infer structure and form a representation thereof. In the scope of this study, we investigate a possible solution in the human brain. We recorded behavioral and electrophysiological data from human participants in a novel variant of the bandit task, where multiple actions lead to multiple outcomes. Crucially, the mapping between actions and outcomes was hidden and not instructed to the participants. Human choice behavior revealed clear hallmarks of credit assignment and learning. Moreover, a computational model which formalizes action selection as the competition between multiple representations of the hidden structure was fit to account for participants data. Starting in a state of uncertainty about the correct representation, the central mechanism of this model is the arbitration of action control towards the representation which minimizes surprise about outcomes. Crucially, single-trial latent-variable analysis reveals that the neural patterns clearly support central quantitative predictions of this surprise minimization model. The results suggest that neural activity is not only related to reinforcement learning under correct as well as incorrect task representations but also reflects central mechanisms of credit assignment and behavioral arbitration.

The electrophysiology of sequential adjustments of dual-task order coordination.

Performing two tasks simultaneously involves the coordination of their processing. This task coordination is particularly required in dual-task situations with varying task orders. When task order switches between subsequent trials, task order coordination leads to task order switch costs in comparison with order repetitions. However, it is open, whether task order coordination is exclusively controlled by the relation of the task orders of the current and the previous trials, or whether additional conditions such as task order before the previous trial leads to a behavioral and neural adjustment of task order coordination. To answer this question, we reanalyzed the data of two previously published experiments with order-cued dual-task paradigms. We did so with regard to whether task order switch costs and the EEG component order-switch positivity in the current dual-task trial would be modulated by order switches vs. repetitions in the previous trial (Trial N-1). In Experiment 1, we found a modulation of the task order switch costs in RTs and response reversals; these costs were reduced after an order switch compared with order repetitions in Trial N-1. In Experiment 2, there were no effects on the task order switch costs in the behavioral data. Nonetheless, we found the order-switch positivity to be strongly modulated by the order transition of the previous trial in both experiments. The order-switch positivity was substantially reduced if the previous trial was an order switch (compared to an order repetition) by itself. This implies that order coordination of dual tasks is adjusted in a gradual way depending on trial's history.

Disentangling task-selection failures from task-execution failures in task switching: an assessment of different paradigms.

Differentiating errors on the basis of the distinct cognitive mechanisms that may have generated them has provided neuropsychologists with useful diagnostic tools. For example, perseverative errors arising from the inability of the patient to set a new criterion for responding are considered one of the hallmarks of cognitive inflexibility. Similarly, in the task-switching paradigm it is possible to distinguish between task-confusion errors, produced by a failure in task selection, and response-confusion errors, arising when the correct task is selected, but the wrong response is given. Nonetheless, only a few studies so far have exploited the existence of different kinds of errors in multitasking situations to inform theories of cognitive flexibility. In the present study, we set out to use a variety of methodologies employed so far in the literature for disentangling errors due to task-selection failure from errors due to task-execution failure. In three experiments, we assessed the capacity of each method to produce error categories that can be mapped as clearly as possible to the cognitive mechanism(s) underlying them using multinomial processing tree modelling. Subsequently, the distinction between task- and response-confusion errors was used to test their differential impact on inhibitory mechanisms in task switching as measured by N-2 repetition costs. Our results are encouraging regarding the possibility of correctly detecting response- and task-selection failures, thus allowing us to assess their differential impact on N-2 repetition costs.

Neural Correlates of Task-order Preparation in Dual Tasks: An EEG Study.

Dual-task scenarios require a coordinated regulation of the processing order of component tasks in light of capacity limitations during response selection. A number of behavioral and neuroimaging findings suggest a distinct set of control processes involved in preparing this task order. In this study, we investigated electrophysiological correlates of task-order preparation in a variant of the overlapping dual-task paradigm with cue-determined task order that resulted in trials with blockwise fixed task order as well as trials with repeated and switched task order in blocks with variable task order. During the cue-stimulus interval, we found an earlier centroparietal order-mixing positivity and a later parietal order-switch positivity. A decoding approach based on multivariate pattern analysis showed that the order-mixing positivity is a necessary prerequisite for successful order selection, whereas the order-switch positivity appears to facilitate the implementation of a new task order after its selection. These correlates of order preparation share striking similarities to commonly found potentials involved in the preparation of individual tasks in the (single-)task-switching paradigm, which is strong empirical support for the account that the underlying preparatory processes are to be considered as higher-level control signals that are implemented independently of specific task representations.

Task Learnability Modulates Surprise but Not Valence Processing for Reinforcement Learning in Probabilistic Choice Tasks.

The goal of temporal difference (TD) reinforcement learning is to maximize outcomes and improve future decision-making. It does so by utilizing a prediction error (PE), which quantifies the difference between the expected and the obtained outcome. In gambling tasks, however, decision-making cannot be improved because of the lack of learnability. On the basis of the idea that TD utilizes two independent bits of information from the PE (valence and surprise), we asked which of these aspects is affected when a task is not learnable. We contrasted behavioral data and ERPs in a learning variant and a gambling variant of a simple two-armed bandit task, in which outcome sequences were matched across tasks. Participants were explicitly informed that feedback could be used to improve performance in the learning task but not in the gambling task, and we predicted a corresponding modulation of the aspects of the PE. We used a model-based analysis of ERP data to extract the neural footprints of the valence and surprise information in the two tasks. Our results revealed that task learnability modulates reinforcement learning via the suppression of surprise processing but leaves the processing of valence unaffected. On the basis of our model and the data, we propose that task learnability can selectively suppress TD learning as well as alter behavioral adaptation based on a flexible cost-benefit arbitration.

Adaptive rescheduling of error monitoring in multitasking.

The concurrent execution of temporally overlapping tasks leads to considerable interference between the subtasks. This also impairs control processes associated with the detection of performance errors. In the present study, we investigated how the human brain adapts to this interference between task representations in such multitasking scenarios. In Experiment 1, participants worked on a dual-tasking paradigm with partially overlapping execution of two tasks (T1 and T2), while we recorded error-related scalp potentials. The error positivity (Pe), a correlate of higher-level error evaluation, was reduced after T1 errors but occurred after a correct T2-response instead. MVPA-based and regression-based single-trial analysis revealed that the immediate Pe and deferred Pe are negatively correlated, suggesting a trial-wise trade-off between immediate and postponed error processing. Experiment 2 confirmed this finding and additionally showed that this result is not due to credit-assignment errors in which a T1 error is falsely attributed to T2. For the first time reporting a Pe that is temporally detached from its eliciting error event by a considerable amount of time, this study illustrates how reliable error detection in dual-tasking is maintained by a mechanism that adaptively schedules error processing, thus demonstrating a remarkable flexibility of the human brain when adapting to multitasking situations.

Cognitive modelling reveals distinct electrophysiological markers of decision confidence and error monitoring.

Is confidence in perceptual decisions generated by the same brain processes as decision itself, or does confidence require metacognitive processes following up on the decision? In a masked orientation task with varying stimulus-onset-asynchrony, we used EEG and cognitive modelling to trace the timing of the neural correlates of confidence. Confidence reported by human observers increased with stimulus-onset-asynchrony in correct and to a lesser degree in incorrect trials, a pattern incompatible with established models of confidence. Electrophysiological activity was associated with confidence in two different time periods, namely 350-500 â€‹ms after stimulus onset and 250-350 â€‹ms after the response. Cognitive modelling revealed that only the activity following on the stimulus exhibited the same statistical regularities as confidence, while the statistical pattern of the activity following the response was incompatible with confidence. It is argued that electrophysiological markers of confidence and error awareness are at least in parts distinct.

The influence of internal models on feedback-related brain activity.

Decision making relies on the interplay between two distinct learning mechanisms, namely habitual model-free learning and goal-directed model-based learning. Recent literature suggests that this interplay is significantly shaped by the environmental structure as represented by an internal model. We employed a modified two-stage but one-decision Markov decision task to investigate how two internal models differing in the predictability of stage transitions influence the neural correlates of feedback processing. Our results demonstrate that fronto-central theta and the feedback-related negativity (FRN), two correlates of reward prediction errors in the medial frontal cortex, are independent of the internal representations of the environmental structure. In contrast, centro-parietal delta and the P3, two correlates possibly reflecting feedback evaluation in working memory, were highly susceptible to the underlying internal model. Model-based analyses of single-trial activity showed a comparable pattern, indicating that while the computation of unsigned reward prediction errors is represented by theta and the FRN irrespective of the internal models, the P3 adapts to the internal representation of an environment. Our findings further substantiate the assumption that the feedback-locked components under investigation reflect distinct mechanisms of feedback processing and that different internal models selectively influence these mechanisms.

The effect of feedback novelty on neural correlates of feedback processing.

It has been suggested that stimulus novelty itself can be rewarding and recent evidence suggests that novelty processing and reward processing share common neural mechanisms. For feedback processing, this can be beneficial as well as detrimental: If novelty lends a rewarding characteristic to a stimulus, then this should particularly decrease the impact of negative feedback. The present study investigated whether such an effect of feedback novelty on feedback processing is reflected in electrophysiological markers of reinforcement learning (feedback-related negativity, FRN) and feedback processing (feedback-P300) in a simple decision-making task. In this task, participants had to chose between two stimuli in a learning trial followed by a novel or a familiar feedback stimulus. Learning from feedback allowed them to optimize their payoff in a later test trial. As expected, we found that the FRN effect, i.e. the difference between the FRN amplitudes after negative and positive feedback, was reduced for novel compared to familiar feedback stimuli. In addition, the amplitude of the feedback-P300 was decreased by feedback novelty, both for the anterior P3a and the posterior P3b. Together, these results indicate that feedback novelty can affect feedback processing as reflected by feedback-related brain activity.

Are errors detected before they occur? Early error sensations revealed by metacognitive judgments on the timing of error awareness.

Errors in choice tasks are not only detected fast and reliably, participants often report that they knew that an error occurred already before a response was produced. These early error sensations stand in contrast with evidence suggesting that the earliest neural correlates of error awareness emerge around 300 ms after erroneous responses. The present study aimed to investigate whether anecdotal evidence for early error sensations can be corroborated in a controlled study in which participants provide metacognitive judgments on the subjective timing of error awareness. In Experiment 1, participants had to report whether they became aware of their errors before or after the response. In Experiment 2, wemeasured confidence in these metacognitive judgments. Our data show that participants report early error sensations with high confidence in the majority of error trials across paradigms and experiments. These results provide first evidence for early error sensations, informing theories of error awareness.

The impact of cue format and cue transparency on task switching performance.

Cues help in retrieving and implementing task-sets, that are actual representations of the to-be performed task in working memory. However, whereas previous studies revealed that the effectiveness of selecting and implementing task-sets based on cues depends on the type of cue (i.e., transparent words vs. arbitrary shapes), it is still unclear which characteristics of cues are responsible for these differences and whether the impact of the cue is bound to task-set retrieval only or also impacts task-set representations. For instance, the amount of interference during actual task performance has been reported to alter dependent on cue type as do preparation gains such as the reduction of switch cost. To investigate the effectiveness of cue characteristics (i.e., cue transparency and cue format), we manipulated those within- and between-participants in three experiments. Main dependent measures were switch costs in reaction times and error rates that occur when participants have to switch task-sets, and thus update working memory content. Our results consistently show beneficial effects of transparent cues for the reduction of switch cost. The influence of cue format was manifest in within-participants manipulation only and was mainly found in error rates. Overall, our data suggest that the amount of interference experienced in actual task performance can be significantly modulated dependent on cue type, suggesting flexible adaptation of the cognitive system to contextual information.

Rapid adaptive adjustments of selective attention following errors revealed by the time course of steady-state visual evoked potentials.

Directing attention to task-relevant stimuli is crucial for successful task performance, but too much attentional selectivity implies that new and unexpected information in the environment remains undetected. A possible mechanism for optimizing this fundamental trade-off could be an error monitoring system that immediately triggers attentional adjustments following the detection of behavioral errors. However, the existence of rapid adaptive post-error adjustments has been controversially debated. While preconscious error processing reflected by an error-related negativity (Ne/ERN) in the event-related potential has been shown to occur within milliseconds after errors, more recent studies concluded that error detection even impairs attentional selectivity and that adaptive adjustments are implemented, if at all, only after errors are consciously detected. Here, we employ steady-state visual evoked potentials elicited by continuously presented stimuli to precisely track the emergence of error-induced attentional adjustments. Our results indicate that errors lead to an immediate reallocation of attention towards task-relevant stimuli, which occurs simultaneously with the Ne/ERN. Single-trial variation of this adjustment was correlated with the Ne/ERN amplitude and predicted adaptive behavioral adjustments on the post-error trial. This suggests that early error monitoring in the medial frontal cortex is directly involved in eliciting adaptive attentional adjustments.

Error-preceding brain activity links neural markers of task preparation to cognitive stability and flexibility.

Balancing stability and flexibility is required to facilitate successful task selection in situations with competing stimuli. Research suggests a set of counteracting control processes that maintains this balance. In the present study, we investigate how two neural correlates of task preparation in event-related potentials (ERPs), the mixing positivity and the switch positivity, can be linked to stability and flexibility in task selection. In a cued task switching paradigm, we analyzed deviations of these ERPs when task confusions occurred, that is, when participants erroneously executed the currently irrelevant task. We found a reduced mixing positivity to be a main source of task confusions in a task environment that required ongoing switches between competing tasks, whereas the switch positivity was uninvolved here. However, an overabundance of this latter component was a source of task confusions in a task environment that required the repetitive execution of the same task, although task switches were not required at all in this condition. These results not only highlight the distinct functional significance of the two preparatory ERPs and show that control processes can be maladaptive in certain contexts. They can also be utilized to locate the mixing positivity and the switch positivity on the stability-flexibility spectrum. Our results are in line with accounts that suggest that a balance between stability and flexibility is facilitated by the concurrent involvement of two control processes. One that manages the top-down bias of the relevant task set and one that increases or decreases competition between alternatively available stimuli.

Sequential conflict resolution under multiple concurrent conflicts: An ERP study.

The purpose of the present study was to determine whether responding to multiple concurrent conflicts results in a simultaneous or sequential conflict resolution. To this end, we measured event-related potentials (ERPs) in a paradigm combining a Stroop and a flanker task. In this paradigm, participants were asked to respond to the color of the central letter while ignoring the meaning of the word (Stroop task) and the color of the flanking letters (flanker task). Trials were either incongruent (i.e., inducing a conflict between two response alternatives) or congruent (i.e., inducing no response conflict) in both tasks, or incongruent in one task and congruent in the other task. The behavioral results showed a smaller Stroop congruency effect (i.e., a smaller difference between Stroop incongruent and congruent trials) for flanker incongruent than for flanker congruent trials, replicating previous findings. The ERP results showed that an early ERP component (i.e., P2) was associated with the resolution of the flanker conflict, whereas a later component (i.e., N450) was associated with the resolution of the Stroop conflict. Together, these findings emphasize a sequential organization of conflict resolution processes in the brain which is adaptive when facing multiple concurrent conflicts.

Adaptive cognitive control attenuates the late positive potential to emotional distractors.

Emotional pictures are inherently prioritized during stimulus perception. While this preferential emotion processing promotes self-preservation and survival, it can be detrimental when it conflicts with current goals and intentions. Recent brain imaging research suggests that the brain resolves such conflicts by suppressing the processing of emotional distractors at the perceptual level. Beyond brain imaging, event-related scalp potential studies in humans have traced preferential emotion processing at distinct temporal stages. Comparing emotional to neutral pictures, an early stage is indexed by the early posterior negativity (EPN) component featuring a relative negativity over posterior sites, while a later stage is associated with the late positive potential (LPP), manifesting as relative positivity over centro-parietal sensors. However, little is known whether emotional response conflict is resolved at each of those processing stages, or whether conflict resolution operates selectively at early or late stages, respectively. The present study assessed EPN and LPP to emotional distractors in an emotional Stroop task as a function of response conflict in the previous trial. Conflict-related processing during the Stroop task was confirmed by a behavioral conflict adaptation effect and modulation of the congruency-sensitive N450 component. Preferential processing of emotional distractors was observed for the EPN as well as the LPP. While the EPN was completely unaffected by conflict in the previous trial, the LPP was selectively reduced subsequent to trials featuring high response conflict. This observation provides support for a conflict-based control of emotion processing and demonstrates that cognitive control acts selectively at specific stages of emotion perception.

Errors can elicit an error positivity in the absence of an error negativity: Evidence for independent systems of human error monitoring.

Errors in human behavior elicit a cascade of brain activity related to performance monitoring and error detection. Whereas the early error-related negativity (Ne/ERN) has been assumed to reflect a fast mismatch or prediction error signal in the medial frontal cortex, the later error positivity (Pe) is viewed as a correlate of conscious error processing. A still open question is whether these components represent two independent systems of error monitoring that rely on different types of information to detect an error. Here, we investigated the prediction that the Ne/ERN but not the Pe requires a representation of the correct response to emerge. To this end, we created a condition in which no information about the correct response was available while error detection was still possible. We hypothesized that a Pe, but no Ne/ERN should be obtained in this case. Participants had to classify targets but ignore flankers that were always associated with an incorrect response. Targets but not flankers were masked with varying target-masking intervals. Crucially, on some trials no target at all was presented, thus preventing the representation of a correct response and the emergence of an Ne/ERN. However, because flankers were easily visible and responses to the flankers were always incorrect, detection of these flanker errors was still possible. In line with predictions of a multiple-systems account, we observed a robust Pe in the absence of an Ne/ERN for these errors. Moreover, this Pe relied on the same neural activity as that on trials with a visible target, as revealed by multivariate pattern analysis. These findings demonstrate that the mechanisms reflected by the two components use different types of information to detect errors, providing evidence for independent systems of human error monitoring.

Effects of feedback reliability on feedback-related brain activity: A feedback valuation account.

Adaptive decision making relies on learning from feedback. Because feedback sometimes can be misleading, optimal learning requires that knowledge about the feedback's reliability be utilized to adjust feedback processing. Although previous research has shown that feedback reliability indeed influences feedback processing, the underlying mechanisms through which this is accomplished remain unclear. Here we propose that feedback processing is adjusted by the adaptive, top-down valuation of feedback. We assume that unreliable feedback is devalued relative to reliable feedback, thus reducing the reward prediction errors that underlie feedback-related brain activity and learning. A crucial prediction of this account is that the effects of feedback reliability are susceptible to contrast effects. That is, the effects of feedback reliability should be enhanced when both reliable and unreliable feedback are experienced within the same context, as compared to when only one level of feedback reliability is experienced. To evaluate this prediction, we measured the event-related potentials elicited by feedback in two experiments in which feedback reliability was varied either within or between blocks. We found that the fronto-central valence effect, a correlate of reward prediction errors during reinforcement learning, was reduced for unreliable feedback. But this result was obtained only when feedback reliability was varied within blocks, thus indicating a contrast effect. This suggests that the adaptive valuation of feedback is one mechanism underlying the effects of feedback reliability on feedback processing.

Common mechanisms in error monitoring and action effect monitoring.

In the present study, we considered error-related brain activity in event-related potentials, to investigate the relationship between error monitoring-that is, the detection and evaluation of erroneous responses-and action effect monitoring-that is, monitoring of the sensory consequences of behavior. To this end, participants worked on a task-switching paradigm that consisted of a free-choice task, in which a puzzle piece had to be attached to an existing one (the prime task), and a subsequent color flanker task (the probe task). We examined whether unexpected action effects in the prime task would affect the subsequent error monitoring in the probe task. We found the neural correlates of error monitoring during the probe task, the error-related negativity as well as the error positivity, to be increased after unexpected action effects in the prime task. In contrast, the neural correlates of visual attention were decreased after unexpected action effects, in line with recent findings on an attenuation of sensory processing after errors. Our results demonstrate a direct link between monitoring processes in the two tasks. We propose that both error monitoring and action effect monitoring rely on a common generic monitoring system related to novelty detection or affective processing. Preactivating this system by means of unexpected action effects increases the sensitivity for detecting an error in the subsequent task.

Stimulation of Dorsolateral Prefrontal Cortex Enhances Adaptive Cognitive Control: A High-Definition Transcranial Direct Current Stimulation Study.

Conflict adaptation is a hallmark effect of adaptive cognitive control and refers to the adjustment of control to the level of previously experienced conflict. Conflict monitoring theory assumes that the dorsolateral prefrontal cortex (DLPFC) is causally involved in this adjustment. However, to date, evidence in humans is predominantly correlational, and heterogeneous with respect to the lateralization of control in the DLPFC. We used high-definition transcranial direct current stimulation (HD-tDCS), which allows for more focal current delivery than conventional tDCS, to clarify the causal involvement of the DLPFC in conflict adaptation. Specifically, we investigated the regional specificity and lateralization of potential beneficial stimulation effects on conflict adaptation during a visual flanker task. One hundred twenty healthy participants were assigned to four HD-tDCS conditions: left or right DLPFC or left or right primary motor cortex (M1). Each group underwent both active and sham HD-tDCS in crossover, double-blind designs. We obtained a sizeable conflict adaptation effect (measured as the modulation of the flanker effect as a function of previous response conflict) in all groups and conditions. However, this effect was larger under active HD-tDCS than under sham stimulation in both DLPFC groups. In contrast, active stimulation had no effect on conflict adaptation in the M1 groups. In sum, the present results indicate that the DLPFC plays a causal role in adaptive cognitive control, but that the involvement of DLPFC in control is not restricted to the left or right hemisphere. Moreover, our study confirms the potential of HD-tDCS to modulate cognition in a regionally specific manner. SIGNIFICANCE STATEMENT: Conflict adaptation is a hallmark effect of adaptive cognitive control. While animal studies have suggested causal involvement of the DLPFC in this phenomenon, such evidence is currently lacking in humans. The present study used high-definition transcranial direct current stimulation (HD-tDCS) to demonstrate that the DLPFC is causally involved in conflict adaptation in humans. Our study confirms a central claim of conflict monitoring theory, which up to now has predominantly relied on correlational studies. Our results further indicate an equal involvement of the left and right DLPFC in adaptive control, whereas stimulation of a control region-the primary motor cortex-had no effect on adaptive control. The study thus confirms the potential of HD-tDCS to modulate cognition in a regionally specific manner.

Neural signatures of adaptive post-error adjustments in visual search.

Errors in speeded choice tasks can lead to post-error adjustments both on the behavioral and on the neural level. There is an ongoing debate whether such adjustments result from adaptive processes that serve to optimize performance or whether they reflect interference from error monitoring or attentional orientation. The present study aimed at identifying adaptive adjustments in a two-stage visual search task, in which participants had to select and subsequently identify a target stimulus presented to the left or right visual hemifield. Target selection and identification can be measured by two distinct event-related potentials, the N2pc and the SPCN. Using a decoder analysis based on multivariate pattern analysis, we were able to isolate the processing stages related to error sources and post-error adjustments. Whereas errors were linked to deviations in the N2pc and the SPCN, only for the N2pc we identified a post-error adjustment, which exhibits key features of source-specific adaptivity. While errors were associated with an increased N2pc, post-error adjustments consisted in an N2pc decrease. We interpret this as an adaptive adjustment of target selection to prevent errors due to disproportionate processing of the task-irrelevant target location. Our study thus provides evidence for adaptive post-error adjustments in visual search.