The neurobehavioral correlates of exploration without learning: Trading off value for explicit, prospective, and variable information gains.

Exploration is typically motivated by gaining information, with previous research showing that potential information gains drive a "directed" type of exploration. Yet, this research usually studies exploration in the context of learning paradigms and does not directly manipulate multiple levels of information gain. Here, we present a task that isolates learning from decision-making and controls the magnitude of prospective information gains. As predicted, participants explore more with larger future information gains. Both value gains and information gains, at a trial-by-trial level, engage the ventromedial prefrontal cortex (vmPFC), the ventral striatum (VStr), the amygdala, the dorsal anterior cingulate cortex (dACC), and the anterior insula (aINS). Moreover, individual sensitivities to value gains and information gains modulate the vmPFC, dACC, and aINS, but the amygdala and VStr are modulated only by individual sensitivities to information gains. Overall, we identify the neural circuitry of information-based exploration and its relationship with inter-individual exploration biases.

Irrelevant Threats Linger and Affect Behavior in High Anxiety.

Threat-related information attracts attention and disrupts ongoing behavior, and particularly so for more anxious individuals. Yet, it is unknown how and to what extent threat-related information leave lingering influences on behavior (e.g., by impeding ongoing learning processes). Here, human male and female participants (N = 47) performed probabilistic reinforcement learning tasks where irrelevant distracting faces (neutral, happy, or fearful) were presented together with relevant monetary feedback. Behavioral modeling was combined with fMRI data (N = 27) to explore the neurocomputational bases of learning relevant and irrelevant information. In two separate studies, individuals with high trait anxiety showed increased avoidance of objects previously paired with the combination of neutral monetary feedback and fearful faces (but not neutral or happy faces). Behavioral modeling revealed that high anxiety increased the integration of fearful faces during feedback learning, and fMRI results (regarded as provisional, because of a relatively small sample size) further showed that variance in the prediction error signal, uniquely accounted for by fearful faces, correlated more strongly with activity in the right DLPFC for more anxious individuals. Behavioral and neuronal dissociations indicated that the threat-related distractors did not simply disrupt learning processes. By showing that irrelevant threats exert long-lasting influences on behavior, our results extend previous research that separately showed that anxiety increases learning from aversive feedbacks and distractibility by threat-related information. Our behavioral results, combined with the proposed neurocomputational mechanism, may help explain how increased exposure to irrelevant affective information contributes to the acquisition of maladaptive behaviors in more anxious individuals.SIGNIFICANCE STATEMENT In modern-day society, people are increasingly exposed to various types of irrelevant information (e.g., intruding social media announcements). Yet, the neurocomputational mechanisms influenced by irrelevant information during learning, and their interactions with increasingly distracted personality types are largely unknown. Using a reinforcement learning task, where relevant feedback is presented together with irrelevant distractors (emotional faces), we reveal an interaction between irrelevant threat-related information (fearful faces) and interindividual anxiety levels. fMRI shows provisional evidence for an interaction between anxiety levels and the coupling between activity in the DLPFC and learning signals specifically elicited by fearful faces. Our study reveals how irrelevant threat-related information may become entrenched in the anxious psyche and contribute to long-lasting abnormal behaviors.

Average reward rates enable motivational transfer across independent reinforcement learning tasks.

Outcomes and feedbacks on performance may influence behavior beyond the context in which it was received, yet it remains unclear what neurobehavioral mechanisms may account for such lingering influences on behavior. The average reward rate (ARR) has been suggested to regulate motivated behavior, and was found to interact with dopamine-sensitive cognitive processes, such as vigilance and associative memory encoding. The ARR could therefore provide a bridge between independent tasks when these are performed in temporal proximity, such that the reward rate obtained in one task could influence performance in a second subsequent task. Reinforcement learning depends on the coding of prediction error signals by dopamine neurons and their downstream targets, in particular the nucleus accumbens. Because these brain regions also respond to changes in ARR, reinforcement learning may be vulnerable to changes in ARR. To test this hypothesis, we designed a novel paradigm in which participants (n = 245) performed two probabilistic reinforcement learning tasks presented in interleaved trials. The ARR was controlled by an "induction" task which provided feedback with a low (p = 0.58), a medium (p = 0.75), or a high probability of reward (p = 0.92), while the impact of ARR on reinforcement learning was tested by a second "reference" task with a constant reward probability (p = 0.75). We find that performance was significantly lower in the reference task when the induction task provided low reward probabilities (i.e., during low levels of ARR), as compared to the medium and high ARR conditions. Behavioral modeling further revealed that the influence of ARR is best described by models which accumulates average rewards (rather than average prediction errors), and where the ARR directly modulates the prediction error signal (rather than affecting learning rates or exploration). Our results demonstrate how affective information in one domain may transfer and affect motivated behavior in other domains. These findings are particularly relevant for understanding mood disorders, but may also inform abnormal behaviors attributed to dopamine dysfunction.

A neural and behavioral trade-off between value and uncertainty underlies exploratory decisions in normative anxiety.

Exploration reduces uncertainty about the environment and improves the quality of future decisions, but at the cost of provisional uncertain and suboptimal outcomes. Although anxiety promotes intolerance to uncertainty, it remains unclear whether and by which mechanisms anxiety relates to exploratory decision-making. We use a dynamic three-armed-bandit task and find that higher trait-anxiety is associated with increased exploration, which in turn harms overall performance. We identify two distinct behavioral sources: first, decisions made by anxious individuals are guided toward reduction of uncertainty; and second, decisions are less guided by immediate value gains. These findings are similar in both loss and gain domains, and further demonstrate that an affective trait relates to exploration and results in an inverse-U-shaped relationship between anxiety and overall performance. Additional imaging data (fMRI) suggests that normative anxiety correlates negatively with the representation of expected-value in the dorsal-anterior-cingulate-cortex, and in contrast, positively with the representation of uncertainty in the anterior-insula. We conclude that a trade-off between value-gains and uncertainty-reduction entails maladaptive decision-making in individuals with higher normal-range anxiety.

Prior Reward Conditioning Dampens Hippocampal and Striatal Responses during an Associative Memory Task.

Offering reward during encoding typically leads to better memory [Adcock, R. A., Thangavel, A., Whitfield-Gabrieli, S.,Knutson, B., & Gabrieli, J. D. E. Reward-motivated learning: Mesolimbic activation precedes memory formation. Neuron, 50, 507-517, 2006]. Whether such memory benefit persists when tested in a different task context remains, however, largely understudied [Wimmer, G. E., & Buechel, C. Reactivation of reward-related patterns from single past episodes supports memory-based decision making. Journal of Neuroscience, 36, 2868-2880, 2016]. Here, we ask whether reward at encoding leads to a generalized advantage across learning episodes, a question of high importance for any everyday life applications, from education to patient rehabilitation. Although we confirmed that offering monetary reward increased responses in the ventral striatum and pleasantness judgments for pictures used as stimuli, this immediate beneficial effect of reward did not carry over to a subsequent and different picture-location association memory task during which no reward was delivered. If anything, a trend for impaired memory accuracy was observed for the initially high-rewarded pictures as compared to low-rewarded ones. In line with this trend in behavioral performance, fMRI activity in reward (i.e., ventral striatum) and in memory (i.e., hippocampus) circuits was reduced during the encoding of new associations using previously highly rewarded pictures (compared to low-reward pictures). These neural effects extended to new pictures from same, previously highly rewarded semantic category. Twenty-four hours later, delayed recall of associations involving originally highly rewarded items was accompanied by decreased functional connectivity between the hippocampus and two brain regions implicated in value-based learning, the ventral striatum and the ventromedial PFC. We conclude that acquired reward value elicits a downward value-adjustment signal in the human reward circuit when reactivated in a novel nonrewarded context, with a parallel disengagement of memory-reward (hippocampal-striatal) networks, likely to undermine new associative learning. Although reward is known to promote learning, here we show how it may subsequently hinder hippocampal and striatal responses during new associative memory formation.

Prediction errors bidirectionally bias time perception.

Time perception and prediction errors are essential for everyday life. We hypothesized that their putative shared circuitry in the striatum might enable these two functions to interact. We show that positive and negative prediction errors bias time perception by increasing and decreasing perceived time, respectively. Imaging and behavioral modeling identify this interaction to occur in the putamen. Depending on context, this interaction may have beneficial or adverse effects.

Re-exploring Mechanisms of Exploration.

Deciding when to exploit what is already known and when to explore new possibilities is crucial for adapting to novel and dynamic environments. Using reinforcement-based decision making, Costa et al. (2019) in this issue of Neuron find that neurons in the amygdala and ventral-striatum differentially signal the benefit from exploring new options and exploiting familiar ones.

Trial-by-Trial Modulation of Associative Memory Formation by Reward Prediction Error and Reward Anticipation as Revealed by a Biologically Plausible Computational Model.

Anticipation and delivery of rewards improves memory formation, but little effort has been made to disentangle their respective contributions to memory enhancement. Moreover, it has been suggested that the effects of reward on memory are mediated by dopaminergic influences on hippocampal plasticity. Yet, evidence linking memory improvements to actual reward computations reflected in the activity of the dopaminergic system, i.e., prediction errors and expected values, is scarce and inconclusive. For example, different previous studies reported that the magnitude of prediction errors during a reinforcement learning task was a positive, negative, or non-significant predictor of successfully encoding simultaneously presented images. Individual sensitivities to reward and punishment have been found to influence the activation of the dopaminergic reward system and could therefore help explain these seemingly discrepant results. Here, we used a novel associative memory task combined with computational modeling and showed independent effects of reward-delivery and reward-anticipation on memory. Strikingly, the computational approach revealed positive influences from both reward delivery, as mediated by prediction error magnitude, and reward anticipation, as mediated by magnitude of expected value, even in the absence of behavioral effects when analyzed using standard methods, i.e., by collapsing memory performance across trials within conditions. We additionally measured trait estimates of reward and punishment sensitivity and found that individuals with increased reward (vs. punishment) sensitivity had better memory for associations encoded during positive (vs. negative) prediction errors when tested after 20 min, but a negative trend when tested after 24 h. In conclusion, modeling trial-by-trial fluctuations in the magnitude of reward, as we did here for prediction errors and expected value computations, provides a comprehensive and biologically plausible description of the dynamic interplay between reward, dopamine, and associative memory formation. Our results also underline the importance of considering individual traits when assessing reward-related influences on memory.

Trait anxiety and post-learning stress do not affect perceptual learning.

While it is well established that stress can modulate declarative learning, very few studies have investigated the influence of stress on non-declarative learning. Here, we studied the influence of post-learning stress, which effectively modulates declarative learning, on perceptual learning of a visual texture discrimination task (TDT). On day one, participants trained for one session with TDT and were instructed that they, at any time, could be exposed to either a high stressor (ice-water; Cold Pressor Test; CPT) or a low stressor (warm water). Participants did not know when or which stressor they would be exposed to. To determine the impact of the stressor on TDT learning, all participants returned the following day to perform another TDT session. Only participants exposed to the high stressor had significantly elevated cortisol levels. However, there was no difference in TDT improvements from day one to day two between the groups. Recent studies suggested that trait anxiety modulates visual perception under anticipation of stressful events. Here, trait anxiety did neither modulate performance nor influence responsiveness to stress. These results do not support a modulatory role for stress on non-declarative perceptual learning.

Different types of feedback change decision criterion and sensitivity differently in perceptual learning.

In (perceptual) learning, performance improves with practice either by changes in sensitivity or decision criterion. Often, changes in sensitivity are regarded as the appropriate measure of learning while changes in criterion are considered unavoidable nuisances. Very little is known about the distinguishing characteristics of both learning types. Here, we show first that block feedback, which affects sensitivity, does not affect criterion. Second, contrary to changes in sensitivity, changes in decision criterion are limited to the training session and do not transfer overnight. Finally, training with biased trial-wise feedback induces a sensitivity change such that a left offset Vernier may be perceived as a right offset Vernier.

About similar characteristics of visual perceptual learning and LTP.

Perceptual learning is an implicit form of learning which induces long-lasting perceptual enhancements. Perceptual learning shows intriguing characteristics. For example, a minimal number of trials per session is needed for learning and the interleaved presentation of more than one stimulus type can hinder learning. Here, we show that these and other characteristics of perceptual learning are very similar to characteristics of long-term potentiation (LTP), the basic mechanism of memory formation. We outline these characteristics and discuss results of electrophysiological experiments which indirectly link LTP and perceptual learning.

Perceptual learning, roving and the unsupervised bias.

Perceptual learning improves perception through training. Perceptual learning improves with most stimulus types but fails when certain stimulus types are mixed during training (roving). This result is surprising because classical supervised and unsupervised neural network models can cope easily with roving conditions. What makes humans so inferior compared to these models? As experimental and conceptual work has shown, human perceptual learning is neither supervised nor unsupervised but reward-based learning. Reward-based learning suffers from the so-called unsupervised bias, i.e., to prevent synaptic "drift", the average reward has to be exactly estimated. However, this is impossible when two or more stimulus types with different rewards are presented during training (and the reward is estimated by a running average). For this reason, we propose no learning occurs in roving conditions. However, roving hinders perceptual learning only for combinations of similar stimulus types but not for dissimilar ones. In this latter case, we propose that a critic can estimate the reward for each stimulus type separately. One implication of our analysis is that the critic cannot be located in the visual system.

Does perceptual learning suffer from retrograde interference?

In motor learning, training a task B can disrupt improvements of performance of a previously learned task A, indicating that learning needs consolidation. An influential study suggested that this is the case also for visual perceptual learning. Using the same paradigm, we failed to reproduce these results. Further experiments with bisection stimuli also showed no retrograde disruption from task B on task A. Hence, for the tasks tested here, perceptual learning does not suffer from retrograde interference.

Anesthesia prevents auditory perceptual learning.

BACKGROUND: An auditory perceptual learning paradigm was used to investigate whether implicit memories are formed during general anesthesia. METHODS: Eighty-seven patients who had an American Society of Anesthesiologists physical status of I-III and were scheduled to undergo an elective surgery with general anesthesia were randomly assigned to one of two groups. One group received auditory stimulation during surgery, whereas the other did not. The auditory stimulation consisted of pure tones presented via headphones. The Bispectral Index level was maintained between 40 and 50 during surgery. To assess learning, patients performed an auditory frequency discrimination task after surgery, and comparisons were made between the groups. General anesthesia was induced with thiopental and maintained with a mixture of fentanyl and sevoflurane. RESULTS: There was no difference in the amount of learning between the two groups (mean +/- SD improvement: stimulated patients 9.2 +/- 11.3 Hz, controls 9.4 +/- 14.1 Hz). There was also no difference in initial thresholds (mean +/- SD initial thresholds: stimulated patients 31.1 +/- 33.4 Hz, controls 28.4 +/- 34.2 Hz). These results suggest that perceptual learning was not induced during anesthesia. No correlation between the bispectral index and the initial level of performance was found (Pearson r = -0.09, P = 0.59). CONCLUSION: Perceptual learning was not induced by repetitive auditory stimulation during anesthesia. This result may indicate that perceptual learning requires top-down processing, which is suppressed by the anesthetic.

Interleaving bisection stimuli - randomly or in sequence - does not disrupt perceptual learning, it just makes it more difficult.

Presenting stimuli of two or more stimulus types randomly interleaved, so called roving, disrupts perceptual learning in many paradigms. Recently, it was shown that no disruption occurs when Gabor stimuli were presented interleaved in sequence, instead of randomly. Here, using bisection stimuli, we found the opposite pattern of results. Presenting bisection stimuli in a sequence disrupted perceptual learning, whereas we found improvement under roving conditions. A meta-analysis showed that parts of this deviation from previous studies is possibly caused by the initial performance level of participants. These results do not prove previous results wrong, they just show that multiple factors play a crucial role in perceptual learning which cannot always be easily controlled for.

Perceptual learning with Chevrons requires a minimal number of trials, transfers to untrained directions, but does not require sleep.

In most models of perceptual learning, the amount of improvement of performance does not depend on the regime of stimulus presentations, but only on the sheer number of trials. Here, we kept the number of stimulus presentations constant while varying the number of trials per session. We show that a minimal number of stimulus presentations per session is necessary, transfer depends strongly on the presentation regime, but sleep has only weak, if at all, effects.

Perceptual learning and roving: Stimulus types and overlapping neural populations.

In perceptual learning, performance usually improves when observers train with one type of stimulus, for example, a bisection stimulus. Roving denotes the situation when, instead of one, two or more types of stimuli are presented randomly interleaved, for example, a bisection stimulus and a vernier. For some combinations of stimulus types, performance improves in roving situations whereas for others it does not. To investigate when roving impedes perceptual learning, we conducted four experiments. Performance improved, for example, when we roved a bisection stimulus and a vernier but not when we roved certain types of bisection stimuli. We propose that roving hinders perceptual learning when the stimulus types are clearly distinct from each other but still excite overlapping but not identical neural populations.