Effects of categorical and numerical feedback on category learning.

Real-world learning signals often come in the form of a continuous range of rewards or punishments, such as receiving more or less money or other reward. However, in laboratory studies, feedback used to examine how humans learn new categories has almost invariably been categorical in nature (i.e. Correct/Incorrect, or A/Not-A). Whether numerical or categorical feedback leads to better learning is an open question. One possibility is that numerical feedback could give more fine-grained information about a category. Alternatively, categorical feedback is more dichotomous, potentially leading to larger error signals. Here we test how feedback impacts category learning by having participants learn to categorize novel line stimuli from either numerical, categorical, or a combination of both types of feedback. Performance was better for categorical relative to the more variable numerical feedback. However, participants were able to learn to effectively categorize from numerical feedback, and providing larger numerical rewards for easier, more representative stimuli was more successful in promoting learning than providing larger rewards for harder to classify stimuli. Simulations and fits of a connectionist model to participants' performance data suggest that categorical feedback promotes better learning by eliciting larger prediction errors than numerical feedback.

Neural regions associated with gain-loss frequency and average reward in older and younger adults.

Research on the biological basis of reinforcement-learning has focused on how brain regions track expected value based on average reward. However, recent work suggests that humans are more attuned to reward frequency. Furthermore, older adults are less likely to use expected values to guide choice than younger adults. This raises the question of whether brain regions assumed to be sensitive to average reward, like the medial and lateral PFC, also track reward frequency, and whether there are age-based differences. Older (60-81 years) and younger (18-30 years) adults performed the Soochow Gambling task, which separates reward frequency from average reward, while undergoing fMRI. Overall, participants preferred options that provided negative net payoffs, but frequent gains. Older adults improved less over time, were more reactive to recent negative outcomes, and showed greater frequency-related activation in several regions, including DLPFC. We also found broader recruitment of prefrontal and parietal regions associated with frequency value and reward prediction errors in older adults, which may indicate compensation. The results suggest greater reliance on average reward for younger adults than older adults.

Preliminary effects of prefrontal tDCS on dopamine-mediated behavior and psychophysiology.

The ability to manipulate dopamine in vivo through non-invasive, reversible mechanisms has the potential to impact clinical, translational, and basic research. Recent PET studies have demonstrated increased dopamine release in the striatum after bifrontal transcranial direct current stimulation (tDCS). We sought to extend this work by examining whether bifrontal tDCS could demonstrate an effect on behavioral and physiological correlates of subcortical dopamine activity. We conducted a preliminary between-subjects study (n = 30) with active and sham tDCS and used spontaneous eye blink rate (EBR), facial attractiveness ratings, and greyscales orienting bias as indirect proxies for dopamine functioning. The initial design and analyses were pre-registered (https://osf.io/gmnpc). Stimulation did not significantly affect any of the three measures, though effect sizes were often moderately large and were all in the predicted directions. Additional exploratory analyses suggested that stimulation's effect on EBR might depend on pre-stimulation dopamine levels. Our results suggest that larger samples than those that are standard in tDCS literature should be used to assess the effect of tDCS on dopamine using indirect measures. Further, exploratory results add to a growing body of work demonstrating the importance of accounting for individual differences in tDCS response.

Acute stress enhances tolerance of uncertainty during decision-making.

Acute stress has been shown to influence reward sensitivity, feedback learning, and risk-taking during decision-making, primarily through activation of the hypothalamic pituitary axis (HPA). However, it is unclear how acute stress affects decision-making among choices that vary in their degree of uncertainty. To address this question, we conducted two experiments in which participants repeatedly chose between two options-a high-uncertainty option that offered highly variable rewards but was advantageous in the long-term, and a low-uncertainty option that offered smaller yet more consistent rewards. The Socially Evaluated Cold Pressor Task (SECPT) was utilized to induce acute stress. Participants in Experiment 1 (N = 114) were exposed to either the SECPT or a warm-water control condition and then completed the decision-making under uncertainty task. Compared to the control condition, those exposed to the acute stress manipulation chose the high-uncertainty option that provided highly variable but larger rewards over the option that provided stable, smaller rewards. Experiment 2 (N = 95) incorporated a salivary cortisol measure. Results replicated the behavioral findings in Experiment 1 and demonstrated that the acute stress manipulation increased salivary cortisol. This work suggests that moderate acute stress is associated with tolerance of outcome variability in contexts that depend on learning to maximize rewards.

Learning reward frequency over reward probability: A tale of two learning rules.

Learning about the expected value of choice alternatives associated with reward is critical for adaptive behavior. Although human choice preferences are affected by the presentation frequency of reward-related alternatives, this may not be captured by some dominant models of value learning, such as the delta rule. In this study, we examined whether reward learning is driven more by learning the probability of reward provided by each option, or how frequently each option has been rewarded, and assess how well models based on average reward (e.g. the delta model) and models based on cumulative reward (e.g. the decay model) can account for choice preferences. In a binary-outcome choice task, participants selected between pairs of options that had reward probabilities of 0.65 (A) versus 0.35 (B) or 0.75 (C) versus 0.25 (D). Crucially, during training there were twice the number of AB trials as CD trials, such that option A was associated with higher cumulative reward, while option C gave higher average reward. Participants then decided between novel combinations of options (e.g., AC). Most participants preferred option A over C, a result predicted by the Decay model, but not the Delta model. We also compared the Delta and Decay models to both more simplified as well as more complex models that assumed additional mechanisms, such as representation of uncertainty. Overall, models that assume learning about cumulative reward provided the best account of the data.

Acute stress improves long-term reward maximization in decision-making under uncertainty.

Acute stress influences reward-seeking tendencies and risky decision-making. However, it is unclear how acute stress influences decision-making in situations in which individuals must learn to either maximize long-term or immediate rewards from experience. Consequently, this study sought to investigate whether acute stress enhances salience of small, immediate or large, delayed rewards on decision-making under uncertainty. The Socially Evaluated Cold Pressor Task (SECPT) was used to induce acute stress. Participants in Experiment 1 (N = 50) were exposed to either the SECPT or a warm-water control condition and then completed a decision-making task in which participants needed to learn to forego immediate rewards in favor of larger delayed rewards. The results demonstrated that acute stress enhanced decisions that maximized long-term, large rewards over immediate, small rewards. Experiment 2 (N = 50) included an assessment of salivary cortisol. Results replicated the behavioral findings in Experiment 1 and demonstrated that the acute stress manipulation increased salivary cortisol, thus providing a potential physiological mechanism for these results. This work suggests that moderate acute stress can improve decision-making under uncertainty that depends on learning to maximize long-term rewards from experience.

A Case of Divergent Predictions Made by Delta and Decay Rule Learning Models.

The Delta and Decay rules are two learning rules used to update expected values in reinforcement learning (RL) models. The delta rule learns average rewards, whereas the decay rule learns cumulative rewards for each option. Participants learned to select between pairs of options that had reward probabilities of .65 (option A) versus .35 (option B) or .75 (option C) versus .25 (option D) on separate trials in a binary-outcome choice task. Crucially, during training there were twice as AB trials as CD trials, therefore participants experienced more cumulative reward from option A even though option C had a higher average reward rate (.75 versus .65). Participants then decided between novel combinations of options (e.g, A versus C). The Decay model predicted more A choices, but the Delta model predicted more C choices, because those respective options had higher cumulative versus average reward values. Results were more in line with the Decay model's predictions. This suggests that people may retrieve memories of cumulative reward to compute expected value instead of learning average rewards for each option.

Gender differences in preference for reward frequency versus reward magnitude in decision-making under uncertainty.

Extensive research has focused on gender differences in intertemporal choices made from description in which participants must choose from multiple options that are specified without ambiguity. However, there has been limited work examining gender differences in intertemporal choices made from experience in which the possible payoffs among choice alternatives are not initially known and can only be gained from experience. Other work suggests that females attend more to reward frequency, whereas males attend more to reward magnitude. However, the tasks used in this research have been complex and did not examine intertemporal decision-making. To specifically test whether females are more sensitive to reward frequency and males are more sensitive to reward magnitude on intertemporal decisions made from experience, we designed a simple choice task in which participants pressed a response button at a time of their own choosing on each of many trials. Faster responses led to smaller, but more frequent rewards, whereas slower responses led to larger, but less frequently given rewards. As predicted, females tended to respond quicker for more certain, smaller rewards than males, supporting our prediction that women attend more to reward frequency whereas men attend more to reward magnitude.