Reliability of gamified reinforcement learning in densely sampled longitudinal assessments.

Reinforcement learning is a core facet of motivation and alterations have been associated with various mental disorders. To build better models of individual learning, repeated measurement of value-based decision-making is crucial. However, the focus on lab-based assessment of reward learning has limited the number of measurements and the test-retest reliability of many decision-related parameters is therefore unknown. In this paper, we present an open-source cross-platform application Influenca that provides a novel reward learning task complemented by ecological momentary assessment (EMA) of current mental and physiological states for repeated assessment over weeks. In this task, players have to identify the most effective medication by integrating reward values with changing probabilities to win (according to random Gaussian walks). Participants can complete up to 31 runs with 150 trials each. To encourage replay, in-game screens provide feedback on the progress. Using an initial validation sample of 384 players (9729 runs), we found that reinforcement learning parameters such as the learning rate and reward sensitivity show poor to fair intra-class correlations (ICC: 0.22-0.53), indicating substantial within- and between-subject variance. Notably, items assessing the psychological state showed comparable ICCs as reinforcement learning parameters. To conclude, our innovative and openly customizable app framework provides a gamified task that optimizes repeated assessments of reward learning to better quantify intra- and inter-individual differences in value-based decision-making over time.

Can't decide how much to EAT? Effort variability for reward is associated with cognitive restraint.

Food intake is inherently variable and often characterized by episodical restraint or overeating (uncontrolled eating). Such heightened variability in intake has been associated with higher variability in the brain response to food reward, but it is an open issue whether comparable associations with elevated variability in reward seeking exist. Here, we assessed whether restraint and uncontrolled eating as markers of trait-like variability in eating are associated with higher intra-individual variability in reward seeking as captured by a cost-benefit paradigm. To test this hypothesis, 81 healthy, overnight-fasting participants (MBMI = 23.0 kg/m2 ± 3.0) completed an effort allocation task (EAT) twice. In the EAT, participants had to exert physical effort to earn monetary and food rewards and indicated levels of wanting through visual analog scales (VAS). As predicted, we found that greater trial-by-trial effort variability was associated with lower scores on cognitive restraint, rp(78) = -0.28, p = .011 (controlled for average effort). In line with previous findings, higher wanting variability was associated with higher BMI, rp(78) = 0.25, p = .026 (controlled for average effort). Collectively, our results support the idea that higher variability in reward seeking is a potential risk factor for eating beyond homeostatic need. Since associations with variability measures of reward exceeded associations with average reward seeking, our findings may indicate that variability in the representation of the reward value could be a crucial aspect driving fluctuations in food intake.

Vagus nerve stimulation boosts the drive to work for rewards.

Interoceptive feedback transmitted via the vagus nerve plays a vital role in motivation by tuning actions according to physiological needs. Whereas vagus nerve stimulation (VNS) reinforces actions in animals, motivational effects elicited by VNS in humans are still largely elusive. Here, we applied non-invasive transcutaneous auricular VNS (taVNS) on the left or right ear while participants exerted effort to earn rewards using a randomized cross-over design (vs. sham). In line with preclinical studies, acute taVNS enhances invigoration of effort, and stimulation on the left side primarily facilitates invigoration for food rewards. In contrast, we do not find conclusive evidence that acute taVNS affects effort maintenance or wanting ratings. Collectively, our results suggest that taVNS enhances reward-seeking by boosting invigoration, not effort maintenance and that the stimulation side affects generalization beyond food reward. Thus, taVNS may enhance the pursuit of prospective rewards which may pave avenues to treat motivational deficiencies.

Stimulation of the vagus nerve reduces learning in a go/no-go reinforcement learning task.

When facing decisions to approach rewards or to avoid punishments, we often figuratively go with our gut, and the impact of metabolic states such as hunger on motivation are well documented. However, whether and how vagal feedback signals from the gut influence instrumental actions is unknown. Here, we investigated the effect of non-invasive transcutaneous auricular vagus nerve stimulation (taVNS) vs. sham (randomized cross-over design) on approach and avoidance behavior using an established go/no-go reinforcement learning paradigm in 39 healthy human participants (23 female) after an overnight fast. First, mixed-effects logistic regression analysis of choice accuracy showed that taVNS acutely impaired decision-making, p = .041. Computational reinforcement learning models identified the cause of this as a reduction in the learning rate through taVNS (∆α = -0.092, pboot = .002), particularly after punishment (∆αPun = -0.081, pboot = .012 vs. ∆αRew =-0.031, pboot = .22). However, taVNS had no effect on go biases, Pavlovian response biases or response time. Hence, taVNS appeared to influence learning rather than action execution. These results highlight a novel role of vagal afferent input in modulating reinforcement learning by tuning the learning rate according to homeostatic needs.

Beyond the average: The role of variable reward sensitivity in eating disorders.

Eating disorders are often characterized by episodes of overeating and undereating. To date, most theories have explained the liability for such episodes by differences in traits such as reward sensitivity or cognitive control. Here, we review the evidence for a more parsimonious account of the waxing and waning in food intake by linking it to state-like variability of alleged traits such as reward sensitivity. To formally demonstrate that our variability model of eating disorders could explain a wide range of observed reward-related behavior, we conducted simulations of value-based choices and learning. These simulations based on well-established computational models of reinforcement learning and Bayesian sequential updating show how variability may arise and manifest in eating behavior. We argue that by reconceptualizing stable traits as distributions over likely states promoting adaptation, our proposed model integrates disparate findings and leads to novel predictions in a quantitative framework. Collectively, these emerging results call for a stronger emphasis on within-person variability to improve mechanistic insights into eating disorders.

Author Correction: The pleasantness of sensory dissonance is mediated by musical style and expertise.

A correction to this article has been published and is linked from the HTML and PDF versions of this paper. The error has been fixed in the paper.

The anterior insula channels prefrontal expectancy signals during affective processing.

Expectancy shapes our perception of impending events. Although such an interplay between cognitive and affective processes is often impaired in mental disorders, it is not well understood how top-down expectancy signals modulate future affect. We therefore track the information flow in the brain during cognitive and affective processing segregated in time using task-specific cross-correlations. Participants in two independent fMRI studies (N1 = 37 & N2 = 55) were instructed to imagine a situation with affective content as indicated by a cue, which was then followed by an emotional picture congruent with expectancy. To correct for intrinsic covariance of brain function, we calculate resting-state cross-correlations analogous to the task. First, using factorial modeling of delta cross-correlations (task-rest) of the first study, we find that the magnitude of expectancy signals in the anterior insula cortex (AIC) modulates the BOLD response to emotional pictures in the anterior cingulate and dorsomedial prefrontal cortex in opposite directions. Second, using hierarchical linear modeling of lagged connectivity, we demonstrate that expectancy signals in the AIC indeed foreshadow this opposing pattern in the prefrontal cortex. Third, we replicate the results in the second study using a higher temporal resolution, showing that our task-specific cross-correlation approach robustly uncovers the dynamics of information flow. We conclude that the AIC arbitrates the recruitment of distinct prefrontal networks during cued picture processing according to triggered expectations. Taken together, our study provides new insights into neuronal pathways channeling cognition and affect within well-defined brain networks. Better understanding of such dynamics could lead to new applications tracking aberrant information processing in mental disorders.

The pleasantness of sensory dissonance is mediated by musical style and expertise.

Western musical styles use a large variety of chords and vertical sonorities. Based on objective acoustical properties, chords can be situated on a dissonant-consonant continuum. While this might to some extent converge with the unpleasant-pleasant continuum, subjective liking might diverge for various chord forms from music across different styles. Our study aimed to investigate how well appraisals of the roughness and pleasantness dimensions of isolated chords taken from real-world music are predicted by Parncutt's established model of sensory dissonance. Furthermore, we related these subjective ratings to style of origin and acoustical features of the chords as well as musical sophistication of the raters. Ratings were obtained for chords deemed representative of the harmonic language of three different musical styles (classical, jazz and avant-garde music), plus randomly generated chords. Results indicate that pleasantness and roughness ratings were, on average, mirror opposites; however, their relative distribution differed greatly across styles, reflecting different underlying aesthetic ideals. Parncutt's model only weakly predicted ratings for all but Classical chords, suggesting that listeners' appraisal of the dissonance and pleasantness of chords bears not only on stimulus-side but also on listener-side factors. Indeed, we found that levels of musical sophistication negatively predicted listeners' tendency to rate the consonance and pleasantness of any one chord as coupled measures, suggesting that musical education and expertise may serve to individuate how these musical dimensions are apprehended.