Modulation of ventral striatal activity by cognitive effort.

Effort discounting theory suggests that the value of a reward should be lower if it was effortful to obtain, whereas contrast theory suggests that the contrast between the costly effort and the reward makes the reward seem more valuable. To test these alternative hypotheses, we used functional magnetic resonance imaging (fMRI) as participants engaged in feedback-based learning that required low or high cognitive effort to obtain positive feedback, while the objective amount of information provided by feedback remained constant. In the low effort condition, a single image was presented with four response options. In the high effort condition, two images were presented, each with two response options, and correct feedback was presented only when participants responded correctly to both of the images. Accuracy was significantly lower for the high effort condition, and all participants reported that the high effort condition was more difficult. A region of the ventral striatum selected for sensitivity to feedback value also showed increased activation to feedback presentation associated with the high effort condition relative to the low effort condition, when controlling for activation from corresponding control conditions where feedback was random. These results suggest that increased cognitive effort produces corresponding increases in positive feedback-related ventral striatum activity, in line with the predictions made by contrast theory. The accomplishment of obtaining a hard-earned intrinsic reward, such as positive feedback, may be particularly likely to promote reward-related brain activity.

Distinguishing informational from value-related encoding of rewarding and punishing outcomes in the human brain.

There is accumulating evidence implicating a set of key brain regions in encoding rewarding and punishing outcomes, including the orbitofrontal cortex, medial prefrontal cortex, ventral striatum, anterior insula, and anterior cingulate. However, it has proved challenging to reach consensus concerning the extent to which different brain areas are involved in differentially encoding rewarding and punishing outcomes. Here, we show that many of the brain areas involved in outcome processing represent multiple outcome components: encoding the value of outcomes (whether rewarding or punishing) and informational coding, i.e. signaling whether a given outcome is rewarding or punishing, ignoring magnitude or experienced utility. In particular, we report informational signals in the lateral orbitofrontal cortex and anterior insular cortex that respond to both rewarding and punishing feedback, even though value-related signals in these areas appear to be selectively driven by punishing feedback. These findings highlight the importance of taking into account features of outcomes other than value when characterising the contributions of different brain regions in outcome processing.

Evidence for model-based computations in the human amygdala during Pavlovian conditioning.

Contemporary computational accounts of instrumental conditioning have emphasized a role for a model-based system in which values are computed with reference to a rich model of the structure of the world, and a model-free system in which values are updated without encoding such structure. Much less studied is the possibility of a similar distinction operating at the level of Pavlovian conditioning. In the present study, we scanned human participants while they participated in a Pavlovian conditioning task with a simple structure while measuring activity in the human amygdala using a high-resolution fMRI protocol. After fitting a model-based algorithm and a variety of model-free algorithms to the fMRI data, we found evidence for the superiority of a model-based algorithm in accounting for activity in the amygdala compared to the model-free counterparts. These findings support an important role for model-based algorithms in describing the processes underpinning Pavlovian conditioning, as well as providing evidence of a role for the human amygdala in model-based inference.

Human dorsal striatal activity during choice discriminates reinforcement learning behavior from the gambler's fallacy.

Reinforcement learning theory has generated substantial interest in neurobiology, particularly because of the resemblance between phasic dopamine and reward prediction errors. Actor-critic theories have been adapted to account for the functions of the striatum, with parts of the dorsal striatum equated to the actor. Here, we specifically test whether the human dorsal striatum--as predicted by an actor-critic instantiation--is used on a trial-to-trial basis at the time of choice to choose in accordance with reinforcement learning theory, as opposed to a competing strategy: the gambler's fallacy. Using a partial-brain functional magnetic resonance imaging scanning protocol focused on the striatum and other ventral brain areas, we found that the dorsal striatum is more active when choosing consistent with reinforcement learning compared with the competing strategy. Moreover, an overlapping area of dorsal striatum along with the ventral striatum was found to be correlated with reward prediction errors at the time of outcome, as predicted by the actor-critic framework. These findings suggest that the same region of dorsal striatum involved in learning stimulus-response associations may contribute to the control of behavior during choice, thereby using those learned associations. Intriguingly, neither reinforcement learning nor the gambler's fallacy conformed to the optimal choice strategy on the specific decision-making task we used. Thus, the dorsal striatum may contribute to the control of behavior according to reinforcement learning even when the prescriptions of such an algorithm are suboptimal in terms of maximizing future rewards.

Error effects in anterior cingulate cortex reverse when error likelihood is high.

Strong error-related activity in medial prefrontal cortex (mPFC) has been shown repeatedly with neuroimaging and event-related potential studies for the last several decades. Multiple theories have been proposed to account for error effects, including comparator models and conflict detection models, but the neural mechanisms that generate error signals remain in dispute. Typical studies use relatively low error rates, confounding the expectedness and the desirability of an error. Here we show with a gambling task and functional magnetic resonance imaging that when losses are more frequent than wins, the mPFC error effect disappears, and moreover, exhibits the opposite pattern by responding more strongly to unexpected wins than losses. These findings provide perspective on recent ERP studies and suggest that mPFC error effects result from a comparison between actual and expected outcomes.

Differentiable contributions of human amygdalar subregions in the computations underlying reward and avoidance learning.

To understand how the human amygdala contributes to associative learning, it is necessary to differentiate the contributions of its subregions. However, major limitations in the techniques used for the acquisition and analysis of functional magnetic resonance imaging (fMRI) data have hitherto precluded segregation of function with the amygdala in humans. Here, we used high-resolution fMRI in combination with a region-of-interest-based normalization method to differentiate functionally the contributions of distinct subregions within the human amygdala during two different types of instrumental conditioning: reward and avoidance learning. Through the application of a computational-model-based analysis, we found evidence for a dissociation between the contributions of the basolateral and centromedial complexes in the representation of specific computational signals during learning, with the basolateral complex contributing more to reward learning, and the centromedial complex more to avoidance learning. These results provide unique insights into the computations being implemented within fine-grained amygdala circuits in the human brain.

Transfer of high domain knowledge to a similar domain.

Researchers have widely examined domain knowledge yet rarely investigate the transfer of knowledge from one domain to another. This study sought to fill in the literature gap concerning the impact of domain knowledge on memory in a similar situation. Specifically, this study examined whether high knowledge of baseball could enhance memory for the similar yet unknown domain of cricket, using a 2 (knowledge) x 2 (prime) design. An interaction occurred, indicating that when primed, baseball knowledge improves memory for cricket events in participants with high baseball knowledge but reduces memory in their low-knowledge counterparts. These results suggest that extensive knowledge in one domain allows it to serve as an organizational framework for incoming information in a similar domain; conversely, priming poorly understood domain knowledge results in negative transfer.

Feedback produces divergence from prospect theory in descriptive choice.

A recent study demonstrated that individuals making experience-based choices underweight small probabilities, in contrast to the overweighting observed in a typical descriptive paradigm. We tested whether trial-by-trial feedback in a repeated descriptive paradigm would engender choices more correspondent with experiential or descriptive paradigms. The results of a repeated gambling task indicated that individuals receiving feedback underweighted small probabilities, relative to their no-feedback counterparts. These results implicate feedback as a critical component during the decision-making process, even in the presence of fully specified descriptive information. A model comparison at the individual-subject level suggested that feedback drove individuals' decision weights toward objective probability weighting.

Building bridges between neural models and complex decision making behaviour.

Diffusion processes, and their discrete time counterparts, random walk models, have demonstrated an ability to account for a wide range of findings from behavioural decision making for which the purely algebraic and deterministic models often used in economics and psychology cannot account. Recent studies that record neural activations in non-human primates during perceptual decision making tasks have revealed that neural firing rates closely mimic the accumulation of preference theorized by behaviourally-derived diffusion models of decision making. This article bridges the expanse between the neurophysiological and behavioural decision making literatures specifically, decision field theory [Busemeyer, J. R. & Townsend, J. T. (1993). Decision field theory: A dynamic-cognitive approach to decision making in an uncertain environment. Psychological Review, 100, 432-459], a dynamic and stochastic random walk theory of decision making, is presented as a model positioned between lower-level neural activation patterns and more complex notions of decision making found in psychology and economics. Potential neural correlates of this model are proposed, and relevant competing models are also addressed.

Decision neuroscience: choices of description and of experience.

A new study suggests that individuals differentially recruit neural regions associated with decision making, depending on whether the information about the options are learned through experience or merely described.

It was nice not seeing you: perceptual learning with rewards in the absence of awareness.

In this issue of Neuron, Seitz et al. show that humans exhibit enhanced perceptual discrimination for visual stimuli that have been repeatedly paired with reward under conditions of suppressed awareness. These findings challenge the view that awareness and focused attention are necessary for perceptual learning.