Valence precedes value in neural encoding of prediction error.

Event-related potentials that follow feedback in reinforcement learning tasks have been proposed to reflect neural encoding of prediction errors. Prior research has shown that in the interval of 240-340 ms multiple different prediction error encodings appear to co-occur, including a value signal carrying signed quantitative prediction error and a valence signal merely carrying sign. The effects used to identify these two encoders, respectively a sign main effect and a sign × size interaction, do not reliably discriminate them. A full discrimination is made possible by comparing tasks in which the reinforcer available on a given trial is set to be either appetitive or aversive against tasks where a trial allows the possibility of either. This study presents a meta-analysis of reinforcement learning experiments, the majority of which presented the possibility of winning or losing money. Value and valence encodings were identified by conventional difference wave methodology but additionally by an analysis of their predicted behavior using a Bayesian analysis that incorporated nulls into the evidence for each encoder. The results suggest that a valence encoding, sensitive only to the available outcomes on the trial at hand precedes a later value encoding sensitive to the outcomes available in the wider experimental context. The implications of this for modeling computational processes of reinforcement learning in humans are discussed.

Reward prediction error in the ERP following unconditioned aversive stimuli.

Reinforcement learning in humans and other animals is driven by reward prediction errors: deviations between the amount of reward or punishment initially expected and that which is obtained. Temporal difference methods of reinforcement learning generate this reward prediction error at the earliest time at which a revision in reward or punishment likelihood is signalled, for example by a conditioned stimulus. Midbrain dopamine neurons, believed to compute reward prediction errors, generate this signal in response to both conditioned and unconditioned stimuli, as predicted by temporal difference learning. Electroencephalographic recordings of human participants have suggested that a component named the feedback-related negativity (FRN) is generated when this signal is carried to the cortex. If this is so, the FRN should be expected to respond equivalently to conditioned and unconditioned stimuli. However, very few studies have attempted to measure the FRN's response to unconditioned stimuli. The present study attempted to elicit the FRN in response to a primary aversive stimulus (electric shock) using a design that varied reward prediction error while holding physical intensity constant. The FRN was strongly elicited, but earlier and more transiently than typically seen, suggesting that it may incorporate other processes than the midbrain dopamine system.

Reward, Salience, and Agency in Event-Related Potentials for Appetitive and Aversive Contexts.

Cognitive architectures tasked with swiftly and adaptively processing biologically important events are likely to classify these on two central axes: motivational salience, that is, those events' importance and unexpectedness, and motivational value, the utility they hold, relative to that expected. Because of its temporal precision, electroencephalography provides an opportunity to resolve processes associated with these two axes. A focus of attention for the last two decades has been the feedback-related negativity (FRN), a frontocentral component occurring 240-340 ms after valenced events that are not fully predicted. Both motivational salience and value are present in such events and competing claims have been made for which of these is encoded by the FRN. The present study suggests that motivational value, in the form of a reward prediction error, is the primary determinant of the FRN in active contexts, while in both passive and active contexts, a weaker and earlier overlapping motivational salience component may be present.

Evidence for parietal reward prediction errors using great grand average meta-analysis.

As a basic principle within the economics of decision-making, reinforcement learning dictates that individuals strive to repeat behaviour that elicits reward, and avoid behaviour that elicits punishment. Neuroeconomics aims to measure reinforcement learning physically in the brain through the use of reward prediction errors: the difference between expected outcome value and actual outcome value following decision-making behaviour. Two electrophysiological components, the frontocentral feedback-related negativity and the more parietal P3, are implicated in outcome processing, but whether these components encode a reward prediction error has been unclear. A source of the unclear literature is likely to be inconsistent quantification of the components. A recent meta-analysis that directly quantified published waveforms rather than using reported effect sizes found strong evidence that the feedback-related negativity encodes a reward prediction error. In the current study, such a meta-analysis was performed on parietal waveforms to establish whether the P3, or parietal areas generally, are sensitive to reward prediction errors. A strong effect was found, both of reward prediction error encoding and simple valence sensitivity at a latency associated with the P3.