Neurons in monkey dorsal raphe nucleus code beginning and progress of step-by-step schedule, reward expectation, and amount of reward outcome in the reward schedule task.

The dorsal raphe nucleus is the major source of serotonin in the brain. It is connected to brain regions related to reward processing, and the neurons show activity related to predicted reward outcome. Clinical observations also suggest that it is important in maintaining alertness and its apparent role in addiction seems to be related to reward processing. Here, we examined whether the neurons in dorsal raphe carry signals about reward outcome and task progress during multitrial schedules. We recorded from 98 single neurons in dorsal raphe of two monkeys. The monkeys perform one, two, or three visual discrimination trials (schedule), obtaining one, two, or three drops of liquid. In the valid cue condition, the length and brightness of a visual cue indicated schedule progress and reward amount, respectively. In the random cue condition, the visual cue was randomly presented with respect to schedule length and reward amount. We found information encoded about (1) schedule onset, (2) reward expectation, (3) reward outcome, and (4) reward amount in the mean firing rates. Information theoretic analysis showed that the temporal variation of the neuronal responses contained additional information related to the progress of the schedule toward the reward rather than only discriminating schedule onset or reward/no reward. When considered in light of all that is known about the raphe in anatomy, physiology, and behavior, the rich encoding about both task progress and predicted reward outcome makes the raphe a strong candidate for providing signals throughout the brain to coordinate persistent goal-seeking behavior.

Stimulus-related activity during conditional associations in monkey perirhinal cortex neurons depends on upcoming reward outcome.

Acquiring the significance of events based on reward-related information is critical for animals to survive and to conduct social activities. The importance of the perirhinal cortex for reward-related information processing has been suggested. To examine whether or not neurons in this cortex represent reward information flexibly when a visual stimulus indicates either a rewarded or unrewarded outcome, neuronal activity in the macaque perirhinal cortex was examined using a conditional-association cued-reward task. The task design allowed us to study how the neuronal responses depended on the animal's prediction of whether it would or would not be rewarded. Two visual stimuli, a color stimulus as Cue1 followed by a pattern stimulus as Cue2, were sequentially presented. Each pattern stimulus was conditionally associated with both rewarded and unrewarded outcomes depending on the preceding color stimulus. We found an activity depending upon the two reward conditions during Cue2, i.e., pattern stimulus presentation. The response appeared after the response dependent upon the image identity of Cue2. The response delineating a specific cue sequence also appeared between the responses dependent upon the identity of Cue2 and reward conditions. Thus, when Cue1 sets the context for whether or not Cue2 indicates a reward, this region represents the meaning of Cue2, i.e., the reward conditions, independent of the identity of Cue2. These results suggest that neurons in the perirhinal cortex do more than associate a single stimulus with a reward to achieve flexible representations of reward information.

Encoding of reward expectation by monkey anterior insular neurons.

The insula, a cortical brain region that is known to encode information about autonomic, visceral, and olfactory functions, has recently been shown to encode information during reward-seeking tasks in both single neuronal recording and functional magnetic resonance imaging studies. To examine the reward-related activation, we recorded from 170 single neurons in anterior insula of 2 monkeys during a multitrial reward schedule task, where the monkeys had to complete a schedule of 1, 2, 3, or 4 trials to earn a reward. In one block of trials a visual cue indicated whether a reward would or would not be delivered in the current trial after the monkey successfully detected that a red spot turned green, and in other blocks the visual cue was random with respect to reward delivery. Over one-quarter of 131 responsive neurons were activated when the current trial would (certain or uncertain) be rewarded if performed correctly. These same neurons failed to respond in trials that were certain, as indicated by the cue, to be unrewarded. Another group of neurons responded when the reward was delivered, similar to results reported previously. The dynamics of population activity in anterior insula also showed strong signals related to knowing when a reward is coming. The most parsimonious explanation is that this activity codes for a type of expected outcome, where the expectation encompasses both certain and uncertain rewards.

Reverse-filtering on extracellular action potential for waveform analysis.

For further understanding the role of serotonergic neurons, unit recordings using behaving primates are increasingly needed. A widely used criterion to identify serotonergic neuron relies on the duration of extracellular action potential (EAP). However, the duration is inaccurate due to the passband limitation needed to carry out the spike sorting. To restore an original waveform, we conducted 1) averaging the EAPs collected from the unfiltered raw signal and 2) reverse-filtering the EAPs collected from the filtered raw signal. The reconstructed waveforms by these analyses well agreed with each other, suggesting that either analysis is applicable to restore the EAPs. Even using multivariate analyses, the reconstructed EAPs in dorsal raphe (DR) could not be divided into different clusters possibly related to neurochemicals, suggesting that the DR neurons in the previous studies that relied on waveform criterion might contain both serotonergic and non-serotonergic neurons. Between DR and orbitofrontal cortex (OFC), there were no differences in the duration of filtered EAPs. However, against the conventional criterion, the duration of the first crest in the unfiltered / restored EAPs in DR that might contain serotonergic neurons was shorter than that in OFC. This raises a possibility that the conventional waveform criterion needs further consideration.

Recency memory effects in Macaques during sequential delayed match-to-sample task with visual noise.

Visual object recognition requires both visual sensory information and memory, and its mechanisms are often studied using old-world monkeys. Wittig et al. (2014, 2016) reported that Rhesus monkeys and humans seem to adopt different strategies in a short-term visual memory task. The Rhesus monkeys seemed to rely on recency of stimulus repetition, whereas humans relied on specific memorization. We conducted experiments using a sequential delayed match-to-sample task with random dot visual noise using Rhesus and Japanese monkeys and found that recency effect was observed in both species. There were differences in the noise effect on behavioral performances across species.

The effect of 5-HT1A receptor antagonist on reward-based decision-making.

When choosing the best action from several alternatives, we compare each value that depends on the balance between benefit and cost. Previous studies have shown that animals and humans with low brain serotonin (5-HT) level tend to choose smaller immediate reward. We used a decision-making schedule task to investigate whether 5-HT1A receptor is responsible for the decisions related to reward. In this task, the monkeys chose either of two different alternatives that were comprised of 1-4 drops of liquid reward (benefit) and 1-4 repeats of a color discrimination trial (workload cost), then executed the chosen schedule. By the administration of 5-HT1A antagonist, WAY100635, the choice tendency did not change, however, the sensitivity to the amount of reward in the schedule part was diminished. The 5-HT1A could have a role in maintaining reward value to keep track with the promised reward rather than modulating workload discounting of reward value.

Neurons in the monkey orbitofrontal cortex mediate reward value computation and decision-making.

Choice reflects the values of available alternatives; more valuable options are chosen more often than less valuable ones. Here we studied whether neuronal responses in orbitofrontal cortex (OFC) reflect the value difference between options, and whether there is a causal link between OFC neuronal activity and choice. Using a decision-making task where two visual stimuli were presented sequentially, each signifying a value, we showed that when the second stimulus appears many neurons encode the value difference between alternatives. Later when the choice occurs, that difference signal disappears and a signal indicating the chosen value emerges. Pharmacological inactivation of OFC neurons coding for choice-related values increases the monkey's latency to make a choice and the likelihood that it will choose the less valuable alternative, when the value difference is small. Thus, OFC neurons code for value information that could be used to directly influence choice.

A comparison of reward-contingent neuronal activity in monkey orbitofrontal cortex and ventral striatum: guiding actions toward rewards.

We have investigated how neuronal activity in the orbitofrontal-ventral striatal circuit is related to reward-directed behavior by comparing activity in these two regions during a visually guided reward schedule task. When a set of visual cues provides information about reward contingency, that is, about whether or not a trial will be rewarded, significant subpopulations of neurons in both orbitofrontal cortex and ventral striatum encode this information. Orbitofrontal and ventral striatal neurons also differentiate between rewarding and non-rewarding trial outcomes, whether or not those outcomes were predicted. The size of the neuronal subpopulation encoding reward contingency is twice as large in orbitofrontal cortex (50% of neurons) as in ventral striatum (26%). Reward-contingency-dependent activity also appears earlier during a trial in orbitofrontal cortex than in ventral striatum. The peak reward-contingency representation in orbitofrontal cortex (31% of neurons), occurs during the wait period, a period of high anticipation prior to any action. The peak ventral striatal representation of reward contingency (18%) occurs during the go period, a time of action. We speculate that signals from orbitofrontal cortex bias ventral striatal activity, and that a flow of reward-contingency information from orbitofrontal cortex to ventral striatum serves to guide actions toward rewards.

Mode changes in activity of single neurons in anterior insular cortex across trials during multi-trial reward schedules.

We previously showed that spike count response distributions in anterior cingulate neurons can be fitted by a mixture of a few Poisson distributions in our reward schedule task. Here we report that the neuronal responses in insular cortex, an area connected to anterior cingulate cortex, can also be nicely fitted. The ratio of Poisson distributions changed with schedule progress, suggesting that neuronal responses in these areas fall into discrete firing modes. More insular neurons show mode changes across the schedules. The selection of firing modes might be related to cognitive processes, but seems independent across the two areas.

Information Accumulation over Time in Monkey Inferior Temporal Cortex Neurons Explains Pattern Recognition Reaction Time under Visual Noise.

We recognize objects even when they are partially degraded by visual noise. We studied the relation between the amount of visual noise (5, 10, 15, 20, or 25%) degrading 8 black-and-white stimuli and stimulus identification in 2 monkeys performing a sequential delayed match-to-sample task. We measured the accuracy and speed with which matching stimuli were identified. The performance decreased slightly (errors increased) as the amount of visual noise increased for both monkeys. The performance remained above 80% correct, even with 25% noise. However, the reaction times markedly increased as the noise increased, indicating that the monkeys took progressively longer to decide what the correct response would be as the amount of visual noise increased, showing that the monkeys trade time to maintain accuracy. Thus, as time unfolds the monkeys act as if they are accumulating the information and/or testing hypotheses about whether the test stimulus is likely to be a match for the sample being held in short-term memory. We recorded responses from 13 single neurons in area TE of the 2 monkeys. We found that stimulus-selective information in the neuronal responses began accumulating when the match stimulus appeared. We found that the greater the amount of noise obscuring the test stimulus, the more slowly stimulus-related information by the 13 neurons accumulated. The noise induced slowing was about the same for both behavior and information. These data are consistent with the hypothesis that area TE neuron population carries information about stimulus identity that accumulates over time in such a manner that it progressively overcomes the signal degradation imposed by adding visual noise.

Characteristics of fast-spiking neurons in the striatum of behaving monkeys.

Inhibitory interneurons are the fundamental constituents of neural circuits that organize network outputs. The striatum as part of the basal ganglia is involved in reward-directed behaviors. However, the role of the inhibitory interneurons in this process remains unclear, especially in behaving monkeys. We recorded the striatal single neuron activity while monkeys performed reward-directed hand or eye movements. Presumed parvalbumin-containing GABAergic interneurons (fast-spiking neurons, FSNs) were identified based on narrow spike shapes in three independent experiments, though they were a small population (4.2%, 42/997). We found that FSNs are characterized by high-frequency and less-bursty discharges, which are distinct from the basic firing properties of the presumed projection neurons (phasically active neurons, PANs). Besides, the encoded information regarding actions and outcomes was similar between FSNs and PANs in terms of proportion of neurons, but the discharge selectivity was higher in PANs than that of FSNs. The coding of actions and outcomes in FSNs and PANs was consistently observed under various behavioral contexts in distinct parts of the striatum (caudate nucleus, putamen, and anterior striatum). Our results suggest that FSNs may enhance the discharge selectivity of postsynaptic output neurons (PANs) in encoding crucial variables for a reward-directed behavior.

Differential encoding of information about progress through multi-trial reward schedules by three groups of ventral striatal neurons.

In the course of daily activity we continually judge whether the goal sought is worth the work that must be done to obtain it. The ventral striatum is thought to play a central role in making such judgments. When reward schedules are used to investigate these judgments ventral striatum neurons show responses near the time of the cue, the bar-release, and/or the reward delivery. We evaluated the type of coding that occurs at these three time points by using codes or factorizations with: (1) two states for reward versus non-reward, (2) four states for the progress in the reward schedule, and (3) six states for all of the states of the schedule, quantified using information theory and ANOVA. For the bar-release- and reward-related responses the percent variance explained was as high for the two states code as with the six states code. The information for the four state code rose slightly but significantly for the bar-release-related neurons. For the cue-related neurons the code with six states carried more information than the simpler codes. Thus, responses at different times appear to play different roles. Responses occurring early in trials differentiate all states, i.e., the path to a reward, whereas those late in trials code knowledge of impending reward.

Self-choice enhances value in reward-seeking in primates.

When an individual chooses one item from two or more alternatives, they compare the values of the expected outcomes. The outcome value can be determined by the associated reward amount, the probability of reward, and the workload required to earn the reward. Rational choice theory states that choices are made to maximize rewards over time, and that the same outcome values lead to an equal likelihood of choices. However, the theory does not distinguish between conditions with the same reward value, even when acquired under different circumstances, and does not always accurately describe real behavior. We have found that allowing a monkey to choose a reward schedule endows the schedule with extra value when compared to performance in an identical schedule that is chosen by another agent (a computer here). This behavior is not consistent with pure rational choice theory. Theoretical analysis using a modified temporal-difference learning model showed an enhanced schedule state value by self-choice. These results suggest that an increased reward value underlies the improved performances by self-choice during reward-seeking behavior.

Small effect of upcoming reward outcomes on visual cue-related neuronal activity in macaque area TE during conditional associations.

Area TE sends dense projections to the perirhinal cortex in macaque monkeys, an area in which we previously observed flexible signals regarding upcoming reward outcomes during a conditional-association cued-reward task. To investigate neuronal processing during the generation of information on upcoming reward outcomes, neuronal activities in area TE were examined. In the task, a color stimulus as Cue 1 and a pattern stimulus as Cue 2 were sequentially presented. Each pattern stimulus indicated both rewarded and unrewarded outcomes depending on the preceding color stimulus. In the activities during Cue 2 presentation, two-way analysis of variance revealed the effect of the interaction between Cue 1 and Cue 2, i.e., reward conditions, in 19 out of 133 neurons recorded in area TE. Of the 19 neurons, 12 also represented a response delineating a specific cue sequence, i.e., a trial-type activity. The latency of the reward-condition dependence in 7 neurons without the trial-type activity was indistinguishable from the latency in neurons without a trial-type activity in the perirhinal cortex. These results suggest that the effect of upcoming reward conditions is small in area TE and that the representation of reward conditions arises in areas beyond the ventral visual pathway, including the perirhinal cortex, during conditional associations.

Differential encoding of factors influencing predicted reward value in monkey rostral anterior cingulate cortex.

BACKGROUND: The value of a predicted reward can be estimated based on the conjunction of both the intrinsic reward value and the length of time to obtain it. The question we addressed is how the two aspects, reward size and proximity to reward, influence the responses of neurons in rostral anterior cingulate cortex (rACC), a brain region thought to play an important role in reward processing. METHODS AND FINDINGS: We recorded from single neurons while two monkeys performed a multi-trial reward schedule task. The monkeys performed 1-4 sequential color discrimination trials to obtain a reward of 1-3 liquid drops. There were two task conditions, a valid cue condition, where the number of trials and reward amount were associated with visual cues, and a random cue condition, where the cue was picked from the cue set at random. In the valid cue condition, the neuronal firing is strongly modulated by the predicted reward proximity during the trials. Information about the predicted reward amount is almost absent at those times. In substantial subpopulations, the neuronal responses decreased or increased gradually through schedule progress to the predicted outcome. These two gradually modulating signals could be used to calculate the effect of time on the perception of reward value. In the random cue condition, little information about the reward proximity or reward amount is encoded during the course of the trial before reward delivery, but when the reward is actually delivered the responses reflect both the reward proximity and reward amount. CONCLUSIONS: Our results suggest that the rACC neurons encode information about reward proximity and amount in a manner that is dependent on utility of reward information. The manner in which the information is represented could be used in the moment-to-moment calculation of the effect of time and amount on predicted outcome value.

Effect of visual noise on pattern recognition.

We recognize objects even when they are partially degraded by visual noise. Using monkeys performing a sequential delayed match-to-sample task, we studied the relation between the amount of visual noise (5, 10, 15, 20 or 25%) degrading the eight black and white stimuli used here, and the accuracy and speed with which matching stimuli were identified. The correct response rate decreased slightly as the amount of visual noise increased for both monkeys. Even at the 25% noise level, the correct response rate was more than 80%, indicating that the monkeys can recognize the pattern they are trying to match when the pattern is masked with visual noise. In contrast, the reaction time to the match stimulus increased substantially as the amount of visual noise increased. Thus, the monkeys appear to be trading time to maintain accuracy, suggesting that the monkeys are accumulating information and/or testing hypotheses about whether the test stimulus is likely to be a match for the sample being held in short-term memory.

Neuronal firing in anterior cingulate neurons changes modes across trials in single states of multitrial reward schedules.

The recorded responses of single neurons often vary considerably in the numbers of spikes emitted across repeats of a single experimental condition. Because of this irregularity and for theoretical convenience the responses are often approximated using a Poisson process. However, it has been frequently pointed out that many details of the responses, including the distribution of spike counts across similar trials, are not consistent with a Poisson process, even an inhomogeneous one. Wiener and Richmond (2003, J Neurosci 23:2394-2406) showed that the spike count distributions could usually be fitted nicely by mixtures of a few (1-3) Poisson distributions, a step they regarded as a computational convenience. Now, we find that a substantial proportion (47%) of the neuronal responses from anterior cingulate cortex, which we conceptualize as part of a system related to the balance between work and reward, have responses with multimodal firing rate distributions. When these distributions are modeled as mixtures of Poisson distributions, the proportions of the different Poisson distributions are related to behavioral state, and might be related to cognitive factors. This suggests that the neurons undergo behaviorally-related mode changes.

Anterior cingulate: single neuronal signals related to degree of reward expectancy.

As monkeys perform schedules containing several trials with a visual cue indicating reward proximity, their error rates decrease as the number of remaining trials decreases, suggesting that their motivation and/or reward expectancy increases as the reward approaches. About one-third of single neurons recorded in the anterior cingulate cortex of monkeys during these reward schedules had responses that progressively changed strength with reward expectancy, an effect that disappeared when the cue was random. Alterations of this progression could be the basis for the changes from normal that are reported in anterior cingulate population activity for obsessive-compulsive disorder and drug abuse, conditions characterized by disturbances in reward expectancy.