Neurons of Macaque Frontal Eye Field Signal Reward-Related Surprise.

The frontal eye field (FEF) plays a well-established role in the control of visual attention. The strength of an FEF neuron's response to a visual stimulus presented in its receptive field is enhanced if the stimulus captures spatial attention by virtue of its salience. A stimulus can be rendered salient by cognitive factors as well as by physical attributes. These include surprise. The aim of the present experiment was to determine whether surprise-induced salience would result in enhanced visual-response strength in the FEF. Toward this end, we monitored neuronal activity in two male monkeys while presenting first a visual cue predicting with high probability that the reward delivered at the end of the trial would be good or bad (large or small) and then a visual cue announcing the size of the impending reward with certainty. The second cue usually confirmed but occasionally violated the expectation set up by the first cue. Neurons responded more strongly to the second cue when it violated than when it confirmed expectation. The increase in the firing rate was accompanied by a decrease in spike-count correlation as expected from capture of attention. Although both good surprise and bad surprise induced enhanced firing, the effects appeared to arise from distinct mechanisms as indicated by the fact that the bad-surprise signal appeared at a longer latency than the good-surprise signal and by the fact that the strength of the two signals varied independently across neurons.

Independent repetition suppression in macaque area V2 and inferotemporal cortex.

When a complexly structured natural image is presented twice in succession, first as adapter and then as test, neurons in area TE of macaque inferotemporal cortex exhibit repetition suppression, responding less strongly to the second presentation than to the first. This phenomenon, which has been studied primarily in TE, might plausibly be argued to arise in TE because TE neurons respond selectively to complex images and thus carry information adequate for determining whether an image is or is not a repeat. However, the idea has never been put to a direct test. To resolve this issue, we monitored neuronal responses to sequences of complex natural images under identical conditions in areas V2 and TE. We found that repetition suppression occurs in both areas. Moreover, in each area, suppression takes the form of a dynamic alteration whereby the initial peak of excitation is followed by a trough and then a rebound of firing rate. To assess whether repetition suppression in either area is transmitted from the other area, we analyzed the timing of the phenomenon and its degree of spatial generalization. Suppression occurs at shorter latency in V2 than in TE. Therefore it is not simply fed back from TE. Suppression occurs in TE but not in V2 under conditions in which the test and adapter are presented in different visual field quadrants. Therefore it is not simply fed forward from V2. We conclude that repetition suppression occurs independently in V2 and TE.NEW & NOTEWORTHY When a complexly structured natural image is presented twice in rapid succession, neurons in inferotemporal area TE exhibit repetition suppression, responding less strongly to the second than to the first presentation. We have explored whether this phenomenon is confined to high-order areas where neurons respond selectively to such images and thus carry information relevant to recognizing a repeat. We have found surprisingly that repetition suppression occurs even in low-order visual area V2.

Contribution of individual features to repetition suppression in macaque inferotemporal cortex.

When an image is presented twice in succession, neurons in area TE of macaque inferotemporal cortex exhibit repetition suppression, responding less strongly to the second presentation than to the first. Suppression is known to occur if the adapter and the test image are subtly different from each other. However, it is not known whether cross suppression occurs between images that are radically different from each other but that share a subset of features. To explore this issue, we measured repetition suppression using colored shapes. On interleaved trials, the test image might be identical to the adapter, might share its shape or color alone, or might differ from it totally. At the level of the neuronal population as a whole, suppression was especially deep when adapter and test were identical, intermediate when they shared only one attribute, and minimal when they shared neither attribute. At the level of the individual neuron, the degree of suppression depended not only on the properties of the two images but also on the preferences of the neuron. Suppression was deeper when the repeated color or shape was preferred by the neuron than when it was not. This effect might arise from feature-specific adaptation or alternatively from adapter-induced fatigue. Both mechanisms conform to the principle that the degree of suppression is determined by the preferences of the neuron.NEW & NOTEWORTHY When an image is presented twice in rapid succession, neurons of inferotemporal cortex exhibit repetition suppression, responding less strongly to the second than to the first presentation. It has been unclear whether this phenomenon depends on the selectivity of the neuron under study. Here, we show that, for a given neuron, suppression is deepest when features preferred by that neuron are repeated. The results argue for a mechanism based either on feature-specific suppression or fatigue.

Neural Correlate of Visual Familiarity in Macaque Area V2.

Neurons in macaque inferotemporal cortex (ITC) respond less strongly to familiar than to novel images. It is commonly assumed that this effect arises within ITC because its neurons respond selectively to complex images and thus encode in an explicit form information sufficient for identifying a particular image as familiar. However, no prior study has examined whether neurons in low-order visual areas selective for local features also exhibit familiarity suppression. To address this issue, we recorded from neurons in macaque area V2 with semichronic microelectrode arrays while monkeys repeatedly viewed a set of large complex natural images. We report here that V2 neurons exhibit familiarity suppression. The effect develops over several days with a trajectory well fitted by an exponential function with a rate constant of ∼100 exposures. Suppression occurs in V2 at a latency following image onset shorter than its reported latency in ITC.SIGNIFICANCE STATEMENT Familiarity suppression, the tendency for neurons to respond less strongly to familiar than novel images, is well known in monkey inferotemporal cortex. Suppression has been thought to arise in inferotemporal cortex because its neurons respond selectively to large complex images and thus explicitly to encode information sufficient for identifying a particular image as familiar. No previous study has explored the possibility that familiarity suppression occurs even in early-stage visual areas where neurons are selective for simple features in confined receptive fields. We now report that neurons in area V2 exhibit familiarity suppression. This finding challenges our current understanding of information processing in V2 as well as our understanding of the mechanisms that underlie familiarity suppression.

Prediction suppression and surprise enhancement in monkey inferotemporal cortex.

Exposing monkeys, over the course of days and weeks, to pairs of images presented in fixed sequence, so that each leading image becomes a predictor for the corresponding trailing image, affects neuronal visual responsiveness in area TE. At the end of the training period, neurons respond relatively weakly to a trailing image when it appears in a trained sequence and, thus, confirms prediction, whereas they respond relatively strongly to the same image when it appears in an untrained sequence and, thus, violates prediction. This effect could arise from prediction suppression (reduced firing in response to the occurrence of a probable event) or surprise enhancement (elevated firing in response to the omission of a probable event). To identify its cause, we compared firing under the prediction-confirming and prediction-violating conditions to firing under a prediction-neutral condition. The results provide strong evidence for prediction suppression and limited evidence for surprise enhancement.NEW & NOTEWORTHY In predictive coding models of the visual system, neurons carry signed prediction error signals. We show here that monkey inferotemporal neurons exhibit prediction-modulated firing, as posited by these models, but that the signal is unsigned. The response to a prediction-confirming image is suppressed, and the response to a prediction-violating image may be enhanced. These results are better explained by a model in which the visual system emphasizes unpredicted events than by a predictive coding model.

In monkeys making value-based decisions, amygdala neurons are sensitive to cue value as distinct from cue salience.

Neurons in the lateral intraparietal (LIP) area of macaque monkey parietal cortex respond to cues predicting rewards and penalties of variable size in a manner that depends on the motivational salience of the predicted outcome (strong for both large reward and large penalty) rather than on its value (positive for large reward and negative for large penalty). This finding suggests that LIP mediates the capture of attention by salient events and does not encode value in the service of value-based decision making. It leaves open the question whether neurons elsewhere in the brain encode value in the identical task. To resolve this issue, we recorded neuronal activity in the amygdala in the context of the task employed in the LIP study. We found that responses to reward-predicting cues were similar between areas, with the majority of reward-sensitive neurons responding more strongly to cues that predicted large reward than to those that predicted small reward. Responses to penalty-predicting cues were, however, markedly different. In the amygdala, unlike LIP, few neurons were sensitive to penalty size, few penalty-sensitive neurons favored large over small penalty, and the dependence of firing rate on penalty size was negatively correlated with its dependence on reward size. These results indicate that amygdala neurons encoded cue value under circumstances in which LIP neurons exhibited sensitivity to motivational salience. However, the representation of negative value, as reflected in sensitivity to penalty size, was weaker than the representation of positive value, as reflected in sensitivity to reward size.NEW & NOTEWORTHY This is the first study to characterize amygdala neuronal responses to cues predicting rewards and penalties of variable size in monkeys making value-based choices. Manipulating reward and penalty size allowed distinguishing activity dependent on motivational salience from activity dependent on value. This approach revealed in a previous study that neurons of the lateral intraparietal (LIP) area encode motivational salience. Here, it reveals that amygdala neurons encode value. The results establish a sharp functional distinction between the two areas.

Prediction suppression in monkey inferotemporal cortex depends on the conditional probability between images.

When monkeys view two images in fixed sequence repeatedly over days and weeks, neurons in area TE of the inferotemporal cortex come to exhibit prediction suppression. The trailing image elicits only a weak response when presented following the leading image that preceded it during training. Induction of prediction suppression might depend either on the contiguity of the images, as determined by their co-occurrence and captured in the measure of joint probability P(A,B), or on their contingency, as determined by their correlation and as captured in the measures of conditional probability P(A|B) and P(B|A). To distinguish between these possibilities, we measured prediction suppression after imposing training regimens that held P(A,B) constant but varied P(A|B) and P(B|A). We found that reducing either P(A|B) or P(B|A) during training attenuated prediction suppression as measured during subsequent testing. We conclude that prediction suppression depends on contingency, as embodied in the predictive relations between the images, and not just on contiguity, as embodied in their co-occurrence.

Macaque monkeys experience visual crowding.

In peripheral vision, objects that are easily discriminated on their own become less discriminable in the presence of surrounding clutter. This phenomenon is known as crowding.The neural mechanisms underlying crowding are not well understood. Better insight might come from single-neuron recording in nonhuman primates, provided they exhibit crowding; however, previous demonstrations of crowding have been confined to humans. In the present study, we set out to determine whether crowding occurs in rhesus macaque monkeys. We found that animals trained to identify a target letter among flankers displayed three hallmarks of crowding as established in humans. First, at a given eccentricity, increasing the spacing between the target and the flankers improved recognition accuracy. Second, the critical spacing, defined as the minimal spacing at which target discrimination was reliable, was proportional to eccentricity. Third, the critical spacing was largely unaffected by object size. We conclude that monkeys, like humans, experience crowding. These findings open the door to studies of crowding at the neuronal level in the monkey visual system.

Image familiarization sharpens response dynamics of neurons in inferotemporal cortex.

Repeated viewing of an image over days and weeks induces a marked reduction in the strength with which neurons in monkey inferotemporal cortex respond to it. The processing advantage that attaches to this reduction is unknown. One possibility is that truncation of the response to a familiar image leaves neurons in a state of readiness to respond to ensuing images and thereby enhances their ability to track rapidly changing displays. We explored this possibility by assessing neuronal responses to familiar and novel images in rapid serial visual displays. Inferotemporal neurons responded more strongly to familiar than to novel images in such displays. The effect was stronger among putative inhibitory neurons than among putative excitatory neurons. A comparable effect occurred at the level of the scalp potential in humans. We conclude that long-term familiarization sharpens the response dynamics of neurons in both monkey and human extrastriate visual cortex.

Statistical learning of serial visual transitions by neurons in monkey inferotemporal cortex.

If monkeys repeatedly, over the course of weeks, view displays in which two images appear in fixed sequence, then neurons of inferotemporal cortex (ITC) come to exhibit prediction suppression. The response to the trailing image is weaker if it follows the leading image with which it was paired during training than if it follows some other leading image. Prediction suppression is a plausible neural mechanism for statistical learning of visual transitions such as has been demonstrated in behavioral studies of human infants and adults. However, in the human studies, subjects are exposed to continuous sequences in which the same image can be both predicted and predicting and statistical dependency can exist between nonadjacent items. The aim of the present study was to investigate whether prediction suppression in ITC develops under such circumstances. To resolve this issue, we exposed monkeys repeatedly to triplets of images presented in fixed order. The results indicate that prediction suppression can be induced by training not only with pairs of images but also with longer sequences.

Intracortical microstimulation of supplementary eye field impairs ability of monkeys to make serially ordered saccades.

Neurons in the supplementary eye field (SEF) of the macaque monkey exhibit rank selectivity, firing differentially as a function of the phase attained during the performance of a task requiring the execution of saccades to a series of objects in fixed order. The activity of these neurons is commonly thought to represent ordinal position in the service of serial-order performance. However, there is little evidence causally linking neuronal activity in the SEF to sequential behavior. To explore the role of the SEF in serial-order performance, we delivered intracortical microstimulation while monkeys performed a task requiring them to make saccades to three objects in a fixed order on each trial. Microstimulation, considered on average across all SEF sites and all phases of the trial, affected saccadic kinematics. In particular, it prolonged the reaction time, increased the peak velocity, and slightly increased the amplitude of saccades. In addition, it interfered with the monkeys' ability to select the target appropriate to a given phase of the trial. The pattern of the errors was such as would be expected if microstimulation shifted the neural representation of ordinal position toward a later phase of the trial.

Response to comment on "In monkeys making value-based decisions, LIP neurons encode cue salience and not action value".

Newsome et al. question neither our key result, that large-penalty cues elicited stronger responses than small-penalty cues, nor our key conclusion, that neurons early in the trial signaled cue salience and not action value. Instead, they focus on subsequent neuronal activity. The patterns of delay-period activity that they note can be explained by reference to experimental methodology.

In monkeys making value-based decisions, LIP neurons encode cue salience and not action value.

In monkeys deciding between alternative saccadic eye movements, lateral intraparietal (LIP) neurons representing each saccade fire at a rate proportional to the value of the reward expected upon its completion. This observation has been interpreted as indicating that LIP neurons encode saccadic value and that they mediate value-based decisions between saccades. Here, we show that LIP neurons representing a given saccade fire strongly not only if it will yield a large reward but also if it will incur a large penalty. This finding indicates that LIP neurons are sensitive to the motivational salience of cues. It is compatible neither with the idea that LIP neurons represent action value nor with the idea that value-based decisions take place in LIP neurons.

Bayesian nonparametric analysis of neuronal intensity rates.

We propose a flexible hierarchical Bayesian nonparametric modeling approach to compare the spiking patterns of neurons recorded under multiple experimental conditions. In particular, we showcase the application of our statistical methodology using neurons recorded from the supplementary eye field region of the brains of two macaque monkeys trained to make delayed eye movements to three different types of targets. The proposed Bayesian methodology can be used to perform either a global analysis, allowing for the construction of posterior comparative intervals over the entire experimental time window, or a pointwise analysis for comparing the spiking patterns locally, in a predetermined portion of the experimental time window. By developing our nonparametric Bayesian model we are able to analyze neuronal data from three or more conditions while avoiding the computational expenses typically associated with more traditional analysis of physiological data.

Statistical learning of visual transitions in monkey inferotemporal cortex.

One of the most fundamental functions of the brain is to predict upcoming events on the basis of the recent past. A closely related function is to signal when a prediction has been violated. The identity of the brain regions that mediate these functions is not known. We set out to determine whether they are implemented at the level of single neurons in the visual system. We gave monkeys prolonged exposure to pairs of images presented in fixed sequence so that each leading image became a strong predictor for the corresponding trailing image. We then monitored the responses of neurons in the inferotemporal cortex to image sequences that obeyed or violated the transitional rules imposed during training. Inferotemporal neurons exhibited a transitional surprise effect, responding much more strongly to unpredicted transitions than to predicted transitions. Thus, neurons even in the visual system make experience-based predictions and react when they fail.

Relation of ordinal position signals to the expectation of reward and passage of time in four areas of the macaque frontal cortex.

Neurons in several areas of the monkey frontal cortex exhibit rank selectivity, firing differentially as a function of the stage attained during the performance of a serial order task. The activity of these neurons is commonly thought to represent ordinal position within the trial. However, they might also be sensitive to factors correlated with ordinal position including time elapsed during the trial (which is greater for each successive stage) and the degree of anticipation of reward (which probably increases at each successive stage). To compare the influences of these factors, we monitored neuronal activity in the supplementary motor area (SMA), presupplementary motor area (pre-SMA), supplementary eye field (SEF), and dorsolateral prefrontal cortex during the performance of a serial order task (requiring a series of saccades in three specified directions), a variable reward task (in which a cue displayed early in the trial indicated whether the reward received at the end of the trial would be large or small), and a long delay task (in which the monkey had simply to maintain fixation during a period of time approximating the duration of an average trial in the serial order task). We found that rank signals were partially correlated with sensitivity to elapsed time and anticipated reward. The connection to elapsed time was strongest in the pre-SMA. The connection to anticipated reward was most pronounced in the SMA and SEF. However, critically, these factors could not fully explain rank selectivity in any of the areas tested.

Responses to compound objects in monkey inferotemporal cortex: the whole is equal to the sum of the discrete parts.

It is commonly thought that neurons in monkey inferotemporal cortex are conjunction selective--that a neuron will respond to an image if and only if it contains a required combination of parts. However, this view is based on the results of experiments manipulating closely adjacent or confluent parts. Neurons may have been sensitive not to the conjunction of parts as such but to the presence of a unique feature created where they abut. Here, we compare responses to two sets of images, one composed of spatially separate and the other of abutting parts. We show that the influences of spatially separate parts combine, to a very close approximation, according to a linear rule. Nonlinearities are more prominent--although still weak--in responses to images composed of abutting parts.

Rank signals in four areas of macaque frontal cortex during selection of actions and objects in serial order.

Neurons in several areas of monkey frontal cortex exhibit ordinal position (rank) selectivity during the performance of serial order tasks. It has been unclear whether rank selectivity or the dependence of rank selectivity on task context varies across the areas of frontal cortex. To resolve this issue, we recorded from neurons in the supplementary motor area (SMA), presupplementary motor area (pre-SMA), supplementary eye field (SEF), and dorsolateral prefrontal cortex (dlPFC) as monkeys performed two oculomotor tasks, one requiring the selection of three actions in sequence and the other requiring the selection of three objects in sequence. We found that neurons representing all ranks were present in all areas. Only to a moderate degree did the prevalence and nature of rank selectivity vary from area to area. The two most prominent inter-area differences involved a lower prevalence of rank selectivity in the dlPFC than in the other areas and a higher proportion of neurons preferring late ranks in the SMA and SEF than in the other areas. Neurons in all four areas are rank generalists in the sense of favoring the same rank in both the serial action task and the serial object task.

Global image dissimilarity in macaque inferotemporal cortex predicts human visual search efficiency.

Finding a target in a visual scene can be easy or difficult depending on the nature of the distractors. Research in humans has suggested that search is more difficult the more similar the target and distractors are to each other. However, it has not yielded an objective definition of similarity. We hypothesized that visual search performance depends on similarity as determined by the degree to which two images elicit overlapping patterns of neuronal activity in visual cortex. To test this idea, we recorded from neurons in monkey inferotemporal cortex (IT) and assessed visual search performance in humans using pairs of images formed from the same local features in different global arrangements. The ability of IT neurons to discriminate between two images was strongly predictive of the ability of humans to discriminate between them during visual search, accounting overall for 90% of the variance in human performance. A simple physical measure of global similarity--the degree of overlap between the coarse footprints of a pair of images--largely explains both the neuronal and the behavioral results. To explain the relation between population activity and search behavior, we propose a model in which the efficiency of global oddball search depends on contrast-enhancing lateral interactions in high-order visual cortex.

Representing the forest before the trees: a global advantage effect in monkey inferotemporal cortex.

Hierarchical stimuli (large shapes composed of small shapes) have long been used to study how humans perceive the global and the local content of a scene--the forest and the trees. Studies using these stimuli have revealed a global advantage effect: humans consistently report global shape faster than local shape. The neuronal underpinnings of this effect remain unclear. Here we demonstrate a correlate and possible mechanism in monkey inferotemporal cortex (IT). Inferotemporal neurons signal the global content of a hierarchical display approximately 30 ms before they signal its local content. This is a specific expression of a general principle, related to spatial scale or spatial frequency rather than to hierarchical level, whereby the representation of a large shape develops in IT before that of a small shape. These findings provide support for a coarse-to-fine model of visual scene representation.