Mixed Selectivity Coding of Content-Temporal Detail by Dorsomedial Posterior Parietal Neurons.

The dorsomedial posterior parietal cortex (dmPPC) is part of a higher-cognition network implicated in elaborate processes underpinning memory formation, recollection, episode reconstruction, and temporal information processing. Neural coding for complex episodic processing is however under-documented. Here, we recorded extracellular neural activities from three male rhesus macaques (Macaca mulatta) and revealed a set of neural codes of "neuroethogram" in the primate parietal cortex. Analyzing neural responses in macaque dmPPC to naturalistic videos, we discovered several groups of neurons that are sensitive to different categories of ethogram items, low-level sensory features, and saccadic eye movement. We also discovered that the processing of category and feature information by these neurons is sustained by the accumulation of temporal information over a long timescale of up to 30 s, corroborating its reported long temporal receptive windows. We performed an additional behavioral experiment with additional two male rhesus macaques and found that saccade-related activities could not account for the mixed neuronal responses elicited by the video stimuli. We further observed monkeys' scan paths and gaze consistency are modulated by video content. Taken altogether, these neural findings explain how dmPPC weaves fabrics of ongoing experiences together in real time. The high dimensionality of neural representations should motivate us to shift the focus of attention from pure selectivity neurons to mixed selectivity neurons, especially in increasingly complex naturalistic task designs.

A One-Shot Shift from Explore to Exploit in Monkey Prefrontal Cortex.

Much animal learning is slow, with cumulative changes in behavior driven by reward prediction errors. When the abstract structure of a problem is known, however, both animals and formal learning models can rapidly attach new items to their roles within this structure, sometimes in a single trial. Frontal cortex is likely to play a key role in this process. To examine information seeking and use in a known problem structure, we trained monkeys in an explore/exploit task, requiring the animal first to test objects for their association with reward, then, once rewarded objects were found, to reselect them on further trials for further rewards. Many cells in the frontal cortex showed an explore/exploit preference aligned with one-shot learning in the monkeys' behavior: the population switched from an explore state to an exploit state after a single trial of learning but partially maintained the explore state if an error indicated that learning had failed. Binary switch from explore to exploit was not explained by continuous changes linked to expectancy or prediction error. Explore/exploit preferences were independent for two stages of the trial: object selection and receipt of feedback. Within an established task structure, frontal activity may control the separate processes of explore and exploit, switching in one trial between the two.SIGNIFICANCE STATEMENT Much animal learning is slow, with cumulative changes in behavior driven by reward prediction errors. When the abstract structure a problem is known, however, both animals and formal learning models can rapidly attach new items to their roles within this structure. To address transitions in neural activity during one-shot learning, we trained monkeys in an explore/exploit task using familiar objects and a highly familiar task structure. When learning was rapid, many frontal neurons showed a binary, one-shot switch between explore and exploit. Within an established task structure, frontal activity may control the separate operations of exploring alternative objects to establish their current role, then exploiting this knowledge for further reward.

Integrated neural dynamics for behavioural decisions and attentional competition in the prefrontal cortex.

In the behaving monkey, complex neural dynamics in the prefrontal cortex contribute to context-dependent decisions and attentional competition. We used demixed principal component analysis to track prefrontal activity dynamics in a cued target detection task. In this task, the animal combined identity of a visual object with a prior instruction cue to determine a target/nontarget decision. From population activity, we extracted principal components for each task feature and examined their time course and sensitivity to stimulus and task variations. For displays containing a single choice object in left or right hemifield, object identity, cue identity and decision were all encoded in population activity, with different dynamics and lateralisation. Object information peaked at 100-200 ms from display onset and was largely confined to the contralateral hemisphere. Cue information was weaker and present even prior to display onset. Integrating information from cue and object, decision information arose more slowly and was bilateral. Individual neurons contributed independently to coding of the three task features. The analysis was then extended to displays with a target in one hemifield and a competing distractor in the other. In this case, the data suggest that each hemisphere initially encoded the identity of the contralateral object. The distractor representation was then rapidly suppressed, with the final target decision again encoded bilaterally. The results show how information is coded along task-related dimensions while competition is resolved and suggest how information flows within and across frontal lobes to implement a learned behavioural decision.

Focused Representation of Successive Task Episodes in Frontal and Parietal Cortex.

Complex cognition is dynamic, with each stage of a task requiring new cognitive processes appropriately linked to stimulus or other content. To investigate control over successive task stages, we recorded neural activity in lateral frontal and parietal cortex as monkeys carried out a complex object selection task, with each trial separated into phases of visual selection and learning from feedback. To study capacity limitation, complexity was manipulated by varying the number of object targets to be learned in each problem. Different task phases were associated with quasi-independent patterns of activity and information coding, with no suggestion of sustained activity linked to a current target. Object and location coding were largely parallel in frontal and inferior parietal cortex, though frontal cortex showed somewhat stronger object representation at feedback, and more sustained location coding at choice. At both feedback and choice, coding precision diminished as task complexity increased, matching a decline in performance. We suggest that, across successive task steps, there is radical but capacity-limited reorganization of frontoparietal activity, selecting different cognitive operations linked to their current targets.

Spatial and temporal distribution of visual information coding in lateral prefrontal cortex.

Prefrontal neurons code many kinds of behaviourally relevant visual information. In behaving monkeys, we used a cued target detection task to address coding of objects, behavioural categories and spatial locations, examining the temporal evolution of neural activity across dorsal and ventral regions of the lateral prefrontal cortex (encompassing parts of areas 9, 46, 45A and 8A), and across the two cerebral hemispheres. Within each hemisphere there was little evidence for regional specialisation, with neurons in dorsal and ventral regions showing closely similar patterns of selectivity for objects, categories and locations. For a stimulus in either visual field, however, there was a strong and temporally specific difference in response in the two cerebral hemispheres. In the first part of the visual response (50-250 ms from stimulus onset), processing in each hemisphere was largely restricted to contralateral stimuli, with strong responses to such stimuli, and selectivity for both object and category. Later (300-500 ms), responses to ipsilateral stimuli also appeared, many cells now responding more strongly to ipsilateral than to contralateral stimuli, and many showing selectivity for category. Activity on error trials showed that late activity in both hemispheres reflected the animal's final decision. As information is processed towards a behavioural decision, its encoding spreads to encompass large, bilateral regions of prefrontal cortex.

Cycles of goal silencing and reactivation underlie complex problem-solving in primate frontal and parietal cortex.

While classic views proposed that working memory (WM) is mediated by sustained firing, recent evidence suggests a contribution of activity-silent states. Within WM, human neuroimaging studies suggest a switch between attentional foreground and background, with only the foregrounded item represented in active neural firing. To address this process at the cellular level, we recorded prefrontal (PFC) and posterior parietal (PPC) neurons in a complex problem-solving task, with monkeys searching for one or two target locations in a first cycle of trials, and retaining them for memory-guided revisits on subsequent cycles. When target locations were discovered, neither frontal nor parietal neurons showed sustained goal-location codes continuing into subsequent trials and cycles. Instead there were sequences of timely goal silencing and reactivation, and following reactivation, sustained states until behavioral response. With two target locations, goal representations in both regions showed evidence of transitions between foreground and background, but the PFC representation was more complete, extending beyond the current trial to include both past and future selections. In the absence of unbroken sustained codes, different neuronal states interact to support maintenance and retrieval of WM representations across successive trials.

Frontal and temporal coding dynamics in successive steps of complex behavior.

Ventrolateral prefrontal cortex (vlPFC), dorsolateral prefrontal cortex (dlPFC), and temporal cortex (TE) all contribute to visual decision-making. Accumulating evidence suggests that vlPFC may play a central role in multiple cognitive operations, perhaps resembling domain-general regions of the human frontal lobe. We trained monkeys in a task calling for learning, retrieval, and spatial selection of rewarded target objects. Recordings of neural activity covered large areas of vlPFC, dlPFC, and TE. Results suggested a central role for vlPFC in each cognitive operation with strong coding of each task feature, while only location was strongly coded in dlPFC and current object identity in TE. During target selection, target location was communicated first from vlPFC to dlPFC, followed by extensive mutual support. In vlPFC, stimulus identities were independently coded in different task operations. The results suggest a central role for the inferior frontal convexity in controlling successive operations of a complex, multi-step task.

Hierarchical coding for sequential task events in the monkey prefrontal cortex.

The frontal lobes play a key role in sequential organization of behavior. Little is known, however, of the way frontal neurons code successive phases of a structured task plan. Using correlational analysis, we asked how a population of frontal cells represents the multiple events of a complex sequential task. Monkeys performed a conventional cue-target association task, with distinct cue, delay, and target phases. Across the population of recorded cells, we examined patterns of activity for different task phases, and in the same phase, for different stimulus objects. The results show hierarchical representation of task events. For different task phases, there were different, approximately orthogonal patterns of activity across the population of neurons. Modulations of each basic pattern encoded stimulus information within each phase. By orthogonal coding, the frontal lobe may control transitions between the discrete steps of a mental program; by correlated coding within each step, similar operations may be applied to different stimulus content.

Target detection by opponent coding in monkey prefrontal cortex.

The pFC plays a key role in flexible, context-specific decision making. One proposal [Machens, C. K., Romo, R., & Brody, C. D. Flexible control of mutual inhibition: A neural model of two-interval discrimination. Science, 307, 1121-1124, 2005] is that prefrontal cells may be dynamically organized into opponent coding circuits, with competitive groups of cells coding opposite behavioral decisions. Here, we show evidence for extensive, temporally evolving opponent organization in the monkey pFC during a cued target detection task. More than a half of all randomly selected cells discriminated stimulus category in this task. The largest set showed target-positive activity, with the strongest responses to the current target, intermediate activity for a nontarget that was a target on other trials, and lowest activity for nontargets never associated with the target category. Second most frequent was a reverse, antitarget pattern. In the ventrolateral frontal cortex, opponent organization was strongly established in phasic responses at stimulus onset; later, such activity was widely spread across dorsolateral and ventrolateral sites. Task-specific organization into opponent cell groups may be a general feature of prefrontal decision making.

Detection of fixed and variable targets in the monkey prefrontal cortex.

Behavioral significance is commonly coded by prefrontal neurons. The significance of a stimulus can be fixed through experience; in complex behavior, however, significance commonly changes with short-term context. To compare these cases, we trained monkeys in 2 versions of visual target detection. In both tasks, animals monitored a series of pictures, making a go response (saccade) at the offset of a specified target picture. In one version, based on "consistent mapping" in human visual search, target and nontarget pictures were fixed throughout training. In the other, based on "varied mapping," a cue at trial onset defined a new target. Building up over the first 1 s following this cue, many cells coded short-term context (cue/target identity) for the current trial. Thereafter, the cell population showed similar coding of behavioral significance in the 2 tasks, with selective early response to targets, and later, sustained activity coding target or nontarget until response. This population similarity was seen despite quite different activity in the 2 tasks for many single cells. At the population level, the results suggest similar prefrontal coding of fixed and short-term behavioral significance.

Distinguishing between parallel and serial processing in visual attention from neurobiological data.

Serial and parallel processing in visual search have been long debated in psychology, but the processing mechanism remains an open issue. Serial processing allows only one object at a time to be processed, whereas parallel processing assumes that various objects are processed simultaneously. Here, we present novel neural models for the two types of processing mechanisms based on analysis of simultaneously recorded spike trains using electrophysiological data from prefrontal cortex of rhesus monkeys while processing task-relevant visual displays. We combine mathematical models describing neuronal attention and point process models for spike trains. The same model can explain both serial and parallel processing by adopting different parameter regimes. We present statistical methods to distinguish between serial and parallel processing based on both maximum likelihood estimates and decoding the momentary focus of attention when two stimuli are presented simultaneously. Results show that both processing mechanisms are in play for the simultaneously recorded neurons, but neurons tend to follow parallel processing in the beginning after the onset of the stimulus pair, whereas they tend to serial processing later on.

Rhesus monkeys mislocalize saccade targets flashed for 100ms around the time of a saccade.

Humans and monkeys mislocalize targets flashed around the time of a saccade. Here, we present data from three monkeys on a double-step task with a 100ms target duration. All three subjects mislocalized targets that were flashed around the time of the first saccade, in spite of long intersaccadic intervals. The error was consistently in the direction opposite that of the saccade, and occurred in some cases when the target presentation was entirely presaccadic. This is inconsistent with a theory invoking a damped representation of eye position, but it is consistent with the hypothesis that it is due to an error in peri-saccadic remapping.

The role of the lateral intraparietal area of the monkey in the generation of saccades and visuospatial attention.

The brain cannot monitor or react towards the entire world at a given time. Instead, using the process of attention, it selects objects in the world for further analysis. Neuronal activity in the monkey intraparietal area has the properties appropriate for a neuronal substrate of attention: instead of all objects being represented in the parietal cortex, only salient objects are. Such objects can be salient because of their physical properties (recently flashed objects or moving objects) or because they can be made important to the animal by virtue of a task. Although lateral intraparietal area (LIP) neurons respond through the delay period of a memory-guided saccade, they also respond in an enhanced manner to distractors flashed during the delay period of a memory-guided saccade being generated to a position outside the receptive field. This activity parallels the monkey's psychophysical attentional process: attention is ordinarily pinned at the goal of a memory-guided saccade, but it shifts briefly to the locus of a task-irrelevant distractor flashed briefly during the delay period and then returns to the goal. Although neurons in LIP have been implicated as being directly involved in the generation of saccadic eye movements, their activity does not predict where, when, or if a saccade will occur. The ensemble of activity in LIP, however, does accurately describe the locus of attention.

Dynamic coding for cognitive control in prefrontal cortex.

Cognitive flexibility is fundamental to adaptive intelligent behavior. Prefrontal cortex has long been associated with flexible cognitive function, but the neurophysiological principles that enable prefrontal cells to adapt their response properties according to context-dependent rules remain poorly understood. Here, we use time-resolved population-level neural pattern analyses to explore how context is encoded and maintained in primate prefrontal cortex and used in flexible decision making. We show that an instruction cue triggers a rapid series of state transitions before settling into a stable low-activity state. The postcue state is differentially tuned according to the current task-relevant rule. During decision making, the response to a choice stimulus is characterized by an initial stimulus-specific population response but evolves to different final decision-related states depending on the current rule. These results demonstrate how neural tuning profiles in prefrontal cortex adapt to accommodate changes in behavioral context. Highly flexible tuning could be mediated via short-term synaptic plasticity.

Dynamic construction of a coherent attentional state in a prefrontal cell population.

Prefrontal cortex has been proposed to show highly adaptive information coding, with neurons dynamically allocated to processing task-relevant information. To track this dynamic allocation in monkey prefrontal cortex, we used time-resolved measures of neural population activity in a simple case of competition between target (behaviorally critical) and nontarget objects in opposite visual hemifields. Early in processing, there were parallel responses to competing inputs, with neurons in each hemisphere dominated by the contralateral stimulus. Later, the nontarget lost control of neural activity, with emerging global control by the behaviorally critical target. The speed of transition reflected the competitive weights of different display elements, occurring most rapidly when relative behavioral significance was well established by training history. In line with adaptive coding, the results show widespread reallocation of prefrontal processing resources as an attentional focus is established.

Effect of background colors on the tuning of color-selective cells in monkey area V4.

When objects are viewed in different illuminants, their color does not change or changes little in spite of significant changes in the wavelength composition of the light reflected from them. In previous studies, we have addressed the physiology underlying this color constancy by recording from cells in areas V1, V2, and V4 of the anesthetized monkey. Truly color-coded cells, ones that respond to a patch of a given color irrespective of the wavelength composition of the light reflected from it, were only found in area V4. In the present study, we have used a different approach to test the responses of V4 cells in both anesthetized and awake behaving monkeys. Stimuli of different colors, embedded within a Mondrian-type multicolored background, were used to identify the chromatic selectivity of neurons. The illumination of the background was then varied, and the tuning of V4 neurons was tested again for each background illumination. With anesthetized monkeys, the psychophysical effect of changing background illumination was inferred from our own experience, whereas in the awake behaving animal, it was directly reported by the monkey. We found that the majority of V4 neurons shifted their color-tuning profile with each change in the background illumination: each time the color of the background on the computer screen was changed so as to simulate a change in illumination, cells shifted their color-tuning function in the direction of the chromaticity component that had been increased. A similar shift was also observed in colored match-to-sample psychometric functions of both human and monkey. The shift in monkey psychometric functions was quantitatively equivalent to the shift in the responses of the corresponding population of cells. We conclude that neurons in area V4 exhibit the property of color constancy and that their response properties are thus able to reflect color perception.

Simultaneous representation of saccade targets and visual onsets in monkey lateral intraparietal area.

The monkey's lateral intraparietal area (LIP) has been associated with attention and saccades. LIP neurons have visual on-responses to objects abruptly appearing in their receptive fields (RFs) and sustained activity preceding saccades to the RF. We studied the relationship between the on-responses and delay activity in LIP using a 'stable-array' task. Monkeys viewed eight distinct, continuously illuminated objects, arranged in a circle with at least one object in the RF. A cue flashed instructing the monkey to make a saccade, after a delay, to the stable object physically matching the cue. The location of the cue was fixed in trial blocks, either in or out of the RF. If the cue was outside the RF, neurons developed delay-period activity tuned for the direction of the saccade target at approximately 190 ms after cue onset. If the cue appeared in the RF, neurons initially responded to cue onset and developed tuning for saccade direction only toward the end of the delay period, 390 ms after cue onset. The cue- and saccade-target responses coexisted throughout a significant portion of the delay period. The results show that visual-on responses and delay-period activity in LIP are functionally separable, and that, although highly selective, the salience representation in LIP can contain more than one object at a time.

The time course of perisaccadic receptive field shifts in the lateral intraparietal area of the monkey.

Neurons in the lateral intraparietal area of the monkey (LIP) have visual receptive fields in retinotopic coordinates when studied in a fixation task. However, in the period immediately surrounding a saccade these receptive fields often shift, so that a briefly flashed stimulus outside the receptive field will drive the neurons if the eye movement will bring the spatial location of that vanished stimulus into the receptive field. This is equivalent to a transient shift of the retinal receptive field. The process enables the monkey brain to process a stimulus in a spatially accurate manner after a saccade, even though the stimulus appeared only before the saccade. We studied the time course of this receptive field shift by flashing a task-irrelevant stimulus for 100 ms before, during, or after a saccade. The stimulus could appear in receptive field as defined by the fixation before the saccade (the current receptive field) or the receptive field as defined by the fixation after the saccade (the future receptive field). We recorded the activity of 48 visually responsive neurons in LIP of three hemispheres of two rhesus monkeys. We studied 45 neurons in the current receptive field task, in which the saccade removed the stimulus from the receptive field. Of these neurons 29/45 (64%) showed a significant decrement of response when the stimulus appeared 250 ms or less before the saccade, as compared with their activity during fixation. The average response decrement was 38% for those cells showing a significant (P < 0.05 by t-test) decrement. We studied 39 neurons in the future receptive field task, in which the saccade brought the spatial location of a recently vanished stimulus into the receptive field. Of these 32/39 (82%) had a significant response to stimuli flashed for 100 ms in the future receptive field, even 400 ms before the saccade. Neurons never responded to stimuli moved by the saccade from a point outside the receptive field to another point outside the receptive field. Neurons did not necessarily show any saccadic suppression for stimuli moved from one part of the receptive field to another by the saccade. Stimuli flashed <250 ms before the saccade-evoked responses in both the presaccadic and the postsaccadic receptive fields, resulting in an increase in the effective receptive field size, an effect that we suggest is responsible for perisaccadic perceptual inaccuracies.