Altered task demands lead to a division of labour for sensory and cognitive processing in the middle temporal area.

Neurons in the primate middle temporal (MT) area signal information about visual motion and work together with the lateral prefrontal cortex (LPFC) to support memory-guided comparisons of visual motion direction. These areas are reciprocally connected, and both contain neurons that signal visual motion direction in the strength of their responses. Previously, LPFC was shown to display marked changes in stimulus coding with altered task demands, including changes in selectivity for motion direction, trial-to-trial variability in responses and comparison effects. Since MT and LPFC are directly interconnected, we sought to determine if MT neurons display similar dependence on task demands. We found that active participation in a motion direction comparison task affected both sensory and nonsensory activity in MT neurons. In fact, neurons that became less selective for motion direction during the active task showed increased signalling for cognitive aspects of the task. This heterogeneity in neural modification with heightened task demands suggests a division of labour in MT, whereby sensory and cognitive signals are both heightened in different subpopulations of neurons.

Linking Neuronal Direction Selectivity to Perceptual Decisions About Visual Motion.

Psychophysical and neurophysiological studies of responses to visual motion have converged on a consistent set of general principles that characterize visual processing of motion information. Both types of approaches have shown that the direction and speed of target motion are among the most important encoded stimulus properties, revealing many parallels between psychophysical and physiological responses to motion. Motivated by these parallels, this review focuses largely on more direct links between the key feature of the neuronal response to motion, direction selectivity, and its utilization in memory-guided perceptual decisions. These links were established during neuronal recordings in monkeys performing direction discriminations, but also by examining perceptual effects of widespread elimination of cortical direction selectivity produced by motion deprivation during development. Other approaches, such as microstimulation and lesions, have documented the importance of direction-selective activity in the areas that are active during memory-guided direction comparisons, area MT and the prefrontal cortex, revealing their likely interactions during behavioral tasks.

Prefrontal Neurons Represent Motion Signals from Across the Visual Field But for Memory-Guided Comparisons Depend on Neurons Providing These Signals.

UNLABELLED: Visual decisions often involve comparisons of sequential stimuli that can appear at any location in the visual field. The lateral prefrontal cortex (LPFC) in nonhuman primates, shown to play an important role in such comparisons, receives information about contralateral stimuli directly from sensory neurons in the same hemisphere, and about ipsilateral stimuli indirectly from neurons in the opposite hemisphere. This asymmetry of sensory inputs into the LPFC poses the question of whether and how its neurons incorporate sensory information arriving from the two hemispheres during memory-guided comparisons of visual motion. We found that, although responses of individual LPFC neurons to contralateral stimuli were stronger and emerged 40 ms earlier, they carried remarkably similar signals about motion direction in the two hemifields, with comparable direction selectivity and similar direction preferences. This similarity was also apparent around the time of the comparison between the current and remembered stimulus because both ipsilateral and contralateral responses showed similar signals reflecting the remembered direction. However, despite availability in the LPFC of motion information from across the visual field, these "comparison effects" required for the comparison stimuli to appear at the same retinal location. This strict dependence on spatial overlap of the comparison stimuli suggests participation of neurons with localized receptive fields in the comparison process. These results suggest that while LPFC incorporates many key aspects of the information arriving from sensory neurons residing in opposite hemispheres, it continues relying on the interactions with these neurons at the time of generating signals leading to successful perceptual decisions. SIGNIFICANCE STATEMENT: Visual decisions often involve comparisons of sequential visual motion that can appear at any location in the visual field. We show that during such comparisons, the lateral prefrontal cortex (LPFC) contains accurate representation of visual motion from across the visual field, supplied by motion processing neurons. However, at the time of comparison, LPFC neurons can only use this information to compute the differences between the stimuli, if stimuli appear at the same retinal location, implicating neurons with localized receptive fields in the comparison process. These findings show that sensory comparisons rely on the interactions between LPFC and sensory neurons that not only supply sensory signals but also actively participate in the comparison of these signals at the time of the decision.

Transitions between Multiband Oscillatory Patterns Characterize Memory-Guided Perceptual Decisions in Prefrontal Circuits.

Neuronal activity in the lateral prefrontal cortex (LPFC) reflects the structure and cognitive demands of memory-guided sensory discrimination tasks. However, we still do not know how neuronal activity articulates in network states involved in perceiving, remembering, and comparing sensory information during such tasks. Oscillations in local field potentials (LFPs) provide fingerprints of such network dynamics. Here, we examined LFPs recorded from LPFC of macaques while they compared the directions or the speeds of two moving random-dot patterns, S1 and S2, separated by a delay. LFP activity in the theta, beta, and gamma bands tracked consecutive components of the task. In response to motion stimuli, LFP theta and gamma power increased, and beta power decreased, but showed only weak motion selectivity. In the delay, LFP beta power modulation anticipated the onset of S2 and encoded the task-relevant S1 feature, suggesting network dynamics associated with memory maintenance. After S2 onset the difference between the current stimulus S2 and the remembered S1 was strongly reflected in broadband LFP activity, with an early sensory-related component proportional to stimulus difference and a later choice-related component reflecting the behavioral decision buildup. Our results demonstrate that individual LFP bands reflect both sensory and cognitive processes engaged independently during different stages of the task. This activation pattern suggests that during elementary cognitive tasks, the prefrontal network transitions dynamically between states and that these transitions are characterized by the conjunction of LFP rhythms rather than by single LFP bands. SIGNIFICANCE STATEMENT: Neurons in the brain communicate through electrical impulses and coordinate this activity in ensembles that pulsate rhythmically, very much like musical instruments in an orchestra. These rhythms change with "brain state," from sleep to waking, but also signal with different oscillation frequencies rapid changes between sensory and cognitive processing. Here, we studied rhythmic electrical activity in the monkey prefrontal cortex, an area implicated in working memory, decision making, and executive control. Monkeys had to identify and remember a visual motion pattern and compare it to a second pattern. We found orderly transitions between rhythmic activity where the same frequency channels were active in all ongoing prefrontal computations. This supports prefrontal circuit dynamics that transitions rapidly between complex rhythmic patterns during structured cognitive tasks.

Unilateral prefrontal lesions impair memory-guided comparisons of contralateral visual motion.

The contribution of the lateral prefrontal cortex (LPFC) to working memory is the topic of active debate. On the one hand, it has been argued that the persistent delay activity in LPFC recorded during some working memory tasks is a reflection of sensory storage, the notion supported by some lesion studies. On the other hand, there is emerging evidence that the LPFC plays a key role in the maintenance of sensory information not by storing relevant visual signals but by allocating visual attention to such stimuli. In this study, we addressed this question by examining the effects of unilateral LPFC lesions during a working memory task requiring monkeys to compare directions of two moving stimuli, separated by a delay. The lesions resulted in impaired thresholds for contralesional stimuli at longer delays, and these deficits were most dramatic when the task required rapid reallocation of spatial attention. In addition, these effects were equally pronounced when the remembered stimuli were at threshold or moved coherently. The contralesional nature of the deficits points to the importance of the interactions between the LPFC and the motion processing neurons residing in extrastriate area MT. Delay-specificity of the deficit supports LPFC involvement in the maintenance stage of the comparison task. However, because this deficit was independent of stimulus features giving rise to the remembered direction and was most pronounced during rapid shifts of attention, its role is more likely to be attending and accessing the preserved motion signals rather than their storage.

A hierarchy of intrinsic timescales across primate cortex.

Specialization and hierarchy are organizing principles for primate cortex, yet there is little direct evidence for how cortical areas are specialized in the temporal domain. We measured timescales of intrinsic fluctuations in spiking activity across areas and found a hierarchical ordering, with sensory and prefrontal areas exhibiting shorter and longer timescales, respectively. On the basis of our findings, we suggest that intrinsic timescales reflect areal specialization for task-relevant computations over multiple temporal ranges.

Linking neural activity to complex decisions.

In the 1990s, seminal work from Newsome and colleagues made it possible to study the neuronal mechanisms of simple perceptual decisions. The key strength of this work was the clear and direct link between neuronal activity and choice processes. Since then, a great deal of research has extended these initial discoveries to more complex forms of decision making, with the goal of bringing the same strength of linkage between neural and psychological processes. Here, we discuss the progress of two such research programs, namely our own, that are aimed at understanding memory-guided decisions and reward-guided decisions. These problems differ in the relevant brain areas, in the progress that has been achieved, and in the extent of broader understanding achieved so far. However, they are unified by the use of theoretical insights about how to link neuronal activity to decisions.

Common rules guide comparisons of speed and direction of motion in the dorsolateral prefrontal cortex.

When a monkey needs to decide whether motion direction of one stimulus is the same or different as that of another held in working memory, neurons in dorsolateral prefrontal cortex (DLPFC) faithfully represent the motion directions being evaluated and contribute to their comparison. Here, we examined whether DLPFC neurons are more generally involved in other types of sensory comparisons. Such involvement would support the existence of generalized sensory comparison mechanisms within DLPFC, shedding light on top-down influences this region is likely to provide to the upstream sensory neurons during comparison tasks. We recorded activity of individual neurons in the DLPFC while monkeys performed a memory-guided decision task in which the important dimension was the speed of two sequentially presented moving random-dot stimuli. We found that many neurons, both narrow-spiking putative local interneurons and broad-spiking putative pyramidal output cells, were speed-selective, with tuning reminiscent of that observed in the motion processing middle temporal (MT) cortical area. Throughout the delay, broad-spiking neurons were more active, showing anticipatory rate modulation and transient periods of speed selectivity. During the comparison stimulus, responses of both cell types were modulated by the speed of the first stimulus, and their activity was highly predictive of the animals' behavioral report. These results are similar to those found for comparisons of motion direction, suggesting the existence of generalized neural mechanisms in the DLPFC subserving the comparison of sensory signals.

Memory-guided sensory comparisons in the prefrontal cortex: contribution of putative pyramidal cells and interneurons.

Comparing two stimuli that occur at different times demands the coordination of bottom-up and top-down processes. It has been hypothesized that the dorsolateral prefrontal (PFC) cortex, the likely source of top-down cortical influences, plays a key role in such tasks, contributing to both maintenance and sensory comparisons. We examined this hypothesis by recording from the PFC of monkeys comparing directions of two moving stimuli, S1 and S2, separated by a memory delay. We determined the contribution of the two principal cell types to these processes by classifying neurons into broad-spiking (BS) putative pyramidal cells and narrow-spiking (NS) putative local interneurons. During the delay, BS cells were more likely to exhibit anticipatory modulation and represent the remembered direction. While this representation was transient, appearing at different times in different neurons, it weakened when direction was not task relevant, suggesting its utility. During S2, both putative cell types showed comparison-related activity modulations. These modulations were of two types, each carried by different neurons, which either preferred trials with stimuli moving in the same direction or trials with stimuli of different directions. These comparison effects were strongly correlated with choice, suggesting their role in circuitry underlying decision making. These results provide the first demonstration of distinct contributions made by principal cell types to memory-guided perceptual decisions. During sensory stimulation both cell types represent behaviorally relevant stimulus features contributing to comparison and decision-related activity. However in the absence of sensory stimulation, putative pyramidal cells dominated, carrying information about the elapsed time and the preceding direction.

Representation of comparison signals in cortical area MT during a delayed direction discrimination task.

Visually guided behavior often involves decisions that are based on evaluating stimuli in the context of those observed previously. Such decisions are made by monkeys comparing two consecutive stimuli, sample and test, moving in the same or opposite directions. We examined whether responses in the motion processing area MT during the comparison phase of this task (test) are modulated by the direction of the preceding stimulus (sample). This modulation, termed comparison signal, was measured by comparing responses to identical test stimuli on trials when it was preceded by sample moving in the same direction (S-trials) with trials when it was preceded by sample moving in a different direction (D-trials). The test always appeared in the neuron's receptive field (RF), whereas sample could appear in the RF or in the contralateral visual field (remote sample). With sample in-RF, we found three types of modulation carried by different sets of neurons: early suppression on S-trials and late enhancement, one on S-trials, and the other on D-trials. Under these conditions, many neurons with and without comparison effects exhibited significant, choice-related activity. Response modulation was also present following the remote sample, even though the information about its direction could only reach MT indirectly via top-down influences. However, unlike on trials with in-RF sample, these signals were dominated by response suppression, shedding light on the contribution of top-down influences to the comparison effects. These results demonstrate that during the task requiring monkeys to compare two directions of motion, MT responses during the comparison phase of this task reflect similarities and differences between the two stimuli, suggesting participation in sensory comparisons. The nature of these signals provides insights into the operation of bottom-up and top-down influences involved in this process.

Trial-to-trial variability of the prefrontal neurons reveals the nature of their engagement in a motion discrimination task.

During motion discrimination tasks, many prefrontal cortex (PFC) neurons are strongly modulated by the behavioral context, suggesting their involvement in sensory discriminations. Recent studies suggest that trial-to-trial variability of spiking activity characteristic of cortical neurons could be a source of information about the state of neurons and their participation in behavioral tasks. We tested this hypothesis by examining the variability of putative pyramidal PFC neurons, a likely source of top-down influences. The variability of these neurons was calculated as a ratio of spike count variance to its mean (fano factor, FF), while monkeys compared the directions of two moving stimuli, sample and test, separated by a delay. We found that the FF tracked consecutive components of the task, dropping rapidly with the onset of stimuli being discriminated and declining more slowly before each salient event of the trial: The sample, the test, and the response. These time-dependent signals were less consistent in direction selective neurons and were largely absent during passive fixation. Furthermore, neurons with test responses that reflected the remembered sample decreased their FF well before the test, revealing the predictive nature of response variability, an effect present only during the active task. The FF was also sensitive to behavioral performance, exhibiting different temporal dynamics on error trials. These changes did not depend on firing rates and were often the only metric correlated with task demands. Our results demonstrate that trial-to-trial variability provides a sensitive measure of the engagement of putative pyramidal PFC neurons in circuits subserving discrimination tasks.

Flexibility of sensory representations in prefrontal cortex depends on cell type.

Discrimination tasks require processing, interpreting, and linking sensory information to the appropriate motor response. We report that neurons in prefrontal cortex (PFC) represent visual motion with precision comparable to cortical neurons at early stages of motion processing, and readily adapt this representation to behavioral context. We found that direction selectivity, recorded while the monkeys discriminated directions, decreased when they judged motion speed and ignored its direction. This decrease was more pronounced in neurons classified as narrow-spiking (NS) putative interneurons than in broad-spiking (BS) putative pyramidal neurons. However, during passive fixation, when the link between motion and its behavioral relevance was removed, both cell types showed a severe selectivity loss. Our results show that flexible sensory representation during active discrimination tasks is achieved in the PFC by a specialized neuronal network of both NS neurons readily adjusting their selectivity to behavioral context, and BS neurons capable of maintaining relatively stable sensory representation.

Directional signals in the prefrontal cortex and in area MT during a working memory for visual motion task.

Neurons in the middle temporal visual area (MT) have been implicated in the perception of visual motion, whereas prefrontal cortex (PFC) neurons have been linked to temporary storage of sensory signals, attentional and executive control of behavior. Using a task that placed demands on both sets of neurons, we investigated their contribution to working memory for visual motion. Monkeys compared the direction of two moving random-dot stimuli, sample and test, separated by a brief memory delay. Neurons in both areas showed robust direction-selective activity during all phases of the task. During the sample, approximately 60% of task-related PFC neurons were direction selective, and this selectivity emerged 40 ms later than in MT. Unlike MT, the PFC responses to sample did not correlate with behavioral choices, but their selectivity was modulated by task demands and diminished on error trials. Reliable directional signals were found in both areas during the memory delay, but these signals were transient rather than sustained by neurons of either area. Responses to the test in both areas were modulated by the remembered sample direction, decreasing when the test direction matched the sample. This decrease arose in the PFC 100 ms later than in MT and was predictive of the forthcoming decision. Our data suggest that neurons in the two regions are functionally connected and make unique contributions to different task components. PFC neurons reflect task-related information about visual motion and represent decisions that may be based, in part, on the comparison in MT between the remembered sample and test.

Area MT neurons respond to visual motion distant from their receptive fields.

Neurons in cortical area MT have localized receptive fields (RF) representing the contralateral hemifield and play an important role in processing visual motion. We recorded the activity of these neurons during a behavioral task in which two monkeys were required to discriminate and remember visual motion presented in the ipsilateral hemifield. During the task, the monkeys viewed two stimuli, sample and test, separated by a brief delay and reported whether they contained motion in the same or in opposite directions. Fifty to 70% of MT neurons were activated by the motion stimuli presented in the ipsilateral hemifield at locations far removed from their classical receptive fields. These responses were in the form of excitation or suppression and were delayed relative to conventional MT responses. Both excitatory and suppressive responses were direction selective, but the nature and the time course of their directionality differed from the conventional excitatory responses recorded with stimuli in the RF. Direction selectivity of the excitatory remote response was transient and early, whereas the suppressive response developed later and persisted after stimulus offset. The presence or absence of these unusual responses on error trials, as well as their magnitude, was affected by the behavioral significance of stimuli used in the task. We hypothesize that these responses represent top-down signals from brain region(s) accessing information about stimuli in the entire visual field and about the behavioral state of the animal. The recruitment of neurons in the opposite hemisphere during processing of behaviorally relevant visual signals reveals a mechanism by which sensory processing can be affected by cognitive task demands.

Working memory in primate sensory systems.

Sensory working memory consists of the short-term storage of sensory stimuli to guide behaviour. There is increasing evidence that elemental sensory dimensions - such as object motion in the visual system or the frequency of a sound in the auditory system - are stored by segregated feature-selective systems that include not only the prefrontal and parietal cortex, but also areas of sensory cortex that carry out relatively early stages of processing. These circuits seem to have a dual function: precise sensory encoding and short-term storage of this information. New results provide insights into how activity in these circuits represents the remembered sensory stimuli.

Training-induced recovery of visual motion perception after extrastriate cortical damage in the adult cat.

Unilateral ibotenic acid lesions of the lateral suprasylvian (LS) cortex severely impair the ability of cats to integrate local motion signals (measured as direction range thresholds) and to extract motion signals from noise (measured as motion signal thresholds) in their contra-lesional visual hemifields. These deficits were found up to several months after the lesions and were limited to thresholds measured with random-dot stimuli, while contrast sensitivity for discriminating the direction of motion of sine-wave gratings remained unaffected. Our goal was to determine whether deficits of complex motion perception could recover and whether the recovery was spontaneous or required retraining. In each cat, a single location in the impaired visual hemifield was selected for visual retraining, which required the animals to discriminate motion direction using random-dot stimuli in which the range of dot directions was varied. Fifteen to 40 days of intensive retraining led to a gradual, complete recovery of motion integration. The recovery was stimulus specific since it did not transfer from direction range to motion signal thresholds, and it was largely restricted to the visual field locations retrained. Delaying the onset of retraining by several days to several months had no significant impact on the extent or rate of recovery. Once recovery was achieved, performance remained stable over a period of several months. These results suggest that recovery of complex visual motion perception after lesions of extrastriate visual cortex is an active process that requires extensive, stimulus- and retinotopically-specific visual retraining.

Activity of neurons in cortical area MT during a memory for motion task.

We recorded the activity of middle temporal (MT) neurons in 2 monkeys while they compared the directions of motion in 2 sequentially presented random-dot stimuli, sample and test, and reported them as the same or different by pressing one of 2 buttons. We found that MT neurons were active not only in response to the sample and test stimuli but also during the 1,500-ms delay separating them. Most neurons showed a characteristic pattern of activity consisting of a small burst of firing early in the delay, followed by a period of suppression and a subsequent increase in firing rate immediately preceding the presentation of the test stimulus. In a third of the neurons, the activity early in the delay not only reflected the direction of the sample stimulus, but was also related to the range of local directions it contained. During the middle of the delay the majority of neurons were suppressed, consistent with a gating mechanism that could be used to ignore task-irrelevant stimuli. Late in the delay, most neurons showed an increase in response, probably in anticipation of the upcoming test. Throughout most of the delay there was a directional signal in the population of MT neurons, manifested by higher firing rates following the sample moving in the antipreferred direction. Whereas some of these effects may be related to sensory adaptation, others are more likely to represent a more active task-related process. These results support the hypothesis that MT neurons actively participate in the successful execution of all aspects of the task requiring processing and remembering visual motion.

Stimulus specificity and temporal dynamics of working memory for visual motion.

When asked to compare two moving stimuli separated by a delay, observers must not only identify stimulus direction but also store it in memory. We examined the properties of this storage mechanism in two macaque monkeys by sequentially presenting two random-dot stimuli, sample and test, in opposite hemifields and introducing a random-motion mask during the delay. The mask interfered with performance only at the precise location of the test, 100-200 ms after the start of the delay, and when its size and speed matched those of the remembered sample. This selective interference suggests that the representation of the motion stimulus in memory preserves its direction, speed, and size and is most fragile shortly after the completion of the encoding phase of the task. This precise preservation of sensory attributes of the motion stimulus suggests that the neural mechanisms involved in the processing of visual motion may also be involved in its storage.