Dynamic and stable population coding of attentional instructions coexist in the prefrontal cortex.

A large body of recent work suggests that neural representations in prefrontal cortex (PFC) are changing over time to adapt to task demands. However, it remains unclear whether and how such dynamic coding schemes depend on the encoded variable and are influenced by anatomical constraints. Using a cued attention task and multivariate classification methods, we show that neuronal ensembles in PFC encode and retain in working memory spatial and color attentional instructions in an anatomically specific manner. Spatial instructions could be decoded both from the frontal eye field (FEF) and the ventrolateral PFC (vlPFC) population, albeit more robustly from FEF, whereas color instructions were decoded more robustly from vlPFC. Decoding spatial and color information from vlPFC activity in the high-dimensional state space indicated stronger dynamics for color, across the cue presentation and memory periods. The change in the color code was largely due to rapid changes in the network state during the transition to the delay period. However, we found that dynamic vlPFC activity contained time-invariant color information within a low-dimensional subspace of neural activity that allowed for stable decoding of color across time. Furthermore, spatial attention influenced decoding of stimuli features profoundly in vlPFC, but less so in visual area V4. Overall, our results suggest that dynamic population coding of attentional instructions within PFC is shaped by anatomical constraints and can coexist with stable subspace coding that allows time-invariant decoding of information about the future target.

Perceptual Texture Dimensions Modulate Neuronal Response Dynamics in Visual Cortical Area V4.

Texture is an important visual attribute for surface pattern discrimination and therefore object segmentation, but the neural bases of texture perception are largely unknown. Previously, we demonstrated that the responses of V4 neurons to naturalistic texture patches are sensitive to four key features of human texture perception: coarseness, directionality, regularity, and contrast. To begin to understand how distinct texture perception emerges from the dynamics of neuronal responses, in 2 macaque monkeys (1 male, 1 female), we investigated the relative contribution of the four texture attributes to V4 responses in terms of the strength and timing of response modulation. We found that the different feature dimensions are associated with different temporal dynamics. Specifically, the response modulation associated with directionality and regularity was significantly delayed relative to that associated with coarseness and contrast, suggesting that the latter are fundamentally simpler feature dimensions. The population of texture-selective neurons could be grouped into multiple clusters based on the combination of feature dimensions encoded, and those subpopulations displayed distinct temporal dynamics characterized by the weighted combinations of multiple features. Finally, we applied a population decoding approach to demonstrate that texture category information can be obtained from short temporal windows across time. These results demonstrate that the representation of different perceptually relevant texture features emerge over time in the responses of V4 neurons. The observed temporal organization provides a framework to interpret how the processing of surface features unfolds in early and midlevel cortical stages, and could ultimately inform the interpretation of perceptual texture dynamics.SIGNIFICANCE STATEMENT To delineate how neuronal responses underlie our ability to perceive visual textures, we related four key perceptual dimensions (coarseness, directionality, regularity, and contrast) of naturalistic textures to the strength and timing of modulation of neuronal responses in area V4, an intermediate stage in the form-processing, ventral visual pathway. Our results provide the first characterization of V4 temporal dynamics for texture encoding along perceptually defined axes.

Task Context Modulates Feature-Selective Responses in Area V4.

Feature selectivity of visual cortical responses measured during passive fixation provides only a partial view of selectivity because it does not account for the influence of cognitive factors. Here we focus on primate area V4 and ask how neuronal tuning is modulated by task engagement. We investigated whether responses to colored shapes during active shape discrimination are simple, stimulus-agnostic, scaled versions of responses during passive fixation, akin to results from attentional studies. Alternatively, responses could be subject to stimulus-specific scaling, that is, responses to different stimuli are modulated differently, resulting in changes in underlying shape/color selectivity. Among 83 well-isolated V4 neurons in two male macaques, only a minority (16 of 83), which were weakly tuned to both shape and color, displayed responses during fixation and discrimination tasks that could be related by stimulus-agnostic scaling. The majority (67 of 83), which were strongly tuned to shape, color, or both, displayed stimulus-dependent response changes during discrimination. For some of these neurons (39 of 83), the shape or color of the stimulus dictated the magnitude of the change, and for others (28 of 83) it was the combination of stimulus shape and color. Importantly, for neurons with one strong and one weak tuning dimension, stimulus-dependent response changes during discrimination were associated with a relative increase in selectivity along the stronger tuning dimension, without changes in tuning peak. These results reveal that more strongly tuned V4 neurons may also be more flexible in their selectivity, and imbalances in selectivity are amplified during active task contexts.SIGNIFICANCE STATEMENT Tuning for stimulus features is typically characterized by recording responses during passive fixation, but cognitive factors, including attention, influence responses in visual cortex. To determine how behavioral engagement influences neuronal responses in area V4, we compared responses to colored shapes during passive fixation and active behavior. For a large fraction of neurons, differences in responses between passive fixation and active behavior depended on the identity of the visual stimulus. For a subgroup of strongly feature-selective neurons, this response modulation was associated with enhanced selectivity for one feature at the expense of selectivity for the other. Such flexibility in tuning strength could improve performance in tasks requiring active judgment of stimuli.

The Paton prize lecture 2021: A colourful experience leading to a reassessment of colour vision and its theories.

In this lecture, given in honour of Sir William Paton, a brilliant scientist and one of Britain's great patrons of biology, I give a personal account of the fundamental issues in colour vision that I have tackled since 1973, when I discovered a cortical zone lying outside the primary visual cortex that is rich in cells with chromatic properties. I do not provide an exhaustive review of colour vision but summarise how my views on colour vision and theories surrounding it have changed in light of that discovery.

Limited Evidence for Sensory Prediction Error Responses in Visual Cortex of Macaques and Humans.

A recent formulation of predictive coding theory proposes that a subset of neurons in each cortical area encodes sensory prediction errors, the difference between predictions relayed from higher cortex and the sensory input. Here, we test for evidence of prediction error responses in spiking responses and local field potentials (LFP) recorded in primary visual cortex and area V4 of macaque monkeys, and in complementary electroencephalographic (EEG) scalp recordings in human participants. We presented a fixed sequence of visual stimuli on most trials, and violated the expected ordering on a small subset of trials. Under predictive coding theory, pattern-violating stimuli should trigger robust prediction errors, but we found that spiking, LFP and EEG responses to expected and pattern-violating stimuli were nearly identical. Our results challenge the assertion that a fundamental computational motif in sensory cortex is to signal prediction errors, at least those based on predictions derived from temporal patterns of visual stimulation.

Perisaccadic encoding of temporal information in macaque area V4.

The accurate processing of temporal information is of critical importance in everyday life. Yet, psychophysical studies in humans have shown that the perception of time is distorted around saccadic eye movements. The neural correlates of this misperception are still poorly understood. Behavioral and neural evidence suggest that it is tightly linked to other known perisaccadic modulations of visual perception. To further our understanding of how temporal processing is affected by saccades, we studied the representations of brief visual time intervals during fixation and saccades in area V4 of two awake macaques. We presented random sequences of vertical bar stimuli and extracted neural responses to double-pulse stimulation at varying interstimulus intervals. Our results show that temporal information about very brief intervals of as brief as 20 ms is reliably represented in the multiunit activity in area V4. Response latencies were not systematically modulated by the saccade. However, a general increase in perisaccadic activity altered the ratio of response amplitudes within stimulus pairs compared with fixation. In line with previous studies showing that the perception of brief time intervals is partly based on response levels, this may be seen as a possible correlate of the perisaccadic misperception of time.NEW & NOTEWORTHY We investigated for the first time how temporal information on very brief timescales is represented in area V4 around the time of saccadic eye movements. Overall, the responses showed an unexpectedly precise representation of time intervals. Our finding of a perisaccadic modulation of relative response amplitudes introduces a new possible correlate of saccade-related perceptual distortions of time.

Searching for object pointers in the visual cortex.

We perceive objects as permanent and stable despite frequent occlusions and eye movements, but their representation in the visual cortex is neither permanent nor stable. Feature selective cells respond only as long as objects are visible, and their responses depend on eye position. We explored the hypothesis that the system maintains object pointers that provide permanence and stability. Pointers should send facilitatory signals to the feature cells of an object, and these signals should persist across temporary occlusions and remap to compensate for image displacements caused by saccades. Here, we searched for such signals in monkey areas V2 and V4 (Macaca mulatta). We developed a new paradigm in which a monkey freely inspects an array of objects in search for reward while some of the objects are being occluded temporarily by opaque drifting strips. Two types of objects were used to manipulate attention. The results were as follows. 1) Eye movements indicated a robust representation of location and type of the occluded objects; 2) in neurons of V4, but not V2, occluded objects produced elevated activity relative to blank condition; 3) the elevation of activity was reduced for objects that had been fixated immediately before the current fixation ('inhibition of return'); and 4) when attended, or when the target of a saccade, visible objects produced enhanced responses in V4, but occluded objects produced no modulation. Although results 1-3 confirm the hypothesis, the absence of modulation under occlusion is not consistent. Further experiments are needed to resolve this discrepancy.NEW & NOTEWORTHY The way we perceive objects as permanent contrasts with the short-lived responses of visual cortical neurons. A theory postulates pointers that give objects continuity, predicting a class of neurons that respond not only to visual objects but also when an occluded object moves into their receptive field. Here, we tested this theory with a novel paradigm in which a monkey freely scans an array of objects while some of them are transiently occluded.

Attention Selectively Gates Afferent Signal Transmission to Area V4.

Selective attention allows focusing on only part of the incoming sensory information. Neurons in the extrastriate visual cortex reflect such selective processing when different stimuli are simultaneously present in their large receptive fields. Their spiking response then resembles the response to the attended stimulus when presented in isolation. Unclear is where in the neuronal pathway attention intervenes to achieve such selective signal routing and processing. To investigate this question, we tagged two equivalent visual stimuli by independent broadband luminance noise and used the spectral coherence of these behaviorally irrelevant signals with the field potential of a local neuronal population in male macaque monkeys' area V4 as a measure for their respective causal influences. This new experimental paradigm revealed that signal transmission was considerably weaker for the not-attended stimulus. Furthermore, our results show that attention does not need to modulate responses in the input populations sending signals to V4 to selectively represent a stimulus, nor do they suggest a change of the V4 neurons' output gain depending on their feature similarity with the stimuli. Our results rather imply that selective attention uses a gating mechanism comprising the synaptic "inputs" that transmit signals from upstream areas into the V4 neurons. A minimal model implementing attention-dependent routing by gamma-band synchrony replicated the attentional gating effect and the signals' spectral transfer characteristics. It supports the proposal that selective interareal gamma-band synchrony subserves signal routing and explains our experimental finding that attention selectively gates signals already at the level of afferent synaptic input.SIGNIFICANCE STATEMENT Depending on the behavioral context, the brain needs to channel the flow of information through its networks of massively interconnected neurons. We designed an experiment that allows to causally assess routing of information originating from an attended object. We found that attention "gates" signals at the interplay between afferent fibers and the local neurons. A minimal model demonstrated that coherent gamma-rhythmic activity (∼60 Hz) between local neurons and their afferent-providing input neurons can realize the gating. Importantly, the attended signals did not need to be amplified already in an earlier processing stage, nor did they get amplified by a simple output response modulation. The method provides a useful tool to study mechanisms of dynamic network configuration underlying cognitive processes.

Image statistics underlying natural texture selectivity of neurons in macaque V4.

Our daily visual experiences are inevitably linked to recognizing the rich variety of textures. However, how the brain encodes and differentiates a plethora of natural textures remains poorly understood. Here, we show that many neurons in macaque V4 selectively encode sparse combinations of higher-order image statistics to represent natural textures. We systematically explored neural selectivity in a high-dimensional texture space by combining texture synthesis and efficient-sampling techniques. This yielded parameterized models for individual texture-selective neurons. The models provided parsimonious but powerful predictors for each neuron's preferred textures using a sparse combination of image statistics. As a whole population, the neuronal tuning was distributed in a way suitable for categorizing textures and quantitatively predicts human ability to discriminate textures. Together, we suggest that the collective representation of visual image statistics in V4 plays a key role in organizing the natural texture perception.

Contour Curvature As an Invariant Code for Objects in Visual Area V4.

UNLABELLED: Size-invariant object recognition-the ability to recognize objects across transformations of scale-is a fundamental feature of biological and artificial vision. To investigate its basis in the primate cerebral cortex, we measured single neuron responses to stimuli of varying size in visual area V4, a cornerstone of the object-processing pathway, in rhesus monkeys (Macaca mulatta). Leveraging two competing models for how neuronal selectivity for the bounding contours of objects may depend on stimulus size, we show that most V4 neurons (∼70%) encode objects in a size-invariant manner, consistent with selectivity for a size-independent parameter of boundary form: for these neurons, "normalized" curvature, rather than "absolute" curvature, provided a better account of responses. Our results demonstrate the suitability of contour curvature as a basis for size-invariant object representation in the visual cortex, and posit V4 as a foundation for behaviorally relevant object codes. SIGNIFICANCE STATEMENT: Size-invariant object recognition is a bedrock for many perceptual and cognitive functions. Despite growing neurophysiological evidence for invariant object representations in the primate cortex, we still lack a basic understanding of the encoding rules that govern them. Classic work in the field of visual shape theory has long postulated that a representation of objects based on information about their bounding contours is well suited to mediate such an invariant code. In this study, we provide the first empirical support for this hypothesis, and its instantiation in single neurons of visual area V4.

Graded Neuronal Modulations Related to Visual Spatial Attention.

UNLABELLED: Studies of visual attention in monkeys typically measure neuronal activity when the stimulus event to be detected occurs at a cued location versus when it occurs at an uncued location. But this approach does not address how neuronal activity changes relative to conditions where attention is unconstrained by cueing. Human psychophysical studies have used neutral cueing conditions and found that neutrally cued behavioral performance is generally intermediate to that of cued and uncued conditions (Posner et al., 1978; Mangun and Hillyard, 1990; Montagna et al., 2009). To determine whether the neuronal correlates of visual attention during neutral cueing are similarly intermediate, we trained macaque monkeys to detect changes in stimulus orientation that were more likely to occur at one location (cued) than another (uncued), or were equally likely to occur at either stimulus location (neutral). Consistent with human studies, performance was best when the location was cued, intermediate when both locations were neutrally cued, and worst when the location was uncued. Neuronal modulations in visual area V4 were also graded as a function of cue validity and behavioral performance. By recording from both hemispheres simultaneously, we investigated the possibility of switching attention between stimulus locations during neutral cueing. The results failed to support a unitary "spotlight" of attention. Overall, our findings indicate that attention-related changes in V4 are graded to accommodate task demands. SIGNIFICANCE STATEMENT: Studies of the neuronal correlates of attention in monkeys typically use visual cues to manipulate where attention is focused ("cued" vs "uncued"). Human psychophysical studies often also include neutrally cued trials to study how attention naturally varies between points of interest. But the neuronal correlates of this neutral condition are unclear. We measured behavioral performance and neuronal activity in cued, uncued, and neutrally cued blocks of trials. Behavioral performance and neuronal responses during neutral cueing were intermediate to those of the cued and uncued conditions. We found no signatures of a single mechanism of attention that switches between stimulus locations. Thus, attention-related changes in neuronal activity are largely hemisphere-specific and graded according to task demands.

Temporally evolving gain mechanisms of attention in macaque area V4.

Cognitive attention and perceptual saliency jointly govern our interaction with the environment. Yet, we still lack a universally accepted account of the interplay between attention and luminance contrast, a fundamental dimension of saliency. We measured the attentional modulation of V4 neurons' contrast response functions (CRFs) in awake, behaving macaque monkeys and applied a new approach that emphasizes the temporal dynamics of cell responses. We found that attention modulates CRFs via different gain mechanisms during subsequent epochs of visually driven activity: an early contrast-gain, strongly dependent on prestimulus activity changes (baseline shift); a time-limited stimulus-dependent multiplicative modulation, reaching its maximal expression around 150 ms after stimulus onset; and a late resurgence of contrast-gain modulation. Attention produced comparable time-dependent attentional gain changes on cells heterogeneously coding contrast, supporting the notion that the same circuits mediate attention mechanisms in V4 regardless of the form of contrast selectivity expressed by the given neuron. Surprisingly, attention was also sometimes capable of inducing radical transformations in the shape of CRFs. These findings offer important insights into the mechanisms that underlie contrast coding and attention in primate visual cortex and a new perspective on their interplay, one in which time becomes a fundamental factor.NEW & NOTEWORTHY We offer an innovative perspective on the interplay between attention and luminance contrast in macaque area V4, one in which time becomes a fundamental factor. We place emphasis on the temporal dynamics of attentional effects, pioneering the notion that attention modulates contrast response functions of V4 neurons via the sequential engagement of distinct gain mechanisms. These findings advance understanding of attentional influences on visual processing and help reconcile divergent results in the literature.

Pooled, but not single-neuron, responses in macaque V4 represent a solution to the stereo correspondence problem.

Binocular disparity is an important cue for depth perception. To correctly represent disparity, neurons must find corresponding visual features between the left- and right-eye images. The visual pathway ascending from V1 to inferior temporal cortex solves the correspondence problem. An intermediate area, V4, has been proposed to be a critical stage in the correspondence process. However, the distinction between V1 and V4 is unclear, because accumulating evidence suggests that the process begins within V1. In this article, we report that the pooled responses in macaque V4, but not responses of individual neurons, represent a solution to the correspondence problem. We recorded single-unit responses of V4 neurons to random-dot stereograms of varying degrees of anticorrelation. To achieve gradual anticorrelation, we reversed the contrast of an increasing proportion of dots as in our previous psychophysical studies, which predicted that the neural correlates of the solution to correspondence problem should gradually eliminate their disparity modulation as the level of anticorrelation increases. Inconsistent with this prediction, the tuning amplitudes of individual V4 neurons quickly decreased to a nonzero baseline with small anticorrelation. By contrast, the shapes of individual tuning curves changed more gradually so that the amplitude of population-pooled responses gradually decreased toward zero over the entire range of graded anticorrelation. We explain these results by combining multiple energy-model subunits. From a comparison with the population-pooled responses in V1, we suggest that disparity representation in V4 is distinctly advanced from that in V1. Population readout of V4 responses provides disparity information consistent with the correspondence solution.

Global selection of saccadic target features by neurons in area v4.

Psychophysical and neurophysiological studies indicate that during the preparation of saccades, visual processing at the target location is facilitated automatically by the deployment of attention. It has been assumed that the neural mechanisms involved in presaccadic shifts of attention are purely spatial in nature. Saccade preparation modulates the visual responses of neurons within extrastriate area V4, where the responses to targets are enhanced and responses to nontargets are suppressed. We tested whether this effect also engages a nonspatial form of modulation. We measured the responses of area V4 neurons to oriented gratings in two monkeys (Macaca mulatta) making delayed saccades to targets distant from the neuronal receptive field (RF). We varied the orientation of both the RF stimulus and the saccadic target. We found that, in addition to the spatial modulation, saccade preparation involves a feature-dependent modulation of V4 neuronal responses. Specifically, we found that the suppression of area V4 responses to nontarget stimuli during the preparation of saccades depends on the features of the saccadic target. Presaccadic suppression was absent when the features of the saccadic target matched the features preferred by individual V4 neurons. This feature-dependent modulation occurred in the absence of any feature-attention task. We show that our observations are consistent with a computational framework in which feature-based effects automatically emerge from saccade-related feedback signals that are spatial in nature.

Microstimulation of area V4 has little effect on spatial attention and on perception of phosphenes evoked in area V1.

Previous transcranial magnetic stimulation (TMS) studies suggested that feedback from higher to lower areas of the visual cortex is important for the access of visual information to awareness. However, the influence of cortico-cortical feedback on awareness and the nature of the feedback effects are not yet completely understood. In the present study, we used electrical microstimulation in the visual cortex of monkeys to test the hypothesis that cortico-cortical feedback plays a role in visual awareness. We investigated the interactions between the primary visual cortex (V1) and area V4 by applying microstimulation in both cortical areas at various delays. We report that the monkeys detected the phosphenes produced by V1 microstimulation but subthreshold V4 microstimulation did not influence V1 phosphene detection thresholds. A second experiment examined the influence of V4 microstimulation on the monkeys' ability to detect the dimming of one of three peripheral visual stimuli. Again, microstimulation of a group of V4 neurons failed to modulate the monkeys' perception of a stimulus in their receptive field. We conclude that conditions exist where microstimulation of area V4 has only a limited influence on visual perception.

Laminar mechanisms of saccadic suppression in primate visual cortex.

Saccadic eye movements are known to cause saccadic suppression, a temporary reduction in visual sensitivity and visual cortical firing rates. While saccadic suppression has been well characterized at the level of perception and single neurons, relatively little is known about the visual cortical networks governing this phenomenon. Here we examine the effects of saccadic suppression on distinct neural subpopulations within visual area V4. We find subpopulation-specific differences in the magnitude and timing of peri-saccadic modulation. Input-layer neurons show changes in firing rate and inter-neuronal correlations prior to saccade onset, and putative inhibitory interneurons in the input layer elevate their firing rate during saccades. A computational model of this circuit recapitulates our empirical observations and demonstrates that an input-layer-targeting pathway can initiate saccadic suppression by enhancing local inhibitory activity. Collectively, our results provide a mechanistic understanding of how eye movement signaling interacts with cortical circuitry to enforce visual stability.

Dissociation in neuronal encoding of object versus surface motion in the primate brain.

A paradox exists in our understanding of motion processing in the primate visual system: neurons in the dorsal motion processing stream often strikingly fail to encode long-range and perceptually salient jumps of a moving stimulus. Psychophysical studies suggest that such long-range motion, which requires integration over more distant parts of the visual field, may be based on higher-order motion processing mechanisms that rely on feature or object tracking. Here, we demonstrate that ventral visual area V4, long recognized as critical for processing static scenes, includes neurons that maintain direction selectivity for long-range motion, even when conflicting local motion is present. These V4 neurons exhibit specific selectivity for the motion of objects, i.e., targets with defined boundaries, rather than the motion of surfaces behind apertures, and are selective for direction of motion over a broad range of spatial displacements and defined by a variety of features. Motion direction at a range of speeds can be accurately decoded on single trials from the activity of just a few V4 neurons. Thus, our results identify a novel motion computation in the ventral stream that is strikingly different from, and complementary to, the well-established system in the dorsal stream, and they support the hypothesis that the ventral stream system interacts with the dorsal stream to achieve the higher level of abstraction critical for tracking dynamic objects.

Bridging neuronal correlations and dimensionality reduction.

Two commonly used approaches to study interactions among neurons are spike count correlation, which describes pairs of neurons, and dimensionality reduction, applied to a population of neurons. Although both approaches have been used to study trial-to-trial neuronal variability correlated among neurons, they are often used in isolation and have not been directly related. We first established concrete mathematical and empirical relationships between pairwise correlation and metrics of population-wide covariability based on dimensionality reduction. Applying these insights to macaque V4 population recordings, we found that the previously reported decrease in mean pairwise correlation associated with attention stemmed from three distinct changes in population-wide covariability. Overall, our work builds the intuition and formalism to bridge between pairwise correlation and population-wide covariability and presents a cautionary tale about the inferences one can make about population activity by using a single statistic, whether it be mean pairwise correlation or dimensionality.

Visual Functions of Primate Area V4.

Area V4-the focus of this review-is a mid-level processing stage along the ventral visual pathway of the macaque monkey. V4 is extensively interconnected with other visual cortical areas along the ventral and dorsal visual streams, with frontal cortical areas, and with several subcortical structures. Thus, it is well poised to play a broad and integrative role in visual perception and recognition-the functional domain of the ventral pathway. Neurophysiological studies in monkeys engaged in passive fixation and behavioral tasks suggest that V4 responses are dictated by tuning in a high-dimensional stimulus space defined by form, texture, color, depth, and other attributes of visual stimuli. This high-dimensional tuning may underlie the development of object-based representations in the visual cortex that are critical for tracking, recognizing, and interacting with objects. Neurophysiological and lesion studies also suggest that V4 responses are important for guiding perceptual decisions and higher-order behavior.

Slow Drift of Neural Activity as a Signature of Impulsivity in Macaque Visual and Prefrontal Cortex.

An animal's decision depends not only on incoming sensory evidence but also on its fluctuating internal state. This state embodies multiple cognitive factors, such as arousal and fatigue, but it is unclear how these factors influence the neural processes that encode sensory stimuli and form a decision. We discovered that, unprompted by task conditions, animals slowly shifted their likelihood of detecting stimulus changes over the timescale of tens of minutes. Neural population activity from visual area V4, as well as from prefrontal cortex, slowly drifted together with these behavioral fluctuations. We found that this slow drift, rather than altering the encoding of the sensory stimulus, acted as an impulsivity signal, overriding sensory evidence to dictate the final decision. Overall, this work uncovers an internal state embedded in population activity across multiple brain areas and sheds further light on how internal states contribute to the decision-making process.