Writing to the Mind's Eye of the Blind.

The implantation of electrodes on the visual cortex of blind individuals could lead to the restoration of a rudimentary form of sight. In this issue of Cell, Beauchamp et al. use electrical stimulation of the visual cortex to create visual perception of shapes.

Inversion of pop-out for a distracting feature dimension in monkey visual cortex.

During visual search, it is important to reduce the interference of distracting objects in the scene. The neuronal responses elicited by the search target stimulus are typically enhanced. However, it is equally important to suppress the representations of distracting stimuli, especially if they are salient and capture attention. We trained monkeys to make an eye movement to a unique "pop-out" shape stimulus among an array of distracting stimuli. One of these distractors had a salient color that varied across trials and differed from the color of the other stimuli, causing it to also pop-out. The monkeys were able to select the pop-out shape target with high accuracy and actively avoided the pop-out color distractor. This behavioral pattern was reflected in the activity of neurons in area V4. Responses to the shape targets were enhanced, while the activity evoked by the pop-out color distractor was only briefly enhanced, directly followed by a sustained period of pronounced suppression. These behavioral and neuronal results demonstrate a cortical selection mechanism that rapidly inverts a pop-out signal to "pop-in" for an entire feature dimension thereby facilitating goal-directed visual search in the presence of salient distractors.

Reflections on the past two decades of neuroscience.

The first issue of Nature Reviews Neuroscience was published 20 years ago, in 2000. To mark this anniversary, in this Viewpoint article we asked a selection of researchers from across the field who have authored pieces published in the journal in recent years for their thoughts on notable and interesting developments in neuroscience, and particularly in their areas of the field, over the past two decades. They also provide some thoughts on current lines of research and questions that excite them.

Recurrent neural networks that learn multi-step visual routines with reinforcement learning.

Many cognitive problems can be decomposed into series of subproblems that are solved sequentially by the brain. When subproblems are solved, relevant intermediate results need to be stored by neurons and propagated to the next subproblem, until the overarching goal has been completed. We will here consider visual tasks, which can be decomposed into sequences of elemental visual operations. Experimental evidence suggests that intermediate results of the elemental operations are stored in working memory as an enhancement of neural activity in the visual cortex. The focus of enhanced activity is then available for subsequent operations to act upon. The main question at stake is how the elemental operations and their sequencing can emerge in neural networks that are trained with only rewards, in a reinforcement learning setting. We here propose a new recurrent neural network architecture that can learn composite visual tasks that require the application of successive elemental operations. Specifically, we selected three tasks for which electrophysiological recordings of monkeys' visual cortex are available. To train the networks, we used RELEARNN, a biologically plausible four-factor Hebbian learning rule, which is local both in time and space. We report that networks learn elemental operations, such as contour grouping and visual search, and execute sequences of operations, solely based on the characteristics of the visual stimuli and the reward structure of a task. After training was completed, the activity of the units of the neural network elicited by behaviorally relevant image items was stronger than that elicited by irrelevant ones, just as has been observed in the visual cortex of monkeys solving the same tasks. Relevant information that needed to be exchanged between subroutines was maintained as a focus of enhanced activity and passed on to the subsequent subroutines. Our results demonstrate how a biologically plausible learning rule can train a recurrent neural network on multistep visual tasks.

Brain2GAN: Feature-disentangled neural encoding and decoding of visual perception in the primate brain.

A challenging goal of neural coding is to characterize the neural representations underlying visual perception. To this end, multi-unit activity (MUA) of macaque visual cortex was recorded in a passive fixation task upon presentation of faces and natural images. We analyzed the relationship between MUA and latent representations of state-of-the-art deep generative models, including the conventional and feature-disentangled representations of generative adversarial networks (GANs) (i.e., z- and w-latents of StyleGAN, respectively) and language-contrastive representations of latent diffusion networks (i.e., CLIP-latents of Stable Diffusion). A mass univariate neural encoding analysis of the latent representations showed that feature-disentangled w representations outperform both z and CLIP representations in explaining neural responses. Further, w-latent features were found to be positioned at the higher end of the complexity gradient which indicates that they capture visual information relevant to high-level neural activity. Subsequently, a multivariate neural decoding analysis of the feature-disentangled representations resulted in state-of-the-art spatiotemporal reconstructions of visual perception. Taken together, our results not only highlight the important role of feature-disentanglement in shaping high-level neural representations underlying visual perception but also serve as an important benchmark for the future of neural coding.

Control of synaptic plasticity in deep cortical networks.

Humans and many other animals have an enormous capacity to learn about sensory stimuli and to master new skills. However, many of the mechanisms that enable us to learn remain to be understood. One of the greatest challenges of systems neuroscience is to explain how synaptic connections change to support maximally adaptive behaviour. Here, we provide an overview of factors that determine the change in the strength of synapses, with a focus on synaptic plasticity in sensory cortices. We review the influence of neuromodulators and feedback connections in synaptic plasticity and suggest a specific framework in which these factors can interact to improve the functioning of the entire network.

Object Selection by Automatic Spreading of Top-Down Attentional Signals in V1.

What is selected when attention is directed to a specific location of the visual field? Theories of object-based attention have suggested that when spatial attention is directed to part of an object, attention does not simply enhance the attended location but automatically spreads to enhance all locations that comprise the object. Here, we tested this hypothesis by reconstructing the distribution of attention from primary visual cortex (V1) population neuronal activity patterns in 24 human adults (17 female) using functional magnetic resonance imaging (fMRI) and population-based receptive field (prf) mapping. We find that attention spreads from a spatially cued location to the underlying object, and enhances all spatial locations that comprise the object. Importantly, this spreading was also evident when the object was not task relevant. These data suggest that attentional selection automatically operates at an object level, facilitating the reconstruction of coherent objects from fragmented representations in early visual cortex.SIGNIFICANCE STATEMENT Object perception is an astonishing feat of the visual system. When visual information about orientation, shape, and color enters through our eyes, it has yet to be integrated into a coherent representation of an object. But which visual features constitute a single object and which features belong to the background? The brain mechanisms underpinning object perception are yet to be understood. We now demonstrate that one candidate mechanism, the successive activation of all parts of an object, occurs in early visual cortex and results in a detailed representation of the object following Gestalt principles. Furthermore, our results suggest that object selection occurs automatically, without involving voluntary control.

The Contribution of AMPA and NMDA Receptors to Persistent Firing in the Dorsolateral Prefrontal Cortex in Working Memory.

Many tasks demand that information is kept online for a few seconds before it is used to guide behavior. The information is kept in working memory as the persistent firing of neurons encoding the memorized information. The neural mechanisms responsible for persistent activity are not yet well understood. Theories attribute an important role to ionotropic glutamate receptors, and it has been suggested that NMDARs are particularly important for persistent firing because they exhibit long time constants. Ionotropic AMPARs have shorter time constants and have been suggested to play a smaller role in working memory. Here we compared the contribution of AMPARs and NMDARs to persistent firing in the dlPFC of male macaque monkeys performing a delayed saccade to a memorized spatial location. We used iontophoresis to eject small amounts of glutamate receptor antagonists, aiming to perturb, but not abolish, neuronal activity. We found that both AMPARs and NMDARs contributed to persistent activity. Blockers of the NMDARs decreased persistent firing associated with the memory of the neuron's preferred spatial location but had comparatively little effect on the representation of the antipreferred location. They therefore decreased the information conveyed by persistent firing about the memorized location. In contrast, AMPAR blockers decreased activity elicited by the memory of both the preferred and antipreferred location, with a smaller effect on the information conveyed by persistent activity. Our results provide new insights into the contribution of AMPARs and NMDARs to persistent activity during working memory tasks.SIGNIFICANCE STATEMENT Working memory enables us to hold on to information that is no longer available to the senses. It relies on the persistent activity of neurons that code for the memorized information, but the detailed mechanisms are not yet well understood. Here we investigated the role of NMDARs and AMPARs in working memory using iontophoresis of antagonists in the PFC of monkeys remembering the location of a visual stimulus for an eye movement response. AMPARs and NMDARs both contributed to persistent activity. NMDAR blockers mostly decreased persistent firing associated with the memory of the neuron's preferred spatial location, whereas AMPAR blockers caused a more general suppression. These results provide new insight into the contribution of AMPARs and NMDARs to working memory.

A Direct Comparison of Spatial Attention and Stimulus-Response Compatibility between Mice and Humans.

Mice are becoming an increasingly popular model for investigating the neural substrates of visual processing and higher cognitive functions. To validate the translation of mouse visual attention and sensorimotor processing to humans, we compared their performance in the same visual task. Mice and human participants judged the orientation of a grating presented on either the right or left side in the visual field. To induce shifts of spatial attention, we varied the stimulus probability on each side. As expected, human participants showed faster RTs and a higher accuracy for the side with a higher probability, a well-established effect of visual attention. The attentional effect was only present in mice when their response was slow. Although the task demanded a judgment of grating orientation, the accuracy of the mice was strongly affected by whether the side of the stimulus corresponded to the side of the behavioral response. This stimulus-response compatibility (Simon) effect was much weaker in humans and only significant for their fastest responses. Both species exhibited a speed-accuracy trade-off in their responses, because slower responses were more accurate than faster responses. We found that mice typically respond very fast, which contributes to the stronger stimulus-response compatibility and weaker attentional effects, which were only apparent in the trials with slowest responses. Humans responded slower and had stronger attentional effects, combined with a weak influence of stimulus-response compatibility, which was only apparent in trials with fast responses. We conclude that spatial attention and stimulus-response compatibility influence the responses of humans and mice but that strategy differences between species determine the dominance of these effects.

Flexible Working Memory Through Selective Gating and Attentional Tagging.

Working memory is essential: it serves to guide intelligent behavior of humans and nonhuman primates when task-relevant stimuli are no longer present to the senses. Moreover, complex tasks often require that multiple working memory representations can be flexibly and independently maintained, prioritized, and updated according to changing task demands. Thus far, neural network models of working memory have been unable to offer an integrative account of how such control mechanisms can be acquired in a biologically plausible manner. Here, we present WorkMATe, a neural network architecture that models cognitive control over working memory content and learns the appropriate control operations needed to solve complex working memory tasks. Key components of the model include a gated memory circuit that is controlled by internal actions, encoding sensory information through untrained connections, and a neural circuit that matches sensory inputs to memory content. The network is trained by means of a biologically plausible reinforcement learning rule that relies on attentional feedback and reward prediction errors to guide synaptic updates. We demonstrate that the model successfully acquires policies to solve classical working memory tasks, such as delayed recognition and delayed pro-saccade/anti-saccade tasks. In addition, the model solves much more complex tasks, including the hierarchical 12-AX task or the ABAB ordered recognition task, both of which demand an agent to independently store and updated multiple items separately in memory. Furthermore, the control strategies that the model acquires for these tasks subsequently generalize to new task contexts with novel stimuli, thus bringing symbolic production rule qualities to a neural network architecture. As such, WorkMATe provides a new solution for the neural implementation of flexible memory control.

Benchmarking laminar fMRI: Neuronal spiking and synaptic activity during top-down and bottom-up processing in the different layers of cortex.

High resolution laminar fMRI is beginning to probe responses in the different layers of cortex. What can we expect this exciting new technique to discover about cortical processing and how can we verify that it is producing an accurate picture of the underlying laminar differences in neural processing? This review will address our knowledge of laminar cortical circuitry gained from electrophysiological studies in macaque monkeys with a focus on the primary visual cortex, as this area has been most often targeted in both laminar electrophysiological and fMRI studies. We will review how recent studies are attempting to verify the accuracy of laminar fMRI by recreating the known laminar profiles of various neural tuning properties. Furthermore, we will examine how feedforward and feedback-related neural processes engage different cortical layers, producing canonical patterns of spiking and synaptic activity as estimated by the analysis of current-source density. These results provide a benchmark for recent studies aiming to examine the profiles of bottom-up and top-down processes with laminar fMRI. Finally, we will highlight particularly useful paradigms and approaches which may help us to understand processing in the different layers of the human cerebral cortex.

The Effects of Context and Attention on Spiking Activity in Human Early Visual Cortex.

Here we report the first quantitative analysis of spiking activity in human early visual cortex. We recorded multi-unit activity from two electrodes in area V2/V3 of a human patient implanted with depth electrodes as part of her treatment for epilepsy. We observed well-localized multi-unit receptive fields with tunings for contrast, orientation, spatial frequency, and size, similar to those reported in the macaque. We also observed pronounced gamma oscillations in the local-field potential that could be used to estimate the underlying spiking response properties. Spiking responses were modulated by visual context and attention. We observed orientation-tuned surround suppression: responses were suppressed by image regions with a uniform orientation and enhanced by orientation contrast. Additionally, responses were enhanced on regions that perceptually segregated from the background, indicating that neurons in the human visual cortex are sensitive to figure-ground structure. Spiking responses were also modulated by object-based attention. When the patient mentally traced a curve through the neurons' receptive fields, the accompanying shift of attention enhanced neuronal activity. These results demonstrate that the tuning properties of cells in the human early visual cortex are similar to those in the macaque and that responses can be modulated by both contextual factors and behavioral relevance. Our results, therefore, imply that the macaque visual system is an excellent model for the human visual cortex.

Texture Segregation Causes Early Figure Enhancement and Later Ground Suppression in Areas V1 and V4 of Visual Cortex.

Segregation of images into figures and background is fundamental for visual perception. Cortical neurons respond more strongly to figural image elements than to background elements, but the mechanisms of figure-ground modulation (FGM) are only partially understood. It is unclear whether FGM in early and mid-level visual cortex is caused by an enhanced response to the figure, a suppressed response to the background, or both.We studied neuronal activity in areas V1 and V4 in monkeys performing a texture segregation task. We compared texture-defined figures with homogeneous textures and found an early enhancement of the figure representation, and a later suppression of the background. Across neurons, the strength of figure enhancement was independent of the strength of background suppression.We also examined activity in the different V1 layers. Both figure enhancement and ground suppression were strongest in superficial and deep layers and weaker in layer 4. The current-source density profiles suggested that figure enhancement was caused by stronger synaptic inputs in feedback-recipient layers 1, 2, and 5 and ground suppression by weaker inputs in these layers, suggesting an important role for feedback connections from higher level areas. These results provide new insights into the mechanisms for figure-ground organization.

Learning a New Selection Rule in Visual and Frontal Cortex.

How do you make a decision if you do not know the rules of the game? Models of sensory decision-making suggest that choices are slow if evidence is weak, but they may only apply if the subject knows the task rules. Here, we asked how the learning of a new rule influences neuronal activity in the visual (area V1) and frontal cortex (area FEF) of monkeys. We devised a new icon-selection task. On each day, the monkeys saw 2 new icons (small pictures) and learned which one was relevant. We rewarded eye movements to a saccade target connected to the relevant icon with a curve. Neurons in visual and frontal cortex coded the monkey's choice, because the representation of the selected curve was enhanced. Learning delayed the neuronal selection signals and we uncovered the cause of this delay in V1, where learning to select the relevant icon caused an early suppression of surrounding image elements. These results demonstrate that the learning of a new rule causes a transition from fast and random decisions to a more considerate strategy that takes additional time and they reveal the contribution of visual and frontal cortex to the learning process.

Simultaneous selection by object-based attention in visual and frontal cortex.

Models of visual attention hold that top-down signals from frontal cortex influence information processing in visual cortex. It is unknown whether situations exist in which visual cortex actively participates in attentional selection. To investigate this question, we simultaneously recorded neuronal activity in the frontal eye fields (FEF) and primary visual cortex (V1) during a curve-tracing task in which attention shifts are object-based. We found that accurate performance was associated with similar latencies of attentional selection in both areas and that the latency in both areas increased if the task was made more difficult. The amplitude of the attentional signals in V1 saturated early during a trial, whereas these selection signals kept increasing for a longer time in FEF, until the moment of an eye movement, as if FEF integrated attentional signals present in early visual cortex. In erroneous trials, we observed an interareal latency difference because FEF selected the wrong curve before V1 and imposed its erroneous decision onto visual cortex. The neuronal activity in visual and frontal cortices was correlated across trials, and this trial-to-trial coupling was strongest for the attended curve. These results imply that selective attention relies on reciprocal interactions within a large network of areas that includes V1 and FEF.

Alpha and gamma oscillations characterize feedback and feedforward processing in monkey visual cortex.

Cognitive functions rely on the coordinated activity of neurons in many brain regions, but the interactions between cortical areas are not yet well understood. Here we investigated whether low-frequency (α) and high-frequency (γ) oscillations characterize different directions of information flow in monkey visual cortex. We recorded from all layers of the primary visual cortex (V1) and found that γ-waves are initiated in input layer 4 and propagate to the deep and superficial layers of cortex, whereas α-waves propagate in the opposite direction. Simultaneous recordings from V1 and downstream area V4 confirmed that γ- and α-waves propagate in the feedforward and feedback direction, respectively. Microstimulation in V1 elicited γ-oscillations in V4, whereas microstimulation in V4 elicited α-oscillations in V1, thus providing causal evidence for the opposite propagation of these rhythms. Furthermore, blocking NMDA receptors, thought to be involved in feedback processing, suppressed α while boosting γ. These results provide new insights into the relation between brain rhythms and cognition.

Reinforcement Learning of Linking and Tracing Contours in Recurrent Neural Networks.

The processing of a visual stimulus can be subdivided into a number of stages. Upon stimulus presentation there is an early phase of feedforward processing where the visual information is propagated from lower to higher visual areas for the extraction of basic and complex stimulus features. This is followed by a later phase where horizontal connections within areas and feedback connections from higher areas back to lower areas come into play. In this later phase, image elements that are behaviorally relevant are grouped by Gestalt grouping rules and are labeled in the cortex with enhanced neuronal activity (object-based attention in psychology). Recent neurophysiological studies revealed that reward-based learning influences these recurrent grouping processes, but it is not well understood how rewards train recurrent circuits for perceptual organization. This paper examines the mechanisms for reward-based learning of new grouping rules. We derive a learning rule that can explain how rewards influence the information flow through feedforward, horizontal and feedback connections. We illustrate the efficiency with two tasks that have been used to study the neuronal correlates of perceptual organization in early visual cortex. The first task is called contour-integration and demands the integration of collinear contour elements into an elongated curve. We show how reward-based learning causes an enhancement of the representation of the to-be-grouped elements at early levels of a recurrent neural network, just as is observed in the visual cortex of monkeys. The second task is curve-tracing where the aim is to determine the endpoint of an elongated curve composed of connected image elements. If trained with the new learning rule, neural networks learn to propagate enhanced activity over the curve, in accordance with neurophysiological data. We close the paper with a number of model predictions that can be tested in future neurophysiological and computational studies.

How attention can create synaptic tags for the learning of working memories in sequential tasks.

Intelligence is our ability to learn appropriate responses to new stimuli and situations. Neurons in association cortex are thought to be essential for this ability. During learning these neurons become tuned to relevant features and start to represent them with persistent activity during memory delays. This learning process is not well understood. Here we develop a biologically plausible learning scheme that explains how trial-and-error learning induces neuronal selectivity and working memory representations for task-relevant information. We propose that the response selection stage sends attentional feedback signals to earlier processing levels, forming synaptic tags at those connections responsible for the stimulus-response mapping. Globally released neuromodulators then interact with tagged synapses to determine their plasticity. The resulting learning rule endows neural networks with the capacity to create new working memory representations of task relevant information as persistent activity. It is remarkably generic: it explains how association neurons learn to store task-relevant information for linear as well as non-linear stimulus-response mappings, how they become tuned to category boundaries or analog variables, depending on the task demands, and how they learn to integrate probabilistic evidence for perceptual decisions.

A unified selection signal for attention and reward in primary visual cortex.

Stimuli associated with high rewards evoke stronger neuronal activity than stimuli associated with lower rewards in many brain regions. It is not well understood how these reward effects influence activity in sensory cortices that represent low-level stimulus features. Here, we investigated the effects of reward information in the primary visual cortex (area V1) of monkeys. We found that the reward value of a stimulus relative to the value of other stimuli is a good predictor of V1 activity. Relative value biases the competition between stimuli, just as has been shown for selective attention. The neuronal latency of this reward value effect in V1 was similar to the latency of attentional influences. Moreover, V1 neurons with a strong value effect also exhibited a strong attention effect, which implies that relative value and top-down attention engage overlapping, if not identical, neuronal selection mechanisms. Our findings demonstrate that the effects of reward value reach down to the earliest sensory processing levels of the cerebral cortex and imply that theories about the effects of reward coding and top-down attention on visual representations should be unified.