Analysis of spiking synchrony in visual cortex reveals distinct types of top-down modulation signals for spatial and object-based attention.

The activity of a border ownership selective (BOS) neuron indicates where a foreground object is located relative to its (classical) receptive field (RF). A population of BOS neurons thus provides an important component of perceptual grouping, the organization of the visual scene into objects. In previous theoretical work, it has been suggested that this grouping mechanism is implemented by a population of dedicated grouping ("G") cells that integrate the activity of the distributed feature cells representing an object and, by feedback, modulate the same cells, thus making them border ownership selective. The feedback modulation by G cells is thought to also provide the mechanism for object-based attention. A recent modeling study showed that modulatory common feedback, implemented by synapses with N-methyl-D-aspartate (NMDA)-type glutamate receptors, accounts for the experimentally observed synchrony in spike trains of BOS neurons and the shape of cross-correlations between them, including its dependence on the attentional state. However, that study was limited to pairs of BOS neurons with consistent border ownership preferences, defined as two neurons tuned to respond to the same visual object, in which attention decreases synchrony. But attention has also been shown to increase synchrony in neurons with inconsistent border ownership selectivity. Here we extend the computational model from the previous study to fully understand these effects of attention. We postulate the existence of a second type of G-cell that represents spatial attention by modulating the activity of all BOS cells in a spatially defined area. Simulations of this model show that a combination of spatial and object-based mechanisms fully accounts for the observed pattern of synchrony between BOS neurons. Our results suggest that modulatory feedback from G-cells may underlie both spatial and object-based attention.

Searching for object pointers in the visual cortex.

We perceive objects as permanent and stable despite frequent occlusions and eye movements, but their representation in the visual cortex is neither permanent nor stable. Feature selective cells respond only as long as objects are visible, and their responses depend on eye position. We explored the hypothesis that the system maintains object pointers that provide permanence and stability. Pointers should send facilitatory signals to the feature cells of an object, and these signals should persist across temporary occlusions and remap to compensate for image displacements caused by saccades. Here, we searched for such signals in monkey areas V2 and V4 (Macaca mulatta). We developed a new paradigm in which a monkey freely inspects an array of objects in search for reward while some of the objects are being occluded temporarily by opaque drifting strips. Two types of objects were used to manipulate attention. The results were as follows. 1) Eye movements indicated a robust representation of location and type of the occluded objects; 2) in neurons of V4, but not V2, occluded objects produced elevated activity relative to blank condition; 3) the elevation of activity was reduced for objects that had been fixated immediately before the current fixation ('inhibition of return'); and 4) when attended, or when the target of a saccade, visible objects produced enhanced responses in V4, but occluded objects produced no modulation. Although results 1-3 confirm the hypothesis, the absence of modulation under occlusion is not consistent. Further experiments are needed to resolve this discrepancy.NEW & NOTEWORTHY The way we perceive objects as permanent contrasts with the short-lived responses of visual cortical neurons. A theory postulates pointers that give objects continuity, predicting a class of neurons that respond not only to visual objects but also when an occluded object moves into their receptive field. Here, we tested this theory with a novel paradigm in which a monkey freely scans an array of objects while some of them are transiently occluded.

Figure-Ground Organization in Natural Scenes: Performance of a Recurrent Neural Model Compared with Neurons of Area V2.

A crucial step in understanding visual input is its organization into meaningful components, in particular object contours and partially occluded background structures. This requires that all contours are assigned to either the foreground or the background (border ownership assignment). While earlier studies showed that neurons in primate extrastriate cortex signal border ownership for simple geometric shapes, recent studies show consistent border ownership coding also for complex natural scenes. In order to understand how the brain performs this task, we developed a biologically plausible recurrent neural network that is fully image computable. Our model uses local edge detector ( B ) cells and grouping ( G ) cells whose activity represents proto-objects based on the integration of local feature information. G cells send modulatory feedback connections to those B cells that caused their activation, making the B cells border ownership selective. We found close agreement between our model and neurophysiological results in terms of the timing of border ownership signals (BOSs) as well as the consistency of BOSs across scenes. We also benchmarked our model on the Berkeley Segmentation Dataset and achieved performance comparable to recent state-of-the-art computer vision approaches. Our proposed model provides insight into the cortical mechanisms of figure-ground organization.

Figure and ground: how the visual cortex integrates local cues for global organization.

Inferring figure-ground organization in two-dimensional images may require different complementary strategies. For isolated objects, it has been shown that mechanisms in visual cortex exploit the overall distribution of contours, but in images of cluttered scenes where the grouping of contours is not obvious, that strategy would fail. However, natural scenes contain local features, specifically contour junctions, that may contribute to the definition of object regions. To study the role of local features in the assignment of border ownership, we recorded single-cell activity from visual cortex in awake behaving Macaca mulatta. We tested configurations perceived as two overlapping figures in which T- and L-junctions depend on the direction of overlap, whereas the overall distribution of contours provides no valid information. While recording responses to the occluding contour, we varied direction of overlap and variably masked some of the critical contour features to determine their influences and their interactions. On average, most features influenced the responses consistently, producing either enhancement or suppression depending on border ownership. Different feature types could have opposite effects even at the same location. Features far from the receptive field produced effects as strong as near features and with the same short latency. Summation was highly nonlinear: any single feature produced more than two-thirds of the effect of all features together. These findings reveal fast and highly specific organization mechanisms, supporting a previously proposed model in which "grouping cells" integrate widely distributed edge signals with specific end-stopped signals to modulate the original edge signals by feedback. NEW & NOTEWORTHY Seeing objects seems effortless, but defining objects in a scene requires sophisticated neural mechanisms. For isolated objects, the visual cortex groups contours based on overall distribution, but this strategy does not work for cluttered scenes. Here, we demonstrate mechanisms that integrate local contour features like T- and L-junctions to resolve clutter. The process is fast, evaluates widely distributed features, and gives any single feature a decisive influence on figure-ground representation.

Figure-ground organization in the visual cortex: does meaning matter?

Figure-ground organization in the visual cortex is generally assumed to be based partly on general rules and partly on specific influences of object recognition in higher centers as found in the temporal lobe. To see if shape familiarity influences figure-ground organization, we tested border ownership-selective neurons in monkey V1/V2 with silhouettes of human and monkey face profiles and "nonsense" silhouettes constructed by mirror-reversing the front part of the profile. We found no superiority of face silhouettes compared with nonsense shapes in eliciting border-ownership signals overall. However, in some neurons, border-ownership signals differed strongly between the two categories consistently across many different profile shapes. Surprisingly, this category selectivity appeared as early as 70 ms after stimulus onset, which is earlier than the typical latency of shape-selective responses but compatible with the earliest face-selective responses in the inferior temporal lobe. Although our results provide no evidence for a delayed top-down influence from object recognition centers, they indicate sophisticated shape categorization mechanisms that are much faster than generally assumed. NEW & NOTEWORTHY A long-standing question is whether low-level sensory representations in cortex are influenced by cognitive "top-down" signals. We studied figure-ground organization in the visual cortex by comparing border-ownership signals for face profiles and matched nonsense shapes. We found no sign of "face superiority" in the population border-ownership signal. However, some neurons consistently differentiated between the face and nonsense categories early on, indicating the presence of shape classification mechanisms that are much faster than previously assumed.

Spike synchrony generated by modulatory common input through NMDA-type synapses.

Common excitatory input to neurons increases their firing rates and the strength of the spike correlation (synchrony) between them. Little is known, however, about the synchronizing effects of modulatory common input. Here, we show that modulatory common input with the slow synaptic kinetics of N-methyl-d-aspartate (NMDA) receptors enhances firing rates and also produces synchrony. Tight synchrony (correlations on the order of milliseconds) always increases with modulatory strength. Unexpectedly, the relationship between strength of modulation and strength of loose synchrony (tens of milliseconds) is not monotonic: The strongest loose synchrony is obtained for intermediate modulatory amplitudes. This finding explains recent neurophysiological results showing that in cortical areas V1 and V2, presumed modulatory top-down input due to contour grouping increases (loose and tight) synchrony but that additional modulatory input due to top-down attention does not change tight synchrony and actually decreases loose synchrony. These neurophysiological findings are understood from our model of integrate-and-fire neurons under the assumption that contour grouping as well as attention lead to additive modulatory common input through NMDA-type synapses. In contrast, circuits with common projections through model α-amino-3-hydroxy-5-methyl-4-isoxazolepropionic acid receptors did not exhibit the paradoxical decrease of synchrony with increased input. Our results suggest that NMDA receptors play a critical role in top-down response modulation in the visual cortex.

Figure-Ground Organization in Visual Cortex for Natural Scenes.

Figure-ground organization and border-ownership assignment are essential for understanding natural scenes. It has been shown that many neurons in the macaque visual cortex signal border-ownership in displays of simple geometric shapes such as squares, but how well these neurons resolve border-ownership in natural scenes is not known. We studied area V2 neurons in behaving macaques with static images of complex natural scenes. We found that about half of the neurons were border-ownership selective for contours in natural scenes, and this selectivity originated from the image context. The border-ownership signals emerged within 70 ms after stimulus onset, only ∼30 ms after response onset. A substantial fraction of neurons were highly consistent across scenes. Thus, the cortical mechanisms of figure-ground organization are fast and efficient even in images of complex natural scenes. Understanding how the brain performs this task so fast remains a challenge.

Figure-ground organization and the emergence of proto-objects in the visual cortex.

A long history of studies of perception has shown that the visual system organizes the incoming information early on, interpreting the 2D image in terms of a 3D world and producing a structure that provides perceptual continuity and enables object-based attention. Recordings from monkey visual cortex show that many neurons, especially in area V2, are selective for border ownership. These neurons are edge selective and have ordinary classical receptive fields (CRF), but in addition their responses are modulated (enhanced or suppressed) depending on the location of a 'figure' relative to the edge in their receptive field. Each neuron has a fixed preference for location on one side or the other. This selectivity is derived from the image context far beyond the CRF. This paper reviews evidence indicating that border ownership selectivity reflects the formation of early object representations ('proto-objects'). The evidence includes experiments showing (1) reversal of border ownership signals with change of perceived object structure, (2) border ownership specific enhancement of responses in object-based selective attention, (3) persistence of border ownership signals in accordance with continuity of object perception, and (4) remapping of border ownership signals across saccades and object movements. Findings 1 and 2 can be explained by hypothetical grouping circuits that sum contour feature signals in search of objectness, and, via recurrent projections, enhance the corresponding low-level feature signals. Findings 3 and 4 might be explained by assuming that the activity of grouping circuits persists and can be remapped. Grouping, persistence, and remapping are fundamental operations of vision. Finding these operations manifest in low-level visual areas challenges traditional views of visual processing. New computational models need to be developed for a comprehensive understanding of the function of the visual cortex.

Spike synchrony reveals emergence of proto-objects in visual cortex.

Neurons at early stages of the visual cortex signal elemental features, such as pieces of contour, but how these signals are organized into perceptual objects is unclear. Theories have proposed that spiking synchrony between these neurons encodes how features are grouped (binding-by-synchrony), but recent studies did not find the predicted increase in synchrony with binding. Here we propose that features are grouped to "proto-objects" by intrinsic feedback circuits that enhance the responses of the participating feature neurons. This hypothesis predicts synchrony exclusively between feature neurons that receive feedback from the same grouping circuit. We recorded from neurons in macaque visual cortex and used border-ownership selectivity, an intrinsic property of the neurons, to infer whether or not two neurons are part of the same grouping circuit. We found that binding produced synchrony between same-circuit neurons, but not between other pairs of neurons, as predicted by the grouping hypothesis. In a selective attention task, synchrony emerged with ignored as well as attended objects, and higher synchrony was associated with faster behavioral responses, as would be expected from early grouping mechanisms that provide the structure for object-based processing. Thus, synchrony could be produced by automatic activation of intrinsic grouping circuits. However, the binding-related elevation of synchrony was weak compared with its random fluctuations, arguing against synchrony as a code for binding. In contrast, feedback grouping circuits encode binding by modulating the response strength of related feature neurons. Thus, our results suggest a novel coding mechanism that might underlie the proto-objects of perception.

A model of proto-object based saliency.

Organisms use the process of selective attention to optimally allocate their computational resources to the instantaneously most relevant subsets of a visual scene, ensuring that they can parse the scene in real time. Many models of bottom-up attentional selection assume that elementary image features, like intensity, color and orientation, attract attention. Gestalt psychologists, however, argue that humans perceive whole objects before they analyze individual features. This is supported by recent psychophysical studies that show that objects predict eye-fixations better than features. In this report we present a neurally inspired algorithm of object based, bottom-up attention. The model rivals the performance of state of the art non-biologically plausible feature based algorithms (and outperforms biologically plausible feature based algorithms) in its ability to predict perceptual saliency (eye fixations and subjective interest points) in natural scenes. The model achieves this by computing saliency as a function of proto-objects that establish the perceptual organization of the scene. All computational mechanisms of the algorithm have direct neural correlates, and our results provide evidence for the interface theory of attention.

Neurophysiological constraints on models of illusory contours.

Illusory contours can appear as interpolation between edges of the stimulus, as in the Kanizsa triangle, or run orthogonal to the inducing elements, as in the Ehrenstein illusion. Single-cell recordings from monkey visual cortex suggest that both are produced by the same mechanism. Neural border ownership coding, on the other hand, which shows a much larger range of context integration, might involve a different mechanism.

Remapping of border ownership in the visual cortex.

We see objects as having continuity although the retinal image changes frequently. How such continuity is achieved is hard to understand, because neurons in the visual cortex have small receptive fields that are fixed on the retina, which means that a different set of neurons is activated every time the eyes move. Neurons in areas V1 and V2 of the visual cortex signal the local features that are currently in their receptive fields and do not show "remapping" when the image moves. However, subsets of neurons in these areas also carry information about global aspects, such as figure-ground organization. Here we performed experiments to find out whether figure-ground organization is remapped. We recorded single neurons in macaque V1 and V2 in which figure-ground organization is represented by assignment of contours to regions (border ownership). We found previously that border-ownership signals persist when a figure edge is switched to an ambiguous edge by removing the context. We now used this paradigm to see whether border ownership transfers when the ambiguous edge is moved across the retina. In the new position, the edge activated a different set of neurons at a different location in cortex. We found that border ownership was transferred to the newly activated neurons. The transfer occurred whether the edge was moved by a saccade or by moving the visual display. Thus, although the contours are coded in retinal coordinates, their assignment to objects is maintained across movements of the retinal image.

A century of Gestalt psychology in visual perception: I. Perceptual grouping and figure-ground organization.

In 1912, Max Wertheimer published his paper on phi motion, widely recognized as the start of Gestalt psychology. Because of its continued relevance in modern psychology, this centennial anniversary is an excellent opportunity to take stock of what Gestalt psychology has offered and how it has changed since its inception. We first introduce the key findings and ideas in the Berlin school of Gestalt psychology, and then briefly sketch its development, rise, and fall. Next, we discuss its empirical and conceptual problems, and indicate how they are addressed in contemporary research on perceptual grouping and figure-ground organization. In particular, we review the principles of grouping, both classical (e.g., proximity, similarity, common fate, good continuation, closure, symmetry, parallelism) and new (e.g., synchrony, common region, element and uniform connectedness), and their role in contour integration and completion. We then review classic and new image-based principles of figure-ground organization, how it is influenced by past experience and attention, and how it relates to shape and depth perception. After an integrated review of the neural mechanisms involved in contour grouping, border ownership, and figure-ground perception, we conclude by evaluating what modern vision science has offered compared to traditional Gestalt psychology, whether we can speak of a Gestalt revival, and where the remaining limitations and challenges lie. A better integration of this research tradition with the rest of vision science requires further progress regarding the conceptual and theoretical foundations of the Gestalt approach, which is the focus of a second review article.

The speed of context integration in the visual cortex.

The observation of figure-ground selectivity in neurons of the visual cortex shows that these neurons can be influenced by the image context far beyond the classical receptive field. To clarify the nature of the context integration mechanism, we studied the latencies of neural edge signals, comparing the emergence of context-dependent definition of border ownership with the onset of local edge definition (contrast polarity; stereoscopic depth order). Single-neuron activity was recorded in areas V1 and V2 of Macaca mulatta under behaviorally induced fixation. Whereas local edge definition emerged immediately (<13 ms) after the edge onset response, the context-dependent signal was delayed by about 30 ms. To see if the context influence was mediated by horizontal fibers within cortex, we measured the latencies of border ownership signals for two conditions in which the relevant context information was located at different distances from the receptive field and compared the latency difference with the difference predicted from horizontal signal propagation. The prediction was based on the increase in cortical distance, computed from the mapping of the test stimuli in the cortex, and the known conduction velocities of horizontal fibers. The measured latencies increased with cortical distance, but much less than predicted by the horizontal propagation hypothesis. Probability calculations showed that an explanation of the context influence by horizontal signal propagation alone is highly unlikely, whereas mechanisms involving back projections from other extrastriate areas are plausible.

Mechanisms of perceptual organization provide auto-zoom and auto-localization for attention to objects.

Visual attention is often understood as a modulatory field acting at early stages of processing, but the mechanisms that direct and fit the field to the attended object are not known. We show that a purely spatial attention field propagating downward in the neuronal network responsible for perceptual organization will be reshaped, repositioned, and sharpened to match the object's shape and scale. Key features of the model are grouping neurons integrating local features into coherent tentative objects, excitatory feedback to the same local feature neurons that caused grouping neuron activation, and inhibition between incompatible interpretations both at the local feature level and at the object representation level.

Representation of object continuity in the visual cortex.

An amazing feature of our visual system is the ability to detect and track objects in the stream of continually changing retinal images. Theories have proposed that the system creates temporary internal representations that persist across changing images, providing continuity. However, how such representations are formed in the brain is not known. Here we examined the time course of the responses of border-ownership-selective neurons in the visual cortex to displays that portray object continuity. We found that the neurons signal border ownership immediately when new objects appear, but when a border that has been assigned to one object is reassigned to another object while the first remains in the display, the initial responses persist. The neurons continue to signal the initial assignment despite the presence of contradicting figure--ground cues. We propose that border ownership selectivity reflects mechanisms that create object continuity.

Analysis of the context integration mechanisms underlying figure-ground organization in the visual cortex.

Most neurons in visual cortex respond to contrast borders and are orientation selective, and some are also selective for which side of a border is figure and which side is ground ("border ownership coding"). These neurons are influenced by the image context far beyond the classical receptive field (CRF) and as early as 25 ms after the onset of activity in the cortex. The nature of the fast context integration mechanism is not well understood. What parts of a figure contribute to the context effect? What is the structure of the "extraclassical surround"? Is the context information propagated through horizontal fibers within cortex or through reciprocal connections via higher-level areas? To address these questions, we studied border ownership modulation with fragmented figures. Neurons were recorded in areas V1 and V2 of Macaca mulatta under behaviorally induced fixation. Test figures were fragmented rectangles. While one edge was centered on the CRF, the presence of the fragments outside the CRF was varied. The surround fragments produced facilitation on the preferred border ownership side as well as suppression on the nonpreferred side, with approximately 80% of the locations contributing on average. Fragments far from the CRF influenced the responses even in the absence of fragments closer to the CRF, and without the extra delay that would incur from propagation through horizontal fibers. Three principally different models are discussed. The results support a model in which the antagonistic surround influences are produced by reentrant signals from a higher-level area.

Short-term memory for figure-ground organization in the visual cortex.

Whether the visual system uses a buffer to store image information and the duration of that storage have been debated intensely in recent psychophysical studies. The long phases of stable perception of reversible figures suggest a memory that persists for seconds. But persistence of similar duration has not been found in signals of the visual cortex. Here, we show that figure-ground signals in the visual cortex can persist for a second or more after the removal of the figure-ground cues. When new figure-ground information is presented, the signals adjust rapidly, but when a figure display is changed to an ambiguous edge display, the signals decay slowly--a behavior that is characteristic of memory devices. Figure-ground signals represent the layout of objects in a scene, and we propose that a short-term memory for object layout is important in providing continuity of perception in the rapid stream of images flooding our eyes.

Synchrony and the binding problem in macaque visual cortex.

We tested the binding-by-synchrony hypothesis which proposes that object representations are formed by synchronizing spike activity between neurons that code features of the same object. We studied responses of 32 pairs of neurons recorded with microelectrodes 3 mm apart in the visual cortex of macaques performing a fixation task. Upon mapping the receptive fields of the neurons, a quadrilateral was generated so that two of its sides were centered in the receptive fields at the optimal orientations. This one-figure condition was compared with a two-figure condition in which the neurons were stimulated by two separate figures, keeping the local edges in the receptive fields identical. For each neuron, we also determined its border ownership selectivity (H. Zhou, H. S. Friedman, & R. von der Heydt, 2000). We examined both synchronization and correlation at nonzero time lag. After correcting for effects of the firing rate, we found that synchrony did not depend on the binding condition. However, finding synchrony in a pair of neurons was correlated with finding border-ownership selectivity in both members of the pair. This suggests that the synchrony reflected the connectivity in the network that generates border ownership assignment. Thus, we have not found evidence to support the binding-by-synchrony hypothesis.