Functional modules for visual scene segmentation in macaque visual cortex.

Segmentation, the computation of object boundaries, is one of the most important steps in intermediate visual processing. Previous studies have reported cells across visual cortex that are modulated by segmentation features, but the functional role of these cells remains unclear. First, it is unclear whether these cells encode segmentation consistently since most studies used only a limited variety of stimulus types. Second, it is unclear whether these cells are organized into specialized modules or instead randomly scattered across the visual cortex: the former would lend credence to a functional role for putative segmentation cells. Here, we used fMRI-guided electrophysiology to systematically characterize the consistency and spatial organization of segmentation-encoding cells across the visual cortex. Using fMRI, we identified a set of patches in V2, V3, V3A, V4, and V4A that were more active for stimuli containing figures compared to ground, regardless of whether figures were defined by texture, motion, luminance, or disparity. We targeted these patches for single-unit recordings and found that cells inside segmentation patches were tuned to both figure-ground and borders more consistently across types of stimuli than cells in the visual cortex outside the patches. Remarkably, we found clusters of cells inside segmentation patches that showed the same border-ownership preference across all stimulus types. Finally, using a population decoding approach, we found that segmentation could be decoded with higher accuracy from segmentation patches than from either color-selective or control regions. Overall, our results suggest that segmentation signals are preferentially encoded in spatially discrete patches.

Olfactory receptor neurons are sensitive to stimulus onset asynchrony: implications for odor source discrimination.

In insects, olfactory receptor neurons (ORNs) are localized in sensilla. Within a sensillum, different ORN types are typically co-localized and exhibit nonsynaptic reciprocal inhibition through ephaptic coupling. This inhibition is hypothesized to aid odor source discrimination in environments where odor molecules (odorants) are dispersed by wind, resulting in turbulent plumes. Under these conditions, odorants from a single source arrive at the ORNs synchronously, while those from separate sources arrive asynchronously. Ephaptic inhibition is expected to be weaker for asynchronous arriving odorants from separate sources, thereby enhancing their discrimination. Previous studies have focused on ephaptic inhibition of sustained ORN responses to constant odor stimuli. This begs the question of whether ephaptic inhibition also affects transient ORN responses and if this inhibition is modulated by the temporal arrival patterns of different odorants. To address this, we recorded co-localized ORNs in the fruit fly Drosophila melanogaster and exposed them to dynamic odorant mixtures. We found reciprocal inhibition, strongly suggesting the presence of ephaptic coupling. This reciprocal inhibition does indeed modulate transient ORN responses and is sensitive to the relative timing of odor stimuli. Notably, the strength of inhibition decreases as the synchrony and correlation between arriving odorants decrease. These results support the hypothesis that ephaptic inhibition aids odor source discrimination.

Figure-ground segmentation based on motion in the archerfish.

Figure-ground segmentation is a fundamental process in visual perception that involves separating visual stimuli into distinct meaningful objects and their surrounding context, thus allowing the brain to interpret and understand complex visual scenes. Mammals exhibit varying figure-ground segmentation capabilities, ranging from primates that can perform well on figure-ground segmentation tasks to rodents that perform poorly. To explore figure-ground segmentation capabilities in teleost fish, we studied how the archerfish, an expert visual hunter, performs figure-ground segmentation. We trained archerfish to discriminate foreground objects from the background, where the figures were defined by motion as well as by discontinuities in intensity and texture. Specifically, the figures were defined by grating, naturalistic texture, and random noise moving in counterphase with the background. The archerfish performed the task well and could distinguish between all three types of figures and grounds. Their performance was comparable to that of primates and outperformed rodents. These findings suggest the existence of a complex visual process in the archerfish visual system that enables the delineation of figures as distinct from backgrounds, and provide insights into object recognition in this animal.

Difficulties with Speech-in-Noise Perception Related to Fundamental Grouping Processes in Auditory Cortex.

In our everyday lives, we are often required to follow a conversation when background noise is present ("speech-in-noise" [SPIN] perception). SPIN perception varies widely-and people who are worse at SPIN perception are also worse at fundamental auditory grouping, as assessed by figure-ground tasks. Here, we examined the cortical processes that link difficulties with SPIN perception to difficulties with figure-ground perception using functional magnetic resonance imaging. We found strong evidence that the earliest stages of the auditory cortical hierarchy (left core and belt areas) are similarly disinhibited when SPIN and figure-ground tasks are more difficult (i.e., at target-to-masker ratios corresponding to 60% rather than 90% performance)-consistent with increased cortical gain at lower levels of the auditory hierarchy. Overall, our results reveal a common neural substrate for these basic (figure-ground) and naturally relevant (SPIN) tasks-which provides a common computational basis for the link between SPIN perception and fundamental auditory grouping.

Spiking model of fixational eye movements and figure-ground segmentation.

We present a model connecting eye movements and cortical state. Its structure includes simulated retinal images, motion detection, feature detectors and layers of spiking neurons. The designed scheme shows how the effect of micro-saccadic scale eye movements can lead to successful figure segregation in a figure-ground paradigm, by inducing changes in the neural dynamics through the time evolution of the inhibition range.

Resolving the Spatial Profile of Figure Enhancement in Human V1 through Population Receptive Field Modeling.

The detection and segmentation of meaningful figures from their background is one of the primary functions of vision. While work in nonhuman primates has implicated early visual mechanisms in this figure-ground modulation, neuroimaging in humans has instead largely ascribed the processing of figures and objects to higher stages of the visual hierarchy. Here, we used high-field fMRI at 7 Tesla to measure BOLD responses to task-irrelevant orientation-defined figures in human early visual cortex (N = 6, four females). We used a novel population receptive field mapping-based approach to resolve the spatial profiles of two constituent mechanisms of figure-ground modulation: a local boundary response, and a further enhancement spanning the full extent of the figure region that is driven by global differences in features. Reconstructing the distinct spatial profiles of these effects reveals that figure enhancement modulates responses in human early visual cortex in a manner consistent with a mechanism of automatic, contextually driven feedback from higher visual areas.SIGNIFICANCE STATEMENT A core function of the visual system is to parse complex 2D input into meaningful figures. We do so constantly and seamlessly, both by processing information about visible edges and by analyzing large-scale differences between figure and background. While influential neurophysiology work has characterized an intriguing mechanism that enhances V1 responses to perceptual figures, we have a poor understanding of how the early visual system contributes to figure-ground processing in humans. Here, we use advanced computational analysis methods and high-field human fMRI data to resolve the distinct spatial profiles of local edge and global figure enhancement in the early visual system (V1 and LGN); the latter is distinct and consistent with a mechanism of automatic, stimulus-driven feedback from higher-level visual areas.

Automatic Encoding of a View-Centered Background Image in the Macaque Temporal Lobe.

Perceptual processing along the ventral visual pathway to the hippocampus (HPC) is hypothesized to be substantiated by signal transformation from retinotopic space to relational space, which represents interrelations among constituent visual elements. However, our visual perception necessarily reflects the first person's perspective based on the retinotopic space. To investigate this two-facedness of visual perception, we compared neural activities in the temporal lobe (anterior inferotemporal cortex, perirhinal and parahippocampal cortices, and HPC) between when monkeys gazed on an object and when they fixated on the screen center with an object in their peripheral vision. We found that in addition to the spatially invariant object signal, the temporal lobe areas automatically represent a large-scale background image, which specify the subject's viewing location. These results suggest that a combination of two distinct visual signals on relational space and retinotopic space may provide the first person's perspective serving for perception and presumably subsequent episodic memory.

Behavioral Evidence and Neural Correlates of Perceptual Grouping by Motion in the Barn Owl.

Perceiving an object as salient from its surround often requires a preceding process of grouping the object and background elements as perceptual wholes. In humans, motion homogeneity provides a strong cue for grouping, yet it is unknown to what extent this occurs in nonprimate species. To explore this question, we studied the effects of visual motion homogeneity in barn owls of both genders, at the behavioral as well as the neural level. Our data show that the coherency of the background motion modulates the perceived saliency of the target object. An object moving in an odd direction relative to other objects attracted more attention when the other objects moved homogeneously compared with when moved in a variety of directions. A possible neural correlate of this effect may arise in the population activity of the intermediate/deep layers of the optic tectum. In these layers, the neural responses to a moving element in the receptive field were suppressed when additional elements moved in the surround. However, when the surrounding elements all moved in one direction (homogeneously moving), they induced less suppression of the response compared with nonhomogeneously moving elements. Moreover, neural responses were more sensitive to the homogeneity of the background motion than to motion-direction contrasts between the receptive field and the surround. The findings suggest similar principles of saliency-by-motion in an avian species as in humans and show a locus in the optic tectum where the underlying neural circuitry may exist.SIGNIFICANCE STATEMENT A critical task of the visual system is to arrange incoming visual information to a meaningful scene of objects and background. In humans, elements that move homogeneously are grouped perceptually to form a categorical whole object. We discovered a similar principle in the barn owl's visual system, whereby the homogeneity of the motion of elements in the scene allows perceptually distinguishing an object from its surround. The novel findings of these visual effects in an avian species, which lacks neocortical structure, suggest that our basic visual perception shares more universal principles across species than presently thought, and shed light on possible brain mechanisms for perceptual grouping.

The latency of a visual evoked potential tracks the onset of decision making.

Encoding of a sensory stimulus is believed to be the first step in perceptual decision making. Previous research has shown that electrical signals recorded from the human brain track evidence accumulation during perceptual decision making (Gold and Shadlen, 2007; O'Connell et al., 2012; Philiastides et al., 2014). In this study we directly tested the hypothesis that the latency of the N200 recorded by EEG (a negative peak occurring between 150 and 275 ms after stimulus presentation in human participants) reflects the visual encoding time (VET) required for completion of figure-ground segregation before evidence accumulation. We show that N200 latencies vary across individuals, are modulated by external visual noise, and increase response time by x milliseconds when they increase by x milliseconds, reflecting a linear regression slope of 1. Simulations of cognitive decision-making theory show that variation in human response times not related to evidence accumulation (non-decision time; NDT), including VET, are tracked by the fastest response times. Evidence that VET is tracked by N200 latencies was found by fitting a linear model between trial-averaged N200 latencies and the 10th percentiles of response times, a model-independent estimate of NDT. Fitting a novel neuro-cognitive model of decision making also yielded a slope of 1 between N200 latency and model-estimated NDT in multiple visual noise conditions, indicating that N200 latencies track the completion of visual encoding and the onset of evidence accumulation. The N200 waveforms were localized to the cortical surface at distributed temporal and extrastriate locations, consistent with a distributed network engaged in figure-ground segregation of the target stimulus.

Border ownership-dependent tilt aftereffect for shape defined by binocular disparity and motion parallax.

Discerning objects from their surrounds (i.e., figure-ground segmentation) in a way that guides adaptive behaviors is a fundamental task of the brain. Neurophysiological work has revealed a class of cells in the macaque visual cortex that may be ideally suited to support this neural computation: border ownership cells (Zhou H, Friedman HS, von der Heydt R. J Neurosci 20: 6594-6611, 2000). These orientation-tuned cells appear to respond conditionally to the borders of objects. A behavioral correlate supporting the existence of these cells in humans was demonstrated with two-dimensional luminance-defined objects (von der Heydt R, Macuda T, Qiu FT. J Opt Soc Am A Opt Image Sci Vis 22: 2222-2229, 2005). However, objects in our natural visual environments are often signaled by complex cues, such as motion and binocular disparity. Thus for border ownership systems to effectively support figure-ground segmentation and object depth ordering, they must have access to information from multiple depth cues with strict depth order selectivity. Here we measured in humans (of both sexes) border ownership-dependent tilt aftereffects after adaptation to figures defined by either motion parallax or binocular disparity. We find that both depth cues produce a tilt aftereffect that is selective for figure-ground depth order. Furthermore, we find that the effects of adaptation are transferable between cues, suggesting that these systems may combine depth cues to reduce uncertainty (Bülthoff HH, Mallot HA. J Opt Soc Am A 5: 1749-1758, 1988). These results suggest that border ownership mechanisms have strict depth order selectivity and access to multiple depth cues that are jointly encoded, providing compelling psychophysical support for their role in figure-ground segmentation in natural visual environments. NEW & NOTEWORTHY Figure-ground segmentation is a critical function that may be supported by "border ownership" neural systems that conditionally respond to object borders. We measured border ownership-dependent tilt aftereffects to figures defined by motion parallax or binocular disparity and found aftereffects for both cues. These effects were transferable between cues but selective for figure-ground depth order, suggesting that the neural systems supporting figure-ground segmentation have strict depth order selectivity and access to multiple depth cues that are jointly encoded.

Figure and ground: how the visual cortex integrates local cues for global organization.

Inferring figure-ground organization in two-dimensional images may require different complementary strategies. For isolated objects, it has been shown that mechanisms in visual cortex exploit the overall distribution of contours, but in images of cluttered scenes where the grouping of contours is not obvious, that strategy would fail. However, natural scenes contain local features, specifically contour junctions, that may contribute to the definition of object regions. To study the role of local features in the assignment of border ownership, we recorded single-cell activity from visual cortex in awake behaving Macaca mulatta. We tested configurations perceived as two overlapping figures in which T- and L-junctions depend on the direction of overlap, whereas the overall distribution of contours provides no valid information. While recording responses to the occluding contour, we varied direction of overlap and variably masked some of the critical contour features to determine their influences and their interactions. On average, most features influenced the responses consistently, producing either enhancement or suppression depending on border ownership. Different feature types could have opposite effects even at the same location. Features far from the receptive field produced effects as strong as near features and with the same short latency. Summation was highly nonlinear: any single feature produced more than two-thirds of the effect of all features together. These findings reveal fast and highly specific organization mechanisms, supporting a previously proposed model in which "grouping cells" integrate widely distributed edge signals with specific end-stopped signals to modulate the original edge signals by feedback. NEW & NOTEWORTHY Seeing objects seems effortless, but defining objects in a scene requires sophisticated neural mechanisms. For isolated objects, the visual cortex groups contours based on overall distribution, but this strategy does not work for cluttered scenes. Here, we demonstrate mechanisms that integrate local contour features like T- and L-junctions to resolve clutter. The process is fast, evaluates widely distributed features, and gives any single feature a decisive influence on figure-ground representation.

Electrophysiological Modulation in an Effort to Complete Illusory Figures: Configuration, Illusory Contour and Closure Effects.

Figure recognition process: From the coarse configuration standing from the background to the closure of a meaningful shape, was investigated by ERP technique. ERP components at different latencies from stimulus onset allowed to tap into the figure recognition process at discrete time-points when different cognitive operations take place. In this study, we present two experiments where the support-ratio (SR) of illusory figures was manipulated to vary continuously the recognition of geometrical figures. In the first experiment three shapes were used to vary the SR and the P1 component (80-130 ms) was modulated by the configuration-effect explained, in part for the first time, with the unbalanced physical stimulation between upper and lower visual field. In the second experiment, we used one shape and varied systematically the SR in a discrimination task. The N1 (130-180 ms) and the N2 (230-270 ms) were modulated by two effects: The Ic-effect, represented by the N1, and the closure-effect, represented by the N2, being larger when the SR was small and the discrimination more difficult with respect to large SRs and easier discrimination. These results showed that figure recognition proceeded from the coarse parsing of the visual scene (configuration-effect), through the completion of a set of illusory borders (Ic-effect) to the final assembling of a meaningful shape (closure-effect).

Uncovering the Spatial Profile of Contour Integration from Fixational Saccades: Evidence for Widespread Processing in V1.

During contour integration, neuronal populations in the primary visual cortex (V1) enhance their responses to the contour while suppressing their responses to the noisy background. However, the spatial extent and profile of these responses are not fully understood. To investigate this question, 2 monkeys were trained on a contour detection task while we measured population responses in V1 using voltage-sensitive dyes. During stimulus presentation the animals made few fixational saccades, and we used their changing gaze position to image and analyze neuronal responses from large part of the stimulus, encoding multiple contour/background elements. We found that contour enhancement was present over the entire contour-mapped areas. The background suppression increased with distance from the contour, extending into background-mapped areas remotely located from the contour. The spatial profile of enhancement and suppression fitted well with a Gaussian model. These results imply that the divergent cortical responses to contour integration are modulated independently and extend over large areas in V1.

Correlations Between Hysteretic Categorical and Continuous Judgments of Perceptual Stimuli Supporting a Unified Dynamical Systems Approach to Perception.

We report from two variants of a figure-ground experiment that is known in the literature to involve a bistable perceptual domain. The first variant was conducted as a two-alternative forced-choice experiment and in doing so tested participants on a categorical measurement scale. The second variant involved a Likert scale measure that was considered to represent a continuous measurement scale. The two variants were conducted as a single within-subjects experiment. Measures of bistability operationalized in terms of hysteresis size scores showed significant positive correlations across the two response conditions. The experimental findings are consistent with a dualistic interpretation of self-organizing perceptual systems when they are described on a macrolevel by means of so-called amplitude equations. This is explicitly demonstrated for a Lotka-Volterra-Haken amplitude equation model of task-related brain activity. As a by-product, the proposed dynamical systems perspective also sheds new light on the anchoring problem of producing numerical, continuous judgments.

When Is Accreting/Deleting Texture Seen as In Front? Interpretation of Depth From Texture Motion.

Standard accounts of accretion/deletion of texture treat it as a definite cue to depth ordering: The accreting/deleting surface is interpreted as being behind the adjoining surface. Froyen, Feldman, and Singh showed that accretion/deletion can also, under certain circumstances, be perceived as a 3D column rotating in front, with the accretion/deletion explained by self-occlusion. These displays differ from traditional accretion/deletion displays in a number of factors, including the presence of figure/ground cues, accretion/deletion on both sides of boundaries, and in the number of distinct regions. In a series of experiments, we systematically manipulated each of these factors in order to determine what factors are actually instrumental in creating the rotating column (accretion/deletion in front) interpretation. In Experiment 1, the width of each region was kept fixed while manipulating the number of regions, and in Experiment 2, the width of the overall display was kept fixed. Observers indicated which set of regions they perceived to be in front. In both experiments, accreting/deleting regions were most likely to be seen in front when geometric figural cues favored a figural interpretation and when textural motion was introduced in all regions (rather than on just one side of each boundary). The number of regions had a relatively small effect (although this effect was larger in Experiment 2). These findings indicate that the geometry of the occluding contour is a critical factor in the interpretation of accretion/deleting, and future models of 3D interpretation involving accretion/deletion must include contour geometry as a key component.

Figure-ground modulation in awake primate thalamus.

Figure-ground discrimination refers to the perception of an object, the figure, against a nondescript background. Neural mechanisms of figure-ground detection have been associated with feedback interactions between higher centers and primary visual cortex and have been held to index the effect of global analysis on local feature encoding. Here, in recordings from visual thalamus of alert primates, we demonstrate a robust enhancement of neuronal firing when the figure, as opposed to the ground, component of a motion-defined figure-ground stimulus is located over the receptive field. In this paradigm, visual stimulation of the receptive field and its near environs is identical across both conditions, suggesting the response enhancement reflects higher integrative mechanisms. It thus appears that cortical activity generating the higher-order percept of the figure is simultaneously reentered into the lowest level that is anatomically possible (the thalamus), so that the signature of the evolving representation of the figure is imprinted on the input driving it in an iterative process.

Consistency of Border-Ownership Cells across Artificial Stimuli, Natural Stimuli, and Stimuli with Ambiguous Contours.

Segmentation and recognition of objects in a visual scene are two problems that are hard to solve separately from each other. When segmenting an ambiguous scene, it is helpful to already know the present objects and their shapes. However, for recognizing an object in clutter, one would like to consider its isolated segment alone to avoid confounds from features of other objects. Border-ownership cells (Zhou et al., 2000) appear to play an important role in segmentation, as they signal the side-of-figure of artificial stimuli. The present work explores the role of border-ownership cells in dorsal macaque visual areas V2 and V3 in the segmentation of natural object stimuli and locally ambiguous stimuli. We report two major results. First, compared with previous estimates, we found a smaller percentage of cells that were consistent across artificial stimuli used previously. Second, we found that the average response of those neurons that did respond consistently to the side-of-figure of artificial stimuli also consistently signaled, as a population, the side-of-figure for borders of single faces, occluding faces and, with higher latencies, even stimuli with illusory contours, such as Mooney faces and natural faces completely missing local edge information. In contrast, the local edge or the outlines of the face alone could not always evoke a significant border-ownership signal. Our results underscore that border ownership is coded by a population of cells, and indicate that these cells integrate a variety of cues, including low-level features and global object context, to compute the segmentation of the scene. SIGNIFICANCE STATEMENT: To distinguish different objects in a natural scene, the brain must segment the image into regions corresponding to objects. The so-called "border-ownership" cells appear to be dedicated to this task, as they signal for a given edge on which side the object is that owns it. Here, we report that individual border-ownership cells are unreliable when tested across a battery of artificial stimuli used previously but can signal border-ownership consistently as a population. We show that these border-ownership population signals are also suited for signaling border-ownership for natural objects and at longer latency, even for stimuli without local edge information. Our results suggest that border-ownership cells integrate both local, low-level and global, high-level cues to segment the scene.

The Representation of Color across the Human Visual Cortex: Distinguishing Chromatic Signals Contributing to Object Form Versus Surface Color.

Many theories of visual object perception assume the visual system initially extracts borders between objects and their background and then "fills in" color to the resulting object surfaces. We investigated the transformation of chromatic signals across the human ventral visual stream, with particular interest in distinguishing representations of object surface color from representations of chromatic signals reflecting the retinal input. We used fMRI to measure brain activity while participants viewed figure-ground stimuli that differed either in the position or in the color contrast polarity of the foreground object (the figure). Multivariate pattern analysis revealed that classifiers were able to decode information about which color was presented at a particular retinal location from early visual areas, whereas regions further along the ventral stream exhibited biases for representing color as part of an object's surface, irrespective of its position on the retina. Additional analyses showed that although activity in V2 contained strong chromatic contrast information to support the early parsing of objects within a visual scene, activity in this area also signaled information about object surface color. These findings are consistent with the view that mechanisms underlying scene segmentation and the binding of color to object surfaces converge in V2.

Texture Segregation Causes Early Figure Enhancement and Later Ground Suppression in Areas V1 and V4 of Visual Cortex.

Segregation of images into figures and background is fundamental for visual perception. Cortical neurons respond more strongly to figural image elements than to background elements, but the mechanisms of figure-ground modulation (FGM) are only partially understood. It is unclear whether FGM in early and mid-level visual cortex is caused by an enhanced response to the figure, a suppressed response to the background, or both.We studied neuronal activity in areas V1 and V4 in monkeys performing a texture segregation task. We compared texture-defined figures with homogeneous textures and found an early enhancement of the figure representation, and a later suppression of the background. Across neurons, the strength of figure enhancement was independent of the strength of background suppression.We also examined activity in the different V1 layers. Both figure enhancement and ground suppression were strongest in superficial and deep layers and weaker in layer 4. The current-source density profiles suggested that figure enhancement was caused by stronger synaptic inputs in feedback-recipient layers 1, 2, and 5 and ground suppression by weaker inputs in these layers, suggesting an important role for feedback connections from higher level areas. These results provide new insights into the mechanisms for figure-ground organization.

Neurons forming optic glomeruli compute figure-ground discriminations in Drosophila.

Many animals rely on visual figure-ground discrimination to aid in navigation, and to draw attention to salient features like conspecifics or predators. Even figures that are similar in pattern and luminance to the visual surroundings can be distinguished by the optical disparity generated by their relative motion against the ground, and yet the neural mechanisms underlying these visual discriminations are not well understood. We show in flies that a diverse array of figure-ground stimuli containing a motion-defined edge elicit statistically similar behavioral responses to one another, and statistically distinct behavioral responses from ground motion alone. From studies in larger flies and other insect species, we hypothesized that the circuitry of the lobula--one of the four, primary neuropiles of the fly optic lobe--performs this visual discrimination. Using calcium imaging of input dendrites, we then show that information encoded in cells projecting from the lobula to discrete optic glomeruli in the central brain group these sets of figure-ground stimuli in a homologous manner to the behavior; "figure-like" stimuli are coded similar to one another and "ground-like" stimuli are encoded differently. One cell class responds to the leading edge of a figure and is suppressed by ground motion. Two other classes cluster any figure-like stimuli, including a figure moving opposite the ground, distinctly from ground alone. This evidence demonstrates that lobula outputs provide a diverse basis set encoding visual features necessary for figure detection.