'Proto-rivalry': how the binocular brain identifies gloss.

Visually identifying glossy surfaces can be crucial for survival (e.g. ice patches on a road), yet estimating gloss is computationally challenging for both human and machine vision. Here, we demonstrate that human gloss perception exploits some surprisingly simple binocular fusion signals, which are likely available early in the visual cortex. In particular, we show that the unusual disparity gradients and vertical offsets produced by reflections create distinctive 'proto-rivalrous' (barely fusible) image regions that are a critical indicator of gloss. We find that manipulating the gradients and vertical components of binocular disparities yields predictable changes in material appearance. Removing or occluding proto-rivalrous signals makes surfaces look matte, while artificially adding such signals to images makes them appear glossy. This suggests that the human visual system has internalized the idiosyncratic binocular fusion characteristics of glossy surfaces, providing a straightforward means of estimating surface attributes using low-level image signals.

Specular reflections and the estimation of shape from binocular disparity.

Binocular stereopsis is a powerful visual depth cue. To exploit it, the brain matches features from the two eyes' views and measures their interocular disparity. This works well for matte surfaces because disparities indicate true surface locations. However, specular (glossy) surfaces are problematic because highlights and reflections are displaced from the true surface in depth, leading to information that conflicts with other cues to 3D shape. Here, we address the question of how the visual system identifies the disparity information created by specular reflections. One possibility is that the brain uses monocular cues to identify that a surface is specular and modifies its interpretation of the disparities accordingly. However, by characterizing the behavior of specular disparities we show that the disparity signals themselves provide key information ("intrinsic markers") that enable potentially misleading disparities to be identified and rejected. We presented participants with binocular views of specular objects and asked them to report perceived depths by adjusting probe dots. For simple surfaces--which do not exhibit intrinsic indicators that the disparities are "wrong"--participants incorrectly treat disparities at face value, leading to erroneous judgments. When surfaces are more complex we find the visual system also errs where the signals are reliable, but rejects and interpolates across areas with large vertical disparities and horizontal disparity gradients. This suggests a general mechanism in which the visual system assesses the origin and utility of sensory signals based on intrinsic markers of their reliability.

Key characteristics of specular stereo.

Because specular reflection is view-dependent, shiny surfaces behave radically differently from matte, textured surfaces when viewed with two eyes. As a result, specular reflections pose substantial problems for binocular stereopsis. Here we use a combination of computer graphics and geometrical analysis to characterize the key respects in which specular stereo differs from standard stereo, to identify how and why the human visual system fails to reconstruct depths correctly from specular reflections. We describe rendering of stereoscopic images of specular surfaces in which the disparity information can be varied parametrically and independently of monocular appearance. Using the generated surfaces and images, we explain how stereo correspondence can be established with known and unknown surface geometry. We show that even with known geometry, stereo matching for specular surfaces is nontrivial because points in one eye may have zero, one, or multiple matches in the other eye. Matching features typically yield skew (nonintersecting) rays, leading to substantial ortho-epipolar components to the disparities, which makes deriving depth values from matches nontrivial. We suggest that the human visual system may base its depth estimates solely on the epipolar components of disparities while treating the ortho-epipolar components as a measure of the underlying reliability of the disparity signals. Reconstructing virtual surfaces according to these principles reveals that they are piece-wise smooth with very large discontinuities close to inflection points on the physical surface. Together, these distinctive characteristics lead to cues that the visual system could use to diagnose specular reflections from binocular information.