Interactions Between 3D Surface Shape and Material Perception.

Our visual systems are remarkably adept at deriving the shape and material properties of surfaces even when only one image of a surface is available. This ability implies that a single image of a surface contains potent information about both surface shape and material. However, from a computational perspective, the problem of deriving surface shape and material is formally ill posed. Any given image could be due to many combinations of shape, material, and illumination. Early computational models required prior knowledge about two of the three scene variables to derive the third. However, such models are biologically implausible because our visual systems are tasked with extracting all relevant scene variables from images simultaneously. This review describes recent progress in understanding how the visual system solves this problem by identifying complex forms of image structure that support its ability to simultaneously derive the shape and material properties of surfaces from images.

Where do the hypotheses come from? Data-driven learning in science and the brain.

Everyone agrees that testing hypotheses is important, but Bowers et al. provide scant details about where hypotheses about perception and brain function should come from. We suggest that the answer lies in considering how information about the outside world could be acquired - that is, learned - over the course of evolution and development. Deep neural networks (DNNs) provide one tool to address this question.

Illusory optical defocus generated by shaded surface texture.

The human visual system is tasked with the problem of extracting information about the world from images that contain a conflated mixture of environmental sources and optical artifacts generated by the focal properties of our eyes. In most contexts, our brains manage to distinguish these sources, but this is not always the case. Recent work showed that shading gradients generated by smooth three-dimensional (3D) surfaces can elicit strong illusory percepts of optical defocus1,2 - the perception of illusory blur is only eliminated when the surface appears attached to self-occluding contours3, surface discontinuities1, or sharp specular reflections1,2, which all generate sharp ('high spatial frequency') image structure. This suggests that it should also be possible to eliminate the illusory blur elicited by shaded surfaces by altering the surface geometry to include small-scale surface relief, which would also generate high-frequency image structure. We report the surprising result here that this manipulation fails to eliminate the perception of blur; the fine texture fails to perceptually 'bind' to the low-frequency image structure when there is a sufficient gap between the spatial scales of the fine and coarse surface structure. These findings suggest that discontinuous 'gaps' in the spatial scale of textures are a segmentation cue the visual system uses to extract multiple causes of image structure.

The role of self-occluding contours in material perception.

The human visual system extracts both the three-dimensional (3D) shape and the material properties of surfaces from single images.1,2,3,4,5,6,7,8,9,10,11,12,13,14 Understanding this remarkable ability is difficult because the problem of extracting both shape and material is formally ill posed: information about one appears to be needed to recover the other.14,15,16,17 Recent work has suggested that a particular class of image contours formed by a surface curving smoothly out of sight (self-occluding contours) contains information that co-specifies both surface shape and material for opaque surfaces.18 However, many natural materials are light permeable (translucent); it is unknown whether there is information along self-occluding contours that can be used to distinguish opaque and translucent materials. Here, we present physical simulations, which demonstrate that variations in intensity generated by opaque and translucent materials are linked to different shape attributes of self-occluding contours. Psychophysical experiments demonstrate that the human visual system exploits the different forms of intensity-shape covariation along self-occluding contours to distinguish opaque and translucent materials. These results provide insight into how the visual system manages to solve the putatively ill-posed problem of extracting both the shape and material properties of 3D surfaces from images.

Perceiving the shape and material properties of 3D surfaces.

Our visual experience of the world relies on the interaction of light with the different substances, surfaces, and objects in our environment. These optical interactions generate images that contain a conflated mixture of different scene variables, which our visual system must somehow disentangle to extract information about the shape and material properties of the world. Such problems have historically been considered to be ill-posed, but recent work suggests that there are complex patterns of covariation in light that co-specify the 3D shape and material properties of surfaces. This work provides new insights into how the visual system acquired the ability to solve problems that have historically been considered intractable.

The role of color in the perception of three-dimensional shape.

The human visual system can derive information about three-dimensional (3D) shape from the structure of light reflected by surfaces. Most research on single static images has focused on the 3D shape information contained in variations of brightness caused by interactions between the illumination and local surface orientation ("shading").1-6 Although color can enhance the recovery of surface shading when color and brightness vary independently,7-9 there is no evidence that color alone provides any information about 3D shape. Here, we show that the wavelength-dependent reflectance of chromatic materials provides information about the 3D shape of translucent materials. We show that different wavelengths of light undergo varying degrees of subsurface light transport, which generates multiple forms of spatial structure: wavelengths that are weakly reflected generate shading-like image structure, linked to 3D surface orientation, whereas wavelengths that penetrate more deeply into the material are primarily constrained by the direction of surface curvature (convexities and concavities).10 Psychophysical experiments demonstrate that the enhanced perception of 3D shape in chromatic translucent surfaces arises from the shading structure generated by weakly reflected wavelengths, which, in turn, generates correlated spatial variations in saturation. These results demonstrate a new functional role for color in the perception of the 3D shape of translucent materials.

Unsupervised learning predicts human perception and misperception of gloss.

Reflectance, lighting and geometry combine in complex ways to create images. How do we disentangle these to perceive individual properties, such as surface glossiness? We suggest that brains disentangle properties by learning to model statistical structure in proximal images. To test this hypothesis, we trained unsupervised generative neural networks on renderings of glossy surfaces and compared their representations with human gloss judgements. The networks spontaneously cluster images according to distal properties such as reflectance and illumination, despite receiving no explicit information about these properties. Intriguingly, the resulting representations also predict the specific patterns of 'successes' and 'errors' in human perception. Linearly decoding specular reflectance from the model's internal code predicts human gloss perception better than ground truth, supervised networks or control models, and it predicts, on an image-by-image basis, illusions of gloss perception caused by interactions between material, shape and lighting. Unsupervised learning may underlie many perceptual dimensions in vision and beyond.

The cospecification of the shape and material properties of light permeable materials.

The problem of extracting the three-dimensional (3D) shape and material properties of surfaces from images is considered to be inherently ill posed. It is thought that a priori knowledge about either 3D shape is needed to infer material properties, or knowledge about material properties are needed to derive 3D shape. Here, we show that there is information in images that cospecify both the material composition and 3D shape of light permeable (translucent) materials. Specifically, we show that the intensity gradients generated by subsurface scattering, the shape of self-occluding contours, and the distribution of specular reflections covary in systematic ways that are diagnostic of both the surface's 3D shape and its material properties. These sources of image covariation emerge from being causally linked to a common environmental source: 3D surface curvature. We show that these sources of covariation take the form of "photogeometric constraints," which link variations in intensity (photometric constraints) to the sign and direction of 3D surface curvature (geometric constraints). We experimentally demonstrate that this covariation generates emergent cues that the visual system exploits to derive the 3D shape and material properties of translucent surfaces and demonstrate the potency of these cues by constructing counterfeit images that evoke vivid percepts of 3D shape and translucency. The concepts of covariation and cospecification articulated herein suggest a principled conceptual path forward for identifying emergent cues that can be used to solve problems in vision that have historically been assumed to be ill posed.

Mid-level vision.

In this primer, Anderson provides an overview of some central topics in mid-level vision, highlighting some recent advances in our understanding of how the human visual system identifies different environmental sources of optical structure.

The perception and misperception of optical defocus, shading, and shape.

The human visual system is tasked with recovering the different physical sources of optical structure that generate our retinal images. Separate research has focused on understanding how the visual system estimates (a) environmental sources of image structure and (b) blur induced by the eye's limited focal range, but little is known about how the visual system distinguishes environmental sources from optical defocus. Here, we present evidence that this is a fundamental perceptual problem and provide insights into how and when the visual system succeeds and fails in solving it. We show that fully focused surface shading can be misperceived as defocused and that optical blur can be misattributed to the material properties and shape of surfaces. We further reveal how these misperceptions depend on the relationship between shading gradients and sharp contours, and conclude that computations of blur are inherently linked to computations of surface shape, material, and illumination.

Irrational contour synthesis.

The mechanisms responsible for generating illusory contours are thought to fulfil an adaptive role in providing estimates of missing contour fragments generated by partial camouflage. One striking apparent counter-example to this view was described in Current Biology 21 (2011) 492-496, which showed that illusory contours could arise in motion displays depicting visible occluding discs occluding and disoccluding thin contours. These motion sequences generate illusory contours even though they play no necessary role in accounting for occlusion and disocclusion of the thin contours. The present work sought to more precisely characterize the quantitative dependence of these 'irrational' contours on the relative contrasts in the image. We show that the perceived strength of the illusory contours generated by these displays depends monotonically on the relative contrast of the occluding and occluded contours and that previous attempts to measure their strength with a method of adjustment appears to be contaminated by response bias. We further show that these illusory contours also arise when the occluding disks are rendered transparent and exhibit similar forms of contrast dependencies. These findings reveal a general methodological problem that can arise using methods of adjustment and provide quantitative data that may be used to identify the neural mechanisms responsible for IC genesis and their perceived strength.

Photogeometric Cues to Perceived Surface Shading.

The human visual system is remarkably adept at extracting the three-dimensional (3D) shape of surfaces from images of smoothly shaded surfaces (shape from shading). Most research into this remarkable perceptual ability has focused on understanding how the visual system derives a specific representation of 3D shape when it is known (or assumed) that shading and self-occluding contours are the sole causes of image structure [1-11]. But there is an even more fundamental problem that must be solved before any such analysis can take place: how does the visual system determine when it's viewing a shaded surface? Here, we present theoretical analyses showing that there is statistically reliable information generated along the bounding contours of smoothly curved surfaces that the visual system uses to identify surface shading. This information can be captured by two photogeometric constraints that link the shape of bounding contours to the distributions of shading intensity along the contours: one that links shading intensity to the local orientations along bounding contours and a second that links shading intensity to bounding contour curvature. We show that these constraints predict the perception of shading for surfaces with smooth self-occluding contours and a widely studied class of bounding contours (planar cuts). The results provide new insights into the information that the visual system exploits to distinguish surface shading from other sources of image structure and offer a coherent explanation of the influence of bounding contours on the perception of surface shading and 3D shape.

Visual Perception: To Curve or Not to Curve.

A popular new illusion shows that the apparent curvature of sinusoidal contours depends on contrast and background luminance. We suggest that the illusion is driven by segmentation mechanisms of human vision, which isolate the contours into smaller segments, some which approximate straight lines, others curves.

Perception and misperception of surface opacity.

A fundamental problem in extracting scene structure is distinguishing different physical sources of image structure. Light reflected by an opaque surface covaries with local surface orientation, whereas light transported through the body of a translucent material does not. This suggests the possibility that the visual system may use the covariation of local surface orientation and intensity as a cue to the opacity of surfaces. We tested this hypothesis by manipulating the contrast of luminance gradients and the surface geometries to which they belonged and assessed how these manipulations affected the perception of surface opacity/translucency. We show that (i) identical luminance gradients can appear either translucent or opaque depending on the relationship between luminance and perceived 3D surface orientation, (ii) illusory percepts of translucency can be induced by embedding opaque surfaces in diffuse light fields that eliminate the covariation between surface orientation and intensity, and (iii) illusory percepts of opacity can be generated when transparent materials are embedded in a light field that generates images where surface orientation and intensity covary. Our results provide insight into how the visual system distinguishes opaque surfaces and light-permeable materials and why discrepancies arise between the perception and physics of opacity and translucency. These results suggest that the most significant information used to compute the perceived opacity and translucency of surfaces arise at a level of representation where 3D shape is made explicit.

What predicts the strength of simultaneous color contrast?

The perceived color of a uniform image patch depends not only on the spectral content of the light that reaches the eye but also on its context. One of the most extensively studied forms of context dependence is a simultaneous contrast display: a center-surround display containing a homogeneous target embedded in a homogenous surround. A number of models have been proposed to account for the chromatic transformations of targets induced by such surrounds, but they were typically derived in the restricted context of experiments using achromatic targets with surrounds that varied along the cardinal axes of color space. There is currently no theoretical consensus that predicts the target color that produces the largest perceived color difference for two arbitrarily chosen surround colors, or what surround would give the largest color induction for an arbitrarily chosen target. Here, we present a method for assessing simultaneous contrast that avoids some of the methodological issues that arise with nulling and matching experiments and diminishes the contribution of temporal adaption. Observers were presented with pairs of center-surround patterns and ordered them from largest to smallest in perceived dissimilarity. We find that the perceived difference for two arbitrarily chosen surrounds is largest when the target falls on the line connecting the two surrounds in color space. We also find that the magnitude of induction is larger for larger differences between chromatic targets and surrounds of the same hue. Our results are consistent with the direction law (Ekroll & Faul, 2012b), and with a generalization of Kirschmann's fourth law, even for viewing conditions that do not favor temporal adaptation.

Perceptual dimensions underlying lightness perception in homogeneous center-surround displays.

Lightness judgments of targets embedded in a homogeneous surround exhibit abrupt steps in perceived lightness at points at which the targets transition from being increments to decrements. This "crispening effect" and the general difficulty of matching low-contrast targets embedded in homogeneous surrounds suggest that a second perceptual dimension in addition to lightness may contribute to the appearance of test patches in these displays. The present study explicitly tested whether two dimensions (lightness and transmittance) could lead to more satisfactory matches than lightness alone in an asymmetric matching task. We also examined whether transmittance matches were more strongly associated with task instructions that had observers match perceived transparency or the perceived edge contrast of the target relative to the surround. We found that matching target lightness in a homogeneous display to that in a textured or rocky display required varying both lightness and transmittance of the test patch on the textured display to obtain the most satisfactory matches. However, observers primarily varied transmittance when instructed to match the perceived contrast of targets against homogeneous surrounds, but not when instructed to match the amount of transparency perceived in the displays. The results suggest that perceived target-surround edge contrast differs between homogeneous and textured displays. Varying the midlevel property of transparency in textured displays provides a natural means for equating both target lightness and the unique appearance of the edge contrast in homogeneous displays.

Motion and texture shape cues modulate perceived material properties.

Specular and matte surfaces can project identical images if the surface geometry and light field are appropriately configured. Our previous work has shown that the visual system can exploit stereopsis and contour cues to 3D shape to disambiguate different surface reflectance interpretations. Here, we test whether material perception depends on information about surface geometry provided by structure from motion and shape from texture. Different surface textures were superimposed on a fixed pattern of luminance gradients to generate two different 3D shape interpretations. Each shape interpretation of the luminance gradients promoted a different experience of surface reflectance and illumination direction, which varied from a specular surface in frontal illumination to a comparatively matte surface in grazing illumination. The shape that appeared most specular exhibited the steepest derivatives of luminance with respect to surface orientation, consistent with physical differences between specular and diffuse reflectance. The effect of apparent shape on perceived reflectance occurred for a variety of surface textures that provided either structure from motion, shape from texture, or both optical sources of shape information. In conjunction with previous findings (Marlow, Todorovic, & Anderson, 2015; Marlow & Anderson, 2015), these results suggest that any cue that provides sufficient information about 3D shape can also be used to derive material properties from the rate that luminance varies as a function of surface curvature.

Where does fitness fit in theories of perception?

Interface theory asserts that neither our perceptual experience of the world nor the scientific constructs used to describe the world are veridical. The primary argument used to uphold this claim is that (1) evolution is driven by a process of natural selection that favors fitness over veridicality, and (2) payoffs do not vary monotonically with truth. I argue that both the arguments used to bolster this claim and the conclusions derived from it are flawed. Interface theory assumes that perception evolved to directly track fitness but fails to consider the role of adaptation on ontogenetic time scales. I argue that the ubiquity of nonmonotonic payoff functions requires that (1) perception tracks "truth" for species that adapt on ontogenetic time scales and (2) that perception should be distinct from utility. These conditions are required to pursue an adaptive strategy to mitigate homeostatic imbalances. I also discuss issues with the interface metaphor, the particular formulation of veridicality that is considered, and the relationship of interface theory to the history of ideas on these topics.

The role of chromatic variance in modulating color appearance.

Chromatictarget patches embedded in a chromatically variegated surround appear less saturated than when they are embedded in an achromatic uniform surround (Brown & MacLeod, 1997), which can be construed as either a form of gamut expansion for targets on uniform surrounds or as a form of gamut compression for targets on variegated surrounds.Ekroll, Faul, and Niederée (2004) suggested that the difference in perceived chromaticity on the two surrounds is caused by a layered scene decomposition, wherein the increased saturation of targets on homogenous surrounds is attributed to a decomposition of a target patch into a chromatically saturated transparent layer overlying an achromatic background.Here, we report asymmetric matching data that show the perceived chromaticity difference observed on the two surrounds depends on the particular direction of chromatic variation applied to the variegated surround. If the chromatic variegated surround has the same or a similar hue to that of the target and the saturation variation of the surround is large compared to the saturation of the target the gamut expansion effect is also large. However, if the variegated surround has a different hue than the hue of the target, the perceived chromaticity difference is small and largely does not depend on the variation in saturation of the surround. These results suggest that a layered scene representation cannot fully explain the gamut expansion effect and suggest that a chromatically tuned contrast gain control mechanism may contribute to the difference in perceived color of targets on achromatic homogeneous surrounds and chromatically variegated surrounds.

Material properties derived from three-dimensional shape representations.

Retinal image structure is due to a complex mixture of physical sources that includes the surface's 3D shape, light-reflectance and transmittance properties, and the light field. The visual system can somehow discriminate between these different sources of image structure and recover information about the objects and surfaces in the scene. There has been significant debate about the nature of the representations that are used to derive surface reflectance properties such as specularity (gloss). Specularity could be derived either directly from 2D image properties or by exploiting information that can only be derived from representations in which 3D shape has been made explicit. We recently provided evidence that 3D shape information can play a critical role in the perception of material specularity, but the shape manipulation in our prior study also significantly changed 2D image properties (Marlow, Todorovic, & Anderson, 2015). Here, we held fixed all monocularly visible 2D image properties and manipulated 3D shape stereoscopically. When binocularly fused, the depicted 3D shapes induced striking transformations in the surfaces' apparent material properties, which vary from matte to 'metallic'. Our psychophysical measurements of perceived specularity reveal that 3D shape information can play a critical role in material perception for both singly-curved surfaces and more complex geometries that curve in two directions. These results provide strong evidence that the perception of material specularity can depend on physical constraints derived from representations in which three-dimensional shape has been made explicit.