Qualitative perception of 3D shape from patterns of luminance curvature.

A new source of information is proposed for the perception of three-dimensional (3D) shape from shading that identifies surface concavities from the curvature of the luminance field. Two experiments measured the abilities of human observers to identify concavities on smoothly curved shaded surfaces depicted with several different patterns of illumination and several different material properties. Observers were required to identify any apparent concavities along designated cross sections of the depicted objects and to mark each concavity with an adjustable dot. To analyze the results, we computed both the surface curvature and the luminance curvature along each image cross section. The results revealed that most responses were in concave regions of the luminance profiles, although they were often shifted in phase relative to the curvature of the depicted surfaces. This pattern of performance was surprisingly robust over large changes in the pattern of illumination or surface material properties. Our analysis predicts that observers should make false alarm responses in regions where a luminance concavity does not correspond to a surface concavity, and our empirical results confirm that prediction.

The many facets of shape.

Shape is an interesting property of objects because it is used in ordinary discourse in ways that seem to have little connection to how it is typically defined in mathematics. The present article describes how the concept of shape can be grounded within Euclidean and non-Euclidean geometry and also to human perception. It considers the formal methods that have been proposed for measuring the differences among shapes and how the performance of those methods compares with shape difference thresholds of human observers. It discusses how different types of shape change can be perceptually categorized. It also evaluates the specific data structures that have been used to represent shape in models of both human and machine vision, and it reviews the psychophysical evidence about the extent to which those models are consistent with human perception. Based on this review of the literature, we argue that shape is not one thing but rather a collection of many object attributes, some of which are more perceptually salient than others. Because the relative importance of these attributes can be context dependent, there is no obvious single definition of shape that is universally applicable in all situations.

Are mirror-symmetric objects of special importance for 3D shape perception? A reply to Sawada and Pizlo (2022).

Yu, Todd, and Petrov (2021) and Yu, Petrov, and Todd (2021) investigated failures of shape constancy that occur when objects are viewed stereoscopically at different distances. Although this result has been reported previously with simple objects such as pyramids or cylinders, we examined more complex objects with bilateral symmetry to test the claim by Li, Sawada, Shi, Kwon, and Pizlo (2011) that the perception of those objects is veridical. Sawada and Pizlo (2022) offer several criticisms of our experiments, but they seem to suggest that the concept of shape is defined by what is computable by their model. If stimuli are used that cannot be discriminated by their model, they are dismissed as degenerate, and tasks that cannot be performed by their model are assumed to be based on something other than shape. This allows them to disregard empirical evidence that is inconsistent with their model. We argue, in contrast, that all reliable aspects of shape perception are deserving of explanation. We also argue that there are many different attributes of shape and many different sources of information about shape that may be relevant in different contexts. It is unlikely that all of them can be explained by a single model.

Failures of stereoscopic shape constancy over changes of viewing distance and size for bilaterally symmetric polyhedra.

Two shape matching experiments examined the effects of viewing distance and object size on observers' judgments of 3D metric shape under binocular viewing. Unlike previous studies on this topic, the stimuli were specifically designed to satisfy the minimal conditions for computing veridical shape from symmetry. Concretely, the stimuli were complex, mirror-symmetric polyhedra whose symmetry planes were oriented at an angle of 45o relative to the line of sight in a shape-matching task. Although it is mathematically possible to accurately compute the 3D shapes of these stimuli using relatively simple algorithms, the results indicated that human observers are unable to do so. Indeed, the apparent shapes of the objects were systematically expanded or compressed in depth as a function of viewing distance, in exactly the same way as has been reported for simpler stimuli that do not satisfy the minimal conditions for an accurate computational analysis. For objects presented at near distances, we also obtained statistically significant effects of object size on observers' shape judgments.

Contours produced by internal specular interreflections provide visual information for the perception of glass materials.

Two experiments are reported that investigated how the perceptual identification of glass is influenced by banding contours formed by internal specular interreflections within glass materials. Observers made material categorization judgments for images depicting glass, chrome, shiny black and shiny white objects, and for contour drawings that were created by edge filtering images of glass, chrome or textured objects. Observers rated each stimulus by adjusting four sliders to indicate their confidence that the depicted material was glass, metal, shiny black, or something else, and these adjustments were constrained so that the sum of all four settings was always 100%. The results revealed that the rendered images were all categorized correctly with a high level of confidence. The contour drawings of glass and textured materials were also categorized correctly with a high level of confidence. However, the contour drawings of chrome materials were miscategorized as glass, with an average confidence rating that was significantly lower than those obtained for the glass contours. It is hypothesized that these different contour types are perceptually distinguished from one another based on how they align with the pattern of surface curvature on an object and the smoothness of the contours.

Effects of illumination on the categorization of shiny materials.

The present research was designed to examine how patterns of illumination influence the perceptual categorization of metal, shiny black, and shiny white materials. The stimuli depicted three possible objects that were illuminated by five possible high-dynamic-range imaging light maps, which varied in their overall distributions of illuminant directions and intensities. The surfaces included a low roughness chrome material, a shiny black material, and a shiny white material with both diffuse and specular components. Observers rated each stimulus by adjusting four sliders to indicate their confidence that the depicted material was metal, shiny black, shiny white, or something else, and these adjustments were constrained so that the sum of all four settings was always 100%. The results revealed that the metal and shiny black categories are easily confused. For example, metal materials with low intensity light maps or a narrow range of illuminant directions are often judged as shiny black, whereas shiny black materials with high intensity light maps or a wide range of illuminant directions are often judged as metal. To discover the visual information on which these judgements are based, we measured several possible image statistics, and we found two that were highly correlated with the observers' confidence ratings in appropriate contexts. We also performed a spherical harmonic analysis on the different light maps to quantitatively predict how they would bias observers' judgments of metal and shiny black surfaces.

Reflections on glass.

An important phenomenon in the study of human perception is the ability of observers to identify different types of surface materials. The present article will consider a wide range of factors that can influence the perceptual identification of glass, including the structural complexity of an object, whether it is hollow or solid, and the pattern of illumination. Several illumination techniques used in the field of photography are described, and examples are provided to show how they interact with structural complexity. A single psychophysical experiment is reported to evaluate the perceptions of naïve observers using a novel categorization task designed to assess potential confusions among multiple material categories. Finally, the paper will enumerate a number of specific image features that are potentially diagnostic for the identification of glass, and it will evaluate their relative importance for human perception.

The visual perception of metal.

The present research was designed to examine how the presence or absence of ambient light influences the appearance of metal. The stimuli depicted three possible objects that were illuminated by three possible patterns of illumination. These were generated by a single point light source, two rectangular area lights, or projecting light onto a translucent white box that contained the object (and the camera) so that the object would be illuminated by ambient light in all directions. The materials were simulated using measured parameters of chrome with four different levels of roughness. Observers rated the metallic appearance and shininess of each depicted object using two sliders. The highest rated appearance of metal and shininess occurred for the surfaces with the lowest roughness in the ambient illumination condition, and these ratings dropped systematically as the roughness was increased. For the objects illuminated by point or area lights, the appearance of metal and shininess were significantly less than in the ambient conditions for the lowest roughness value, and significantly greater than in the ambient condition for intermediate values of roughness. We also included a control condition depicting objects with a shiny plastic reflectance function that had both diffuse and specular components. These objects were rated as highly shiny but they did not appear metallic. A theoretical hypothesis is proposed that the defining characteristic of metal (as opposed to black plastic) is the presence of specular sheen over most of the visible surface area.

3D Shape Perception in Posterior Cortical Atrophy: A Visual Neuroscience Perspective.

Posterior cortical atrophy (PCA) is a rare focal neurodegenerative syndrome characterized by progressive visuoperceptual and visuospatial deficits, most often due to atypical Alzheimer's disease (AD). We applied insights from basic visual neuroscience to analyze 3D shape perception in humans affected by PCA. Thirteen PCA patients and 30 matched healthy controls participated, together with two patient control groups with diffuse Lewy body dementia (DLBD) and an amnestic-dominant phenotype of AD, respectively. The hierarchical study design consisted of 3D shape processing for 4 cues (shading, motion, texture, and binocular disparity) with corresponding 2D and elementary feature extraction control conditions. PCA and DLBD exhibited severe 3D shape-processing deficits and AD to a lesser degree. In PCA, deficient 3D shape-from-shading was associated with volume loss in the right posterior inferior temporal cortex. This region coincided with a region of functional activation during 3D shape-from-shading in healthy controls. In PCA patients who performed the same fMRI paradigm, response amplitude during 3D shape-from-shading was reduced in this region. Gray matter volume in this region also correlated with 3D shape-from-shading in AD. 3D shape-from-disparity in PCA was associated with volume loss slightly more anteriorly in posterior inferior temporal cortex as well as in ventral premotor cortex. The findings in right posterior inferior temporal cortex and right premotor cortex are consistent with neurophysiologically based models of the functional anatomy of 3D shape processing. However, in DLBD, 3D shape deficits rely on mechanisms distinct from inferior temporal structural integrity. SIGNIFICANCE STATEMENT: Posterior cortical atrophy (PCA) is a neurodegenerative syndrome characterized by progressive visuoperceptual dysfunction and most often an atypical presentation of Alzheimer's disease (AD) affecting the ventral and dorsal visual streams rather than the medial temporal system. We applied insights from fundamental visual neuroscience to analyze 3D shape perception in PCA. 3D shape-processing deficits were affected beyond what could be accounted for by lower-order processing deficits. For shading and disparity, this was related to volume loss in regions previously implicated in 3D shape processing in the intact human and nonhuman primate brain. Typical amnestic-dominant AD patients also exhibited 3D shape deficits. Advanced visual neuroscience provides insight into the pathogenesis of PCA that also bears relevance for vision in typical AD.

Can a Bayesian analysis account for systematic errors in judgments of 3D shape from texture? A reply to Saunders and Chen.

Saunders and Chen (2015) have recently proposed a Bayesian model of the perception of 3D shape from texture that they claim is superior to an alternative model based on scaling contrast that was originally proposed by Todd, Thaler, Dijkstra, Koenderink, and Kappers (2007). This commentary will review a variety of empirical findings that are relevant to this issue, and it will also evaluate how well these findings can be explained by different possible models that have been proposed in the literature. The results will demonstrate that the scaling contrast model can account for almost all of the factors that can influence apparent shape from texture with just two free parameters. The Bayesian model of Saunders and Chen has a greater number of free parameters than the scaling contrast model, yet it is not sufficiently developed to make quantitative predictions about any of these factors. Moreover, there are several other processes that would be necessary to actually implement their model that are not mentioned in their theoretical discussion. These include some mechanism for measuring veridical slant from texture, and a mechanism for computing global 3D shape from local estimates of optical slant.

The darker-is-deeper heuristic for the perception of 3D shape from shading: Is it perceptually or ecologically valid?

The darker-is-deeper heuristic was originally proposed by Langer and Zucker (1994) for approximating 3D shape from shading under conditions of diffuse illumination that typically occur for outdoor scenes on a cloudy day, and it is based on the assumption that vignetting is the primary source of luminance variation under those conditions. It was later rejected as a model of human perception by Langer and Bülthoff (2000), because points in concavities that appear to be the deepest are most often located on local luminance maxima. Despite that result, this heuristic has continued to be described in the literature as a viable model of human perception (e.g., Chen & Tyler, 2015; Tyler, 1998), based entirely on the appearance of image intensity gratings, which have little or no connection to real 3D surfaces or patterns of illumination. In this article we will present a large number of examples to show what actually happens when surfaces are viewed under directional and diffuse illuminations. The results will highlight a number of well-known phenomena in addition to vignetting that can influence the pattern of shading on a surface under diffuse illumination, and they will also demonstrate that the darker-is-deeper heuristic is generally invalid for all types of illumination, except in unusual circumstances.

The effects of smooth occlusions and directions of illumination on the visual perception of 3-D shape from shading.

Human observers made local orientation judgments of smoothly shaded surfaces illuminated from different directions by large area lights, both with and without visible smooth occlusion contours. Test-retest correlations between the first and second halves of the experiment revealed that observers' judgments were highly reliable, with a residual error of only 2%. Over 88% of the variance between observers' judgments and the simulated objects could be accounted for by an affine correlation, but there was also a systematic nonaffine component that accounted for approximately 10% of the perceptual error. The presence or absence of visible smooth occlusion contours had a negligible effect on performance, but there was a small effect of the illumination direction, such that the response surfaces were sheared slightly toward the light source. These shearing effects were much smaller, however, than the effects produced by changes in illumination on the overall pattern of luminance or luminance gradients. Implications of these results for current models of estimating 3-D shape from shading are considered.

On the relative detectability of configural properties.

A match-to-sample shape-discrimination task was employed to measure the detectability of different types of transformations. To create the foils for this task, the standard object could be altered by adding a small hole (a topological property), adding small bumps to straight edges (a projective property), changing the relative orientations of parallel contours (an affine property), or stretching the standard object to alter its aspect ratio (a Euclidean property). The results revealed that the relative perceptual salience of different types of shape change is consistent with the Klein hierarchy of geometries. That is to say observers were most sensitive to changes in topological structure, followed by changes in projective, affine, and Euclidean structure, respectively. The predicted patterns of performance among the different conditions were computed using a wide variety of commonly used shape-difference metrics, but none of them had a significant positive correlation with the observers' thresholds.

Anterior regions of monkey parietal cortex process visual 3D shape.

The intraparietal cortex is involved in the control of visually guided actions, like reach-to-grasp movements, which require extracting the 3D shape and position of objects from 2D retinal images. Using fMRI in behaving monkeys, we investigated the role of the intraparietal cortex in processing stereoscopic information for recovering the depth structure and the position in depth of objects. We found that while several areas (CIP, LIP, and AIP on the lateral bank; PIP and MIP on the medial bank) are activated by stereoscopic stimuli, AIP and an adjoining portion of LIP are sensitive only to depth structure. Furthermore, only these two regions are sensitive to both the depth structure and the 2D shape of small objects. These results indicate that extracting 3D spatial information from stereo involves several intraparietal areas, among which AIP and anterior LIP are more specifically engaged in extracting the 3D shape of objects.

The selectivity of neurons in the macaque fundus of the superior temporal area for three-dimensional structure from motion.

Motion is a potent cue for the perception of three-dimensional (3D) shape in primates, but little is known about its underlying neural mechanisms. Guided by recent functional magnetic resonance imaging results, we tested neurons in the fundus of the superior temporal sulcus (FST) area of two macaque monkeys (Macaca mulatta, one male) using motion-defined surface patches with various 3D shapes such as slanted planes, saddles, or cylinders. The majority of the FST neurons (>80%) were selective for stimuli depicting specific shapes, and all the surfaces tested were represented among the selective FST neurons. Importantly, this selectivity tolerated changes in speed, position, size, or between binocular and monocular presentations. This tolerance demonstrates that the 3D structure-from-motion (3D-SFM) selectivity of FST neurons is a higher-order selectivity, which cannot be reduced to a lower-order speed selectivity. The 3D-SFM selectivity of FST neurons was unaffected by removal of the opposed-motion cue that supplemented the speed gradient cue in the standard stimuli. When tested with the same standard stimuli, fewer neurons in the middle temporal/visual 5 (MT/V5) area were selective than FST neurons. In addition, selective MT/V5 neurons represented fewer types of surfaces and were less tolerant of stimulus changes than FST neurons. Overall, these results indicate that FST neurons code motion-defined 3D shape fragments, underscoring the central role of FST in processing 3D-SFM.

The perception of 3D shape from planar cut contours.

A new computational analysis is described for estimating 3D shapes from orthographic images of surfaces that are textured with planar cut contours. For any given contour pattern, this model provides a family of possible interpretations that are all related by affine scaling and shearing transformations in depth, depending on the specific values of its free parameters that are used to compute the shape estimate. Two psychophysical experiments were performed in an effort to compare the model predictions with observers' judgments of 3D shape for developable and non-developable surfaces. The results reveal that observers' perceptions can be systematically distorted by affine scaling and shearing transformations in depth and that the magnitude and direction of these distortions vary systematically with the 3D orientations of the contour planes.

The visual identification of relational categories.

An experiment was performed to investigate the ability of human observers to identify configural relations among three dots. Four stimulus categories were defined on the basis of whether or not the dots were arranged collinearly and whether or not the central dot was equally spaced relative to the two flanking dots. Observers were initially trained with feedback to identify these categories at a single orientation with a fixed uniform background, and then they were tested with variable orientations and backgrounds without feedback. The results revealed almost perfect generalization. We also simulated the same task using a recent feature hierarchy model (J. Mutch & D. G. Lowe, 2008) that is among the most successful for object recognition. This model performed well for fixed orientations and backgrounds, but it could not accurately identify these categories with variable orientations and backgrounds even when given training with those conditions. These findings suggest that feature hierarchy models represent the spatial relations within an image quite differently than human observers.

Bilateral Symmetry Has No Effect on Stereoscopic Shape Judgments.

A single experiment is reported that measured the apparent stereoscopic shapes of symmetric and asymmetric objects at different viewing distances. The symmetric stimuli were specifically designed to satisfy the minimal conditions for computing veridical shape from symmetry. That is to say, they depicted complex, bilaterally symmetric, plane-faced polyhedra whose symmetry planes were oriented at an angle of 45° relative to the line of sight. The asymmetric stimuli were distorted versions of the symmetric ones in which the 3D position of each vertex was randomly displaced. Prior theoretical analyses have shown that it is mathematically possible to compute the 3D shapes of symmetric stimuli under these conditions, but those algorithms are useless for asymmetric objects. The results revealed that the apparent shapes of both types of objects were expanded or compressed in depth as a function of viewing distance, in exactly the same way as has been reported in many other studies, and that the presence or absence of symmetry had no detectable effect on performance.

The processing of three-dimensional shape from disparity in the human brain.

Three-dimensional (3D) shape is important for the visual control of grasping and manipulation and for object recognition. Although there has been some progress in our understanding of how 3D shape is extracted from motion and other monocular cues, little is known of how the human brain extracts 3D shape from disparity, commonly regarded as the strongest depth cue. Previous fMRI studies in the awake monkey have established that the interaction between stereo (present or absent) and the order of disparity (zero or second order) constitutes the MR signature of regions housing second-order disparity-selective neurons (Janssen et al., 2000; Srivastava et al., 2006; Durand et al., 2007; Joly et al., 2007). Testing the interaction between stereo and order of disparity in a large cohort of human subjects, revealed the involvement of five IPS regions (VIPS/V7*, POIPS, DIPSM, DIPSA, and phAIP), as well as V3 and the V3A complex in occipital cortex, the posterior inferior temporal gyrus (ITG), and ventral premotor cortex (vPrCS) in the extraction and processing of 3D shape from stereo. Control experiments ruled out attention and convergence eye movements as confounding factors. Many of these regions, DIPSM, DIPSA, phAIP, and probably posterior ITG and ventral premotor cortex, correspond to monkey regions with similar functionality, whereas the evolutionarily new or modified regions are located in occipital (the V3A complex) and occipitoparietal cortex (VIPS/V7* and POIPS). Interestingly, activity in these occipital regions correlates with the depth amplitude perceived by the subjects in the 3D surfaces used as stimuli in these fMRI experiments.

The perception of 3D shape from texture based on directional width gradients.

A new computational analysis is described that is capable of estimating the 3D shapes of continuously curved surfaces with anisotropic textures that are viewed with negligible perspective. This analysis assumes that the surface texture is homogeneous, and it makes specific predictions about how the apparent shape of a surface should be distorted in cases where that assumption is violated. Two psychophysical experiments are reported in an effort to test those predictions, and the results confirm that observers' ordinal shape judgments are consistent with what would be expected based on the model. The limitations of this analysis are also considered, and a complimentary model is discussed that is only appropriate for surfaces viewed with large amounts of perspective.