Peripheral material perception.

Humans can rapidly identify materials, such as wood or leather, even within a complex visual scene. Given a single image, one can easily identify the underlying "stuff," even though a given material can have highly variable appearance; fabric comes in unlimited variations of shape, pattern, color, and smoothness, yet we have little trouble categorizing it as fabric. What visual cues do we use to determine material identity? Prior research suggests that simple "texture" features of an image, such as the power spectrum, capture information about material properties and identity. Few studies, however, have tested richer and biologically motivated models of texture. We compared baseline material classification performance to performance with synthetic textures generated from the Portilla-Simoncelli model and several common image degradations. The textures retain statistical information but are otherwise random. We found that performance with textures and most degradations was well below baseline, suggesting insufficient information to support foveal material perception. Interestingly, modern research suggests that peripheral vision might use a statistical, texture-like representation. In a second set of experiments, we found that peripheral performance is more closely predicted by texture and other image degradations. These findings delineate the nature of peripheral material classification.

Vagueness and volume: Testing the perception of depth in images with linear, sharp, or blurred contours.

In European painting, a transition took place where artists started to consciously introduce blurred or soft contours in their works. There may have been several reasons for this. One suggestion in art historical literature is that this may have been done to create a stronger sense of volume in the depicted figures or objects. Here we describe four experiments in which we tried to test whether soft or blurred contours do indeed enhance a sense volume or depth. In the first three experiments, we found that, for both paintings and abstract shapes, three dimensionality was actually decreased instead of increased for blurred (and line) contours, in comparison with sharp contours. In the last experiment, we controlled for the position of the blur (on the lit or dark side) and found that blur on the lit side evoked a stronger impression of three dimensionality. Overall, the experiments robustly show that an art historical conjecture that a blurred contour increases three dimensionality is not granted. Because the blurred contours can be found in many established art works such as from Leonardo and Vermeer, there must be other rationales behind this use than the creation of a stronger sense of volume or depth.

Time-of-day perception in paintings.

The spectral shape, irradiance, direction, and diffuseness of daylight vary regularly throughout the day. The variations in illumination and their effect on the light reflected from objects may in turn provide visual information as to the time of day. We suggest that artists' color choices for paintings of outdoor scenes might convey this information and that therefore the time of day might be decoded from the colors of paintings. Here we investigate whether human viewers' estimates of the depicted time of day in paintings correlate with their image statistics, specifically chromaticity and luminance variations. We tested time-of-day perception in 17th- to 20th-century Western European paintings via two online rating experiments. In Experiment 1, viewers' ratings from seven time choices varied significantly and largely consistently across paintings but with some ambiguity between morning and evening depictions. Analysis of the relationship between image statistics and ratings revealed correlations with the perceived time of day: higher "morningness" ratings associated with higher brightness, contrast, and saturation and darker yellow/brighter blue hues; "eveningness" with lower brightness, contrast, and saturation and darker blue/brighter yellow hues. Multiple linear regressions of extracted principal components yielded a predictive model that explained 76% of the variance in time-of-day perception. In Experiment 2, viewers rated paintings as morning or evening only; rating distributions differed significantly across paintings, and image statistics predicted people's perceptions. These results suggest that artists used different color palettes and patterns to depict different times of day, and the human visual system holds consistent assumptions about the variation of natural light depicted in paintings.

Zooming in on style: Exploring style perception using details of paintings.

Most studies on the perception of style have used whole scenes/entire paintings; in our study, we isolated a single motif (an apple) to reduce or even eliminate the influence of composition, iconography, and other contextual information. In this article, we empirically address two fundamental questions of the existence (Experiment 1) and description (Experiment 2) of style. We chose 48 cut-outs of mostly Western European paintings (15th to 21st century) that showed apples. In Experiment 1, 415 unique participants completed online triplet similarity tasks. Multidimensional scaling (MDS) reached a nonrandom three-dimensional (3D) embedding, showing that participants are able to judge stylistic differences in a systematic way. We also found a strong correlation between creation year and embedding, both a linear correlation with Dimension 2, and a rotational correlation in the first two dimensions. To interpret the embedding further, in Experiment 2, we fitted three color statistics and nine attribute ratings (glossiness, three-dimensionality, convincingness, brush coarseness, etc.) to the 3D perceptual style space. Results showed that Dimension 1 is associated with spatial attributes (Smoothness, Brushstroke coarseness) and Convincingness, Dimension 2 is related to Hue, and Dimension 3 is related to Chroma. The results suggest that texture and color are two important variables for style perception. By isolating the motifs, we could exclude higher levels of information such as composition and context. Interestingly, the results reinforce previous findings using whole scenes, suggesting that style can already be perceived in sometimes very small fragments of paintings.

Materials In Paintings (MIP): An interdisciplinary dataset for perception, art history, and computer vision.

In this paper, we capture and explore the painterly depictions of materials to enable the study of depiction and perception of materials through the artists' eye. We annotated a dataset of 19k paintings with 200k+ bounding boxes from which polygon segments were automatically extracted. Each bounding box was assigned a coarse material label (e.g., fabric) and half was also assigned a fine-grained label (e.g., velvety, silky). The dataset in its entirety is available for browsing and downloading at materialsinpaintings.tudelft.nl. We demonstrate the cross-disciplinary utility of our dataset by presenting novel findings across human perception, art history and, computer vision. Our experiments include a demonstration of how painters create convincing depictions using a stylized approach. We further provide an analysis of the spatial and probabilistic distributions of materials depicted in paintings, in which we for example show that strong patterns exists for material presence and location. Furthermore, we demonstrate how paintings could be used to build more robust computer vision classifiers by learning a more perceptually relevant feature representation. Additionally, we demonstrate that training classifiers on paintings could be used to uncover hidden perceptual cues by visualizing the features used by the classifiers. We conclude that our dataset of painterly material depictions is a rich source for gaining insights into the depiction and perception of materials across multiple disciplines and hope that the release of this dataset will drive multidisciplinary research.

Soft like velvet and shiny like satin: Perceptual material signatures of fabrics depicted in 17th century paintings.

Dutch 17th century painters were masters in depicting materials and their properties in a convincing way. Here, we studied the perception of the material signatures and key image features of different depicted fabrics, like satin and velvet. We also tested whether the perception of fabrics depicted in paintings related to local or global cues, by cropping the stimuli. In Experiment 1, roughness, warmth, softness, heaviness, hairiness, and shininess were rated for the stimuli shown either full figure or cropped. In the full figure, all attributes except shininess were rated higher for velvet, whereas shininess was rated higher for satin. This distinction was less clear in the cropped condition, and some properties were perceived significantly different between the two conditions. In Experiment 2 we tested whether this difference was due to the choice of the cropped area. On the basis of the results of Experiment 1, shininess and softness were rated for multiple crops from each fabric. Most crops from the same fabric differed significantly in shininess, but not in softness perception. Perceived shininess correlated positively with the mean luminance of the crops and the highlights' coverage. Experiment 1 showed that painted velvet and satin triggered distinct perceptions, indicative of robust material signatures of the two fabrics. The results of Experiment 2 suggest that the presence of local image cues affects the perception of optical properties like shininess, but not mechanical properties such as softness.

If painters give you lemons, squeeze the knowledge out of them. A study on the visual perception of the translucent and juicy appearance of citrus fruits in paintings.

Citrus fruits are characterized by a juicy and translucent interior, important properties that drive material recognition and food acceptance. Yet, a thorough understanding of their visual perception is still missing. Using citrus fruits depicted in 17th-century paintings as stimuli, we ran three rating experiments. In Experiment 1, participants rated the perceived similarity in translucency or juiciness of the fruits. In Experiment 2, different groups of participants rated one image feature from a list obtained in a preliminary experiment. In Experiment 3, translucency and juiciness were rated. We constructed two-dimensional perceptual spaces for both material properties and fitted the ratings of the image features into the spaces to interpret them. "Highlights," "peeled side," "bumpiness," and "color saturation" fit the juiciness space best and were high for the highly juicy stimuli. "Peeled side," "intensity of light gradient," "highlights," and "color saturation" were the most salient features of the translucency space, being high for the highly translucent stimuli. The same image features were also indicated in a 17th-century painting manual for material depiction (Beurs, 1692; Beurs, in press). Altogether, we disclosed the expertise of painters with regard to material perception by identifying the image features that trigger a visual impression of juiciness and translucency in citrus fruits.

Painterly depiction of material properties.

Painters are masters of depiction and have learned to evoke a clear perception of materials and material attributes in a natural, three-dimensional setting, with complex lighting conditions. Furthermore, painters are not constrained by reality, meaning that they could paint materials without exactly following the laws of nature, while still evoking the perception of materials. Paintings have to our knowledge not been studied on a big scale from a material perception perspective. In this article, we studied the perception of painted materials and their attributes by using human annotations to find instances of 15 materials, such as wood, stone, fabric, etc. Participants made perceptual judgments about 30 unique segments of these materials for 10 material attributes, such as glossiness, roughness, hardness, etc. We found that participants were able to perform this task well while being highly consistent. Participants, however, did not consistently agree with each other, and the measure of consistency depended on the material attribute being perceived. Additionally, we found that material perception appears to function independently of the medium of depiction-the results of our principal component analysis agreed well with findings in former studies for photographs and computer renderings.

Understanding gloss perception through the lens of art: Combining perception, image analysis, and painting recipes of 17th century painted grapes.

To understand the key image features that we use to infer the glossiness of materials, we analyzed the pictorial shortcuts used by 17th century painters to imitate the optical phenomenon of specular reflections when depicting grapes. Gloss perception of painted grapes was determined via a rating experiment. We computed the contrast, blurriness, and coverage of the grapes' highlights in the paintings' images, inspired by Marlow and Anderson (2013). The highlights were manually segmented from the images, and next the features contrast, coverage, and blurriness were semiautomatically quantified using self-defined algorithms. Multiple linear regressions of contrast and blurriness resulted in a predictive model that could explain 69% of the variance in gloss perception. No effect was found for coverage. These findings are in agreement with the instructions to render glossiness of grapes contained in a 17th century painting manual (Beurs, 1692/in press), suggesting that painting practice embeds knowledge about key image features that trigger specific material percepts.

Visual communication of how fabrics feel.

Although product photos and movies are abundantly present in online shopping environments, little is known about how much of the real product experience they capture. While previous studies have shown that movies or interactive imagery give users the impression that these communication forms are more effective, there are no studies addressing this issue quantitatively. We used nine different samples of jeans, because in general fabrics represent a large and interesting product category and specifically because jeans can visually be rather similar while haptically be rather different. In the first experiment we let observers match a haptic stimulus to a visual representation and found that movies were more informative about how objects would feel than photos. In a second experiment we wanted to confirm this finding by using a different experimental paradigm that we deemed a better general paradigm for future studies on this topic: correlations of pairwise similarity ratings. However, the beneficial effect of the movies was absent when using this new paradigm. In the third experiment we investigated this issue by letting people visually observe other people in making haptic similarity judgments. Here, we did find a significant correlation between haptic and visual data. Together, the three experiments suggest that there is a small but significant effect of movies over photos (Experiment 1) but at the same time a significant difference between visual representations and visually perceiving products in reality (Experiments 2 and 3). This finding suggests a substantial theoretical potential for decreasing the gap between virtual and real product presentation.

Context mitigates crowding: Peripheral object recognition in real-world images.

Object recognition is often conceived of as proceeding by segmenting an object from its surround, then integrating its features. In turn, peripheral vision's sensitivity to clutter, known as visual crowding, has been framed as due to a failure to restrict that integration to features belonging to the object. We hand-segment objects from their background, and find that rather than helping peripheral recognition, this impairs it when compared to viewing the object in its real-world context. Context is in fact so important that it alone (no visible target object) is just as informative, in our experiments, as seeing the object alone. Finally, we find no advantage to separately viewing the context and segmented object. These results, taken together, suggest that we should not think of recognition as ideally operating on pre-segmented objects, nor of crowding as the failure to do so.

Ways of Viewing Pictorial Plasticity.

The plastic effect is historically used to denote various forms of stereopsis. The vivid impression of depth often associated with binocular stereopsis can also be achieved in other ways, for example, using a synopter. Accounts of this go back over a hundred years. These ways of viewing all aim to diminish sensorial evidence that the picture is physically flat. Although various viewing modes have been proposed in the literature, their effects have never been compared. In the current study, we compared three viewing modes: monocular blur, synoptic viewing, and free viewing (using a placebo synopter). By designing a physical embodiment that was indistinguishable for the three experimental conditions, we kept observers naïve with respect to the differences between them; 197 observers participated in an experiment where the three viewing modes were compared by performing a rating task. Results indicate that synoptic viewing causes the largest plastic effect. Monocular blur scores lower than synoptic viewing but is still rated significantly higher than the baseline conditions. The results strengthen the idea that synoptic viewing is not due to a placebo effect. Furthermore, monocular blur has been verified for the first time as a way of experiencing the plastic effect, although the effect is smaller than synoptic viewing. We discuss the results with respect to the theoretical basis for the plastic effect. We show that current theories are not described with sufficient details to explain the differences we found.

Highlight shapes and perception of gloss for real and photographed objects.

Gloss perception strongly depends on the three-dimensional shape and the illumination of the object under consideration. In this study we investigated the influence of the spatial structure of the illumination on gloss perception. A diffuse light box in combination with differently shaped masks was used to produce a set of six simple and complex highlight shapes. The geometry of the simple highlight shapes was inspired by conventional artistic practice (e.g., ring flash for photography, window shape for painting and disk or square for cartoons). In the box we placed spherical stimuli that were painted in six degrees of glossiness. This resulted in a stimulus set of six highlight shapes and six gloss levels, a total of 36 stimuli. We performed three experiments of which two took place using digital photographs on a computer monitor and one with the real spheres in the light box. The observers had to perform a comparison task in which they chose which of two stimuli was glossiest and a rating task in which they rated the glossiness. The results show that, perhaps surprisingly, more complex highlight shapes were perceived to produce a less glossy appearance than simple highlight shapes such as a disk or square. These findings were confirmed for both viewing conditions, on a computer display and in a real setting. The results show that variations in the spatial structure of "rather simple" illumination of the "extended source" type highlight influences perceived glossiness.

Effects of surface reflectance on local second order shape estimation in dynamic scenes.

In dynamic scenes, relative motion between the object, the observer, and/or the environment projects as dynamic visual information onto the retina (optic flow) that facilitates 3D shape perception. When the object is diffusely reflective, e.g. a matte painted surface, this optic flow is directly linked to object shape, a property found at the foundations of most traditional shape-from-motion (SfM) schemes. When the object is specular, the corresponding specular flow is related to shape curvature, a regime change that challenges the visual system to determine concurrently both the shape and the distortions of the (sometimes unknown) environment reflected from its surface. While human observers are able to judge the global 3D shape of most specular objects, shape-from-specular-flow (SFSF) is not veridical. In fact, recent studies have also shown systematic biases in the perceived motion of such objects. Here we focus on the perception of local shape from specular flow and compare it to that of matte-textured rotating objects. Observers judged local surface shape by adjusting a rotation and scale invariant shape index probe. Compared to shape judgments of static objects we find that object motion decreases intra-observer variability in local shape estimation. Moreover, object motion introduces systematic changes in perceived shape between matte-textured and specular conditions. Taken together, this study provides a new insight toward the contribution of motion and surface material to local shape perception.

A new view through Alberti's window.

In his famous treatise on perspective, Alberti compared picture perception with looking through a window. Although Alberti himself was more concerned with picture production than perception, the window metaphor is still widely used to describe picture perception. By performing depth perception experiments, we investigated whether Alberti's hypothesis makes sense in a geometrical fashion. If pictures are regarded as windows, the locus of objects with equal depth should be similar for pictorial and real space-ideally, spherical. Furthermore, if the loci of equidistance are indeed similar for real and pictorial space, their difference should be flat. We designed two experiments to investigate this claim. In the first experiment, a pairwise depth comparison task was used to compute the global perceived depth structure of a complex scene. We found that perception of the real space is more accurate and less ambiguous than pictorial space. More interestingly, we found that the relative differences between these two spaces (locus of relative equidistance) are curved, which contradicts the window hypothesis. In the second experiment, we wanted to measure the absolute locus of equidistance that we believed was diagnostic for the difference between real and pictorial space perception. We found that under normal circumstances, the distribution of equally perceived depths is curved in real space, and relatively flat in pictorial space. However, we also found exceptions. For example, viewing real space with one eye yielded similar results as normal pictorial space perception. We conclude that Alberti's hypothesis needs a revision.

Perception of length to width relations of city squares.

In this paper, we focus on how people perceive the aspect ratio of city squares. Earlier research has focused on distance perception but not so much on the perceived aspect ratio of the surrounding space. Furthermore, those studies have focused on "open" spaces rather than urban areas enclosed by walls, houses and filled with people, cars, etc. In two experiments, we therefore measured, using a direct and an indirect method, the perceived aspect ratio of five city squares in the historic city center of Delft, the Netherlands. We also evaluated whether the perceived aspect ratio of city squares was affected by the position of the observer on the square. In the first experiment, participants were asked to set the aspect ratio of a small rectangle such that it matched the perceived aspect ratio of the city square. In the second experiment, participants were asked to estimate the length and width of the city square separately. In the first experiment, we found that the perceived aspect ratio was in general lower than the physical aspect ratio. However, in the second experiment, we found that the calculated ratios were close to veridical except for the most elongated city square. We conclude therefore that the outcome depends on how the measurements are performed. Furthermore, although indirect measurements are nearly veridical, the perceived aspect ratio is an underestimation of the physical aspect ratio when measured in a direct way. Moreover, the perceived aspect ratio also depends on the location of the observer. These results may be beneficial to the design of large open urban environments, and in particular to rectangular city squares.

Relative flattening between velvet and matte 3D shapes: evidence for similar shape-from-shading computations.

Among other cues, the visual system uses shading to infer the 3D shape of objects. The shading pattern depends on the illumination and reflectance properties (BRDF). In this study, we compared 3D shape perception between identical shapes with different BRDFs. The stimuli were photographed 3D printed random smooth shapes that were either painted matte gray or had a gray velvet layer. We used the gauge figure task (J. J. Koenderink, A. J. van Doorn, & A. M. L. Kappers, 1992) to quantify 3D shape perception. We found that the shape of velvet objects was systematically perceived to be flatter than the matte objects. Furthermore, observers' judgments were more similar for matte shapes than for velvet shapes. Lastly, we compared subjective with veridical reliefs and found large systematic differences: Both matte and velvet shapes were perceived more flat than the actual shape. The isophote pattern of a flattened Lambertian shape resembles the isophote pattern of an unflattened velvet shape. We argue that the visual system uses a similar shape-from-shading computation for matte and velvet objects that partly discounts material properties.

Depth in box spaces.

Human observers adjust the frontal view of a wireframe box on a computer screen so as to look equally deep and wide, so that in the intended setting the box looks like a cube. Perspective cues are limited to the size-distance effect, since all angles are fixed. Both the size on the screen, and the viewing distance from the observer to the screen were varied. All observers prefer a template view of a cube over a veridical rendering, independent of picture size and viewing distance. If the rendering shows greater or lesser foreshortening than the template, the box appears like a long corridor or a shallow slab, that is, like a 'deformed' cube. Thus observers ignore 'veridicality'. This does not fit an 'inverse optics' model. We discuss a model of 'vision as optical user interface'.

Probing pictorial relief: from experimental design to surface reconstruction.

The perception of pictorial surfaces has been studied quantitatively for more than 20 years. During this time, the "gauge figure method" has been shown to be a fast and intuitive method to quantify pictorial relief. In this method, observers have to adjust the attitude of a gauge figure such that it appears to lie flat on a surface in pictorial space. Although the method has received substantial attention in the literature and has become increasingly popular, a clear, step-by-step description has not been published yet. In this article, a detailed description of the method is provided: stimulus and sample preparation, performing the experiment, and reconstructing a 3-D surface from the experimental data. Furthermore, software (written in PsychToolbox) based on this description is provided in an online supplement. This report serves three purposes: First, it facilitates experimenters who want to use the gauge figure task but have been unable to design it, due to the lack of information in the literature. Second, the detailed description can facilitate the design of software for various other platforms, possibly Web-based. Third, the method described in this article is extended to objects with holes and inner contours. This class of objects have not yet been investigated with the gauge figure task.

Cast shadows in wide perspective.

We investigated the apparent spatial layout of cast shadows up to very wide fields of view. We presented up to 130 degrees wide images in which two 'flat poles' were standing on a green lawn under a cloudless blue sky on a sunny day. The poles threw sharp cast shadows on the green, of which one was fixed. The observer's task was to adjust the azimuth of the shadow of the other pole such that it fitted the scene. The source elevation was kept constant. The two cast shadows are, of course, parallel in physical space, but generically not in the picture plane because of the wide perspective. We found that observers made huge systematic errors, indicating that, generically, they fail to account for these perspective effects. The systematic deviations could be well described by a weighted linear combination of the directions in the picture plane and in the physical space, with weights that depended on the positions of, and distance between, the poles.