Peripheral material perception.

Humans can rapidly identify materials, such as wood or leather, even within a complex visual scene. Given a single image, one can easily identify the underlying "stuff," even though a given material can have highly variable appearance; fabric comes in unlimited variations of shape, pattern, color, and smoothness, yet we have little trouble categorizing it as fabric. What visual cues do we use to determine material identity? Prior research suggests that simple "texture" features of an image, such as the power spectrum, capture information about material properties and identity. Few studies, however, have tested richer and biologically motivated models of texture. We compared baseline material classification performance to performance with synthetic textures generated from the Portilla-Simoncelli model and several common image degradations. The textures retain statistical information but are otherwise random. We found that performance with textures and most degradations was well below baseline, suggesting insufficient information to support foveal material perception. Interestingly, modern research suggests that peripheral vision might use a statistical, texture-like representation. In a second set of experiments, we found that peripheral performance is more closely predicted by texture and other image degradations. These findings delineate the nature of peripheral material classification.

Vagueness and volume: Testing the perception of depth in images with linear, sharp, or blurred contours.

In European painting, a transition took place where artists started to consciously introduce blurred or soft contours in their works. There may have been several reasons for this. One suggestion in art historical literature is that this may have been done to create a stronger sense of volume in the depicted figures or objects. Here we describe four experiments in which we tried to test whether soft or blurred contours do indeed enhance a sense volume or depth. In the first three experiments, we found that, for both paintings and abstract shapes, three dimensionality was actually decreased instead of increased for blurred (and line) contours, in comparison with sharp contours. In the last experiment, we controlled for the position of the blur (on the lit or dark side) and found that blur on the lit side evoked a stronger impression of three dimensionality. Overall, the experiments robustly show that an art historical conjecture that a blurred contour increases three dimensionality is not granted. Because the blurred contours can be found in many established art works such as from Leonardo and Vermeer, there must be other rationales behind this use than the creation of a stronger sense of volume or depth.

Time-of-day perception in paintings.

The spectral shape, irradiance, direction, and diffuseness of daylight vary regularly throughout the day. The variations in illumination and their effect on the light reflected from objects may in turn provide visual information as to the time of day. We suggest that artists' color choices for paintings of outdoor scenes might convey this information and that therefore the time of day might be decoded from the colors of paintings. Here we investigate whether human viewers' estimates of the depicted time of day in paintings correlate with their image statistics, specifically chromaticity and luminance variations. We tested time-of-day perception in 17th- to 20th-century Western European paintings via two online rating experiments. In Experiment 1, viewers' ratings from seven time choices varied significantly and largely consistently across paintings but with some ambiguity between morning and evening depictions. Analysis of the relationship between image statistics and ratings revealed correlations with the perceived time of day: higher "morningness" ratings associated with higher brightness, contrast, and saturation and darker yellow/brighter blue hues; "eveningness" with lower brightness, contrast, and saturation and darker blue/brighter yellow hues. Multiple linear regressions of extracted principal components yielded a predictive model that explained 76% of the variance in time-of-day perception. In Experiment 2, viewers rated paintings as morning or evening only; rating distributions differed significantly across paintings, and image statistics predicted people's perceptions. These results suggest that artists used different color palettes and patterns to depict different times of day, and the human visual system holds consistent assumptions about the variation of natural light depicted in paintings.

Quantifying the spatial, temporal, angular and spectral structure of effective daylight in perceptually meaningful ways.

We present a method to capture the 7-dimensional light field structure, and translate it into perceptually-relevant information. Our spectral cubic illumination method quantifies objective correlates of perceptually relevant diffuse and directed light components, including their variations over time, space, in color and direction, and the environment's response to sky and sunlight. We applied it "in the wild", capturing how light on a sunny day differs between light and shadow, and how light varies over sunny and cloudy days. We discuss the added value of our method for capturing nuanced lighting effects on scene and object appearance, such as chromatic gradients.

Zooming in on style: Exploring style perception using details of paintings.

Most studies on the perception of style have used whole scenes/entire paintings; in our study, we isolated a single motif (an apple) to reduce or even eliminate the influence of composition, iconography, and other contextual information. In this article, we empirically address two fundamental questions of the existence (Experiment 1) and description (Experiment 2) of style. We chose 48 cut-outs of mostly Western European paintings (15th to 21st century) that showed apples. In Experiment 1, 415 unique participants completed online triplet similarity tasks. Multidimensional scaling (MDS) reached a nonrandom three-dimensional (3D) embedding, showing that participants are able to judge stylistic differences in a systematic way. We also found a strong correlation between creation year and embedding, both a linear correlation with Dimension 2, and a rotational correlation in the first two dimensions. To interpret the embedding further, in Experiment 2, we fitted three color statistics and nine attribute ratings (glossiness, three-dimensionality, convincingness, brush coarseness, etc.) to the 3D perceptual style space. Results showed that Dimension 1 is associated with spatial attributes (Smoothness, Brushstroke coarseness) and Convincingness, Dimension 2 is related to Hue, and Dimension 3 is related to Chroma. The results suggest that texture and color are two important variables for style perception. By isolating the motifs, we could exclude higher levels of information such as composition and context. Interestingly, the results reinforce previous findings using whole scenes, suggesting that style can already be perceived in sometimes very small fragments of paintings.

Soft like velvet and shiny like satin: Perceptual material signatures of fabrics depicted in 17th century paintings.

Dutch 17th century painters were masters in depicting materials and their properties in a convincing way. Here, we studied the perception of the material signatures and key image features of different depicted fabrics, like satin and velvet. We also tested whether the perception of fabrics depicted in paintings related to local or global cues, by cropping the stimuli. In Experiment 1, roughness, warmth, softness, heaviness, hairiness, and shininess were rated for the stimuli shown either full figure or cropped. In the full figure, all attributes except shininess were rated higher for velvet, whereas shininess was rated higher for satin. This distinction was less clear in the cropped condition, and some properties were perceived significantly different between the two conditions. In Experiment 2 we tested whether this difference was due to the choice of the cropped area. On the basis of the results of Experiment 1, shininess and softness were rated for multiple crops from each fabric. Most crops from the same fabric differed significantly in shininess, but not in softness perception. Perceived shininess correlated positively with the mean luminance of the crops and the highlights' coverage. Experiment 1 showed that painted velvet and satin triggered distinct perceptions, indicative of robust material signatures of the two fabrics. The results of Experiment 2 suggest that the presence of local image cues affects the perception of optical properties like shininess, but not mechanical properties such as softness.

If painters give you lemons, squeeze the knowledge out of them. A study on the visual perception of the translucent and juicy appearance of citrus fruits in paintings.

Citrus fruits are characterized by a juicy and translucent interior, important properties that drive material recognition and food acceptance. Yet, a thorough understanding of their visual perception is still missing. Using citrus fruits depicted in 17th-century paintings as stimuli, we ran three rating experiments. In Experiment 1, participants rated the perceived similarity in translucency or juiciness of the fruits. In Experiment 2, different groups of participants rated one image feature from a list obtained in a preliminary experiment. In Experiment 3, translucency and juiciness were rated. We constructed two-dimensional perceptual spaces for both material properties and fitted the ratings of the image features into the spaces to interpret them. "Highlights," "peeled side," "bumpiness," and "color saturation" fit the juiciness space best and were high for the highly juicy stimuli. "Peeled side," "intensity of light gradient," "highlights," and "color saturation" were the most salient features of the translucency space, being high for the highly translucent stimuli. The same image features were also indicated in a 17th-century painting manual for material depiction (Beurs, 1692; Beurs, in press). Altogether, we disclosed the expertise of painters with regard to material perception by identifying the image features that trigger a visual impression of juiciness and translucency in citrus fruits.

Painterly depiction of material properties.

Painters are masters of depiction and have learned to evoke a clear perception of materials and material attributes in a natural, three-dimensional setting, with complex lighting conditions. Furthermore, painters are not constrained by reality, meaning that they could paint materials without exactly following the laws of nature, while still evoking the perception of materials. Paintings have to our knowledge not been studied on a big scale from a material perception perspective. In this article, we studied the perception of painted materials and their attributes by using human annotations to find instances of 15 materials, such as wood, stone, fabric, etc. Participants made perceptual judgments about 30 unique segments of these materials for 10 material attributes, such as glossiness, roughness, hardness, etc. We found that participants were able to perform this task well while being highly consistent. Participants, however, did not consistently agree with each other, and the measure of consistency depended on the material attribute being perceived. Additionally, we found that material perception appears to function independently of the medium of depiction-the results of our principal component analysis agreed well with findings in former studies for photographs and computer renderings.

Understanding gloss perception through the lens of art: Combining perception, image analysis, and painting recipes of 17th century painted grapes.

To understand the key image features that we use to infer the glossiness of materials, we analyzed the pictorial shortcuts used by 17th century painters to imitate the optical phenomenon of specular reflections when depicting grapes. Gloss perception of painted grapes was determined via a rating experiment. We computed the contrast, blurriness, and coverage of the grapes' highlights in the paintings' images, inspired by Marlow and Anderson (2013). The highlights were manually segmented from the images, and next the features contrast, coverage, and blurriness were semiautomatically quantified using self-defined algorithms. Multiple linear regressions of contrast and blurriness resulted in a predictive model that could explain 69% of the variance in gloss perception. No effect was found for coverage. These findings are in agreement with the instructions to render glossiness of grapes contained in a 17th century painting manual (Beurs, 1692/in press), suggesting that painting practice embeds knowledge about key image features that trigger specific material percepts.

Visual communication of how fabrics feel.

Although product photos and movies are abundantly present in online shopping environments, little is known about how much of the real product experience they capture. While previous studies have shown that movies or interactive imagery give users the impression that these communication forms are more effective, there are no studies addressing this issue quantitatively. We used nine different samples of jeans, because in general fabrics represent a large and interesting product category and specifically because jeans can visually be rather similar while haptically be rather different. In the first experiment we let observers match a haptic stimulus to a visual representation and found that movies were more informative about how objects would feel than photos. In a second experiment we wanted to confirm this finding by using a different experimental paradigm that we deemed a better general paradigm for future studies on this topic: correlations of pairwise similarity ratings. However, the beneficial effect of the movies was absent when using this new paradigm. In the third experiment we investigated this issue by letting people visually observe other people in making haptic similarity judgments. Here, we did find a significant correlation between haptic and visual data. Together, the three experiments suggest that there is a small but significant effect of movies over photos (Experiment 1) but at the same time a significant difference between visual representations and visually perceiving products in reality (Experiments 2 and 3). This finding suggests a substantial theoretical potential for decreasing the gap between virtual and real product presentation.

Highlight shapes and perception of gloss for real and photographed objects.

Gloss perception strongly depends on the three-dimensional shape and the illumination of the object under consideration. In this study we investigated the influence of the spatial structure of the illumination on gloss perception. A diffuse light box in combination with differently shaped masks was used to produce a set of six simple and complex highlight shapes. The geometry of the simple highlight shapes was inspired by conventional artistic practice (e.g., ring flash for photography, window shape for painting and disk or square for cartoons). In the box we placed spherical stimuli that were painted in six degrees of glossiness. This resulted in a stimulus set of six highlight shapes and six gloss levels, a total of 36 stimuli. We performed three experiments of which two took place using digital photographs on a computer monitor and one with the real spheres in the light box. The observers had to perform a comparison task in which they chose which of two stimuli was glossiest and a rating task in which they rated the glossiness. The results show that, perhaps surprisingly, more complex highlight shapes were perceived to produce a less glossy appearance than simple highlight shapes such as a disk or square. These findings were confirmed for both viewing conditions, on a computer display and in a real setting. The results show that variations in the spatial structure of "rather simple" illumination of the "extended source" type highlight influences perceived glossiness.

Material perception across different media-comparing perceived attributes in oil paintings and engravings.

We investigated the influence of the medium on the perception of depicted objects and materials. Oil paintings and their reproductions in engravings were chosen because they are vastly distinctive media while having completely identical content. A total of 15 pairs were collected, consisting of 88 fragments depicting different materials, including fabric, skin, wood and metal. Besides the original condition, we created three manipulations to understand the effect of colour (a greyscale version) and contrast (equalised histograms towards both painting and engraving). We performed rating experiments on five attributes: three-dimensionality, glossiness, convincingness, smoothness and softness. An average of 25 participants finished each of the 20 online experimental sessions (five attributes X four conditions). Besides clear correlations between the two media, the differences mainly show in their means (different levels of perceived attributes) and standard deviations (perceived range). In most sessions, paintings depict a wider range than engravings. In addition, it was the histogram equalisation (global contrast) that made the most impact on perceived attributes, rather than colour removal. This suggests that engravers compensated for the lack of colour by exploiting the possibilities of local contrast.

Relative flattening between velvet and matte 3D shapes: evidence for similar shape-from-shading computations.

Among other cues, the visual system uses shading to infer the 3D shape of objects. The shading pattern depends on the illumination and reflectance properties (BRDF). In this study, we compared 3D shape perception between identical shapes with different BRDFs. The stimuli were photographed 3D printed random smooth shapes that were either painted matte gray or had a gray velvet layer. We used the gauge figure task (J. J. Koenderink, A. J. van Doorn, & A. M. L. Kappers, 1992) to quantify 3D shape perception. We found that the shape of velvet objects was systematically perceived to be flatter than the matte objects. Furthermore, observers' judgments were more similar for matte shapes than for velvet shapes. Lastly, we compared subjective with veridical reliefs and found large systematic differences: Both matte and velvet shapes were perceived more flat than the actual shape. The isophote pattern of a flattened Lambertian shape resembles the isophote pattern of an unflattened velvet shape. We argue that the visual system uses a similar shape-from-shading computation for matte and velvet objects that partly discounts material properties.

Probing pictorial relief: from experimental design to surface reconstruction.

The perception of pictorial surfaces has been studied quantitatively for more than 20 years. During this time, the "gauge figure method" has been shown to be a fast and intuitive method to quantify pictorial relief. In this method, observers have to adjust the attitude of a gauge figure such that it appears to lie flat on a surface in pictorial space. Although the method has received substantial attention in the literature and has become increasingly popular, a clear, step-by-step description has not been published yet. In this article, a detailed description of the method is provided: stimulus and sample preparation, performing the experiment, and reconstructing a 3-D surface from the experimental data. Furthermore, software (written in PsychToolbox) based on this description is provided in an online supplement. This report serves three purposes: First, it facilitates experimenters who want to use the gauge figure task but have been unable to design it, due to the lack of information in the literature. Second, the detailed description can facilitate the design of software for various other platforms, possibly Web-based. Third, the method described in this article is extended to objects with holes and inner contours. This class of objects have not yet been investigated with the gauge figure task.

Materials In Paintings (MIP): An interdisciplinary dataset for perception, art history, and computer vision.

In this paper, we capture and explore the painterly depictions of materials to enable the study of depiction and perception of materials through the artists' eye. We annotated a dataset of 19k paintings with 200k+ bounding boxes from which polygon segments were automatically extracted. Each bounding box was assigned a coarse material label (e.g., fabric) and half was also assigned a fine-grained label (e.g., velvety, silky). The dataset in its entirety is available for browsing and downloading at materialsinpaintings.tudelft.nl. We demonstrate the cross-disciplinary utility of our dataset by presenting novel findings across human perception, art history and, computer vision. Our experiments include a demonstration of how painters create convincing depictions using a stylized approach. We further provide an analysis of the spatial and probabilistic distributions of materials depicted in paintings, in which we for example show that strong patterns exists for material presence and location. Furthermore, we demonstrate how paintings could be used to build more robust computer vision classifiers by learning a more perceptually relevant feature representation. Additionally, we demonstrate that training classifiers on paintings could be used to uncover hidden perceptual cues by visualizing the features used by the classifiers. We conclude that our dataset of painterly material depictions is a rich source for gaining insights into the depiction and perception of materials across multiple disciplines and hope that the release of this dataset will drive multidisciplinary research.

Cross-modal visuo-haptic mental rotation: comparing objects between senses.

The simple experience of a coherent percept while looking and touching an object conceals an intriguing issue: different senses encode and compare information in different modality-specific reference frames. We addressed this problem in a cross-modal visuo-haptic mental rotation task. Two objects in various orientations were presented at the same spatial location, one visually and one haptically. Participants had to identify the objects as same or different. The relative angle between viewing direction and hand orientation was manipulated (Aligned versus Orthogonal). In an additional condition (Delay), a temporal delay was introduced between haptic and visual explorations while the viewing direction and the hand orientation were orthogonal to each other. Whereas the phase shift of the response time function was close to 0 degrees in the Aligned condition, we observed a consistent phase shift in the hand's direction in the Orthogonal condition. A phase shift, although reduced, was also found in the Delay condition. Counterintuitively, these results mean that seen and touched objects do not need to be physically aligned for optimal performance to occur. The present results suggest that the information about an object is acquired in separate visual and hand-centered reference frames, which directly influence each other and which combine in a time-dependent manner.

Illusory gloss on Lambertian surfaces.

It has recently been shown that an increase of the relief height of a glossy surface positively correlates with the perceived level of gloss (Y.-H. Ho, M. S. Landy, & L. T. Maloney, 2008). In the study presented here we investigated whether this relation could be explained by the finding that glossiness perception correlates with the skewness of the luminance histogram (I. Motoyoshi, S. Nishida, L. Sharan, & E. H. Adelson, 2007). First, we formally derived a general relation between the depth range of a Lambertian surface, the illumination direction and the associated image intensity transformation. From this intensity transformation we could numerically simulate the relation between relief stretch and the skewness statistic. This relation predicts that skewness increases with increasing surface depth. Furthermore, it predicts that the correlation between skewness and illumination can be either positive or negative, depending on the depth range. We experimentally tested whether changes in the depth range and illumination direction alter the appearance. We indeed find a convincingly strong illusory gloss effect on stretched Lambertian surfaces. However, the results could not be fully explained by the skewness hypothesis. We reinterpreted our results in the context of the bas-relief ambiguity (P. N. Belhumeur, D. J. Kriegman, & L. Yuille, 1999) and show that this model qualitatively predicts illusory highlights on locations that differ from actual specular highlight locations with increasing illumination direction.

Haptic curvature contrast in raised lines and solid shapes.

It is known that our senses are influenced by contrast effects and aftereffects. For haptic perception, the curvature aftereffect has been studied in depth but little is known about curvature contrast. In this study we let observers explore two shapes simultaneously. The shape felt by the index finger could either be flat or convexly curved. The curvature at the thumb was varied to quantify the curvature of a subjectively flat shape. We found that when the index finger was presented with a convex shape, a flat shape at the thumb was also perceived to be convex.The effect is rather strong, on average 20% of the contrasting curvature. The contrast effect was present for both raised line stimuli and solid shapes. Movement measurements revealed that the curvature of the path taken by the metacarpus (part of the hand that connects the fingers) was approximately the average of the path curvatures taken by the thumb and index finger. A failure to correct for the movement of the hand could explain the contrast effect.

Haptic perception disambiguates visual perception of 3D shape.

We studied the influence of haptics on visual perception of three-dimensional shape. Observers were shown pictures of an oblate spheroid in two different orientations. A gauge-figure task was used to measure their perception of the global shape. In the first two sessions only vision was used. The results showed that observers made large errors and interpreted the oblate spheroid as a sphere. They also misinterpreted the rotated oblate spheroid for a prolate spheroid. In two subsequent sessions observers were allowed to touch the stimulus while performing the task. The visual input remained unchanged: the observers were looking at the picture and could not see their hands. The results revealed that observers perceived a shape that was different from the vision-only sessions and closer to the veridical shape. Whereas, in general, vision is subject to ambiguities that arise from interpreting the retinal projection, our study shows that haptic input helps to disambiguate and reinterpret the visual input more veridically.

Look what I have felt: unidentified haptic line drawings are identified after sketching.

The difficulty that observers experience when trying to identify a raised line drawing by touch is still largely unexplained. In this article, we show that observers who are unable to haptically identify a raised line drawing are suddenly able to do so after they have sketched on paper what they have in their mind. We conducted three experiments: first of all we show that this effect is robust; in the second experiment, we show that identification-after-sketching is caused by visual inspection of the sketch, and not caused by feedback in general; and in the third we show that sketches which were identified by the observers who produced them, were also identified by completely naive viewers. These experiments demonstrate that during raised line drawing identification the mental capacities required to interpret the stimulus seem to be inadequate: although enough pictorial information was present to produce a sketch which could even be identified by naive viewers, the stimulus could not be identified by haptic and mental processing alone. Furthermore, we investigated whether increasing the haptic perceptual field by using two hands instead of one hand had an influence on identification performance. We did indeed find that using two hands significantly increased identification. We use both results to discuss the underlying mechanisms of haptic raised line drawing identification.