Probing the Link Between Vision and Language in Material Perception Using Psychophysics and Unsupervised Learning.

We can visually discriminate and recognize a wide range of materials. Meanwhile, we use language to express our subjective understanding of visual input and communicate relevant information about the materials. Here, we investigate the relationship between visual judgment and language expression in material perception to understand how visual features relate to semantic representations. We use deep generative networks to construct an expandable image space to systematically create materials of well-defined and ambiguous categories. From such a space, we sampled diverse stimuli and compared the representations of materials from two behavioral tasks: visual material similarity judgments and free-form verbal descriptions. Our findings reveal a moderate but significant correlation between vision and language on a categorical level. However, analyzing the representations with an unsupervised alignment method, we discover structural differences that arise at the image-to-image level, especially among materials morphed between known categories. Moreover, visual judgments exhibit more individual differences compared to verbal descriptions. Our results show that while verbal descriptions capture material qualities on the coarse level, they may not fully convey the visual features that characterize the material's optical properties. Analyzing the image representation of materials obtained from various pre-trained data-rich deep neural networks, we find that human visual judgments' similarity structures align more closely with those of the text-guided visual-semantic model than purely vision-based models. Our findings suggest that while semantic representations facilitate material categorization, non-semantic visual features also play a significant role in discriminating materials at a finer level. This work illustrates the need to consider the vision-language relationship in building a comprehensive model for material perception. Moreover, we propose a novel framework for quantitatively evaluating the alignment and misalignment between representations from different modalities, leveraging information from human behaviors and computational models.

Unsupervised learning reveals interpretable latent representations for translucency perception.

Humans constantly assess the appearance of materials to plan actions, such as stepping on icy roads without slipping. Visual inference of materials is important but challenging because a given material can appear dramatically different in various scenes. This problem especially stands out for translucent materials, whose appearance strongly depends on lighting, geometry, and viewpoint. Despite this, humans can still distinguish between different materials, and it remains unsolved how to systematically discover visual features pertinent to material inference from natural images. Here, we develop an unsupervised style-based image generation model to identify perceptually relevant dimensions for translucent material appearances from photographs. We find our model, with its layer-wise latent representation, can synthesize images of diverse and realistic materials. Importantly, without supervision, human-understandable scene attributes, including the object's shape, material, and body color, spontaneously emerge in the model's layer-wise latent space in a scale-specific manner. By embedding an image into the learned latent space, we can manipulate specific layers' latent code to modify the appearance of the object in the image. Specifically, we find that manipulation on the early-layers (coarse spatial scale) transforms the object's shape, while manipulation on the later-layers (fine spatial scale) modifies its body color. The middle-layers of the latent space selectively encode translucency features and manipulation of such layers coherently modifies the translucency appearance, without changing the object's shape or body color. Moreover, we find the middle-layers of the latent space can successfully predict human translucency ratings, suggesting that translucent impressions are established in mid-to-low spatial scale features. This layer-wise latent representation allows us to systematically discover perceptually relevant image features for human translucency perception. Together, our findings reveal that learning the scale-specific statistical structure of natural images might be crucial for humans to efficiently represent material properties across contexts.

An Open-Source Cognitive Test Battery to Assess Human Attention and Memory.

Cognitive test batteries are widely used in diverse research fields, such as cognitive training, cognitive disorder assessment, or brain mechanism understanding. Although they need flexibility according to their usage objectives, most test batteries are not available as open-source software and are not be tuned by researchers in detail. The present study introduces an open-source cognitive test battery to assess attention and memory, using a javascript library, p5.js. Because of the ubiquitous nature of dynamic attention in our daily lives, it is crucial to have tools for its assessment or training. For that purpose, our test battery includes seven cognitive tasks (multiple-objects tracking, enumeration, go/no-go, load-induced blindness, task-switching, working memory, and memorability), common in cognitive science literature. By using the test battery, we conducted an online experiment to collect the benchmark data. Results conducted on 2 separate days showed the high cross-day reliability. Specifically, the task performance did not largely change with the different days. Besides, our test battery captures diverse individual differences and can evaluate them based on the cognitive factors extracted from latent factor analysis. Since we share our source code as open-source software, users can expand and manipulate experimental conditions flexibly. Our test battery is also flexible in terms of the experimental environment, i.e., it is possible to experiment either online or in a laboratory environment.

Crystal or jelly? Effect of color on the perception of translucent materials with photographs of real-world objects.

Translucent materials are ubiquitous in nature (e.g. teeth, food, and wax), but our understanding of translucency perception is limited. Previous work in translucency perception has mainly used monochromatic rendered images as stimuli, which are restricted by their diversity and realism. Here, we measure translucency perception with photographs of real-world objects. Specifically, we use three behavior tasks: binary classification of "translucent" versus "opaque," semantic attribute rating of perceptual qualities (see-throughness, glossiness, softness, glow, and density), and material categorization. Two different groups of observers finish the three tasks with color or grayscale images. We find that observers' agreements depend on the physical material properties of the objects such that translucent materials generate more interobserver disagreements. Further, there are more disagreements among observers in the grayscale condition in comparison to that in the color condition. We also discover that converting images to grayscale substantially affects the distributions of attribute ratings for some images. Furthermore, ratings of see-throughness, glossiness, and glow could predict individual observers' binary classification of images in both grayscale and color conditions. Last, converting images to grayscale alters the perceived material categories for some images such that observers tend to misjudge images of food as non-food and vice versa. Our result demonstrates that color is informative about material property estimation and recognition. Meanwhile, our analysis shows that mid-level semantic estimation of material attributes might be closely related to high-level material recognition. We also discuss individual differences in our results and highlight the importance of such consideration in material perception.

Visual discrimination of optical material properties: A large-scale study.

Complex visual processing involved in perceiving the object materials can be better elucidated by taking a variety of research approaches. Sharing stimulus and response data is an effective strategy to make the results of different studies directly comparable and can assist researchers with different backgrounds to jump into the field. Here, we constructed a database containing several sets of material images annotated with visual discrimination performance. We created the material images using physically based computer graphics techniques and conducted psychophysical experiments with them in both laboratory and crowdsourcing settings. The observer's task was to discriminate materials on one of six dimensions (gloss contrast, gloss distinctness of image, translucent vs. opaque, metal vs. plastic, metal vs. glass, and glossy vs. painted). The illumination consistency and object geometry were also varied. We used a nonverbal procedure (an oddity task) applicable for diverse use cases, such as cross-cultural, cross-species, clinical, or developmental studies. Results showed that the material discrimination depended on the illuminations and geometries and that the ability to discriminate the spatial consistency of specular highlights in glossiness perception showed larger individual differences than in other tasks. In addition, analysis of visual features showed that the parameters of higher order color texture statistics can partially, but not completely, explain task performance. The results obtained through crowdsourcing were highly correlated with those obtained in the laboratory, suggesting that our database can be used even when the experimental conditions are not strictly controlled in the laboratory. Several projects using our dataset are underway.

Discounting mechanism underlies extinction illusion.

Humans can perceive a coherent visual scene despite a low spatial resolution in peripheral vision. How does the visual system determine whether an object exists in the periphery? We addressed this question by focusing on the extinction illusion in which a disk becomes subjectively invisible when presented at the intersection of grids. We hypothesized that the disk would go unnoticed when the stimuli with and without the disk produced the same strength of visual signals. The visual system would miss the disk by confounding the target signals with the intersection signals that should be discounted. Computational analysis revealed that the energy ratio between the stimuli with and without the disk decreased with stimulus eccentricity and such energy ratio could successfully explain the observer's d' to detect the disk. These results indicate that the discounting mechanism relying on stimulus energy determines the awareness toward a peripheral object.

A Computational Mechanism for Seeing Dynamic Deformation.

Human observers perceptually discriminate the dynamic deformation of materials in the real world. However, the psychophysical and neural mechanisms responsible for the perception of dynamic deformation have not been fully elucidated. By using a deforming bar as the stimulus, we showed that the spatial frequency of deformation was a critical determinant of deformation perception. Simulating the response of direction-selective units (i.e., MT pattern motion cells) to stimuli, we found that the perception of dynamic deformation was well explained by assuming a higher-order mechanism monitoring the spatial pattern of direction responses. Our model with the higher-order mechanism also successfully explained the appearance of a visual illusion wherein a static bar apparently deforms against a tilted drifting grating. In particular, it was the lower spatial frequencies in this pattern that strongly contributed to the deformation perception. Finally, by manipulating the luminance of the static bar, we observed that the mechanism for the illusory deformation was more sensitive to luminance than contrast cues.

Motion Perception: From Detection to Interpretation.

Visual motion processing can be conceptually divided into two levels. In the lower level, local motion signals are detected by spatiotemporal-frequency-selective sensors and then integrated into a motion vector flow. Although the model based on V1-MT physiology provides a good computational framework for this level of processing, it needs to be updated to fully explain psychophysical findings about motion perception, including complex motion signal interactions in the spatiotemporal-frequency and space domains. In the higher level, the velocity map is interpreted. Although there are many motion interpretation processes, we highlight the recent progress in research on the perception of material (e.g., specular reflection, liquid viscosity) and on animacy perception. We then consider possible linking mechanisms of the two levels and propose intrinsic flow decomposition as the key problem. To provide insights into computational mechanisms of motion perception, in addition to psychophysics and neurosciences, we review machine vision studies seeking to solve similar problems.

Material and shape perception based on two types of intensity gradient information.

Visual estimation of the material and shape of an object from a single image includes a hard ill-posed computational problem. However, in our daily life we feel we can estimate both reasonably well. The neural computation underlying this ability remains poorly understood. Here we propose that the human visual system uses different aspects of object images to separately estimate the contributions of the material and shape. Specifically, material perception relies mainly on the intensity gradient magnitude information, while shape perception relies mainly on the intensity gradient order information. A clue to this hypothesis was provided by the observation that luminance-histogram manipulation, which changes luminance gradient magnitudes but not the luminance-order map, effectively alters the material appearance but not the shape of an object. In agreement with this observation, we found that the simulated physical material changes do not significantly affect the intensity order information. A series of psychophysical experiments further indicate that human surface shape perception is robust against intensity manipulations provided they do not disturb the intensity order information. In addition, we show that the two types of gradient information can be utilized for the discrimination of albedo changes from highlights. These findings suggest that the visual system relies on these diagnostic image features to estimate physical properties in a distal world.

Visual wetness perception based on image color statistics.

Color vision provides humans and animals with the abilities to discriminate colors based on the wavelength composition of light and to determine the location and identity of objects of interest in cluttered scenes (e.g., ripe fruit among foliage). However, we argue that color vision can inform us about much more than color alone. Since a trichromatic image carries more information about the optical properties of a scene than a monochromatic image does, color can help us recognize complex material qualities. Here we show that human vision uses color statistics of an image for the perception of an ecologically important surface condition (i.e., wetness). Psychophysical experiments showed that overall enhancement of chromatic saturation, combined with a luminance tone change that increases the darkness and glossiness of the image, tended to make dry scenes look wetter. Theoretical analysis along with image analysis of real objects indicated that our image transformation, which we call the wetness enhancing transformation, is consistent with actual optical changes produced by surface wetting. Furthermore, we found that the wetness enhancing transformation operator was more effective for the images with many colors (large hue entropy) than for those with few colors (small hue entropy). The hue entropy may be used to separate surface wetness from other surface states having similar optical properties. While surface wetness and surface color might seem to be independent, there are higher order color statistics that can influence wetness judgments, in accord with the ecological statistics. The present findings indicate that the visual system uses color image statistics in an elegant way to help estimate the complex physical status of a scene.

Human perception of subresolution fineness of dense textures based on image intensity statistics.

We are surrounded by many textures with fine dense structures, such as human hair and fabrics, whose individual elements are often finer than the spatial resolution limit of the visual system or that of a digitized image. Here we show that human observers have an ability to visually estimate subresolution fineness of those textures. We carried out a psychophysical experiment to show that observers could correctly discriminate differences in the fineness of hair-like dense line textures even when the thinnest line element was much finer than the resolution limit of the eye or that of the display. The physical image analysis of the textures, along with a theoretical analysis based on the central limit theorem, indicates that as the fineness of texture increases and the number of texture elements per resolvable unit increases, the intensity contrast of the texture decreases and the intensity histogram approaches a Gaussian shape. Subsequent psychophysical experiments showed that these image features indeed play critical roles in fineness perception; i.e., lowering the contrast made artificial and natural textures look finer, and this effect was most evident for textures with unimodal Gaussian-like intensity distributions. These findings indicate that the human visual system is able to estimate subresolution texture fineness on the basis of diagnostic image features correlated with subresolution fineness, such as the intensity contrast and the shape of the intensity histogram.

Stain on texture: perception of a dark spot having a blurred edge on textured backgrounds.

When distinguishing illumination from reflectance edges, both edge blurriness and textural continuity across an edge are generally used as cues to promote the illumination-edge interpretation. However, when these cues were combined, i.e., when a dark spot having a blurred edge was placed on textured backgrounds, we unexpectedly found that the spot appears stained or painted rather than differently illuminated ("stain on texture" phenomenon). This phenomenon suggests a disruptive interaction between the visual processing of blurred edges and background texture. Our experiments showed that middle spatial-frequency components of background texture play a critical role in producing this interaction. Specifically, when a textured background had relatively stronger energy in middle spatial-frequency bands, the dark spot having a blurred edge on the textured background was perceived as differing in reflectance. The findings are discussed in view of multiple levels of visual processes: one mainly concerns low-level features such as spatial-frequency components and another is a higher-level process that takes into account the likelihood of spatial configurations in natural scenes, such as "spot shadow" in which the shadow is isolated and the shadow caster is out of sight.

Spatial organization affects lightness perception on articulated surrounds.

The articulation effect refers to a change in lightness contrast induced by adding small patches of different luminances to a uniform background surrounding a target in a lightness contrast display. This study investigated how local luminance signals are integrated to generate the articulation effect. We asked whether spatial organization due to perceptual grouping can influence the articulation effect even when the spatially averaged luminance of the surrounds is held constant. Grouping factors used were common-fate motion (Experiment 1), similarity of orientation (Experiment 2), and synchrony (Experiment 3). Results of all experiments consistently showed that the articulation effect was larger when the target was strongly grouped with the articulation patches. These findings provide converging evidence for the effects of spatial organization on the articulation effect. Moreover, they suggest that lightness computation underlying the articulation effect depends on a middle-level representation in which perceptual organization is at least partially established. The changes in lightness perception due to spatial organization could be accounted for by the double-anchoring theory of lightness (Bressan, 2006b).

Local computation of lightness on articulated surrounds.

Lightness of a grey target on a uniform light (or dark) surround changes by articulating the surround (articulation effect). To elucidate the processing of lightness underlying the articulation effect, the present study introduced transparency over a dark surround and investigated its effects on lightness of the target. The transparency was produced by adding a contiguous external field to the dark surround while keeping local stimulus configuration constant. Results showed that the target lightness did not change on the articulated surround when a dark transparent filter was perceived over the target, although it did on the uniform surround. These results suggest that image decomposition into a transparent filter and an underlying surface does not necessarily change lightness of the surface if the surface is articulated. Moreover, the present study revealed that articulating the surround does not always enhance lightness contrast; it can reduce the contrast effect when the target luminance is not the highest within the surround. These findings are consistent with the theoretical view that lightness perception on articulated surfaces is determined locally within a spatially limited region, and they also place a constraint on how the luminance distribution within the limited region is scaled.