PsyPhy: A Psychophysics Driven Evaluation Framework for Visual Recognition.

By providing substantial amounts of data and standardized evaluation protocols, datasets in computer vision have helped fuel advances across all areas of visual recognition. But even in light of breakthrough results on recent benchmarks, it is still fair to ask if our recognition algorithms are doing as well as we think they are. The vision sciences at large make use of a very different evaluation regime known as Visual Psychophysics to study visual perception. Psychophysics is the quantitative examination of the relationships between controlled stimuli and the behavioral responses they elicit in experimental test subjects. Instead of using summary statistics to gauge performance, psychophysics directs us to construct item-response curves made up of individual stimulus responses to find perceptual thresholds, thus allowing one to identify the exact point at which a subject can no longer reliably recognize the stimulus class. In this article, we introduce a comprehensive evaluation framework for visual recognition models that is underpinned by this methodology. Over millions of procedurally rendered 3D scenes and 2D images, we compare the performance of well-known convolutional neural networks. Our results bring into question recent claims of human-like performance, and provide a path forward for correcting newly surfaced algorithmic deficiencies.

Individual Aesthetic Preferences for Faces Are Shaped Mostly by Environments, Not Genes.

Although certain characteristics of human faces are broadly considered more attractive (e.g., symmetry, averageness), people also routinely disagree with each other on the relative attractiveness of faces. That is, to some significant degree, beauty is in the "eye of the beholder." Here, we investigate the origins of these individual differences in face preferences using a twin design, allowing us to estimate the relative contributions of genetic and environmental variation to individual face attractiveness judgments or face preferences. We first show that individual face preferences (IP) can be reliably measured and are readily dissociable from other types of attractiveness judgments (e.g., judgments of scenes, objects). Next, we show that individual face preferences result primarily from environments that are unique to each individual. This is in striking contrast to individual differences in face identity recognition, which result primarily from variations in genes [1]. We thus complete an etiological double dissociation between two core domains of social perception (judgments of identity versus attractiveness) within the same visual stimulus (the face). At the same time, we provide an example, rare in behavioral genetics, of a reliably and objectively measured behavioral characteristic where variations are shaped mostly by the environment. The large impact of experience on individual face preferences provides a novel window into the evolution and architecture of the social brain, while lending new empirical support to the long-standing claim that environments shape individual notions of what is attractive.

Perceptual Annotation: Measuring Human Vision to Improve Computer Vision.

For many problems in computer vision, human learners are considerably better than machines. Humans possess highly accurate internal recognition and learning mechanisms that are not yet understood, and they frequently have access to more extensive training data through a lifetime of unbiased experience with the visual world. We propose to use visual psychophysics to directly leverage the abilities of human subjects to build better machine learning systems. First, we use an advanced online psychometric testing platform to make new kinds of annotation data available for learning. Second, we develop a technique for harnessing these new kinds of information-"perceptual annotations"-for support vector machines. A key intuition for this approach is that while it may remain infeasible to dramatically increase the amount of data and high-quality labels available for the training of a given system, measuring the exemplar-by-exemplar difficulty and pattern of errors of human annotators can provide important information for regularizing the solution of the system at hand. A case study for the problem face detection demonstrates that this approach yields state-of-the-art results on the challenging FDDB data set.