The visual experience dataset: Over 200 recorded hours of integrated eye movement, odometry, and egocentric video.

We introduce the Visual Experience Dataset (VEDB), a compilation of more than 240 hours of egocentric video combined with gaze- and head-tracking data that offer an unprecedented view of the visual world as experienced by human observers. The dataset consists of 717 sessions, recorded by 56 observers ranging from 7 to 46 years of age. This article outlines the data collection, processing, and labeling protocols undertaken to ensure a representative sample and discusses the potential sources of error or bias within the dataset. The VEDB's potential applications are vast, including improving gaze-tracking methodologies, assessing spatiotemporal image statistics, and refining deep neural networks for scene and activity recognition. The VEDB is accessible through established open science platforms and is intended to be a living dataset with plans for expansion and community contributions. It is released with an emphasis on ethical considerations, such as participant privacy and the mitigation of potential biases. By providing a dataset grounded in real-world experiences and accompanied by extensive metadata and supporting code, the authors invite the research community to use and contribute to the VEDB, facilitating a richer understanding of visual perception and behavior in naturalistic settings.

The role of texture summary statistics in material recognition from drawings and photographs.

Material depictions in artwork are useful tools for revealing image features that support material categorization. For example, artistic recipes for drawing specific materials make explicit the critical information leading to recognizable material properties (Di Cicco, Wjintjes, & Pont, 2020) and investigating the recognizability of material renderings as a function of their visual features supports conclusions about the vocabulary of material perception. Here, we examined how the recognition of materials from photographs and drawings was affected by the application of the Portilla-Simoncelli texture synthesis model. This manipulation allowed us to examine how categorization may be affected differently across materials and image formats when only summary statistic information about appearance was retained. Further, we compared human performance to the categorization accuracy obtained from a pretrained deep convolutional neural network to determine if observers' performance was reflected in the network. Although we found some similarities between human and network performance for photographic images, the results obtained from drawings differed substantially. Our results demonstrate that texture statistics play a variable role in material categorization across rendering formats and material categories and that the human perception of material drawings is not effectively captured by deep convolutional neural networks trained for object recognition.

The Fork-and-Knife Illusion.

We describe a transparency illusion that can be observed with an ordinary metal knife and fork. Placed in the correct configuration relative to the fork, the metal knife appears transparent, with some observers experiencing a bistable percept in which transparency alternates with reflective appearance. The effect is related to other illusory percepts that follow from careful placement of mirrored surfaces, but to our knowledge, it is unique in that the key feature of the illusion is how the mirrored surface (in this case, the knife) is perceived rather than how a mirror induces altered perception of other objects and surfaces. We describe conditions that do and do not affect the strength of the illusion and point out its connections to previously reported phenomena.

Neural sensitivity to natural image statistics changes during middle childhood.

Natural images have properties that adults' behavioral and neural responses are sensitive to, but the development of this sensitivity is not clear. Behaviorally, children acquire adult-like sensitivity to natural image statistics during middle childhood (Ellemberg et al., 2012), but infants exhibit sensitivity to deviations of natural image structure (Balas & Woods, 2014). We used event-related potentials (ERPs) to examine sensitivity to natural image statistics during childhood at distinct processing stages (the P1 and N1 components). We presented children (5-10 years old) and adults with natural images varying in positive/negative contrast, and natural/synthetic texture appearance to compare electrophysiological responses to images that did or did not violate natural statistics. We hypothesized that children would acquire sensitivity to these deviations late in middle childhood. Instead, we observed significant responses to unnatural contrast and texture statistics at the N1 in all age groups. At the P1, however, only young children exhibited sensitivity to contrast polarity. The latter effect suggests greater sensitivity earlier in development to some violations of natural image statistics. We discuss these results in terms of changing patterns of invariant texture processing during middle childhood and ongoing refinement of the representations supporting natural image perception.

Children's use of local and global visual features for material perception.

Adults can rapidly recognize material properties in natural images, and children's performance in material categorization tasks suggests that this ability develops slowly during childhood. In the current study, we further examined the information children use to recognize materials during development by asking how the use of local versus global visual features for material perception changes in middle childhood. We recruited adults and 5- to 10-year-old children for three experiments that required participants to distinguish between shape-matched images of real and artificial food. Accurate performance in this task requires participants to distinguish between a wide range of material properties characteristic of each category, thus testing material perception abilities broadly. In two tasks, we applied distinct methods of image scrambling (block scrambling and diffeomorphic scrambling) to parametrically disrupt global appearance while preserving features in small spatial neighborhoods. In the third task, we used image blurring to parametrically disrupt local feature visibility. Our key question was whether or not participant age affected performance differently when local versus global appearance was disrupted. We found that although image blur led to disproportionately poorer performance in young children, this effect was reduced or absent when diffeomorphic scrambling was used. We interpret this outcome as evidence that the ability to recruit large-scale visual features for material perception may develop slowly during middle childhood.

Children are sensitive to mutual information in intermediate-complexity face and non-face features.

Understanding developmental changes in children's use of specific visual information for recognizing object categories is essential for understanding how experience shapes recognition. Research on the development of face recognition has focused on children's use of low-level information (e.g. orientation sub-bands), or high-level information. In face categorization tasks, adults also exhibit sensitivity to intermediate complexity features that are diagnostic of the presence of a face. Do children also use intermediate complexity features for categorizing faces and objects, and, if so, how does their sensitivity to such features change during childhood? Intermediate-complexity features bridge the gap between low- and high-level processing: they have computational benefits for object detection and segmentation, and are known to drive neural responses in the ventral visual system. Here, we have investigated the developmental trajectory of children's sensitivity to diagnostic category information in intermediate-complexity features. We presented children (5-10 years old) and adults with image fragments of faces (Experiment 1) and cars (Experiment 2) varying in their mutual information, which quantifies a fragment's diagnosticity of a specific category. Our goal was to determine whether children were sensitive to the amount of mutual information in these fragments, and if their information usage is different from adults. We found that despite better overall categorization performance in adults, all children were sensitive to fragment diagnosticity in both categories, suggesting that intermediate representations of appearance are established early in childhood. Moreover, children's usage of mutual information was not limited to face fragments, suggesting the extracting intermediate-complexity features is a process that is not specific only to faces. We discuss the implications of our findings for developmental theories of face and object recognition.

The Effects of Blur and Inversion on the Recognition of Ambient Face Images.

When viewing unfamiliar faces that vary in expressions, angles, and image quality, observers make many recognition errors. Specifically, in unconstrained identity-sorting tasks, observers struggle to cope with variation across different images of the same person while succeeding at telling different people apart. The use of ambient face images in this simple card-sorting task reveals the magnitude of these face recognition errors and suggests a useful platform to reexamine the nature of face processing using naturalistic stimuli. In the present study, we chose to investigate the impact of two basic stimulus manipulations (image blur and face inversion) on identity sorting with ambient images. Although these manipulations are both known to affect face processing when well-controlled, frontally viewed face images are used, examining how they affect performance for ambient images is an important step toward linking the large body of research using controlled face images to more ecologically valid viewing conditions. Briefly, we observed a high cost of image blur regardless of blur magnitude, and a strong inversion effect that affected observers' sensitivity to extrapersonal variability but did not affect the number of unique identities they estimated were present in the set of images presented to them.

Neural sensitivity to natural texture statistics in infancy.

During infancy, vision becomes tuned to environmental statistics. For example, infant face recognition "narrows" in response to the frequency of face categories in the visual world, inducing out-group effects that disadvantage other-race, other-species, and other-age face recognition. There are many other low-level statistical regularities in visual experience that infants may also become tuned to during this period. In particular, natural scenes have lawful properties that adults and children are sensitive to. To what extent do infants become tuned to these regularities during the first year of life? In particular, do infants exhibit evidence of perceptual narrowing that excludes atypical images from fluent processing? We examined this question by measuring 6- and 9-month-old infants' event-related potentials (ERPs) to natural and artificial textures created by: (a) Disrupting local statistics via contrast negation, (b) Disrupting global statistics via parametric texture synthesis, or (c) both of these. We predicted that younger infants' would be sensitive to both manipulations of natural appearance, but that older infants might not distinguish between different kinds of atypical images. Instead, we found that sensitivity to synthetic appearance is only evident late in infancy. We discuss what these results imply for our understanding of visual statistical learning in infancy.

School-age children's neural sensitivity to horizontal orientation energy in faces.

Face processing mechanisms are tuned to specific low-level features including mid-range spatial frequencies and horizontal orientation energy. Behaviorally, adult observers are more effective at face recognition tasks when these information channels are available. Neural responses to face images also reflect these information biases: Face-sensitive ERP components respond preferentially to face images that contain horizontal orientation energy. How does neural tuning of face representations to horizontal information develop? Behavioral results show that this information bias increases over time such that younger children have a reduced bias favoring horizontally-filtered faces that increases with age. In the present study, we chose to investigate how neural sensitivity to these low-level features develops in the same age range, using ERP as a means of studying children and adults. Specifically, we examined how both face-sensitive ERP components (the P100 and N170) changed their responses to faces and non-faces as a function of age and orientation energy. Briefly, we found that the latency of the P100 and N170 component across age groups was consistent with the gradual emergence of a bias favoring horizontal orientation energy during middle childhood. The amplitude of the N170 component, however, exhibited a more complicated developmental profile that does not easily map onto previous behavioral results obtained from children in the same age ranges.

Children's use of visual summary statistics for material categorization.

Although adults' ability to recognize materials from complex natural images has been well characterized, we still know very little about the development of material perception. When do children exhibit adult-like abilities to categorize materials? What visual features do they use to do so as a function of age and material category? In the present study, we attempted to address both of these issues in two experiments that we administered to school-age children (5-10 years old) and adults. In both tasks, we asked our participants to categorize natural materials (metal, stone, water, and wood) using original images of these materials as well as synthetic images made with the Portilla-Simoncelli algorithm. By including synthetic images in our stimulus set, we were able to assess both how material categorization develops during childhood and how visual summary statistics are recruited for material perception across age groups. We observed that when asked to provide category labels for individual images (Experiment 1), young children were disproportionately bad at categorizing some materials after they were synthesized, suggesting material-specific changes in information use over the course of development. However, when asked to match real and synthetic images according to material category without labeling (Experiment 2), these effects were weakened. We conclude that while children have adult-like abilities to encode and compare images based on summary statistics, the mapping between summary statistics and category labels undergoes prolonged development during childhood.

Children (but not adults) judge similarity in own- and other-race faces by the color of their skin.

Both face shape and pigmentation are diagnostic cues for face identification and categorization. In particular, both shape and pigmentation contribute to observers' categorization of faces by race. Although many theoretical accounts of the behavioral other-race effect either explicitly or implicitly depend on differential use of visual information as a function of category expertise, there is little evidence that observers do in fact differentially rely on distinct visual cues for own- and other-race faces. In the current study, we examined how Asian and Caucasian children (4-6 years of age) and adults use three-dimensional shape and two-dimensional pigmentation to make similarity judgments of White, Black, and Asian faces. Children in this age range are capable of making category judgments about race but also are sufficiently plastic with regard to the behavioral other-race effect that it seems as though their representations of facial appearance across different categories are still emerging. Using a simple match-to-sample similarity task, we found that children tend to use pigmentation to judge facial similarity more than adults and also that own-group versus other-group category membership appears to influence how quickly children learn to use shape information more readily. Therefore, we suggest that children continue to adjust how different visual information is weighted during early and middle childhood and that experience with faces affects the speed at which adult-like weightings are established.

Orientation biases for facial emotion recognition during childhood and adulthood.

Facial emotion recognition develops slowly, with continuing changes in performance observable up to 10 years of age and beyond. In the current study, we chose to examine how the use of specific low-level visual features for emotion recognition may change during childhood. Adults exhibit information biases for face recognition; specific spatial frequency and orientation sub-bands make a larger contribution to recognition than others. This means that depending on the specific task (e.g., identification, emotion recognition), participants will perform worse when some features are removed from the original image and better when those features are included. One example of such an information bias for face recognition is the differential contribution of horizontal orientation energy relative to vertical orientation energy; adult participants are better able to recognize faces and categorize their emotional expressions when horizontal information is included than when only vertical information is included. Although several recent studies have demonstrated various ways in which horizontal orientation energy (and so-called "bar-codes" for face appearance) contribute to adult face processing, there have been as yet no studies describing how such a bias emerges developmentally that may offer insight into the mechanisms underlying the slow development of facial emotion recognition. In the current study, we compared children's (5- and 6-year-olds and 7- and 8-year-olds) and adults' performance in a simple emotion categorization task using orientation-filtered faces to determine the extent to which horizontal and vertical orientation energy contributed to recognition as a function of age. We found that although all three participant groups exhibited a clear bias favoring the use of horizontal orientation energy, the nature of this bias differed as a function of age. Specifically, 5- and 6-year-olds exhibited a disproportionate performance cost when vertical orientation energy was all that was available relative to when stimuli were limited to horizontal orientation energy. One feature of the development of facial emotion recognition, thus, appears to be the capability to use suboptimal or weakly diagnostic information to support recognition.

Domain-specific development of face memory but not face perception.

How does the remarkable human ability for face recognition arise over development? Competing theories have proposed either late maturity (beyond 10 years) or early maturity (before 5 years), but have not distinguished between perceptual and memory aspects of face recognition. Here, we demonstrate a perception-memory dissociation. We compare rate of development for (adult, human) faces versus other social stimuli (bodies), other discrete objects (cars), and other categories processed in discrete brain regions (scenes, bodies), from 5 years to adulthood. For perceptual discrimination, performance improved with age at the same rate for faces and all other categories, indicating no domain-specific development. In contrast, face memory increased more strongly than non-face memory, indicating domain-specific development. The results imply that each theory is partly true: the late maturity theory holds for face memory, and the early maturity theory for face perception.

Children's neural response to contrast-negated faces is species specific.

Face recognition abilities develop dramatically during the first year of life, but comparatively little is known about the nature of face-specific perceptual development during early childhood. Face-specific effects of image appearance on recognition, including face inversion and contrast negation, are a useful means of understanding the functional properties of face perception developmentally. Here, we examined the generality of the impact of contrast negation on face perception during early childhood using event-related potentials (ERPs). Specifically, we recorded continuous electroencephalography (EEG) while adult participants and children between 4 and 6 years of age viewed human and non-human primate faces presented in either positive or negative contrast. We examined both the P100 and N170 components to determine whether or not sensitivity to contrast polarity was evident in face-sensitive components during early childhood and also whether or not that sensitivity was specific to species category. We found evidence of a species-specific effect of contrast negation at the N170, suggesting that by early childhood some aspects of face-specific processing have been restricted to a relatively narrow class of face stimuli. However, this effect is of the opposite sign relative to adults, suggesting that there is continued maturation of face-specific processing during childhood.

Is visual information use during facial emotion recognition related to eating disorder symptoms in college-aged men and women? An experimental study.

BACKGROUND: Previous studies of emotion recognition abilities of people with eating disorders used accuracy to identify performance deficits for these individuals. The current study examined eating disorder symptom severity as a function of emotion categorization abilities, using a visual cognition paradigm that offers insights into how emotional faces may be categorized, as opposed to just how well these faces are categorized. METHODS: Undergraduate students (N = 87, 50 women, 34 men, 3 non-binary) completed the Bubbles task and a standard emotion categorization task, as well as a set of questionnaires assessing their eating disorder symptomology and comorbid disorders. We examined the relationship between visual information use (assessed via Bubbles) and eating disorder symptomology (EDDS) while controlling for anxiety (STAI), depression (BDI-II), alexithymia (TAS), and emotion regulation difficulties (DERS-sf). RESULTS: Overall visual information use (i.e. how well participants used facial features important for accurate emotion categorization) was not significantly related to eating disorder symptoms, despite producing interpretable patterns for each emotion category. Emotion categorization accuracy was also not related to eating disorder symptoms. CONCLUSIONS: Results from this study must be interpreted with caution, given the non-clinical sample. Future research may benefit from comparing visual information use in patients with an eating disorder and healthy controls, as well as employing designs focused on specific emotion categories, such as anger.

Men and women with severe eating disorder symptoms may find it harder to identify and describe emotions than people with less severe eating disorder symptoms. However, previous work makes it difficult to determine why emotion recognition deficits exist, and what underlying abilities or strategies are actually different due to a deficit. In addition to a typical emotion recognition task (emotion categorization), this study used the Bubbles task, which allowed us to determine which parts of an image are important for emotion recognition, and whether participants used these parts during the task. In 87 undergraduate students (47 female; 49 with clinically-significant eating disorder symptoms), there was no significant relationship between task performance and eating disorder symptom severity, before and after controlling for the relationship with other comorbid disorders. Our results imply that emotion recognition deficits are unlikely to be an important mechanism underlying eating disorder pathology in participants with a range of eating disorders symptoms.

Not the norm: Face likeness is not the same as similarity to familiar face prototypes.

Face images depicting the same individual can differ substantially from one another. Ecological variation in pose, expression, lighting, and other sources of appearance variability complicates the recognition and matching of unfamiliar faces, but acquired familiarity leads to the ability to cope with these challenges. Among the many ways that face of the same individual can vary, some images are judged to be better likenesses of familiar individuals than others. Simply put, these images look more like the individual under consideration than others. But what does it mean for an image to be a better likeness than another? Does likeness entail typicality, or is it something distinct from this? We examined the relationship between the likeness of face images and the similarity of those images to average images of target individuals using a set of famous faces selected for reciprocal familiarity/unfamiliarity across US and UK participants. We found that though likeness judgments are correlated with similarity-to-prototype judgments made by both familiar and unfamiliar participants, this correlation was smaller than the correlation between similarity judgments made by different participant groups. This implies that while familiarity weakens the relationship between likeness and similarity-to-prototype judgments, it does not change similarity-to-prototype judgments to the same degree.

Putting people in context: ERP responses to bodies in natural scenes.

The N190 is a body-sensitive ERP component that responds to images of human bodies in different poses. In natural settings, bodies vary in posture and appear within complex, cluttered environments, frequently with other people. In many studies, however, such variability is absent. How does the N190 response change when observers see images that incorporate these sources of variability? In two experiments (N = 16 each), we varied the natural appearance of upright and inverted bodies to examine how the N190 amplitude, latency, and the Body-Inversion Effect (BIE) were affected by natural variability. In Experiment 1, we varied the number of people present in upright and inverted naturalistic scenes such that only one body, a subitizable number of bodies, or a "crowd" was present. In Experiment 2, we varied the natural body appearance by presenting bodies either as silhouettes or with photographic detail. Further, we varied the natural background appearance by either removing it or presenting individual bodies within a rich environment. Using component-based analyses of the N190, we found that the number of bodies in a scene reduced the N190 amplitude, but didn't affect the BIE (Experiment 1). Naturalistic body and background appearance (Experiment 2) also affected the N190, such that component amplitude was dramatically reduced by naturalistic appearance. To complement this analysis, we examined the contribution of spatiotemporal features (i.e., electrode × time point amplitude) via SVM decoding. This technique allows us to examine which timepoints across the entire waveform contribute the most to successful decoding of body orientation in each condition. This analysis revealed that later timepoints (after 300ms) contribute most to successful orientation decoding. These results demonstrate that natural appearance variability affects body processing at the N190 and that later ERP components may make important contributions to body processing in natural scenes.

Bayesian face recognition and perceptual narrowing in face-space.

During the first year of life, infants' face recognition abilities are subject to 'perceptual narrowing', the end result of which is that observers lose the ability to distinguish previously discriminable faces (e.g. other-race faces) from one another. Perceptual narrowing has been reported for faces of different species and different races, in developing humans and primates. Though the phenomenon is highly robust and replicable, there have been few efforts to model the emergence of perceptual narrowing as a function of the accumulation of experience with faces during infancy. The goal of the current study is to examine how perceptual narrowing might manifest as statistical estimation in 'face-space', a geometric framework for describing face recognition that has been successfully applied to adult face perception. Here, I use a computer vision algorithm for Bayesian face recognition to study how the acquisition of experience in face-space and the presence of race categories affect performance for own and other-race faces. Perceptual narrowing follows from the establishment of distinct race categories, suggesting that the acquisition of category boundaries for race is a key computational mechanism in developing face expertise.

Thin-slice perception develops slowly.

Body language and facial gesture provide sufficient visual information to support high-level social inferences from "thin slices" of behavior. Given short movies of nonverbal behavior, adults make reliable judgments in a large number of tasks. Here we find that the high precision of adults' nonverbal social perception depends on the slow development, over childhood, of sensitivity to subtle visual cues. Children and adult participants watched short silent clips in which a target child played with Lego blocks either in the (off-screen) presence of an adult or alone. Participants judged whether the target was playing alone or not; that is, they detected the presence of a social interaction (from the behavior of one participant in that interaction). This task allowed us to compare performance across ages with the true answer. Children did not reach adult levels of performance on this task until 9 or 10 years of age, and we observed an interaction between age and video reversal. Adults and older children benefitted from the videos being played in temporal sequence, rather than reversed, suggesting that adults (but not young children) are sensitive to natural movement in social interactions.

A summary statistic representation in peripheral vision explains visual search.

Vision is an active process: We repeatedly move our eyes to seek out objects of interest and explore our environment. Visual search experiments capture aspects of this process, by having subjects look for a target within a background of distractors. Search speed often correlates with target-distractor discriminability; search is faster when the target and distractors look quite different. However, there are notable exceptions. A given discriminability can yield efficient searches (where the target seems to "pop-out") as well as inefficient ones (where additional distractors make search significantly slower and more difficult). Search is often more difficult when finding the target requires distinguishing a particular configuration or conjunction of features. Search asymmetries abound. These puzzling results have fueled three decades of theoretical and experimental studies. We argue that the key issue in search is the processing of image patches in the periphery, where visual representation is characterized by summary statistics computed over a sizable pooling region. By quantifying these statistics, we predict a set of classic search results, as well as peripheral discriminability of crowded patches such as those found in search displays.