Organization of pRF size along the AP axis of the hippocampus and adjacent medial temporal cortex is related to specialization for scenes versus faces.

The hippocampus is largely recognized for its integral contributions to memory processing. By contrast, its role in perceptual processing remains less clear. Hippocampal properties vary along the anterior-posterior (AP) axis. Based on past research suggesting a gradient in the scale of features processed along the AP extent of the hippocampus, the representations have been proposed to vary as a function of granularity along this axis. One way to quantify such granularity is with population receptive field (pRF) size measured during visual processing, which has so far received little attention. In this study, we compare the pRF sizes within the hippocampus to its activation for images of scenes versus faces. We also measure these functional properties in surrounding medial temporal lobe (MTL) structures. Consistent with past research, we find pRFs to be larger in the anterior than in the posterior hippocampus. Critically, our analysis of surrounding MTL regions, the perirhinal cortex, entorhinal cortex, and parahippocampal cortex shows a similar correlation between scene sensitivity and larger pRF size. These findings provide conclusive evidence for a tight relationship between the pRF size and the sensitivity to image content in the hippocampus and adjacent medial temporal cortex.

Neither Enhanced Nor Lost: The Unique Role of Attention in Children's Neural Representations.

A defining feature of children's cognition is the especially slow development of their attention. Despite a rich behavioral literature characterizing the development of attention, little is known about how developing attentional abilities modulate neural representations in children. This information is critical to understanding how attentional development shapes the way children process information. One possibility is that attention might be less likely to shape neural representations in children as compared with adults. In particular, representations of attended items may be less likely to be enhanced relative to unattended items. To investigate this possibility, we measured brain activity using fMRI while children (seven to nine years; male and female) and adults (21-31 years; male and female) performed a one-back task in which they were directed to attend to either motion direction or an object in a display where both were present. We used multivoxel pattern analysis to compare decoding accuracy of attended and unattended information. Consistent with attentional enhancement, we found higher decoding accuracy for task-relevant information (i.e., objects in the object-attended condition) than for task-irrelevant information (i.e., motion in the object-attended condition) in adults' visual cortices. However, in children's visual cortices, both task-relevant and task-irrelevant information were decoded equally well. What is more, whole-brain analysis showed that the children represented task-irrelevant information more than adults in multiple regions across the brain, including the prefrontal cortex. These findings show that (1) attention does not modulate neural representations in the child visual cortex, and (2) developing brains can, and do, represent more information than mature brains.SIGNIFICANCE STATEMENT Children have been shown to struggle with maintaining their attention to specific information, and at the same time, can show better learning of "distractors." While these are critical properties of childhood, their underlying neural mechanisms are unknown. To fill in this critical knowledge gap, we explored how attention shapes what is represented in children's and adults' brains using fMRI while both were asked to focus on just one of two things (objects and motion). We found that unlike adults, who prioritize the information they were asked to focus on, children represent both what they were asked to prioritize and what they were asked to ignore. This shows that attention has a fundamentally different impact on children's neural representations.

Changing What You Like: Modifying Contour Properties Shifts Aesthetic Valuations of Scenes.

To what extent do aesthetic experiences arise from the human ability to perceive and extract meaning from visual features? Ordinary scenes, such as a beach sunset, can elicit a sense of beauty in most observers. Although it appears that aesthetic responses can be shared among humans, little is known about the cognitive mechanisms that underlie this phenomenon. We developed a contour model of aesthetics that assigns values to visual properties in scenes, allowing us to predict aesthetic responses in adults from around the world. Through a series of experiments, we manipulate contours to increase or decrease aesthetic value while preserving scene semantic identity. Contour manipulations directly shift subjective aesthetic judgments. This provides the first experimental evidence for a causal relationship between contour properties and aesthetic valuation. Our findings support the notion that visual regularities underlie the human capacity to derive pleasure from visual information.

Memorability of line drawings of scenes: the role of contour properties.

Why are some images more likely to be remembered than others? Previous work focused on the influence of global, low-level visual features as well as image content on memorability. To better understand the role of local, shape-based contours, we here investigate the memorability of photographs and line drawings of scenes. We find that the memorability of photographs and line drawings of the same scenes is correlated. We quantitatively measure the role of contour properties and their spatial relationships for scene memorability using a Random Forest analysis. To determine whether this relationship is merely correlational or if manipulating these contour properties causes images to be remembered better or worse, we split each line drawing into two half-images, one with high and the other with low predicted memorability according to the trained Random Forest model. In a new memorability experiment, we find that the half-images predicted to be more memorable were indeed remembered better, confirming a causal role of shape-based contour features, and, in particular, T junctions in scene memorability. We performed a categorization experiment on half-images to test for differential access to scene content. We found that half-images predicted to be more memorable were categorized more accurately. However, categorization accuracy for individual images was not correlated with their memorability. These results demonstrate that we can measure the contributions of individual contour properties to scene memorability and verify their causal involvement with targeted image manipulations, thereby bridging the gap between low-level features and scene semantics in our understanding of memorability.

Anger is red, sadness is blue: Emotion depictions in abstract visual art by artists and non-artists.

Through the manipulation of color and form, visual abstract art is often used to convey feelings and emotions. Here, we explored how colors and lines are used to express basic emotions and whether non-artists express emotions through art in similar ways as trained artists. Both artists and non-artists created abstract color drawings and line drawings depicting six emotions (i.e., anger, disgust, fear, joy, sadness, and wonder). To test whether people represented basic emotions in similar ways, we computationally predicted the emotion of a given drawing by comparing it to a set of references created by averaging across all other participants' drawings within each emotion category. We found that prediction accuracy was higher for color drawings than line drawings and higher for color drawings by non-artists than by artists. In a behavioral experiment, we found that people (N = 242) could also accurately infer emotions, showing the same pattern of results as our computational predictions. Further computational analyses of the drawings revealed systematic use of certain colors and line features to depict each basic emotion (e.g., anger is generally redder and more densely drawn than other emotions, sadness is more blue and contains more vertical lines). Taken together, these results imply that abstract color and line drawings are able to convey certain emotions based on their visual features, which are also used by human observers to understand the intended emotional connotation of abstract artworks.

Neural Representations in the Prefrontal Cortex Are Task Dependent for Scene Attributes But Not for Scene Categories.

Natural scenes deliver rich sensory information about the world. Decades of research has shown that the scene-selective network in the visual cortex represents various aspects of scenes. However, less is known about how such complex scene information is processed beyond the visual cortex, such as in the prefrontal cortex. It is also unknown how task context impacts the process of scene perception, modulating which scene content is represented in the brain. In this study, we investigate these questions using scene images from four natural scene categories, which also depict two types of scene attributes, temperature (warm or cold), and sound level (noisy or quiet). A group of healthy human subjects from both sexes participated in the present study using fMRI. In the study, participants viewed scene images under two different task conditions: temperature judgment and sound-level judgment. We analyzed how these scene attributes and categories are represented across the brain under these task conditions. Our findings show that scene attributes (temperature and sound level) are only represented in the brain when they are task relevant. However, scene categories are represented in the brain, in both the parahippocampal place area and the prefrontal cortex, regardless of task context. These findings suggest that the prefrontal cortex selectively represents scene content according to task demands, but this task selectivity depends on the types of scene content: task modulates neural representations of scene attributes but not of scene categories.SIGNIFICANCE STATEMENT Research has shown that visual scene information is processed in scene-selective regions in the occipital and temporal cortices. Here, we ask how scene content is processed and represented beyond the visual brain, in the prefrontal cortex (PFC). We show that both scene categories and scene attributes are represented in PFC, with interesting differences in task dependency: scene attributes are only represented in PFC when they are task relevant, but scene categories are represented in PFC regardless of the task context. Together, our work shows that scene information is processed beyond the visual cortex, and scene representation in PFC reflects how adaptively our minds extract relevant information from a scene.

Scene wheels: Measuring perception and memory of real-world scenes with a continuous stimulus space.

Precisely characterizing mental representations of visual experiences requires careful control of experimental stimuli. Recent work leveraging such stimulus control has led to important insights; however, these findings are constrained to simple visual properties like color and line orientation. There remains a critical methodological barrier to characterizing perceptual and mnemonic representations of realistic visual experiences. Here, we introduce a novel method to systematically control visual properties of natural scene stimuli. Using generative adversarial networks (GANs), a state-of-the-art deep learning technique for creating highly realistic synthetic images, we generated scene wheels in which continuously changing visual properties smoothly transition between meaningful realistic scenes. To validate the efficacy of scene wheels, we conducted two behavioral experiments that assess perceptual and mnemonic representations attained from the scene wheels. In the perceptual validation experiment, we tested whether the continuous transition of scene images along the wheel is reflected in human perceptual similarity judgment. The perceived similarity of the scene images correspondingly decreased as distances between the images increase on the wheel. In the memory experiment, participants reconstructed to-be-remembered scenes from the scene wheels. Reconstruction errors for these scenes resemble error distributions observed in prior studies using simple stimulus properties. Importantly, perceptual similarity judgment and memory precision varied systematically with scene wheel radius. These findings suggest our novel approach offers a window into the mental representations of naturalistic visual experiences.

Concavity as a diagnostic feature of visual scenes.

Despite over two decades of research on the neural mechanisms underlying human visual scene, or place, processing, it remains unknown what exactly a "scene" is. Intuitively, we are always inside a scene, while interacting with the outside of objects. Hence, we hypothesize that one diagnostic feature of a scene may be concavity, portraying "inside", and predict that if concavity is a scene-diagnostic feature, then: 1) images that depict concavity, even non-scene images (e.g., the "inside" of an object - or concave object), will be behaviorally categorized as scenes more often than those that depict convexity, and 2) the cortical scene-processing system will respond more to concave images than to convex images. As predicted, participants categorized concave objects as scenes more often than convex objects, and, using functional magnetic resonance imaging (fMRI), two scene-selective cortical regions (the parahippocampal place area, PPA, and the occipital place area, OPA) responded significantly more to concave than convex objects. Surprisingly, we found no behavioral or neural differences between images of concave versus convex buildings. However, in a follow-up experiment, using tightly-controlled images, we unmasked a selective sensitivity to concavity over convexity of scene boundaries (i.e., walls) in PPA and OPA. Furthermore, we found that even highly impoverished line drawings of concave shapes are behaviorally categorized as scenes more often than convex shapes. Together, these results provide converging behavioral and neural evidence that concavity is a diagnostic feature of visual scenes.

Children automatically abstract categorical regularities during statistical learning.

Statistical learning allows us to discover myriad structures in our environment, which is saturated with information at many different levels-from items to categories. How do children learn different levels of information-about regularities that pertain to items and the categories they come from-and how does this differ from adults? Studies on category learning and memory have suggested that children may be more focused on items than adults. If this is also the case for statistical learning, children may not extract and learn the multi-level regularities that adults can. We report three experiments showing that children and adults extract both item- and category-level regularities in statistical learning. In Experiments 1 and 2, we show that both children and adults can learn structure at the item and category levels when they are measured independently. In Experiment 3, we show that both children and adults learn about categories even when exposure does not require this: both are able to generalize their learning from the item to the category level. Results indicate that statistical learning operates across multi-level structure in children and adults alike, enabling generalization of learning from specific items to exemplars from categories of those items that observers have never seen. Even though children may be more focused on items during other forms of learning, they learn about categories from item-level input during statistical learning.

The Development of Attention to Objects and Scenes: From Object-Biased to Unbiased.

Selective attention is the ability to focus on goal-relevant information while filtering out irrelevant information. This work examined the development of selective attention to natural scenes and objects with a rapid serial visual presentation paradigm. Children (N = 69, ages 4-6 years) and adults (N = 80) were asked to attend to either objects or scenes, while ignoring the other type of stimulus. A multinomial processing tree model was used to decompose selective attention into focusing and filtering components. The results suggest that attention is object-biased in children, due to difficulty filtering attention to goal-irrelevant objects, whereas attention in adults is relatively unbiased. The findings suggest important developmental asymmetries in selective attention to scenes and objects.

Modality-Independent Coding of Scene Categories in Prefrontal Cortex.

Natural environments convey information through multiple sensory modalities, all of which contribute to people's percepts. Although it has been shown that visual or auditory content of scene categories can be decoded from brain activity, it remains unclear how humans represent scene information beyond a specific sensory modality domain. To address this question, we investigated how categories of scene images and sounds are represented in several brain regions. A group of healthy human subjects (both sexes) participated in the present study, where their brain activity was measured with fMRI while viewing images or listening to sounds of different real-world environments. We found that both visual and auditory scene categories can be decoded not only from modality-specific areas, but also from several brain regions in the temporal, parietal, and prefrontal cortex (PFC). Intriguingly, only in the PFC, but not in any other regions, categories of scene images and sounds appear to be represented in similar activation patterns, suggesting that scene representations in PFC are modality-independent. Furthermore, the error patterns of neural decoders indicate that category-specific neural activity patterns in the middle and superior frontal gyri are tightly linked to categorization behavior. Our findings demonstrate that complex scene information is represented at an abstract level in the PFC, regardless of the sensory modality of the stimulus.SIGNIFICANCE STATEMENT Our experience in daily life includes multiple sensory inputs, such as images, sounds, or scents from the surroundings, which all contribute to our understanding of the environment. Here, for the first time, we investigated where and how in the brain information about the natural environment from multiple senses is merged to form modality-independent representations of scene categories. We show direct decoding of scene categories across sensory modalities from patterns of neural activity in the prefrontal cortex (PFC). We also conclusively tie these neural representations to human categorization behavior by comparing patterns of errors between a neural decoder and behavior. Our findings suggest that PFC is a central hub for integrating sensory information and computing modality-independent representations of scene categories.

Spatial relationships between contours impact rapid scene classification.

Photographs and line drawings of natural scenes are easily classified even when the image is only briefly visible to the observer. Contour junctions and points of high curvature have been shown to be important for perceptual organization (Attneave, 1954; Biederman, 1987) and have been proposed to be influential in rapid scene classification (Walther & Shen, 2014). Here, we manipulate the junctions in images, either randomly translating them, or selectively removing or maintaining them. Observers were better at classifying images when the contours were randomly translated (disrupting the junctions) than when the junctions were randomly shifted (partially disrupting contour information). Moreover, observers were better at classifying a scene when shown only segments between junctions, than when shown only the junctions, with the middle segments removed. These results suggest that categorizing line drawings of real-world scenes does not solely rely on junction statistics. The spatial locations of the junctions are important, as well as their relationships with one another. Furthermore, the segments between junctions appear to facilitate scene classification, possibly due to their involvement in symmetry relationships with other contour segments.

Contour junctions underlie neural representations of scene categories in high-level human visual cortex.

Humans efficiently grasp complex visual environments, making highly consistent judgments of entry-level category despite their high variability in visual appearance. How does the human brain arrive at the invariant neural representations underlying categorization of real-world environments? We here show that the neural representation of visual environments in scene-selective human visual cortex relies on statistics of contour junctions, which provide cues for the three-dimensional arrangement of surfaces in a scene. We manipulated line drawings of real-world environments such that statistics of contour orientations or junctions were disrupted. Manipulated and intact line drawings were presented to participants in an fMRI experiment. Scene categories were decoded from neural activity patterns in the parahippocampal place area (PPA), the occipital place area (OPA) and other visual brain regions. Disruption of junctions but not orientations led to a drastic decrease in decoding accuracy in the PPA and OPA, indicating the reliance of these areas on intact junction statistics. Accuracy of decoding from early visual cortex, on the other hand, was unaffected by either image manipulation. We further show that the correlation of error patterns between decoding from the scene-selective brain areas and behavioral experiments is contingent on intact contour junctions. Finally, a searchlight analysis exposes the reliance of visually active brain regions on different sets of contour properties. Statistics of contour length and curvature dominate neural representations of scene categories in early visual areas and contour junctions in high-level scene-selective brain regions.

Dissociation of salience-driven and content-driven spatial attention to scene category with predictive decoding of gaze patterns.

Scene content is thought to be processed quickly and efficiently to bias subsequent visual exploration.Does scene content bias spatial attention during task-free visual exploration of natural scenes?If so, is this bias driven by patterns of physical salience or content-driven biases formed through previous encounters with similar scenes? We conducted two eye-tracking experiments to address these questions. Using a novel gaze decoding method, we show that fixation patterns predict scene category during free exploration. Additionally, we isolate salience-driven contributions using computational salience maps and content-driven contributions using gaze-restricted fixation data. We find distinct time courses for salience-driven and content-driven effects. The influence of physical salience peaked initially but quickly fell off at 600 ms past stimulus onset. The influence of content effects started at chance and steadily increased over the 2000 ms after stimulus onset. The combination of these two components significantly explains the time course of gaze allocation during free exploration.

Contribution of malocclusion and female facial attractiveness to smile esthetics evaluated by eye tracking.

INTRODUCTION: There is disagreement in the literature concerning the importance of the mouth in overall facial attractiveness. Eye tracking provides an objective method to evaluate what people see. The objective of this study was to determine whether dental and facial attractiveness alters viewers' visual attention in terms of which area of the face (eyes, nose, mouth, chin, ears, or other) is viewed first, viewed the greatest number of times, and viewed for the greatest total time (duration) using eye tracking. METHODS: Seventy-six viewers underwent 1 eye tracking session. Of these, 53 were white (49% female, 51% male). Their ages ranged from 18 to 29 years, with a mean of 19.8 years, and none were dental professionals. After being positioned and calibrated, they were shown 24 unique female composite images, each image shown twice for reliability. These images reflected a repaired unilateral cleft lip or 3 grades of dental attractiveness similar to those of grades 1 (near ideal), 7 (borderline treatment need), and 10 (definite treatment need) as assessed in the aesthetic component of the Index of Orthodontic Treatment Need (AC-IOTN). The images were then embedded in faces of 3 levels of attractiveness: attractive, average, and unattractive. During viewing, data were collected for the first location, frequency, and duration of each viewer's gaze. RESULTS: Observer reliability ranged from 0.58 to 0.92 (intraclass correlation coefficients) but was less than 0.07 (interrater) for the chin, which was eliminated from the study. Likewise, reliability for the area of first fixation was kappa less than 0.10 for both intrarater and interrater reliabilities; the area of first fixation was also removed from the data analysis. Repeated-measures analysis of variance showed a significant effect (P <0.001) for level of attractiveness by malocclusion by area of the face. For both number of fixations and duration of fixations, the eyes overwhelmingly were most salient, with the mouth receiving the second most visual attention. At times, the mouth and the eyes were statistically indistinguishable in viewers' gazes of fixation and duration. As the dental attractiveness decreased, the visual attention increased on the mouth, approaching that of the eyes. AC-IOTN grade 10 gained the most attention, followed by both AC-IOTN grade 7 and the cleft. AC-IOTN grade 1 received the least amount of visual attention. Also, lower dental attractiveness (AC-IOTN 7 and AC-IOTN 10) received more visual attention as facial attractiveness increased. CONCLUSIONS: Eye tracking indicates that dental attractiveness can alter the level of visual attention depending on the female models' facial attractiveness when viewed by laypersons.

Nonaccidental properties underlie human categorization of complex natural scenes.

Humans can categorize complex natural scenes quickly and accurately. Which scene properties enable people to do this with such apparent ease? We extracted structural properties of contours (orientation, length, curvature) and contour junctions (types and angles) from line drawings of natural scenes. All of these properties contain information about scene categories that can be exploited computationally. However, when we compared error patterns from computational scene categorization with those from a six-alternative forced-choice scene-categorization experiment, we found that only junctions and curvature made significant contributions to human behavior. To further test the critical role of these properties, we perturbed junctions in line drawings by randomly shifting contours and found a significant decrease in human categorization accuracy. We conclude that scene categorization by humans relies on curvature as well as the same nonaccidental junction properties used for object recognition. These properties correspond to the visual features represented in area V2.

Simple line drawings suffice for functional MRI decoding of natural scene categories.

Humans are remarkably efficient at categorizing natural scenes. In fact, scene categories can be decoded from functional MRI (fMRI) data throughout the ventral visual cortex, including the primary visual cortex, the parahippocampal place area (PPA), and the retrosplenial cortex (RSC). Here we ask whether, and where, we can still decode scene category if we reduce the scenes to mere lines. We collected fMRI data while participants viewed photographs and line drawings of beaches, city streets, forests, highways, mountains, and offices. Despite the marked difference in scene statistics, we were able to decode scene category from fMRI data for line drawings just as well as from activity for color photographs, in primary visual cortex through PPA and RSC. Even more remarkably, in PPA and RSC, error patterns for decoding from line drawings were very similar to those from color photographs. These data suggest that, in these regions, the information used to distinguish scene category is similar for line drawings and photographs. To determine the relative contributions of local and global structure to the human ability to categorize scenes, we selectively removed long or short contours from the line drawings. In a category-matching task, participants performed significantly worse when long contours were removed than when short contours were removed. We conclude that global scene structure, which is preserved in line drawings, plays an integral part in representing scene categories.

Brief category learning distorts perceptual space for complex scenes.

The formation of categories is known to distort perceptual space: representations are pushed away from category boundaries and pulled toward categorical prototypes. This phenomenon has been studied with artificially constructed objects, whose feature dimensions are easily defined and manipulated. How such category-induced perceptual distortions arise for complex, real-world scenes, however, remains largely unknown due to the technical challenge of measuring and controlling scene features. We address this question by generating realistic scene images from a high-dimensional continuous space using generative adversarial networks and using the images as stimuli in a novel learning task. Participants learned to categorize the scene images along arbitrary category boundaries and later reconstructed the same scenes from memory. Systematic biases in reconstruction errors closely tracked each participant's subjective category boundaries. These findings suggest that the perception of global scene properties is warped to align with a newly learned category structure after only a brief learning experience.

Shape-Based Measures Improve Scene Categorization.

Converging evidence indicates that deep neural network models that are trained on large datasets are biased toward color and texture information. Humans, on the other hand, can easily recognize objects and scenes from images as well as from bounding contours. Mid-level vision is characterized by the recombination and organization of simple primary features into more complex ones by a set of so-called Gestalt grouping rules. While described qualitatively in the human literature, a computational implementation of these perceptual grouping rules is so far missing. In this article, we contribute a novel set of algorithms for the detection of contour-based cues in complex scenes. We use the medial axis transform (MAT) to locally score contours according to these grouping rules. We demonstrate the benefit of these cues for scene categorization in two ways: (i) Both human observers and CNN models categorize scenes most accurately when perceptual grouping information is emphasized. (ii) Weighting the contours with these measures boosts performance of a CNN model significantly compared to the use of unweighted contours. Our work suggests that, even though these measures are computed directly from contours in the image, current CNN models do not appear to extract or utilize these grouping cues.

Experience transforms crossmodal object representations in the anterior temporal lobes.

Combining information from multiple senses is essential to object recognition, core to the ability to learn concepts, make new inferences, and generalize across distinct entities. Yet how the mind combines sensory input into coherent crossmodal representations - the crossmodal binding problem - remains poorly understood. Here, we applied multi-echo fMRI across a 4-day paradigm, in which participants learned three-dimensional crossmodal representations created from well-characterized unimodal visual shape and sound features. Our novel paradigm decoupled the learned crossmodal object representations from their baseline unimodal shapes and sounds, thus allowing us to track the emergence of crossmodal object representations as they were learned by healthy adults. Critically, we found that two anterior temporal lobe structures - temporal pole and perirhinal cortex - differentiated learned from non-learned crossmodal objects, even when controlling for the unimodal features that composed those objects. These results provide evidence for integrated crossmodal object representations in the anterior temporal lobes that were different from the representations for the unimodal features. Furthermore, we found that perirhinal cortex representations were by default biased toward visual shape, but this initial visual bias was attenuated by crossmodal learning. Thus, crossmodal learning transformed perirhinal representations such that they were no longer predominantly grounded in the visual modality, which may be a mechanism by which object concepts gain their abstraction.