The time-course of real-world scene perception: Spatial and semantic processing.

Real-world scene perception unfolds remarkably quickly, yet the underlying visual processes are poorly understood. Space-centered theory maintains that a scene's spatial structure (e.g., openness, mean depth) can be rapidly recovered from low-level image statistics. In turn, the statistical relationship between a scene's spatial properties and semantic content allows for semantic identity to be inferred from its layout. We tested this theory by investigating (1) the temporal dynamics of spatial and semantic perception in real-world scenes, and (2) dependencies between spatial and semantic judgments. Participants viewed backward-masked images for 13.3 to 106.7 ms, and identified the semantic (e.g., beach, road) or spatial structure (e.g., open, closed-off) category. We found no temporal precedence of spatial discrimination relative to semantic discrimination. Computational analyses further suggest that, instead of using spatial layout to infer semantic categories, humans exploit semantic information to discriminate spatial structure categories. These findings challenge traditional 'bottom-up' views of scene perception.

Deep learning models fail to capture the configural nature of human shape perception.

A hallmark of human object perception is sensitivity to the holistic configuration of the local shape features of an object. Deep convolutional neural networks (DCNNs) are currently the dominant models for object recognition processing in the visual cortex, but do they capture this configural sensitivity? To answer this question, we employed a dataset of animal silhouettes and created a variant of this dataset that disrupts the configuration of each object while preserving local features. While human performance was impacted by this manipulation, DCNN performance was not, indicating insensitivity to object configuration. Modifications to training and architecture to make networks more brain-like did not lead to configural processing, and none of the networks were able to accurately predict trial-by-trial human object judgements. We speculate that to match human configural sensitivity, networks must be trained to solve a broader range of object tasks beyond category recognition.

Clinical utilization of a sous vide device in the acute rewarming of frostbitten extremities.

The current standard of care for acute frostbite rewarming is the use of a circulating warm water bath at a temperature of 37 °C to 39 °C. There is no standardized method to achieve this. Manual management of a warm water bath can be inefficient and time consuming. This case describes the clinical use of a sous vide cooking device to create and maintain a circulating warm water bath to rewarm acute frostbite. A 34 year-old male presented to the emergency department with acute frostbite. Each of the patient's feet were placed in a water bath with a sous vide device attached to the side of the basin and set to 38 °C. Temperatures were recorded every 2 m from 2 thermometers. Once target temperature was achieved, the extremities were rewarmed for 30 m. The water baths required an average of 25 m to reach target temperature and maintained the target temperature within ±1 °C for the duration of the rewarming. The extremities were clinically thawed in one session and there were no adverse events. The patient was seen by plastic and vascular surgery and admitted to the hospital for conservative management. He was discharged on hospital day 3 and did not require any amputations. A sous vide device can be used clinically to heat and maintain a water bath and successfully rewarm frostbitten extremities in one 30 m cycle. No adverse events were reported and providers rated this as a convenient method of water bath management.

Scaling depth from shadow offset.

When an object casts a shadow on a background surface, both the offset of the shadow and the blur of its penumbra are potential cues to the distance between the object and the background. However, the shadow offset and blur are also affected by the direction and angular extent of the light source and these are often unknown. This means that the observer must make some assumptions about the illumination, the expected distribution of depth, or the relation between offset and depth in order to use shadows to make distance judgments. Here, we measure human judgments of perceived depth over a range of shadow offsets, blurs, and lighting directions to gain insight into this internal model. We find that distance judgments are relatively unaffected by blur or light direction, whereas the shadow offset has a strong and linear effect. The data are consistent with two models, a generic shadow-to-depth model and a Bayesian model.

Ice hockey spectators use contextual cues to guide predictive eye movements.

Eye movements are an integral part of human visual perception. They allow us to have a small foveal region with exquisite acuity and at the same time a large visual field. For a long time, eye movements were regarded as machine-like behaviors in response to visual stimulation1, but over the past few decades it has been convincingly shown that expectations, intended actions, rewards and many other cognitive factors can have profound effects on the way we move our eyes2-4. In order to be useful, our oculomotor system must minimize delay with respect to the dynamic events in the visual scene. The ability to do so has been demonstrated in situations where we are in control of these events, for example when we are making a sandwich or tea5, and when we are active participants, for example when hitting a cricket ball6. But what about scenes with complex dynamics that we do not control or directly take part in, like a hockey game we are watching as a spectator? A semantic influence on gaze fixation location during viewing of tennis videos has been suggested before7. Here we use carefully annotated hockey videos to show that the brain is indeed able to exploit the semantic context of the game to anticipate the continuous motion of the puck, leading to eye movements that are fundamentally different than when following exactly the same motion without any context.

Category systems for real-world scenes.

Categorization performance is a popular metric of scene recognition and understanding in behavioral and computational research. However, categorical constructs and their labels can be somewhat arbitrary. Derived from exhaustive vocabularies of place names (e.g., Deng et al., 2009), or the judgements of small groups of researchers (e.g., Fei-Fei, Iyer, Koch, & Perona, 2007), these categories may not correspond with human-preferred taxonomies. Here, we propose clustering by increasing the rand index via coordinate ascent (CIRCA): an unsupervised, data-driven clustering method for deriving ground-truth scene categories. In Experiment 1, human participants organized 80 stereoscopic images of outdoor scenes from the Southampton-York Natural Scenes (SYNS) dataset (Adams et al., 2016) into discrete categories. In separate tasks, images were grouped according to i) semantic content, ii) three-dimensional spatial structure, or iii) two-dimensional image appearance. Participants provided text labels for each group. Using the CIRCA method, we determined the most representative category structure and then derived category labels for each task/dimension. In Experiment 2, we found that these categories generalized well to a larger set of SYNS images, and new observers. In Experiment 3, we tested the relationship between our category systems and the spatial envelope model (Oliva & Torralba, 2001). Finally, in Experiment 4, we validated CIRCA on a larger, independent dataset of same-different category judgements. The derived category systems outperformed the SUN taxonomy (Xiao, Hays, Ehinger, Oliva, & Torralba, 2010) and an alternative clustering method (Greene, 2019). In summary, we believe this novel categorization method can be applied to a wide range of datasets to derive optimal categorical groupings and labels from psychophysical judgements of stimulus similarity.

The role of global cues in the perceptual grouping of natural shapes.

Perceptual grouping of the bounding contours of objects is a crucial step in visual scene understanding and object recognition. The standard perceptual model for this task, supported by a convergence of physiological and psychophysical evidence, is based upon an association field that governs local grouping, and a Markov or transitivity assumption that allows global contours to be inferred solely from these local cues. However, computational studies suggest that these local cues may not be sufficient for reliable identification of object boundaries in natural scenes. Here we employ a novel psychophysical method to assess the potential role of more global factors in the perceptual grouping of natural object contours. Observers were asked to detect briefly presented fragmented target contours in oriented element noise. We employed natural animal shape stimuli, which in addition to local grouping cues possess global regularities that could potentially be exploited to guide grouping and thereby improve target detection performance. To isolate the role of these global regularities we contrasted performance with open and closed control target stimuli we call local metamers, as they afford the same local grouping cues as animal shapes. We found that performance for closed metamers exceeded performance for open metamers, while performance for animal targets exceeded both, indicating that global grouping cues represented in higher visual areas codetermine the association between orientation signals coded in early visual cortex. These results demand a revision to the standard model for perceptual grouping of contours to accommodate feedback from higher visual areas coding global shape properties.

Shape from Contour: Computation and Representation.

The human visual system reliably extracts shape information from complex natural scenes in spite of noise and fragmentation caused by clutter and occlusions. A fast, feedforward sweep through ventral stream involving mechanisms tuned for orientation, curvature, and local Gestalt principles produces partial shape representations sufficient for simpler discriminative tasks. More complete shape representations may involve recurrent processes that integrate local and global cues. While feedforward discriminative deep neural network models currently produce the best predictions of object selectivity in higher areas of the object pathway, a generative model may be required to account for all aspects of shape perception. Research suggests that a successful model will account for our acute sensitivity to four key perceptual dimensions of shape: topology, symmetry, composition, and deformation.

Frequency tuning of shape perception revealed by classification image analysis.

Classification image analysis is a powerful technique for elucidating linear detection and discrimination mechanisms, but it has primarily been applied to contrast detection. Here we report a novel classification image methodology for identifying linear mechanisms underlying shape discrimination. Although prior attempts to apply classification image methods to shape perception have been confined to simple radial shapes, the method proposed here can be applied to general 2-D (planar) shapes of arbitrary complexity, including natural shapes. Critical to the method is the projection of each target shape onto a Fourier descriptor (FD) basis set, which allows the essential perceptual features of each shape to be represented by a relatively small number of coefficients. We demonstrate that under this projection natural shapes are low pass, following a relatively steep power law. To efficiently identify the observer's classification template, we employ a yes/no paradigm and match the spectral density of the stimulus noise in FD space to the power law density of the target shape. The proposed method generates linear template models for animal shape detection that are predictive of human judgments. These templates are found to be biased away from the ideal, overly weighting lower frequencies. This low-pass bias suggests that higher frequency shape processing relies on nonlinear mechanisms.

The Southampton-York Natural Scenes (SYNS) dataset: Statistics of surface attitude.

Recovering 3D scenes from 2D images is an under-constrained task; optimal estimation depends upon knowledge of the underlying scene statistics. Here we introduce the Southampton-York Natural Scenes dataset (SYNS: https://syns.soton.ac.uk), which provides comprehensive scene statistics useful for understanding biological vision and for improving machine vision systems. In order to capture the diversity of environments that humans encounter, scenes were surveyed at random locations within 25 indoor and outdoor categories. Each survey includes (i) spherical LiDAR range data (ii) high-dynamic range spherical imagery and (iii) a panorama of stereo image pairs. We envisage many uses for the dataset and present one example: an analysis of surface attitude statistics, conditioned on scene category and viewing elevation. Surface normals were estimated using a novel adaptive scale selection algorithm. Across categories, surface attitude below the horizon is dominated by the ground plane (0° tilt). Near the horizon, probability density is elevated at 90°/270° tilt due to vertical surfaces (trees, walls). Above the horizon, probability density is elevated near 0° slant due to overhead structure such as ceilings and leaf canopies. These structural regularities represent potentially useful prior assumptions for human and machine observers, and may predict human biases in perceived surface attitude.

Recurrent Processing in the Formation of Shape Percepts.

The human visual system must extract reliable object information from cluttered visual scenes several times per second, and this temporal constraint has been taken as evidence that the underlying cortical processing must be strictly feedforward. Here we use a novel rapid reinforcement paradigm to probe the temporal dynamics of the neural circuit underlying rapid object shape perception and thus test this feedforward assumption. Our results show that two shape stimuli are optimally reinforcing when separated in time by ∼60 ms, suggesting an underlying recurrent circuit with a time constant (feedforward + feedback) of 60 ms. A control experiment demonstrates that this is not an attentional cueing effect. Instead, it appears to reflect the time course of feedback processing underlying the rapid perceptual organization of shape. SIGNIFICANCE STATEMENT: Human and nonhuman primates can spot an animal shape in complex natural scenes with striking speed, and this has been taken as evidence that the underlying cortical mechanisms are strictly feedforward. Using a novel paradigm to probe the dynamics of shape perception, we find that two shape stimuli are optimally reinforcing when separated in time by 60 ms, suggesting a fast but recurrent neural circuit. This work (1) introduces a novel method for probing the temporal dynamics of cortical circuits underlying perception, (2) provides direct evidence against the feedforward assumption for rapid shape perception, and (3) yields insight into the role of feedback connections in the object pathway.

Effects of specular highlights on perceived surface convexity.

Shading is known to produce vivid perceptions of depth. However, the influence of specular highlights on perceived shape is unclear: some studies have shown that highlights improve quantitative shape perception while others have shown no effect. Here we ask how specular highlights combine with Lambertian shading cues to determine perceived surface curvature, and to what degree this is based upon a coherent model of the scene geometry. Observers viewed ambiguous convex/concave shaded surfaces, with or without highlights. We show that the presence/absence of specular highlights has an effect on qualitative shape, their presence biasing perception toward convex interpretations of ambiguous shaded objects. We also find that the alignment of a highlight with the Lambertian shading modulates its effect on perceived shape; misaligned highlights are less likely to be perceived as specularities, and thus have less effect on shape perception. Increasing the depth of the surface or the slant of the illuminant also modulated the effect of the highlight, increasing the bias toward convexity. The effect of highlights on perceived shape can be understood probabilistically in terms of scene geometry: for deeper objects and/or highly slanted illuminants, highlights will occur on convex but not concave surfaces, due to occlusion of the illuminant. Given uncertainty about the exact object depth and illuminant direction, the presence of a highlight increases the probability that the surface is convex.

A century of Gestalt psychology in visual perception: I. Perceptual grouping and figure-ground organization.

In 1912, Max Wertheimer published his paper on phi motion, widely recognized as the start of Gestalt psychology. Because of its continued relevance in modern psychology, this centennial anniversary is an excellent opportunity to take stock of what Gestalt psychology has offered and how it has changed since its inception. We first introduce the key findings and ideas in the Berlin school of Gestalt psychology, and then briefly sketch its development, rise, and fall. Next, we discuss its empirical and conceptual problems, and indicate how they are addressed in contemporary research on perceptual grouping and figure-ground organization. In particular, we review the principles of grouping, both classical (e.g., proximity, similarity, common fate, good continuation, closure, symmetry, parallelism) and new (e.g., synchrony, common region, element and uniform connectedness), and their role in contour integration and completion. We then review classic and new image-based principles of figure-ground organization, how it is influenced by past experience and attention, and how it relates to shape and depth perception. After an integrated review of the neural mechanisms involved in contour grouping, border ownership, and figure-ground perception, we conclude by evaluating what modern vision science has offered compared to traditional Gestalt psychology, whether we can speak of a Gestalt revival, and where the remaining limitations and challenges lie. A better integration of this research tradition with the rest of vision science requires further progress regarding the conceptual and theoretical foundations of the Gestalt approach, which is the focus of a second review article.

Local visual energy mechanisms revealed by detection of global patterns.

A central goal of visual neuroscience is to relate the selectivity of individual neurons to perceptual judgments, such as detection of a visual pattern at low contrast or in noise. Since neurons in early areas of visual cortex carry information only about a local patch of the image, detection of global patterns must entail spatial pooling over many such neurons. Physiological methods provide access to local detection mechanisms at the single-neuron level but do not reveal how neural responses are combined to determine the perceptual decision. Behavioral methods provide access to perceptual judgments of a global stimulus but typically do not reveal the selectivity of the individual neurons underlying detection. Here we show how the existence of a nonlinearity in spatial pooling does allow properties of these early mechanisms to be estimated from behavioral responses to global stimuli. As an example, we consider detection of large-field sinusoidal gratings in noise. Based on human behavioral data, we estimate the length and width tuning of the local detection mechanisms and show that it is roughly consistent with the tuning of individual neurons in primary visual cortex of primate. We also show that a local energy model of pooling based on these estimated receptive fields is much more predictive of human judgments than competing models, such as probability summation. In addition to revealing underlying properties of early detection and spatial integration mechanisms in human cortex, our findings open a window on new methods for relating system-level perceptual judgments to neuron-level processing.

Oriented texture detection: ideal observer modelling and classification image analysis.

Perception of visual texture flows contributes to object segmentation, shape perception, and object recognition. To better understand the visual mechanisms underlying texture flow perception, we studied the factors limiting detection of simple forms of texture flows composed of local dot dipoles (Glass patterns) and related stimuli. To provide a benchmark for human performance, we derived an ideal observer for this task. We found that human detection thresholds were 8.0 times higher than ideal. We considered three factors that might account for this performance gap: (1) false matches between dipole dots (correspondence errors), (2) loss of sensitivity with increasing eccentricity, and (3) local orientation bandwidth. To estimate the effect of correspondence errors, we compared detection of Glass patterns with detection of matched line-segment stimuli, where no correspondence uncertainty exists. We found that eliminating correspondence errors reduced human thresholds by a factor of 1.8. We used a novel form of classification image analysis to directly estimate loss of sensitivity with eccentricity and local orientation bandwidth. Incorporating the eccentricity effects into the ideal observer model increased ideal thresholds by a factor of 2.9. Interestingly, estimated orientation bandwidth increased ideal thresholds by only 8%. Taking all three factors into account, human thresholds were only 58% higher than model thresholds. Our findings suggest that correspondence errors and eccentricity losses account for the great majority of the perceptual loss in the visual processing of Glass patterns.

Cue dynamics underlying rapid detection of animals in natural scenes.

Humans are known to be good at rapidly detecting animals in natural scenes. Evoked potential studies indicate that the corresponding neural signals can emerge in the brain within 150 msec of stimulus onset (S. Thorpe, D. Fize, & C. Marlot, 1996) and eye movements toward animal targets can be initiated in roughly the same timeframe (H. Kirchner & S. J. Thorpe, 2006). Given the speed of this discrimination, it has been suggested that the underlying visual mechanisms must be relatively simple and feedforward, but in fact little is known about these mechanisms. A key step is to understand the visual cues upon which these mechanisms rely. Here we investigate the role and dynamics of four potential cues: two-dimensional boundary shape, texture, luminance, and color. Results suggest that the fastest mechanisms underlying animal detection in natural scenes use shape as a principal discriminative cue, while somewhat slower mechanisms integrate these rapidly computed shape cues with image texture cues. Consistent with prior studies, we find little role for luminance and color cues throughout the time course of visual processing, even though information relevant to the task is available in these signals.

Visual short-term memory for natural scenes: Effects of eccentricity.

It is well established that a range of basic visual acuities and sensitivities decline with retinal eccentricity due in part to a decline in spatial sampling in the retina. However, it is also known that not all peripheral deficits can be explained entirely by such low-level factors, suggesting a specialization of central vision for certain visual tasks. Here, we examine visual short-term memory for natural scenes and ask whether low-level factors can fully account for variations in performance across the visual field. We measure local recognition performance as a function of eccentricity for both coherent and scrambled natural scenes. We find that while spatial coherence substantially increases recognition rates for targets near fixation, the benefit of spatial coherence vanishes in the periphery. These results suggest that low-level factors cannot fully explain the decline in visual short-term memory for natural scenes in the periphery and that mechanisms selective for global configuration are largely confined to the central visual field.

Tied factor analysis for face recognition across large pose differences.

Face recognition algorithms perform very unreliably when the pose of the probe face is different from the gallery face: typical feature vectors vary more with pose than with identity. We propose a generative model that creates a one-to-many mapping from an idealized "identity" space to the observed data space. In identity space, the representation for each individual does not vary with pose. We model the measured feature vector as being generated by a pose-contingent linear transformation of the identity variable in the presence of Gaussian noise. We term this model "tied" factor analysis. The choice of linear transformation (factors) depends on the pose, but the loadings are constant (tied) for a given individual. We use the EM algorithm to estimate the linear transformations and the noise parameters from training data. We propose a probabilistic distance metric which allows a full posterior over possible matches to be established. We introduce a novel feature extraction process and investigate recognition performance using the FERET, XM2VTS and PIE databases. Recognition performance compares favourably to contemporary approaches.