The Sabanci University Dynamic Face Database (SUDFace): Development and validation of an audiovisual stimulus set of recited and free speeches with neutral facial expressions.

Faces convey a wide range of information, including one's identity, and emotional and mental states. Face perception is a major research topic in many research fields, such as cognitive science, social psychology, and neuroscience. Frequently, stimuli are selected from a range of available face databases. However, even though faces are highly dynamic, most databases consist of static face stimuli. Here, we introduce the Sabanci University Dynamic Face (SUDFace) database. The SUDFace database consists of 150 high-resolution audiovisual videos acquired in a controlled lab environment and stored with a resolution of 1920 × 1080 pixels at a frame rate of 60 Hz. The multimodal database consists of three videos of each human model in frontal view in three different conditions: vocalizing two scripted texts (conditions 1 and 2) and one Free Speech (condition 3). The main focus of the SUDFace database is to provide a large set of dynamic faces with neutral facial expressions and natural speech articulation. Variables such as face orientation, illumination, and accessories (piercings, earrings, facial hair, etc.) were kept constant across all stimuli. We provide detailed stimulus information, including facial features (pixel-wise calculations of face length, eye width, etc.) and speeches (e.g., duration of speech and repetitions). In two validation experiments, a total number of 227 participants rated each video on several psychological dimensions (e.g., neutralness and naturalness of expressions, valence, and the perceived mental states of the models) using Likert scales. The database is freely accessible for research purposes.

High confidence and low accuracy in redundancy masking.

Visual scenes typically contain redundant information. One mechanism by which the visual system compresses such redundancies is 'redundancy masking' - the reduction of the perceived number of items in repeating patterns. For example, when presented with three lines in the periphery, observers frequently report only two lines. Redundancy masking is strong in radial arrangements and absent in tangential arrangements. Previous studies suggested that redundancy-masked percepts predominate in stimuli susceptible to redundancy masking. Here, we investigated whether strong redundancy masking is associated with high confidence in perceptual judgements. Observers viewed three to seven radially or tangentially arranged lines at 10° eccentricity. They first indicated the number of lines, and then rated their confidence in their responses. As expected, redundancy masking was strong in radial arrangements and weak in tangential arrangements. Importantly, with radial arrangements, observers were more confident in their responses when redundancy masking occurred (i.e., lower number of lines reported) than when it did not occur (i.e., correct number of lines reported). Hence, observers reported higher confidence for erroneous than for correct judgments. In contrast, with tangential arrangements, observers were similarly confident in their responses whether redundancy masking occurred or not. The inversion of confidence in the radial condition (higher confidence when accuracy was low and lower confidence when accuracy was high) suggests that redundancy-masked appearance trumps 'veridical' perception. The often-reported richness of visual consciousness may partly be due to overconfidence in erroneous judgments in visual scenes that are subject to redundancy masking.

Multidimensional feature interactions in visual crowding: When configural cues eliminate the polarity advantage.

Crowding occurs when surrounding objects (flankers) impair target perception. A key property of crowding is the weaker interference when target and flankers strongly differ on a given dimension. For instance, identification of a target letter is usually superior with flankers of opposite versus the same contrast polarity as the target (the "polarity advantage"). High performance when target-flanker similarity is low has been attributed to the ungrouping of target and flankers. Here, we show that configural cues can override the usual advantage of low target-flanker similarity, and strong target-flanker grouping can reduce - instead of exacerbate - crowding. In Experiment 1, observers were presented with line triplets in the periphery and reported the tilt (left or right) of the central line. Target and flankers had the same (uniform condition) or opposite contrast polarity (alternating condition). Flanker configurations were either upright (||), unidirectionally tilted (\\ or //), or bidirectionally tilted (\/ or /\). Upright flankers yielded stronger crowding than unidirectional flankers, and weaker crowding than bidirectional flankers. Importantly, our results revealed a clear interaction between contrast polarity and flanker configuration. Triplets with upright and bidirectional flankers, but not unidirectional flankers, showed the polarity advantage. In Experiments 2 and 3, we showed that emergent features and redundancy masking (i.e. the reduction of the number of perceived items in repeating configurations) made it easier to discriminate between uniform triplets when flanker tilts were unidirectional (but not when bidirectional). We propose that the spatial configurations of uniform triplets with unidirectional flankers provided sufficient task-relevant information to enable a similar performance as with alternating triplets: strong-target flanker grouping alleviated crowding. We suggest that features which modulate crowding strength can interact non-additively, limiting the validity of typical crowding rules to contexts where only single, independent dimensions determine the effects of target-flanker similarity.

Atypical visual field asymmetries in redundancy masking.

Redundancy masking is the reduction of the perceived number of items in repeating patterns. It shares a number of characteristics with crowding, the impairment of target identification in visual clutter. Crowding strongly depends on the location of the target in the visual field. For example, it is stronger in the upper compared to the lower visual field and is usually weakest on the horizontal meridian. This pattern of visual field asymmetries is common in spatial vision, as revealed by tasks measuring, for example, spatial resolution and contrast sensitivity. Here, to characterize redundancy masking and reveal its similarities to and differences from other spatial tasks, we investigated whether redundancy masking shows the same typical visual field asymmetries. Observers were presented with three to six radially arranged lines at 10° eccentricity at one of eight locations around fixation and were asked to report the number of lines. We found asymmetries that differed pronouncedly from those found in crowding. Redundancy masking did not differ between upper and lower visual fields. Importantly, redundancy masking was stronger on the horizontal meridian than on the vertical meridian, the opposite of what is usually found in crowding. These results show that redundancy masking diverges from crowding in regard to visual field asymmetries, suggesting different underlying mechanisms of redundancy masking and crowding. We suggest that the observed atypical visual field asymmetries in redundancy masking are due to the superior extraction of regularity and a more pronounced compression of visual space on the horizontal compared to the vertical meridian.

Capsule networks as recurrent models of grouping and segmentation.

Classically, visual processing is described as a cascade of local feedforward computations. Feedforward Convolutional Neural Networks (ffCNNs) have shown how powerful such models can be. However, using visual crowding as a well-controlled challenge, we previously showed that no classic model of vision, including ffCNNs, can explain human global shape processing. Here, we show that Capsule Neural Networks (CapsNets), combining ffCNNs with recurrent grouping and segmentation, solve this challenge. We also show that ffCNNs and standard recurrent CNNs do not, suggesting that the grouping and segmentation capabilities of CapsNets are crucial. Furthermore, we provide psychophysical evidence that grouping and segmentation are implemented recurrently in humans, and show that CapsNets reproduce these results well. We discuss why recurrence seems needed to implement grouping and segmentation efficiently. Together, we provide mutually reinforcing psychophysical and computational evidence that a recurrent grouping and segmentation process is essential to understand the visual system and create better models that harness global shape computations.

Geometrically restricted image descriptors: A method to capture the appearance of shape.

Shape perception varies depending on many factors. For example, presenting a stimulus in the periphery often yields a different appearance compared with its foveal presentation. However, how exactly shape appearance is altered under different conditions remains elusive. One reason for this is that studies typically measure identification performance, leaving details about target appearance unknown. The lack of appearance-based methods and general challenges to quantify appearance complicate the investigation of shape appearance. Here, we introduce Geometrically Restricted Image Descriptors (GRIDs), a method to investigate the appearance of shapes. Stimuli in the GRID paradigm are shapes consisting of distinct line elements placed on a grid by connecting grid nodes. Each line is treated as a discrete target. Observers are asked to capture target appearance by placing lines on a freely viewed response grid. We used GRIDs to investigate the appearance of letters and letter-like shapes. Targets were presented at 10° eccentricity in the right visual field. Gaze-contingent stimulus presentation was used to prevent eye movements to the target. The data were analyzed by quantifying the differences between targets and response in regard to overall accuracy, element discriminability, and several distinct error types. Our results show how shape appearance can be captured by GRIDs, and how a fine-grained analysis of stimulus parts provides quantifications of appearance typically not available in standard measures of performance. We propose that GRIDs are an effective tool to investigate the appearance of shapes.

Variations in crowding, saccadic precision, and spatial localization reveal the shared topology of spatial vision.

Visual sensitivity varies across the visual field in several characteristic ways. For example, sensitivity declines sharply in peripheral (vs. foveal) vision and is typically worse in the upper (vs. lower) visual field. These variations can affect processes ranging from acuity and crowding (the deleterious effect of clutter on object recognition) to the precision of saccadic eye movements. Here we examine whether these variations can be attributed to a common source within the visual system. We first compared the size of crowding zones with the precision of saccades using an oriented clock target and two adjacent flanker elements. We report that both saccade precision and crowded-target reports vary idiosyncratically across the visual field with a strong correlation across tasks for all participants. Nevertheless, both group-level and trial-by-trial analyses reveal dissociations that exclude a common representation for the two processes. We therefore compared crowding with two measures of spatial localization: Landolt-C gap resolution and three-dot bisection. Here we observe similar idiosyncratic variations with strong interparticipant correlations across tasks despite considerably finer precision. Hierarchical regression analyses further show that variations in spatial precision account for much of the variation in crowding, including the correlation between crowding and saccades. Altogether, we demonstrate that crowding, spatial localization, and saccadic precision show clear dissociations, indicative of independent spatial representations, whilst nonetheless sharing idiosyncratic variations in spatial topology. We propose that these topological idiosyncrasies are established early in the visual system and inherited throughout later stages to affect a range of higher-level representations.

Redundancy masking: The loss of repeated items in crowded peripheral vision.

Crowding is the deterioration of target identification in the presence of neighboring objects. Recent studies using appearance-based methods showed that the perceived number of target elements is often diminished in crowding. Here we introduce a related type of diminishment in repeating patterns (sets of parallel lines), which we term "redundancy masking." In four experiments, observers were presented with arrays of small numbers of lines centered at 10° eccentricity. The task was to indicate the number of lines. In Experiment 1, spatial characteristics of redundancy masking were examined by varying the inter-line spacing. We found that redundancy masking decreased with increasing inter-line spacing and ceased at spacings of approximately 0.25 times the eccentricity. In Experiment 2, we assessed whether the strength of redundancy masking differed between radial and tangential arrangements of elements as it does in crowding. Redundancy masking was strong with radially arranged lines (horizontally arranged vertical lines), and absent with tangentially arranged lines (vertically arranged horizontal lines). In Experiment 3, we investigated whether target size (line width and length) modulated redundancy masking. There was an effect of width: Thinner lines yielded stronger redundancy masking. We did not find any differences between the tested line lengths. In Experiment 4, we varied the regularity of the line arrays by vertically or horizontally jittering the positions of the lines. Redundancy masking was strongest with regular spacings and weakened with decreasing regularity. Our experiments show under which conditions whole items are lost in crowded displays, and how this redundancy masking resembles-and partly diverges from-crowded identification. We suggest that redundancy masking is a contributor to the deterioration of performance in crowded displays with redundant patterns.

Letters Lost: Capturing Appearance in Crowded Peripheral Vision Reveals a New Kind of Masking.

Peripheral vision is strongly limited by crowding, the deleterious influence of flanking items on target perception. Distinguishing what is seen from what is merely inferred in crowding is difficult because task demands and prior knowledge may influence observers' reports. Here, we used a standard identification task in which participants were susceptible to these influences, and to minimize them, we used a free-report-and-drawing paradigm. Three letters were presented in the periphery. In Experiment 1, 10 participants were asked to identify the central target letter. In Experiment 2, 25 participants freely named and drew what they saw. When three identical letters were presented, performance was almost perfect in Experiment 1, but it was very poor in Experiment 2, in which most participants reported only two letters. Our study reveals limitations of standard crowding paradigms and uncovers a hitherto unrecognized effect that we call redundancy masking.

Emergent features in the crowding zone: When target-flanker grouping surmounts crowding.

Crowding is the impairment of target identification when the target is surrounded by nearby flankers. Two hallmarks of crowding are that it is stronger when the flankers are close to the target and when the target strongly groups with the flankers. Here we show the opposite of both. A chevron target (pointing up or down) was presented at 8° eccentricity in the right visual field. It was surrounded by four flankers. Three of the flankers varied (pointing left or right). The fourth, the critical flanker (CF), was fixed in one orientation (left, right, up, down), yielding different configurations with the target. The CF's distance to the target was varied. Target identification depended strongly on the distance and the orientation of the CF. Remarkably, when the target and the CF grouped into a good configuration and elicited an emergent feature, performance was high if the CF was close to the target. This effect was particularly strong when participants were informed about the different CF-target configurations before the experiment. Reducing crowding and grouping by asynchronous presentation of the CF and the other items abolished the effect. When participants reported the entire configuration of the CF and the target, performance rapidly decreased with increasing spacing when the CF and the target were different but not when they were the same, indicating different spatial extents of the corresponding grouping processes. Our results show that the features emerging from the configurations of the target and a flanker strongly modulate crowding. Strong target-flanker grouping can benefit performance.

Appearance changes and error characteristics in crowding revealed by drawings.

Peripheral vision is strongly limited by crowding: Targets that are easily recognized in isolation are unrecognizable when flanked by close-by objects. Crowding does not only impair target recognition but also changes appearance. Here we investigated appearance changes and errors in crowding by letting observers draw crowded stimuli. Observers drew stimuli presented at 6° and 12° eccentricity. Stimuli consisted of characters and letter-like symbols. Targets were presented with either a flanker on each side or in isolation. To characterize appearance changes and errors in crowding, we developed a scoring system that captured differences between the drawings and the stimuli. The resulting drawings revealed strong appearance changes under crowding. Importantly, our results reveal crowding errors that are usually not shown in standard crowding paradigms. We found high rates of element Omissions and element Truncations, indicating a central role of target "diminishment" in crowding. Furthermore, we show that a subset of the observed element Omissions and Additions was possibly caused by feature migration. Relatively high rates of position errors, in particular element Translations, reflected the often reported location uncertainty in crowding. Virtually no complete target-flanker substitutions were observed. We suggest a new classification system for errors in crowding, and propose drawing as a useful appearance-based method to investigate crowding.

Crowding, grouping, and object recognition: A matter of appearance.

In crowding, the perception of a target strongly deteriorates when neighboring elements are presented. Crowding is usually assumed to have the following characteristics. (a) Crowding is determined only by nearby elements within a restricted region around the target (Bouma's law). (b) Increasing the number of flankers can only deteriorate performance. (c) Target-flanker interference is feature-specific. These characteristics are usually explained by pooling models, which are well in the spirit of classic models of object recognition. In this review, we summarize recent findings showing that crowding is not determined by the above characteristics, thus, challenging most models of crowding. We propose that the spatial configuration across the entire visual field determines crowding. Only when one understands how all elements of a visual scene group with each other, can one determine crowding strength. We put forward the hypothesis that appearance (i.e., how stimuli look) is a good predictor for crowding, because both crowding and appearance reflect the output of recurrent processing rather than interactions during the initial phase of visual processing.

How color, regularity, and good Gestalt determine backward masking.

The strength of visual backward masking depends on the stimulus onset asynchrony (SOA) between target and mask. Recently, it was shown that the conjoint spatial layout of target and mask is as crucial as SOA. Particularly, masking strength depends on whether target and mask group with each other. The same is true in crowding where the global spatial layout of the flankers and target-flanker grouping determine crowding strength. Here, we presented a vernier target followed by different flanker configurations at varying SOAs. Similar to crowding, masking of a red vernier target was strongly reduced for arrays of 10 green compared with 10 red flanking lines. Unlike crowding, single green lines flanking the red vernier showed strong masking. Irregularly arranged flanking lines yielded stronger masking than did regularly arranged lines, again similar to crowding. While cuboid flankers reduced crowding compared with single lines, this was not the case in masking. We propose that, first, masking is reduced when the flankers are part of a larger spatial structure. Second, spatial factors counteract color differences between the target and the flankers. Third, complex Gestalts, such as cuboids, seem to need longer processing times to show ungrouping effects as observed in crowding. Strong parallels between masking and crowding suggest similar underlying mechanism; however, temporal factors in masking additionally modulate performance, acting as an additional grouping cue.

Foveal target repetitions reduce crowding.

Crowding is the limitation of peripheral vision by clutter. Objects that are easily identified when presented in isolation are hard to identify when presented flanked by similar close-by objects. It is often assumed that the signal of a crowded target is irretrievably lost because it is combined with the signals of the flankers. Here, we asked whether a target signal can be enhanced (or retrieved) by items presented far outside the crowding region. We investigated whether remote items matching a peripheral, crowded target enhanced discrimination compared to remote items not matching the target. In Experiment 1, we presented the remote item at different locations in the visual field and found that, when presented in the fovea, a matching remote item improved target discrimination compared to a nonmatching remote item. In Experiment 2, we varied stimulus onset asynchronies between target and remote items and found a strong effect when the remote item was presented simultaneously with the target. The effect diminished (or was absent) with increasing temporal separation. In Experiment 3, we asked whether semantic knowledge of a target was sufficient to improve target discrimination and found that this was not the case. We conclude that crowded target signals are not irretrievably lost. Rather, their accurate recognition is facilitated in the presence of remote items that match the target. We suggest that long-range grouping mechanisms underlie this "uncrowding" effect.

Crowding is reduced by onset transients in the target object (but not in the flankers).

In peripheral vision, objects that are visible in isolation become difficult to identify in clutter. This crowding effect is typically strong when objects are similar in a given dimension (e.g., color) and weak when they differ. Here we examine the selectivity of crowding for temporal differences-namely, the transient signals associated with object onsets and offsets. Observers judged the orientation of a peripheral Gabor target surrounded by four flankers. Midway through each trial, selected elements "blinked" off and on again. Performance was poor (crowding was strong) when all Gabors blinked simultaneously or when only the flankers blinked. In contrast, performance improved dramatically when the target alone blinked despite the continued presence of the flankers. This asymmetric release from crowding occurs across a range of blink durations and target-flanker separations. A similar release was found when the target onset was delayed relative to the flanker onsets, though varying the target offset had little effect. This suggests that blinks (composed of offset and onset events) reduce crowding specifically because they separate target and flanker onsets. Finally, with luminance pedestals added to the Gabors, crowding was reduced by blinks in the target pedestal only when the target Gabor was present; pedestal blinks before/after the stimulus Gabors (as precues/postcues) had no effect. That is, transients do not simply cue the target location. The asymmetry of this effect (reduced crowding with target transients, no effect with flanker transients) also precludes explanations based on similarity or grouping. We attribute our findings to the isolation of the target in transient (vs. sustained) visual channels.

Attention in redundancy masking.

Peripheral vision is limited due to several factors, such as visual resolution, crowding, and attention. When attention is not directed towards a stimulus, detection, discrimination, and identification are often compromised. Recent studies have found a new phenomenon that strongly limits peripheral vision, "redundancy masking". In redundancy masking, the number of perceived items in repeating patterns is reduced. For example, when presenting three lines in the peripheral visual field and asking participants to report the number of lines, often only two lines are reported. Here, we investigated what role attention plays in redundancy masking. If redundancy masking was due to limited attention to the target, it should be stronger when less attention is allocated to the target, and absent when attention is maximally focused on the target. Participants were presented with line arrays and reported the number of lines in three cueing conditions (i.e., single cue, double cue, and no cue). Redundancy masking was observed in all cueing conditions, with observers reporting fewer lines than presented in the single, double, and no cue conditions. These results suggest that redundancy masking is not due to limited attention. The number of lines reported was closer to the correct number of lines in the single compared to the double and the no cue conditions, suggesting that reduced attention additionally compromised stimulus discrimination, and replicating typical effects of diminished attention. Taken together, our results suggest that the extent of attention to peripherally presented stimuli modulates discrimination performance, but does not account for redundancy masking.

The effect of processing partial information in dynamic face perception.

Face perception is a major topic in vision research. Most previous research has concentrated on (holistic) spatial representations of faces, often with static faces as stimuli. However, faces are highly dynamic stimuli containing important temporal information. How sensitive humans are regarding temporal information in dynamic faces is not well understood. Studies investigating temporal information in dynamic faces usually focus on the processing of emotional expressions. However, faces also contain relevant temporal information without any strong emotional expression. To investigate cues that modulate human sensitivity to temporal order, we utilized muted dynamic neutral face videos in two experiments. We varied the orientation of the faces (upright and inverted) and the presence/absence of eye blinks as partial dynamic cues. Participants viewed short, muted, monochromic videos of models vocalizing a widely known text (National Anthem). Videos were played either forward (in the correct temporal order) or backward. Participants were asked to determine the direction of the temporal order for each video, and (at the end of the experiment) whether they had understood the speech. We found that face orientation, and the presence/absence of an eye blink affected sensitivity, criterion (bias) and reaction time: Overall, sensitivity was higher for upright compared to inverted faces, and in the condition where an eye blink was present compared to the condition without an eye blink. Reaction times were mostly faster in the conditions with higher sensitivity. A bias to report inverted faces as 'backward' observed in Experiment I, where upright and inverted faces were presented randomly interleaved within each block, was absent when presenting upright and inverted faces in different blocks in Experiment II. Language comprehension results revealed that there was higher sensitivity when understanding the speech compared to not understanding the speech in both experiments. Taken together, our results showed higher sensitivity with upright compared to inverted faces, suggesting that the perception of dynamic, task-relevant information was superior with the canonical orientation of the faces. Furthermore, partial information coming from eye blinks, in addition to mouth movements, seemed to play a significant role in dynamic face perception, both when faces were presented upright and inverted. We suggest that studying the perception of facial dynamics beyond emotional expressions will help us to better understand the mechanisms underlying the temporal integration of facial information from different -partial and holistic- sources, and that our results show how different strategies, depending on the available information, are employed by human observers when judging the temporal order of faces.

When crowding of crowding leads to uncrowding.

In object recognition, features are thought to be processed in a hierarchical fashion from low-level analysis (edges and lines) to complex figural processing (shapes and objects). Here, we show that figural processing determines low-level processing. Vernier offset discrimination strongly deteriorated when we embedded a vernier in a square. This is a classic crowding effect. Surprisingly, crowding almost disappeared when additional squares were added. We propose that figural interactions between the squares precede low-level suppression of the vernier by the single square, contrary to hierarchical models of object recognition.

Foveal vision determines the perceived emotion of face ensembles.

People can extract summary statistical information from groups of similar objects, an ability called ensemble perception. However, not every object in a group is weighted equally. For example, in ensemble emotion perception, faces far from fixation were weighted less than faces close to fixation. Yet the contribution of foveal input in ensemble emotion perception is still unclear. In two experiments, groups of faces with varying emotions were presented for 100 ms at three different eccentricities (0°, 3°, 8°). Observers reported the perceived average emotion of the group. In two conditions, stimuli consisted of a central face flanked by eight faces (flankers) (central-present condition) and eight faces without the central face (central-absent condition). In the central-present condition, the emotion of the central face was either congruent or incongruent with that of the flankers. In Experiment 1, flanker emotions were uniform (identical flankers); in Experiment 2 they were varied. In both experiments, performance in the central-present condition was superior at 3° compared to 0° and 8°. At 0°, performance was superior in the central-absent (i.e., no foveal input) compared to the central-present condition. Poor performance in the central-present condition was driven by the incongruent condition where the foveal face strongly biased responses. At 3° and 8°, performance was comparable between central-present and central-absent conditions. Our results showed how foveal input determined the perceived emotion of face ensembles, suggesting that ensemble perception fails when salient target information is available in central vision.