Searching for serial dependencies in the brain.

Identifying the neural correlates of visual serial dependence has lagged behind the behavioral understanding. A new study in PLOS Biology provides a model of interpreting the complex relationship between physiology and behavior in studies of serial dependence.

Super recognizers: Increased sensitivity or reduced biases? Insights from serial dependence.

Super recognizers (SRs) are people that exhibit a naturally occurring superiority for processing facial identity. Despite the increase of SR research, the mechanisms underlying their exceptional abilities remain unclear. Here, we investigated whether the enhanced facial identity processing of SRs could be attributed to the lack of sequential effects, such as serial dependence. In serial dependence, perception of stimulus features is assimilated toward stimuli presented in previous trials. This constant error in visual perception has been proposed as a mechanism that promotes perceptual stability in everyday life. We hypothesized that an absence of this constant source of error in SRs could account for their superior processing-potentially in a domain-general fashion. We tested SRs (n = 17) identified via a recently proposed diagnostic framework (Ramon, 2021) and age-matched controls (n = 20) with two experiments probing serial dependence in the face and shape domains. In each experiment, observers were presented with randomly morphed face identities or shapes and were asked to adjust a face's identity or a shape to match the stimulus they saw. We found serial dependence in controls and SRs alike, with no difference in its magnitude across groups. Interestingly, we found that serial dependence impacted the performance of SRs more than that of controls. Taken together, our results show that enhanced face identity processing skills in SRs cannot be attributed to the lack of serial dependence. Rather, serial dependence, a beneficial nested error in our visual system, may in fact further stabilize the perception of SRs and thus enhance their visual processing proficiency.

Serial dependence in visual perception: A meta-analysis and review.

Positive sequential dependencies are phenomena in which actions, perception, decisions, and memory of features or objects are systematically biased toward visual experiences from the recent past. Among many labels, serial dependencies have been referred to as priming, sequential dependencies, sequential effects, or serial effects. Despite extensive research on the topic, the field still lacks an operational definition of what counts as serial dependence. In this meta-analysis, we review the vast literature on serial dependence and quantitatively assess its key diagnostic characteristics across several different domains of visual perception. The meta-analyses fully characterize serial dependence in orientation, face, and numerosity perception. They show that serial dependence is defined by four main kinds of tuning: serial dependence decays with time (temporal-tuning), it depends on relative spatial location (spatial-tuning), it occurs only between similar features and objects (feature-tuning), and it is modulated by attention (attentional-tuning). We also review studies of serial dependence that report single observer data, highlighting the importance of individual differences in serial dependence. Finally, we discuss a range of outstanding questions and novel research avenues that are prompted by the meta-analyses. Together, the meta-analyses provide a full characterization of serial dependence as an operationally defined family of visual phenomena, and they outline several of the key diagnostic criteria for serial dependence that should serve as guideposts for future research.

Capsule networks as recurrent models of grouping and segmentation.

Classically, visual processing is described as a cascade of local feedforward computations. Feedforward Convolutional Neural Networks (ffCNNs) have shown how powerful such models can be. However, using visual crowding as a well-controlled challenge, we previously showed that no classic model of vision, including ffCNNs, can explain human global shape processing. Here, we show that Capsule Neural Networks (CapsNets), combining ffCNNs with recurrent grouping and segmentation, solve this challenge. We also show that ffCNNs and standard recurrent CNNs do not, suggesting that the grouping and segmentation capabilities of CapsNets are crucial. Furthermore, we provide psychophysical evidence that grouping and segmentation are implemented recurrently in humans, and show that CapsNets reproduce these results well. We discuss why recurrence seems needed to implement grouping and segmentation efficiently. Together, we provide mutually reinforcing psychophysical and computational evidence that a recurrent grouping and segmentation process is essential to understand the visual system and create better models that harness global shape computations.

Global and high-level effects in crowding cannot be predicted by either high-dimensional pooling or target cueing.

In visual crowding, the perception of a target deteriorates in the presence of nearby flankers. Traditionally, target-flanker interactions have been considered as local, mostly deleterious, low-level, and feature specific, occurring when information is pooled along the visual processing hierarchy. Recently, a vast literature of high-level effects in crowding (grouping effects and face-holistic crowding in particular) led to a different understanding of crowding, as a global, complex, and multilevel phenomenon that cannot be captured or explained by simple pooling models. It was recently argued that these high-level effects may still be captured by more sophisticated pooling models, such as the Texture Tiling model (TTM). Unlike simple pooling models, the high-dimensional pooling stage of the TTM preserves rich information about a crowded stimulus and, in principle, this information may be sufficient to drive high-level and global aspects of crowding. In addition, it was proposed that grouping effects in crowding may be explained by post-perceptual target cueing. Here, we extensively tested the predictions of the TTM on the results of six different studies that highlighted high-level effects in crowding. Our results show that the TTM cannot explain any of these high-level effects, and that the behavior of the model is equivalent to a simple pooling model. In addition, we show that grouping effects in crowding cannot be predicted by post-perceptual factors, such as target cueing. Taken together, these results reinforce once more the idea that complex target-flanker interactions determine crowding and that crowding occurs at multiple levels of the visual hierarchy.

Selective age-related changes in orientation perception.

Orientation perception is a fundamental property of the visual system and an important basic processing stage for visual scene perception. Neurophysiological studies have found broader tuning curves and increased noise in orientation-selective neurons of senescent monkeys and cats, results that suggest an age-related decline in orientation perception. However, behavioral studies in humans have found no evidence for such decline, with performance being comparable for younger and older participants in orientation detection and discrimination tasks. Crucially, previous behavioral studies assessed performance for cardinal orientation only, and it is well known that the human visual system prefers cardinal over oblique orientations, a phenomenon called the oblique effect. We hypothesized that age-related changes depend on the orientation tested. In two experiments, we investigated orientation discrimination and reproduction for a large range of cardinal and oblique orientations in younger and older adults. We found substantial age-related decline for oblique but not for cardinal orientations, thus demonstrating that orientation perception selectively declines for oblique orientations. Taken together, our results serve as the missing link between previous neurophysiological and human behavioral studies on orientation perception in healthy aging.

Visual crowding in driving.

Visual crowding-the deleterious influence of nearby objects on object recognition-is considered to be a major bottleneck for object recognition in cluttered environments. Although crowding has been studied for decades with static and artificial stimuli, it is still unclear how crowding operates when viewing natural dynamic scenes in real-life situations. For example, driving is a frequent and potentially fatal real-life situation where crowding may play a critical role. In order to investigate the role of crowding in this kind of situation, we presented observers with naturalistic driving videos and recorded their eye movements while they performed a simulated driving task. We found that the saccade localization on pedestrians was impacted by visual clutter, in a manner consistent with the diagnostic criteria of crowding (Bouma's rule of thumb, flanker similarity tuning, and the radial-tangential anisotropy). In order to further confirm that altered saccadic localization is a behavioral consequence of crowding, we also showed that crowding occurs in the recognition of cluttered pedestrians in a more conventional crowding paradigm. We asked participants to discriminate the gender of pedestrians in static video frames and found that the altered saccadic localization correlated with the degree of crowding of the saccade targets. Taken together, our results provide strong evidence that crowding impacts both recognition and goal-directed actions in natural driving situations.

What crowding can tell us about object representations.

In crowding, perception of a target usually deteriorates when flanking elements are presented next to the target. Surprisingly, adding further flankers can lead to a release from crowding. In previous work we showed that, for example, vernier offset discrimination at 9° of eccentricity deteriorated when a vernier was embedded in a square. Adding further squares improved performance. The more squares presented, the better the performance, extending across 20° of the visual field. Here, we show that very similar results hold true for shapes other than squares, including unfamiliar, irregular shapes. Hence, uncrowding is not restricted to simple and familiar shapes. Our results provoke the question of whether any type of shape is represented at any location in the visual field. Moreover, small changes in the orientation of the flanking shapes led to strong increases in crowding strength. Hence, highly specific shape-specific interactions across large parts of the visual field determine vernier acuity.

Intact crowding and temporal masking in dyslexia.

Phonological deficits in dyslexia are well documented. However, there is an ongoing discussion about whether visual deficits limit the reading skills of people with dyslexia. Here, we investigated visual crowding and backward masking. We presented a Vernier (i.e., two vertical bars slightly offset to the left or right) and asked observers to indicate the offset direction. Vernier stimuli are visually similar to letters and are strongly affected by crowding, even in the fovea. To increase task difficulty, Verniers are often followed by a mask (i.e., backward masking). We measured Vernier offset discrimination thresholds for the basic Vernier task, under crowding, and under backward masking, in students with dyslexia (n = 19) and age and intelligence matched students (n = 27). We found no group differences in any of these conditions. Controls with fast visual processing (good backward masking performance), were faster readers. By contrast, no such correlation was found among the students with dyslexia, suggesting that backward masking does not limit their reading efficiency. These findings indicate that neither elevated crowding nor elevated backward masking pose a bottleneck to reading skills of people with dyslexia.

Crowding, grouping, and object recognition: A matter of appearance.

In crowding, the perception of a target strongly deteriorates when neighboring elements are presented. Crowding is usually assumed to have the following characteristics. (a) Crowding is determined only by nearby elements within a restricted region around the target (Bouma's law). (b) Increasing the number of flankers can only deteriorate performance. (c) Target-flanker interference is feature-specific. These characteristics are usually explained by pooling models, which are well in the spirit of classic models of object recognition. In this review, we summarize recent findings showing that crowding is not determined by the above characteristics, thus, challenging most models of crowding. We propose that the spatial configuration across the entire visual field determines crowding. Only when one understands how all elements of a visual scene group with each other, can one determine crowding strength. We put forward the hypothesis that appearance (i.e., how stimuli look) is a good predictor for crowding, because both crowding and appearance reflect the output of recurrent processing rather than interactions during the initial phase of visual processing.

No priming for global motion in crowding.

There is psychophysical evidence that low-level priming, e.g., from oriented gratings, as well as high-level semantic priming, survives crowding. We investigated priming for global translational motion in crowded and noncrowded conditions. The results indicated that reliable motion priming occurs in the noncrowded condition, but motion priming does not survive crowding. Crowding persisted despite variations in the direction of the flankers with respect to the prime's direction. Motion priming was still absent under crowding when 85% of the flankers moved in the same direction as the prime. Crowding also persisted despite variations in the speed of the flankers relative to the prime even when the flankers' speed was four times slower than the speed of the prime. However, a priming effect was evident when the prime's spatial location was precued and its distance to the flankers increased, suggesting a release from crowding. These results suggest that transient attention induced by precueing the spatial location of the prime may improve subjects' ability to discriminate its direction. Spatial cueing could act to decrease the integration field, thereby diminishing the influence of nearby distracters. In an additional experiment in which we used fewer flankers, we found a priming effect under conditions in which the interelement distance varied between flankers and prime. Overall, the results suggest that motion priming is strongly affected by crowding, but transient attention can partially retrieve such facilitation.

Release of crowding by pattern completion.

In crowding, target perception deteriorates in the presence of flanking elements. Crowding is classically explained by low-level mechanisms such as pooling or feature substitution. However, we have previously shown that perceptual grouping between the target and flankers, rather than low-level mechanisms, determines crowding. There are many grouping cues that can determine crowding, such as low- and high-level feature similarity, low- and high-level pattern regularity, and good Gestalt. Here we show that pattern completion, another grouping cue that is important for crowding in foveal vision, is also important in peripheral vision. We also describe computer simulations that show how pattern completion, and crowding in general, can be partly explained by recurrent processing.

How color, regularity, and good Gestalt determine backward masking.

The strength of visual backward masking depends on the stimulus onset asynchrony (SOA) between target and mask. Recently, it was shown that the conjoint spatial layout of target and mask is as crucial as SOA. Particularly, masking strength depends on whether target and mask group with each other. The same is true in crowding where the global spatial layout of the flankers and target-flanker grouping determine crowding strength. Here, we presented a vernier target followed by different flanker configurations at varying SOAs. Similar to crowding, masking of a red vernier target was strongly reduced for arrays of 10 green compared with 10 red flanking lines. Unlike crowding, single green lines flanking the red vernier showed strong masking. Irregularly arranged flanking lines yielded stronger masking than did regularly arranged lines, again similar to crowding. While cuboid flankers reduced crowding compared with single lines, this was not the case in masking. We propose that, first, masking is reduced when the flankers are part of a larger spatial structure. Second, spatial factors counteract color differences between the target and the flankers. Third, complex Gestalts, such as cuboids, seem to need longer processing times to show ungrouping effects as observed in crowding. Strong parallels between masking and crowding suggest similar underlying mechanism; however, temporal factors in masking additionally modulate performance, acting as an additional grouping cue.

Intact Serial Dependence in Schizophrenia: Evidence from an Orientation Adjustment Task.

BACKGROUND AND HYPOTHESIS: For a long time, it was proposed that schizophrenia (SCZ) patients rely more on sensory input and less on prior information, potentially leading to reduced serial dependence-ie, a reduced influence of prior stimuli in perceptual tasks. However, existing evidence is constrained to a few paradigms, and whether reduced serial dependence reflects a general characteristic of the disease remains unclear. STUDY DESIGN: We investigated serial dependence in 26 SCZ patients and 27 healthy controls (CNT) to evaluate the influence of prior stimuli in a classic visual orientation adjustment task, a paradigm not previously tested in this context. STUDY RESULTS: As expected, the CNT group exhibited clear serial dependence, with systematic biases toward the orientation of stimuli shown in the preceding trials. Serial dependence in SCZ patients was largely comparable to that in the CNT group. CONCLUSIONS: These findings challenge the prevailing notion of reduced serial dependence in SCZ, suggesting that observed differences between healthy CNT and patients may depend on aspects of perceptual or cognitive processing that are currently not understood.

When crowding of crowding leads to uncrowding.

In object recognition, features are thought to be processed in a hierarchical fashion from low-level analysis (edges and lines) to complex figural processing (shapes and objects). Here, we show that figural processing determines low-level processing. Vernier offset discrimination strongly deteriorated when we embedded a vernier in a square. This is a classic crowding effect. Surprisingly, crowding almost disappeared when additional squares were added. We propose that figural interactions between the squares precede low-level suppression of the vernier by the single square, contrary to hierarchical models of object recognition.

Three's a crowd: Fast ensemble perception of first impressions of trustworthiness.

Trustworthiness impressions are fundamental social judgements with far-reaching consequences in many aspects of society, including criminal justice, leadership selection and partner preferences. Thus far, most research has focused on facial characteristics that make a face individually appear more or less trustworthy. However, in everyday life, faces are not always perceived in isolation but are often encountered in crowds. It has been proposed that we deal with the large amount of facial information in a group by extracting summary statistics of the crowd, a phenomenon called ensemble perception. Prior research showed that ensemble perception occurs for various facial features, such as emotional expression, facial identity, and attractiveness. Here, we investigated whether observers can integrate the level of trustworthiness from multiple faces to extract an average impression of the crowd. Across four studies, participants were presented with crowds of faces and were asked to report their average level of trustworthiness with an adjustment (Experiment 1) and a rating task (Experiments 2 and 3). Participants were able to extract an ensemble perception of trustworthiness impressions from multiple faces. Moreover, observers were able to form a summary statistic of trustworthiness impressions from a group of faces as quickly as 250 ms (Experiment 4). Taken together, these results demonstrate that ensemble perception can occur at the level of impressions of trustworthiness. Thus, these critical social judgements not only occur for individual faces but are also integrated into a unique ensemble impression of crowds. Our findings contribute to the development of a more ecological approach to the study of trust impressions, since they provide an understanding of trustworthiness judgements not only on an individual level, but on a much broader social group level. Furthermore, our results drive forward new theory because they demonstrate for the first time that ensemble representations cover a broad range of phenomena than previously recognized, including complex high-level facial trait judgements such as trustworthiness impressions.

Serial Dependence in Dermatological Judgments.

Serial Dependence is a ubiquitous visual phenomenon in which sequentially viewed images appear more similar than they actually are, thus facilitating an efficient and stable perceptual experience in human observers. Although serial dependence is adaptive and beneficial in the naturally autocorrelated visual world, a smoothing perceptual experience, it might turn maladaptive in artificial circumstances, such as medical image perception tasks, where visual stimuli are randomly sequenced. Here, we analyzed 758,139 skin cancer diagnostic records from an online app, and we quantified the semantic similarity between sequential dermatology images using a computer vision model as well as human raters. We then tested whether serial dependence in perception occurs in dermatological judgments as a function of image similarity. We found significant serial dependence in perceptual discrimination judgments of lesion malignancy. Moreover, the serial dependence was tuned to the similarity in the images, and it decayed over time. The results indicate that relatively realistic store-and-forward dermatology judgments may be biased by serial dependence. These findings help in understanding one potential source of systematic bias and errors in medical image perception tasks and hint at useful approaches that could alleviate the errors due to serial dependence.

Serial dependence in perception across naturalistic generative adversarial network-generated mammogram.

Purpose: Human perception and decisions are biased toward previously seen stimuli. This phenomenon is known as serial dependence and has been extensively studied for the last decade. Recent evidence suggests that clinicians' judgments of mammograms might also be impacted by serial dependence. However, the stimuli used in previous psychophysical experiments on this question, consisting of artificial geometric shapes and healthy tissue backgrounds, were unrealistic. We utilized realistic and controlled generative adversarial network (GAN)-generated radiographs to mimic images that clinicians typically encounter. Approach: Mammograms from the digital database for screening mammography (DDSM) were utilized to train a GAN. This pretrained GAN was then adopted to generate a large set of authentic-looking simulated mammograms: 20 circular morph continuums, each with 147 images, for a total of 2940 images. Using these stimuli in a standard serial dependence experiment, participants viewed a random GAN-generated mammogram on each trial and subsequently matched the GAN-generated mammogram encountered using a continuous report. The characteristics of serial dependence from each continuum were analyzed. Results: We found that serial dependence affected the perception of all naturalistic GAN-generated mammogram morph continuums. In all cases, the perceptual judgments of GAN-generated mammograms were biased toward previously encountered GAN-generated mammograms. On average, perceptual decisions had 7% categorization errors that were pulled in the direction of serial dependence. Conclusions: Serial dependence was found even in the perception of naturalistic GAN-generated mammograms created by a GAN. This supports the idea that serial dependence could, in principle, contribute to decision errors in medical image perception tasks.

Grouping, pooling, and when bigger is better in visual crowding.

In crowding, perception of a target is strongly deteriorated by nearby elements. Crowding is often explained by pooling models predicting that adding flankers increases crowding. In contrast, the centroid hypothesis proposes that adding flankers decreases crowding--"bigger is better." In foveal vision, we have recently shown that adding flankers can increase or decrease crowding depending on whether the target groups or ungroups from the flankers. We have further shown how configural effects, such as good and global Gestalt, determine crowding. Foveal and peripheral crowding do not always reveal the same characteristics. Here, we show that the very same grouping and Gestalt results of foveal vision are also found in the periphery. These results can neither be explained by simple pooling nor by centroid models. We discuss when bigger is better and how grouping might shape crowding.

Ensemble perception: Stacking the hay to find the needle.

The visual clutter we constantly encounter in the world limits object recognition, a phenomenon known as visual crowding. A new study shows that ensemble perception counters this by condensing redundant information into summary statistical representations, which thus releases visual crowding's effect on individual objects.