Masking, crowding, and grouping: Connecting low and mid-level vision.

An important task for vision science is to build a unitary framework of low- and mid-level vision. As a step on this way, our study examined differences and commonalities between masking, crowding and grouping-three processes that occur through spatial interactions between neighbouring elements. We measured contrast thresholds as functions of inter-element spacing and eccentricity for Gabor detection, discrimination and contour integration, using a common stimulus grid consisting of nine Gabor elements. From these thresholds, we derived a) the baseline contrast necessary to perform each task and b) the spatial extent over which task performance was stable. This spatial window can be taken as an indicator of field size, where elements that fall within a putative field are readily combined. We found that contrast thresholds were universally modulated by inter-element distance, with a shallower and inverted effect for grouping compared with masking and crowding. Baseline contrasts for detecting stimuli and discriminating their properties were positively linked across the tested retinal locations (parafovea and near periphery), whereas those for integrating elements and discriminating their properties were negatively linked. Meanwhile, masking and crowding spatial windows remained uncorrelated across eccentricity, although they were correlated across participants. This suggests that the computation performed by each type of visual field operates over different distances that co-varies across observers, but not across retinal locations. Contrast-processing units may thus lie at the core of the shared idiosyncrasies across tasks reported in many previous studies, despite the fundamental differences in the extent of their spatial windows.

Characterizing the in-out asymmetry in visual crowding.

An object's processing is impaired by the presence of nearby clutter. Several distinct mechanisms, such as masking and visual crowding, are thought to contribute to such flanker-induced interference. It is therefore important to determine which mechanism is operational in any given situation. Previous studies have proposed that the in-out asymmetry (IOA), where a peripheral flanker interferes with the target more than a foveal flanker, is diagnostic of crowding. However, several studies have documented inconsistencies in the occurrence of this asymmetry, particularly at locations beyond the horizontal meridian, casting doubt on its ability to delineate crowding. In this study, to determine if IOA is diagnostic of crowding, we extensively charted its properties. We asked a relatively large set of participants (n = 38) to identify a briefly presented peripheral letter flanked by a single inward or outward letter at one of four locations. We also manipulated target location uncertainty and attentional allocation by blocking, randomizing or pre-cueing the target location. Using multilevel Bayesian regression analysis, we found robust IOA at all locations, although its strength was modulated by target location, location uncertainty, and attentional allocation. Our findings suggest that IOA can be an excellent marker of crowding, to the extent that it is not observed in other flanker-interference mechanisms, such as masking.

Emergence of crowding: The role of contrast and orientation salience.

Crowding causes difficulties in judging attributes of an object surrounded by other objects. We investigated crowding for stimuli that isolated either S-cone or luminance mechanisms or combined them. By targeting different retinogeniculate mechanisms with contrast-matched stimuli, we aim to determine the earliest site at which crowding emerges. Discrimination was measured in an orientation judgment task where Gabor targets were presented parafoveally among flankers. In the first experiment, we assessed flanked and unflanked orientation discrimination thresholds for pure S-cone and achromatic stimuli and their combinations. In the second experiment, to capture individual differences, we measured unflanked detection and orientation sensitivity, along with performance under flanker interference for stimuli containing luminance only or combined with S-cone contrast. We confirmed that orientation sensitivity was lower for unflanked S-cone stimuli. When flanked, the pattern of results for S-cone stimuli was the same as for achromatic stimuli with comparable (i.e. low) contrast levels. We also found that flanker interference exhibited a genuine signature of crowding only when orientation discrimination threshold was reliably surpassed. Crowding, therefore, emerges at a stage that operates on signals representing task-relevant featural (here, orientation) information. Because luminance and S-cone mechanisms have very different spatial tuning properties, it is most parsimonious to conclude that crowding takes place at a neural processing stage after they have been combined.

Too little, too late, and in the wrong place: Alpha band activity does not reflect an active mechanism of selective attention.

Selective attention focuses visual processing on relevant stimuli in order to allow for adaptive behaviour despite an abundance of distracting information. It has been proposed that increases in alpha band (8-12 â€‹Hz) amplitude reflect an active mechanism for distractor suppression. If this were the case, increases in alpha band amplitude should be succeeded by a decrease in distractor processing. Surprisingly, this connection has not been tested directly; specifically, studies that have investigated changes in alpha band after attention-directing cues have not directly assessed the neuronal processing of distractors. We concurrently recorded alpha activity and steady-state visual evoked potentials (SSVEPs) to assess the processing of target and distractor stimuli. In two experiments, participants covertly shifted attention to one of two letter streams (left or right) to detect infrequent target letters 'X' while ignoring the other stream. In line with previous findings, alpha band amplitudes contralateral to the unattended location increased compared to a pre-cue baseline. However, there was no suppression of SSVEP amplitudes elicited by unattended stimuli, while there was a pronounced enhancement of SSVEPs elicited by attended stimuli. Furthermore, and crucially, changes in alpha band amplitude during attention shifts did not precede those in SSVEPs and hit rates in both experiments, indicating that changes in alpha band amplitudes are likely to be a consequence of attention shifts rather than the other way around. We conclude that these findings contradict the notion that alpha band activity reflects mechanisms that have a causal role in the allocation of selective attention.

A simple rule to describe interactions between visual categories.

Humans can rapidly categorise visual objects when presented in isolation. However, in everyday life we encounter multiple objects at the same time. Far less is known about how simultaneously active object representations interact. We examined such interactions by asking participants to categorise a target object at the basic (Experiment 1) or the superordinate (Experiment 2) level while the representation of another object was still active. We found that the "prime" object strongly modulated the response to the target implying that the prime's category was rapidly and automatically accessed, influencing subsequent categorical processing. Using drift diffusion modelling, we show that a prime, whose category is different from that of the target, interferes with target processing primarily during the evidence accumulation stage. This suggests that the state of category-processing neurons is altered by an active representation and this modifies the processing of other categories. Interestingly, the strength of interference increases with the similarity between the distractor and the target category. Considering these results and previous studies, we propose a general principle that category interactions are determined by the distance from a distractor's representation to the target's task-relevant categorical boundary. We argue that this principle arises from the specific architectural organisation of categories in the brain.

Conscious updating is a rhythmic process.

As the visual world changes, its representation in our consciousness must be constantly updated. Given that the external changes are continuous, it appears plausible that conscious updating is continuous as well. Alternatively, this updating could be periodic, if, for example, its implementation at the neural level relies on oscillatory activity. The flash-lag illusion, where a briefly presented flash in the vicinity of a moving object is misperceived to lag behind the moving object, is a useful tool for studying the dynamics of conscious updating. Here, we show that the trial-by-trial variability in updating, measured by the flash-lag effect (FLE), is highly correlated with the phase of spontaneous EEG oscillations in occipital (5-10 Hz) and frontocentral (12-20 Hz) cortices just around the reference event (flash onset). Further, the periodicity in each region independently influences the updating process, suggesting a two-stage periodic mechanism. We conclude that conscious updating is not continuous; rather, it follows a rhythmic pattern.

The temporal evolution of coarse location coding of objects: evidence for feedback.

Objects occupy space. How does the brain represent the spatial location of objects? Retinotopic early visual cortex has precise location information but can only segment simple objects. On the other hand, higher visual areas can resolve complex objects but only have coarse location information. Thus coarse location of complex objects might be represented by either (a) feedback from higher areas to early retinotopic areas or (b) coarse position encoding in higher areas. We tested these alternatives by presenting various kinds of first- (edge-defined) and second-order (texture) objects. We applied multivariate classifiers to the pattern of EEG amplitudes across the scalp at a range of time points to trace the temporal dynamics of coarse location representation. For edge-defined objects, peak classification performance was high and early and thus attributable to the retinotopic layout of early visual cortex. For texture objects, it was low and late. Crucially, despite these differences in peak performance and timing, training a classifier on one object and testing it on others revealed that the topography at peak performance was the same for both first- and second-order objects. That is, the same location information, encoded by early visual areas, was available for both edge-defined and texture objects at different time points. These results indicate that locations of complex objects such as textures, although not represented in the bottom-up sweep, are encoded later by neural patterns resembling the bottom-up ones. We conclude that feedback mechanisms play an important role in coarse location representation of complex objects.

Categorical membership modulates crowding: evidence from characters.

Visual crowding is generally thought to affect recognition mostly or only at the level of feature combination. Calling this assertion into question, recent studies have shown that if a target object and its flankers belong to different categories crowding is weaker than if they belong to the same category. Nevertheless, these results can be explained in terms of featural differences between categories. The current study tests if category-level (i.e., high-level) interference in crowding occurs when featural differences are controlled for. First, replicating previous results, we found lower critical spacing for targets and flankers belonging to different categories. Second, we observed the same, albeit weaker, category-specific effect when objects in both categories had the exact same feature set, suggesting that category-specific effects persist even when featural differences are fully controlled for. Third, we manipulated the semantic content of the flankers while keeping their feature set constant, by using upright or rotated objects, and found that meaning modulated crowding. An exclusively feature-based account of crowding would predict no differences due to such changes in meaning. We conclude that crowding results from not only the well-documented feature-level interactions but also additional interactions at a level where objects are grouped by meaning.

The Bouma law of crowding, revised: critical spacing is equal across parts, not objects.

Crowding is the inability to identify an object among flankers in the periphery. It is due to inappropriate incorporation of features from flanking objects in perception of the target. Crowding is characterized by measuring critical spacing, the minimum distance needed between a target and flankers to allow recognition. The existing Bouma law states that, at a given point and direction in the visual field, critical spacing, measured from the center of a target object to the center of a similar flanking object, is the same for all objects (Pelli & Tillman, 2008). Because flipping an object about its center preserves its center-to-center spacing to other objects, according to the Bouma law, crowding should be unaffected. However, because crowding is a result of feature combination, the location of features within an object might matter. In a series of experiments, we find that critical spacing is affected by the location of features within the flanker. For some flankers, a flip greatly reduces crowding even though it maintains target-flanker spacing and similarity. Our results suggest that the existing Bouma law applies to simple one-part objects, such as a single roman letter or a Gabor patch. Many objects consist of multiple parts; for example, a word is composed of multiple letters that crowd each other. To cope with such complex objects, we revise the Bouma law to say that critical spacing is equal across parts, rather than objects. This accounts for old and new findings.

The time course of Temporal Binding in social and nonsocial interactions.

Temporal Binding (TB) is the subjective compression of action-effect intervals. While the effects of nonsocial actions are highly predictable, it is not the case when interacting with conspecifics, who often act under their own volition, at a time of their choosing. Given the relative differences in action-effect predictability in non-social and social interactions, it is plausible that TB and its properties differ across these situations. To examine this, in two experiments, we compared the time course of TB in social and nonsocial interactions, systematically varying action-effect intervals (200-2,100 ms). Participants were told they were (a) interacting with another person via a live webcam, who was in fact a confederate (social condition), (b) interacting with pre-recorded videos (nonsocial condition), or (c) observing two pre-recorded videos (control condition; Experiment 2). Results across experiments showed greater TB for social compared to nonsocial conditions, and the difference was proportional to the action-effect intervals. Further, in Experiment 1, TB was consistently observed throughout the experiment for social interactions, whereas nonsocial TB decreased from the first to the second half of the experiment. In Experiment 2, the nonsocial condition did not differ from control, whereas the social condition did, exhibiting enhanced binding. We argue these results suggest that the sociality of an interaction modulates the 'internal clock' of time perception.

Fundamental units of numerosity estimation.

Humans can approximately enumerate a large number of objects at a single glance. While several mechanisms have been proposed to account for this ability, the fundamental units over which they operate remain unclear. Previous studies have argued that estimation mechanisms act only on topologically distinct units or on units formed by spatial grouping cues such as proximity and connectivity, but not on units grouped by similarity. Over four experiments, we tested this claim by systematically assessing and demonstrating that similarity grouping leads to underestimation, just as spatial grouping does. Ungrouped objects with the same low-level properties as grouped objects did not cause underestimation. Further, the underestimation caused by spatial and similarity grouping was additive, suggesting that these grouping processes operate independently. These findings argue against the proposal that estimation mechanisms operate solely on topological units. Instead, we conclude that estimation processes act on representations constructed after Gestalt grouping principles, whether similarity based or spatial, have organised incoming visual input.

Neural and behavioral signatures of the multidimensionality of manipulable object processing.

Understanding how we recognize objects requires unravelling the variables that govern the way we think about objects and the neural organization of object representations. A tenable hypothesis is that the organization of object knowledge follows key object-related dimensions. Here, we explored, behaviorally and neurally, the multidimensionality of object processing. We focused on within-domain object information as a proxy for the decisions we typically engage in our daily lives - e.g., identifying a hammer in the context of other tools. We extracted object-related dimensions from subjective human judgments on a set of manipulable objects. We show that the extracted dimensions are cognitively interpretable and relevant - i.e., participants are able to consistently label them, and these dimensions can guide object categorization; and are important for the neural organization of knowledge - i.e., they predict neural signals elicited by manipulable objects. This shows that multidimensionality is a hallmark of the organization of manipulable object knowledge.

Visual field asymmetries in numerosity processing.

A small number of objects can be rapidly and accurately enumerated, whereas a larger number of objects can only be approximately enumerated. These subitizing and estimation abilities, respectively, are both spatial processes relying on extracting information across spatial locations. Nevertheless, whether and how these processes vary across visual field locations remains unknown. Here, we examined if enumeration displays asymmetries around the visual field. Experiment 1 tested small number (1-6) enumeration at cardinal and non-cardinal peripheral locations while manipulating the spacing among the objects. Experiment 2 examined enumeration at cardinal locations in more detail while minimising crowding. Both experiments demonstrated a Horizontal-Vertical Asymmetry (HVA) where performance was better along the horizontal axis relative to the vertical. Experiment 1 found that this effect was modulated by spacing with stronger asymmetry at closer spacing. Experiment 2 revealed further asymmetries: a Vertical Meridian Asymmetry (VMA) with better enumeration on the lower vertical meridian than on the upper and a Horizontal Meridian Asymmetry (HMA) with better enumeration along the left horizontal meridian than along the right. All three asymmetries were evident for both subitizing and estimation. HVA and VMA have been observed in a range of visual tasks, indicating that they might be inherited from early visual constraints. However, HMA is observed primarily in mid-level tasks, often involving attention. These results suggest that while enumeration processes can be argued to inherit low-level visual constraints, the findings are, parsimoniously, consistent with visual attention playing a role in both subitizing and estimation.

The same binding in contour integration and crowding.

Binding of features helps object recognition in contour integration but hinders it in crowding. In contour integration, aligned adjacent objects group together to form a path. In crowding, flanking objects make the target unidentifiable. However, to date, the two tasks have only been studied separately. K. A. May and R. F. Hess (2007) suggested that the same binding mediates both tasks. To test this idea, we ask observers to perform two different tasks with the same stimulus. We present oriented grating patches that form a "snake letter" in the periphery. Observers report either the identity of the whole letter (contour integration task) or the phase of one of the grating patches (crowding task). We manipulate the strength of binding between gratings by varying the alignment between them, i.e., the Gestalt goodness of continuation, measured as "wiggle." We find that better alignment strengthens binding, which improves contour integration and worsens crowding. Observers show equal sensitivity to alignment in these two very different tasks, suggesting that the same binding mechanism underlies both phenomena. It has been claimed that grouping among flankers reduces their crowding of the target. Instead, we find that these published cases of weak crowding are due to weak binding resulting from target-flanker misalignment. We conclude that crowding is mediated solely by the grouping of flankers with the target and is independent of grouping among flankers.

Bullet trains and steam engines: exogenous attention zips but endogenous attention chugs along.

Analyzing a scene requires shifting attention from object to object. Although several studies have attempted to determine the speed of these attentional shifts, there are large discrepancies in their estimates. Here, we adapt a method pioneered by T. A. Carlson, H. Hogendoorn, and F. A. J. Verstraten (2006) that directly measures pure attentional shift times. We also test if attentional shifts can be handled in parallel by the independent resources available in the two cortical hemispheres. We present 10 "clocks," with single revolving hands, in a ring around fixation. Observers are asked to report the hand position on one of the clocks at the onset of a transient cue. The delay between the reported time and the veridical time at cue onset can be used to infer processing and attentional shift times. With this setup, we use a novel subtraction method that utilizes different combinations of exogenous and endogenous cues to determine shift times for both types of attention. In one experiment, subjects shift attention to an exogenously cued clock (baseline condition) in one block, and in other blocks, subjects perform one further endogenous shift to a nearby clock (test condition). In another experiment, attention is endogenously cued to one clock (baseline condition), and on other trials, an exogenous cue further shifts attention to a nearby clock (test condition). Subtracting report delays in the baseline condition from those obtained in the test condition allows us to isolate genuine attentional shift times. In agreement with previous studies, our results reveal that endogenous attention is much slower than exogenous attention (endogenous: 250 ms; exogenous: 100 ms). Surprisingly, the dependence of shift time on distance is minimal for exogenous attention, whereas it is steep for endogenous attention. In the final experiment, we find that endogenous shifts are faster across hemifields than within a hemifield suggesting that the two hemispheres can simultaneously process at least parts of these shifts.

Subitizing object parts reveals a second stage of individuation.

Humans can efficiently individuate a small number of objects. This subitizing ability is thought to be a consequence of limited attentional resources. However, how and what is selected during the individuation process remain outstanding questions. We investigated these in four experiments by examining if parts of objects are enumerated as efficiently as distinct objects in the presence and absence of distractor objects. We found that distractor presence reduced subitizing efficiency. Crucially, parts connected to multiple objects were enumerated less efficiently than independent objects or parts connected to a single object. These results argue against direct individuation of parts and show that objecthood plays a fundamental role in individuation. Objects are selected first and their components are selected in subsequent steps. This reveals that individuation operates sequentially over multiple levels.

Social Agency as a continuum.

Sense of Agency, the phenomenology associated with causing one's own actions and corresponding effects, is a cornerstone of human experience. Social Agency can be defined as the Sense of Agency experienced in any situation in which the effects of our actions are related to a conspecific. This can be implemented as the other's reactions being caused by our action, joint action modulating our Sense of Agency, or the other's mere social presence influencing our Sense of Agency. It is currently an open question how such Social Agency can be conceptualized and how it relates to its nonsocial variant. This is because, compared with nonsocial Sense of Agency, the concept of Social Agency has remained oversimplified and underresearched, with disparate empirical paradigms yielding divergent results. Reviewing the empirical evidence and the commonalities and differences between different instantiations of Social Agency, we propose that Social Agency can be conceptualized as a continuum, in which the degree of cooperation is the key dimension that determines our Sense of Agency, and how it relates to nonsocial Sense of Agency. Taking this perspective, we review how the different factors that typically influence Sense of Agency affect Social Agency, and in the process highlight outstanding empirical questions within the field. Finally, concepts from wider research areas are discussed in relation to the ecological validity of Social Agency paradigms, and we provide recommendations for future methodology.

Clustering leads to underestimation of numerosity, but crowding is not the cause.

Humans have the remarkable ability to rapidly estimate the number of objects in a visual scene without relying on counting, something referred to as the number sense. It has been well documented that the more clustered the elements are, the lower their perceived numerosity is. A recent account of this observation is the crowdinghypothesis, which posits that the perceived underestimation is driven by visual crowding: the inability to recognise objects in clutter. Crowding can impair individuation of the elements, which would explain the underestimation. Here, we tested the crowding hypothesis by assessing numerosity estimation and crowding for the same stimulus configurations in the same participants. Experiment 1 compared the two tasks when numerosity can be considered to be estimated directly by the visual system (reference patch density = 0.12 items/deg2), while Experiment 2 used high density stimuli (density = 0.88 items/deg2), where numerosity may be estimated indirectly. In both cases, we found that spacing and similarity between elements affected estimation and crowding tasks in markedly different ways. These results are incompatible with a crowding account of numerosity underestimation and point to separate mechanisms for object identification and number estimation, although grouping may play a moderating role in both cases.

Response selection modulates crowding: a cautionary tale for invoking top-down explanations.

Object recognition in the periphery is limited by clutter. This phenomenon of visual crowding is ameliorated when the objects are dissimilar. This effect of inter-object similarity has been extensively studied for low-level features and is thought to reflect bottom-up processes. Recently, crowding was also found to be reduced when objects belonged to explicitly distinct groups; that is, crowding was weak when they had low group membership similarity. It has been claimed that top-down knowledge is necessary to explain this effect of group membership, implying that the effect of similarity on crowding cannot be a purely bottom-up process. We tested the claim that the effect of group membership relies on knowledge in two experiments and found that neither explicit knowledge about differences in group membership nor the possibility of acquiring knowledge about target identities is necessary to produce the effects. These results suggest that top-down processes need not be invoked to explain the effect of group membership. Instead, we suggest that differences in flanker reportability that emerge from the differences in group membership are the source of the effect. That is, when targets and flankers are sampled from distinct groups, flankers cannot be inadvertently reported, leading to fewer errors and hence weaker crowding. Further, we argue that this effect arises at the stage of response selection. This conclusion is well supported by an analytical model based on these principles. We conclude that previously observed effects in crowding attributed to top-down or higher level processes might instead be due to post-perceptual response selection strategies.

Crowding in humans is unlike that in convolutional neural networks.

Object recognition is a primary function of the human visual system. It has recently been claimed that the highly successful ability to recognise objects in a set of emergent computer vision systems-Deep Convolutional Neural Networks (DCNNs)-can form a useful guide to recognition in humans. To test this assertion, we systematically evaluated visual crowding, a dramatic breakdown of recognition in clutter, in DCNNs and compared their performance to extant research in humans. We examined crowding in three architectures of DCNNs with the same methodology as that used among humans. We manipulated multiple stimulus factors including inter-letter spacing, letter colour, size, and flanker location to assess the extent and shape of crowding in DCNNs. We found that crowding followed a predictable pattern across architectures that was different from that in humans. Some characteristic hallmarks of human crowding, such as invariance to size, the effect of target-flanker similarity, and confusions between target and flanker identities, were completely missing, minimised or even reversed. These data show that DCNNs, while proficient in object recognition, likely achieve this competence through a set of mechanisms that are distinct from those in humans. They are not necessarily equivalent models of human or primate object recognition and caution must be exercised when inferring mechanisms derived from their operation.