Evidence against a speed limit in multiple-object tracking.

Everyday tasks often require us to keep track of multiple objects in dynamic scenes. Past studies show that tracking becomes more difficult as objects move faster. In the present study, we show that this trade-off may not be due to increased speed itself but may, instead, be due to the increased crowding that usually accompanies increases in speed. Here, we isolate changes in speed from variations in crowding, by projecting a tracking display either onto a small area at the center of a hemispheric projection dome or onto the entire dome. Use of the larger display increased retinal image size and object speed by a factor of 4 but did not increase interobject crowding. Results showed that tracking accuracy was equally good in the large-display condition, even when the objects traveled far into the visual periphery. Accuracy was also not reduced when we tested object speeds that limited performance in the small-display condition. These results, along with a reinterpretation of past studies, suggest that we might be able to track multiple moving objects as fast as we can a single moving object, once the effect of object crowding is eliminated.

Visual indexes, preconceptual objects, and situated vision.

This paper argues that a theory of situated vision, suited for the dual purposes of object recognition and the control of action, will have to provide something more than a system that constructs a conceptual representation from visual stimuli: it will also need to provide a special kind of direct (preconceptual, unmediated) connection between elements of a visual representation and certain elements in the world. Like natural language demonstratives (such as 'this' or 'that') this direct connection allows entities to be referred to without being categorized or conceptualized. Several reasons are given for why we need such a preconceptual mechanism which individuates and keeps track of several individual objects in the world. One is that early vision must pick out and compute the relation among several individual objects while ignoring their properties. Another is that incrementally computing and updating representations of a dynamic scene requires keeping track of token individuals despite changes in their properties or locations. It is then noted that a mechanism meeting these requirements has already been proposed in order to account for a number of disparate empirical phenomena, including subitizing, search-subset selection and multiple object tracking (Pylyshyn et al., Canadian Journal of Experimental Psychology 48(2) (1994) 260). This mechanism, called a visual index or FINST, is briefly discussed and it is argued that viewing it as performing a demonstrative or preconceptual reference function has far-reaching implications not only for a theory of situated vision, but also for suggesting a new way to look at why the primitive individuation of visual objects, or proto-objects, is so central in computing visual representations. Indexing visual objects is also, according to this view, the primary means for grounding visual concepts and is a potentially fruitful way to look at the problem of visual integration across time and across saccades, as well as to explain how infants' numerical capacity might arise.

What is a visual object? Evidence from target merging in multiple object tracking.

The notion that visual attention can operate over visual objects in addition to spatial locations has recently received much empirical support, but there has been relatively little empirical consideration of what can count as an 'object' in the first place. We have investigated this question in the context of the multiple object tracking paradigm, in which subjects must track a number of independently and unpredictably moving identical items in a field of identical distractors. What types of feature clusters can be tracked in this manner? In other words, what counts as an 'object' in this task? We investigated this question with a technique we call target merging: we alter tracking displays so that distinct target and distractor locations appear perceptually to be parts of the same object by merging pairs of items (one target with one distractor) in various ways - for example, by connecting item locations with a simple line segment, by drawing the convex hull of the two items, and so forth. The data show that target merging makes the tracking task far more difficult to varying degrees depending on exactly how the items are merged. The effect is perceptually salient, involving in some conditions a total destruction of subjects' capacity to track multiple items. These studies provide strong evidence for the object-based nature of tracking, confirming that in some contexts attention must be allocated to objects rather than arbitrary collections of features. In addition, the results begin to reveal the types of spatially organized scene components that can be independently attended as a function of properties such as connectedness, part structure, and other types of perceptual grouping.

Tracking an object through feature space.

Visual attention allows an observer to select certain visual information for specialized processing. Selection is readily apparent in 'tracking' tasks where even with the eyes fixed, observers can track a target as it moves among identical distractor items. In such a case, a target is distinguished by its spatial trajectory. Here we show that one can keep track of a stationary item solely on the basis of its changing appearance--specified by its trajectory along colour, orientation, and spatial frequency dimensions--even when a distractor shares the same spatial location. This ability to track through feature space bears directly on competing theories of attention, that is, on whether attention can select locations in space, features such as colour or shape, or particular visual objects composed of constellations of visual features. Our results affirm, consistent with a growing body of psychophysical and neurophysiological evidence, that attention can indeed select specific visual objects. Furthermore, feature-space tracking extends the definition of visual object to include not only items with well defined spatio-temporal trajectories, but also those with well defined featuro-temporal trajectories.

Multiple object tracking and attentional processing.

How are attentional priorities set when multiple stimuli compete for access to the limited-capacity visual attention system? According to Pylyshyn (1989) and Yantis and Johnson (1990), a small number of visual objects can be preattentively indexed or tagged and thereby accessed more rapidly by a subsequent attentional process (e.g., the traditional "spotlight of attention"). In the present study, we used the multiple object tracking methodology of Pylyshyn and Storm (1988) to investigate the relation between what we call "visual indexing" and attentional processing. Participants visually tracked a subset of a set of identical, independently randomly moving objects in a display (the targets), and made a speeded identification response when they noticed a target or a nontarget (distractor) object undergo a subtle form transformation. We found that target form changes were identified more rapidly than nontarget form changes, and that the speed of responding to target form changes was unaffected by the number of nontargets in the display when the form-changing targets were successfully tracked. We also found that this enhanced processing only applied to the targets themselves and not to nearby nontarget distractors, showing that the allocation of a broadened region of visual attention (as in the zoom-lens model of attentional allocation) could not account for these findings. These results confirm that visual indexing bestows a processing priority to a number of objects in the visual field.

Tracking multiple items through occlusion: clues to visual objecthood.

In three experiments, subjects attempted to track multiple items as they moved independently and unpredictably about a display. Performance was not impaired when the items were briefly (but completely) occluded at various times during their motion, suggesting that occlusion is taken into account when computing enduring perceptual objecthood. Unimpaired performance required the presence of accretion and deletion cues along fixed contours at the occluding boundaries. Performance was impaired when items were present on the visual field at the same times and to the same degrees as in the occlusion conditions, but disappeared and reappeared in ways which did not implicate the presence of occluding surfaces (e.g., by imploding and exploding into and out of existence instead of accreting and deleting along a fixed contour). Unimpaired performance did not require visible occluders (i.e., Michotte's tunnel effect) or globally consistent occluder positions. We discuss implications of these results for theories of objecthood in visual attention.

Is vision continuous with cognition? The case for cognitive impenetrability of visual perception.

Although the study of visual perception has made more progress in the past 40 years than any other area of cognitive science, there remain major disagreements as to how closely vision is tied to cognition. This target article sets out some of the arguments for both sides (arguments from computer vision, neuroscience, psychophysics, perceptual learning, and other areas of vision science) and defends the position that an important part of visual perception, corresponding to what some people have called early vision, is prohibited from accessing relevant expectations, knowledge, and utilities in determining the function it computes--in other words, it is cognitively impenetrable. That part of vision is complex and involves top-down interactions that are internal to the early vision system. Its function is to provide a structured representation of the 3-D surfaces of objects sufficient to serve as an index into memory, with somewhat different outputs being made available to other systems such as those dealing with motor control. The paper also addresses certain conceptual and methodological issues raised by this claim, such as whether signal detection theory and event-related potentials can be used to assess cognitive penetration of vision. A distinction is made among several stages in visual processing, including, in addition to the inflexible early-vision stage, a pre-perceptual attention-allocation stage and a post-perceptual evaluation, selection, and inference stage, which accesses long-term memory. These two stages provide the primary ways in which cognition can affect the outcome of visual perception. The paper discusses arguments from computer vision and psychology showing that vision is "intelligent" and involves elements of "problem solving." The cases of apparently intelligent interpretation sometimes cited in support of this claim do not show cognitive penetration; rather, they show that certain natural constraints on interpretation, concerned primarily with optical and geometrical properties of the world, have been compiled into the visual system. The paper also examines a number of examples where instructions and "hints" are alleged to affect what is seen. In each case it is concluded that the evidence is more readily assimilated to the view that when cognitive effects are found, they have a locus outside early vision, in such processes as the allocation of focal attention and the identification of the stimulus.

Multiple-location access in vision: evidence from illusory line motion.

Four experiments with undergraduates used illusory line motion (ILM) to contrast Z. W. Pylyshyn's (1989) FINST theory of spatial indexing with predictions made by unitary attention models. Multiple-onset stimuli were able to cause ILM at disparate, noncontiguous spatial locations. Consistent with gradient explanations of ILM and with FINST theory predictions, varying line-drawing speed and the number of stimuli revealed a decrease in ILM and a capacity limitation, respectively. Modeling analyses suggested a limit in the number of locations (5-7) that could elicit the illusion. Requiring participants to report the locations of all stimuli exhibiting illusory motion in a specified direction suggested parallel access to between 2 and 5 display locations simultaneously. The results of all 4 experiments were predicted by FINST theory but not by a broad class of unitary attention hypotheses.

Searching through subsets: a test of the visual indexing hypothesis.

This paper presents three experiments investigating the claim that the visual system utilizes a primitive indexing mechanism (sometimes called FINSTS; Pylyshyn, 1989) to make non-contiguous features directly accessible for further visual processing. This claim is investigated using a variant of the conjunction search task in which subjects search among a subset of the items in a conjunction search display for targets defined by a conjunction of colour and orientation. The members of the subset were identified by virtue of the late onset of the objects' place-holders. The cued subset was manipulated to include either homogeneous distractors or mixed distractors. Observers were able to select a subset of three items from among fifteen for further processing (Experiment 1); furthermore, a reaction time advantage for homogeneous subsets over mixed subsets was observed, indicating that more than one of the subset is selected for further specialized processing. The homogeneous subset advantage held for subsets of two to five items (Experiment 2), and the time required to process the cued subset did not increase with increased dispersion of the items (Experiment 3). These results support the basic claim of the indexing theory: the claim that multiple visual indexes are used in selecting objects for visual processing.

Multiple parallel access in visual attention.

It is widely accepted that there exists a region or locus of maximal resource allocation in visual perception--sometimes referred to as the spotlight of attention. We have argued that even if there is a single locus of processing, there must be multiple loci of parallel access--several places in the visual field must be indexed at once and these indexes can be used to determine where attention is allocated. We have carried out a variety of studies to support these ideas, including experiments showing that subjects can track multiple independent moving targets in a field of identical distractors, that the enhanced ability to detect changes occurring on these targets does not accrue to nontargets nor to items lying inside the convex polygon that they form (so that a zoom-lens of attention does not fit the data). We have used a visual search paradigm to show that (serial or parallel) search can be confined to a subset of indexed items and the layout of these items is of little importance. We have also studied the phenomenon known as subitizing and have shown that subitizing occurs only when items can be preattentively individuated and in those cases location precuing has little effect, compared with when counting occurs, which suggests that subitizing may be carried out by counting active indexes rather than items in the visual field. And finally we have run studies showing that a certain motion effect that is sensitive to attention can occur at multiple precued loci. We believe that this evidence suggests that there is an early preattentive stage in vision where a small number of salient items in the visual field are indexed and thereby made readily accessible for a variety of visual tasks.

Some primitive mechanisms of spatial attention.

Our approach to studying the architecture of mind has been to look for certain extremely simple mechanisms which we have good reason to suspect must exist, and to confirm these empirically. We have been concerned primarily with certain low-level mechanisms in vision which allow the visual system to simultaneously index items at multiple spatial locations, and have developed a provisional model (called the FINST model) of these mechanisms. Among the studies we have carried out to support these ideas are ones showing that subjects can track multiple independent moving targets in a field of identical distractors, and that their ability to track these targets and detect changes occurring on them does not generalize to non-targets or to items lying inside the convex polygon that they form (so that a zoom lens of attention does not fit the data). We have used a visual search paradigm to show that (serial or parallel) search can be confined to a subset of indexed items and the layout of these items is of little importance. We have also carried out a large number of studies on the phenomenon known as subitizing and have shown that subitizing occurs only when items can be preattentively individuated and in those cases location precuing has little effect, compared with when counting occurs, which suggests that subitizing may be carried out by counting active indexes rather than items in the visual field. And finally we have run studies showing that a certain motion effect which is sensitive to attention can occur at multiple precued loci. We believe that taken as a whole the evidence is most parsimoniously accounted for in terms of the hypothesis that there is an early preattentive stage in vision where a small number of salient items in the visual field are indexed and thereby made readily accessible for a variety of visual tasks.

Why are small and large numbers enumerated differently? A limited-capacity preattentive stage in vision.

"Subitizing," the process of enumeration when there are fewer than 4 items, is rapid (40-100 ms/item), effortless, and accurate. "Counting," the process of enumeration when there are more than 4 items, is slow (250-350 ms/item), effortful, and error-prone. Why is there a difference in the way the small and large numbers of items are enumerated? A theory of enumeration is proposed that emerges from a general theory of vision, yet explains the numeric abilities of preverbal infants, children, and adults. We argue that subitizing exploits a limited-capacity parallel mechanism for item individuation, the FINST mechanism, associated with the multiple target tracking task (Pylyshyn, 1989; Pylyshyn & Storm, 1988). Two kinds of evidence support the claim that subitizing relies on preattentive information, whereas counting requires spatial attention. First, whenever spatial attention is needed to compute a spatial relation (cf. Ullman, 1984) or to perform feature integration (cf. Treisman & Gelade, 1980), subitizing does not occur (Trick & Pylyshyn, 1993a). Second, the position of the attentional focus, as manipulated by cue validity, has a greater effect on counting than subitizing latencies (Trick & Pylyshyn, 1993b).

What enumeration studies can show us about spatial attention: evidence for limited capacity preattentive processing.

Subitizing, the enumeration of 1-4 items, is rapid (40-120 ms/item) and accurate. Counting, the enumeration of 5 items or more, is slow (250-350 ms/item) and error-prone. Why are small numbers of items enumerated differently from large numbers of items? It is suggested that subitizing relies on a preattentive mechanism. Ss could subitize heterogeneously sized multicontour items but not concentric multicontour items, which require attentional processing because preattentive gestalt processes misgroup contours from different items to form units. Similarly, Ss could subitize target items among distractors but only if the targets and distractors differed by a feature, a property derived through preattentive analysis. Thus, subitizing must rely on a mechanism that can handle a few items at once, which operates before attention but after preattentive operations of feature detection and grouping.

The role of location indexes in spatial perception: a sketch of the FINST spatial-index model.

This paper hypothesizes a resource-limited mechanism, called a FINST, for individuating or indexing visual features, as distinct from encoding their type or location. FINSTs have the property that they index features in a way that is transparent to their retinal location, and hence under certain conditions succeed in "pointing to" scene locations. The basic assumption is that no operation upon sets of features can occur unless all the features to which the operation applies are first FINSTed. A number of applications of this hypothesis are explored in this paper, including applications to phenomena such as the spatial stability of visual percepts, the ability to track several independently moving targets in parallel, the ability to detect a class of spatial relations requiring the use of "visual routines", various mental imagery phenomena, and the ability to encode complex shapes for recognition. In addition, the possibility is examined that such indexes might be used to bind perceived locations to arguments in motor commands, thereby allowing some forms of perceptual-motor coordination. Several additional assumptions are introduced for this purpose, including the postulation of other indexes (called ANCHORS) for non-visually sensed locations. It is assumed that ANCHORS can be bound to FINSTs, thus allowing cross-referencing of visually detected locations to locations given within a proprioceptive or motor-command frame of reference.

Tracking multiple independent targets: evidence for a parallel tracking mechanism.

There is considerable evidence that visual attention is concentrated at a single locus in the visual field, and that this locus can be moved independent of eye movements. Two studies are reported which suggest that, while certain aspects of attention require that locations be scanned serially, at least one operation may be carried out in parallel across several independent loci in the visual field. That is the operation of indexing features and tracking their identity. The studies show that: (a) subjects are able to track a subset of up to 5 objects in a field of 10 identical randomly-moving objects in order to distinguish a change in a target from a change in a distractor; and (b) when the speed and distance parameters of the display are designed so that, on the basis of some very conservative assumptions about the speed of attention movement and encoding times, the predicted performance of a serial scanning and updating algorithm would not exceed about 40% accuracy, subjects still manage to do the task with 87% accuracy. These findings are discussed in relation to an earlier, and independently motivated model of feature-binding--called the FINST model--which posits a primitive identity maintenance mechanism that indexes and tracks a limited number of visual objects in parallel. These indexes are hypothesized to serve the function of binding visual features prior to subsequent pattern recognition.

Spatio-temporal parameters and the three-dimensionality of apparent motion: evidence for two types of processing.

The minimum ISI required for perceiving apparent motion in depth was measured as a function of the 2D separation of stimuli and the physical separation of stimuli in depth. It was found that temporal thresholds increased as a function of the separation of stimuli in depth. This supports the results of previous research indicating that the perceived three-dimensionality of apparent motion in depth increases with ISI. In addition, the rate of threshold increase was significantly greater in displays with short 2D separations of stimuli than in displays with large 2D separations. This robust functional dissociation of thresholds indicates that the short-range system may be involved in the processing of apparent motion in depth in the former case.