From vision to action: experiments and models of steering control during driving.

Experienced drivers performed simple steering maneuvers in the absence of continuous visual input. Experiments conducted in a driving simulator assessed drivers' performance of lane corrections during brief visual occlusion and examined the visual cues that guide steering. The dependence of steering behavior on heading, speed, and lateral position at the start of the maneuver was measured. Drivers adjusted steering amplitude with heading and performed the maneuver more rapidly at higher speeds. These dependencies were unaffected by a 1.5-s visual occlusion at the start of the maneuver. Longer occlusions resulted in severe performance degradation. Two steering control models were developed to account for these findings. In the 1st, steering actions were coupled to perceptual variables such as lateral position and heading. In the 2nd, drivers pursued a virtual target in the scene. Both models yielded behavior that closely matches that of human drivers.

Recovering heading for visually-guided navigation.

We present a model for recovering the direction of heading of an observer who is moving relative to a scene that may contain self-moving objects. The model builds upon an algorithm proposed by Rieger and Lawton, based on earlier work by Longuet-Higgins and Prazdny. The algorithm uses velocity differences computed in regions of high depth variation to locate the focus of expansion, which indicates the observer's heading direction. We relate the behavior of the model to psychophysical observations regarding the ability of human observers to judge heading direction, and show how the model copes with self-moving objects in the environment.

Recovering three-dimensional structure from motion with surface reconstruction.

This paper addresses the computational role that the construction of a complete surface representation may play in the recovery of 3-D structure from motion. We first discuss the need to integrate surface reconstruction with the structure-from-motion process, both on computational and perceptual grounds. We then present a model that combines a feature-based structure-from-motion algorithm with a smooth surface interpolation mechanism. This model allows multiple surfaces to be represented in a given viewing direction, incorporates constraints on surface structure from object boundaries, and segregates image features onto multiple surfaces on the basis of their 2-D image motion. We present the results of computer simulations that relate the qualitative behavior of this model to psychophysical observations. In a companion paper, we discuss further perceptual observations regarding the possible role of surface reconstruction in the human recovery of 3-D structure from motion.

Structure-from-motion: perceptual evidence for surface interpolation.

Dynamic random-dot displays representing a rotating cylinder were used to investigate surface interpolation in the perception of structure-from-motion (SFM) in humans. Surface interpolation refers to a process in which a complete surface in depth is reconstructed from the object depth values extracted at the stimulus features. Surface interpolation will assign depth values even in parts of the object that contain no features. Such a "fill-in" process should make the detection of featureless stimulus areas ("holes") difficult. Indeed, we demonstrate that such holes in our rotating cylinder can be as wide as one-quarter of the stimulus before subjects can reliably detect their presence. Subjects were presented with a variation on the rotating cylinder in which all dots were oscillating either in synchrony or asynchronously. Subjects perceive a rigidly rotating cylinder even when such a percept is not in agreement with the physical stimulus. To reconcile this discrepancy between actual and perceived stimulus we propose that individual points contribute to a surface based object representation and that in this process the visual system looses access to the identity of the individual features that make up the surface. Finally we are able to explain a variety of previously documented perceptual peculiarities in the perception of structure-from-motion by arguing that the perceptual interpretation of the object's boundaries influences the surface interpolation process. These findings offer strong perceptual evidence for a process of surface interpolation and are also physiologically plausible given results from recordings in awake behaving monkey cortical areas V1 and MT. The companion paper demonstrates how such a surface interpolation process can be incorporated into a structure-from-motion algorithm and how object boundaries can influence the perception of structure-from-motion as has been demonstrated before and in this paper.

Comments on ``digital step edges from zero crossings of second directional derivatives''.

In a recent paper,1 Haralick published an edge detection scheme that was supported, in part, by an evaluation against the Prewitt and the Marr-Hildreth (¿2G) operators. This evaluation led to the conclusion that Haralick's method performed the best, and the ¿2G operator performed the worst. The implementation of the ¿2G operator, on which this evaluation was based, differed significantly from that used by Marr and Hildreth. Evaluation of the performance of the Marr-Hildreth implementation of the ¿2G operator on similar images shows that this edge detection method in fact performs comparably to the Prewitt and Haralick operators.

The computation of the velocity field.

The organization of movement in the changing retinal image provides a valuable source of information for analysing the environment in terms of objects, their motion in space, and their three-dimensional structure. A description of this movement is not provided to our visual system directly, however; it must be inferred from the pattern of changing intensity that reaches the eye. This paper examines the problem of motion measurement, which we formulate as the computation of an instantaneous two-dimensional velocity field from the changing image. Initial measurements of motion take place at the location of significant intensity changes. These measurements provide only one component of local velocity, and must be integrated to compute the two-dimensional velocity field. A fundamental problem for this integration stage is that the velocity field is not determined uniquely from information available in the changing image. We formulate an additional constraint of smoothness of the velocity field, based on the physical assumption that surfaces are generally smooth, which allows the computation of a unique velocity field. A theoretical analysis of the conditions under which this computation yields the correct velocity field suggests that the solution is physically plausible. Empirical studies show the predictions of this computation to be consistent with human motion perception.

Computational studies of the extraction of visual spatial information from binocular and motion cues.

This paper reviews some of the contributions that work in computational vision has made to the study of biological vision systems. We concentrate on two areas where there has been strong interaction between computational and experimental studies: the use of binocular stereo to recover the distances to surfaces in space, and the recovery of the three-dimensional shape of objects from relative motion in the image. With regard to stereo, we consider models proposed for solving the stereo correspondence problem, focussing on the way in which physical properties of the world constrain possible methods of solution. We also show how critical observations regarding human stereo vision have helped to shape these models. With regard to the recovery of structure from motion, we focus on how the constraint of object rigidity has been used in computational models of this process.

Incremental rigidity scheme for recovering structure from motion: position-based versus velocity-based formulations.

Perceptual studies suggest that the visual system may use a rigidity assumption in its recovery of three-dimensional structure from motion. Ullman [Perception 13, 255 (1984)] recently proposed a computational scheme that uses this assumption to recover the structure of rigid and nonrigid objects in motion. The scheme assumes the input to be discrete positions of elements in motion, under orthographic projection. We present formulations of Ullman's method that use velocity information and perspective projection in the recovery of structure. Theoretical and computer analyses show that the velocity-based formulations provide a rough estimate of structure quickly but are not robust over an extended time. The stable long-term recovery of structure requires disparate views of moving objects.

Human heading judgments in the presence of moving objects.

When moving toward a stationary scene, people judge their heading quite well from visual information alone. Much experimental and modeling work has been presented to analyze how people judge their heading for stationary scenes. However, in everyday life, we often move through scenes that contain moving objects. Most models have difficulty computing heading when moving objects are in the scene, and few studies have examined how well humans perform in the presence of moving objects. In this study, we tested how well people judge their heading in the presence of moving objects. We found that people perform remarkably well under a variety of conditions. The only condition that affects an observer's ability to judge heading accurately consists of a large moving object crossing the observer's path. In this case, the presence of the object causes a small bias in the heading judgments. For objects moving horizontally with respect to the observer, this bias is in the object's direction of motion. These results present a challenge for computational models.

Differential effects of shared attention on perception of heading and 3-D object motion.

When a person moves in a straight line through a stationary environment, the images of object surfaces move in a radial pattern away from a single point. This point, known as the focus of expansion (FOE), corresponds to the person's direction of motion. People judge their heading from image motion quite well in this situation. They perform most accurately when they can see the region around the FOE, which contains the most useful information for this task. Furthermore, a large moving object in the scene has no effect on observer heading judgments unless it obscures the FOE. Therefore, observers may obtain the most accurate heading judgments by focusing their attention on the region around the FOE. However, in many situations (e.g., driving), the observer must pay attention to other moving objects in the scene (e.g., cars and pedestrians) to avoid collisions. These objects may be located far from the FOE in the visual field. We tested whether people can accurately judge their heading and the three-dimensional (3-D) motion of objects while paying attention to one or the other task. The results show that differential allocation of attention affects people's ability to judge 3-D object motion much more than it affects their ability to judge heading. This suggests that heading judgments are computed globally, whereas judgments about object motion may require more focused attention.

The perceptual buildup of three-dimensional structure from motion.

We present a set of psychophysical experiments that measure the accuracy of perceived three-dimensional (3-D) structure derived from relative motion in the changing two-dimensional image. The experiments are motivated in part by a computational model proposed by Ullman (1984), called the incremental rigidity scheme, in which an accurate 3-D structure is built up incrementally, by considering images of moving objects over an extended time period. Our main conclusions are: First, the human visual system can derive an accurate model of the relative depths of moving points, even in the presence of noise in their image positions; second, the accuracy of the 3-D model improves with time, eventually reaching a plateau; and third, the 3-D structure currently perceived appears to depend on previous 3-D models. Through computer simulations, we relate the results of our psychophysical experiments with the predictions of Ullman's model.