Processing 3D form and 3D motion: respective contributions of attention-based and stimulus-driven activity.

This study aims at segregating the neural substrate for the 3D-form and 3D-motion attributes in structure-from-motion perception, and at disentangling the stimulus-driven and endogenous-attention-driven processing of these attributes. Attention and stimulus were manipulated independently: participants had to detect the transitions of one attribute--form, 3D motion or colour--while the visual stimulus underwent successive transitions of all attributes. We compared the BOLD activity related to form and 3D motion in three conditions: stimulus-driven processing (unattended transitions), endogenous attentional selection (task) or both stimulus-driven processing and attentional selection (attended transitions). In all conditions, the form versus 3D-motion contrasts revealed a clear dorsal/ventral segregation. However, while the form-related activity is consistent with previously described shape-selective areas, the activity related to 3D motion does not encompass the usual "visual motion" areas, but rather corresponds to a high-level motion system, including IPL and STS areas. Second, we found a dissociation between the neural processing of unattended attributes and that involved in endogenous attentional selection. Areas selective for 3D-motion and form showed either increased activity at transitions of these respective attributes or decreased activity when subjects' attention was directed to a competing attribute. We propose that both facilitatory and suppressive mechanisms of attribute selection are involved depending on the conditions driving this selection. Therefore, attentional selection is not limited to an increased activity in areas processing stimulus properties, and may unveil different functional localization from stimulus modulation.

Measurement of the visual contribution to postural steadiness from the COP movement: methodology and reliability.

We studied the reliability of different measures of the visual contribution to postural steadiness by recording the postural sway during standing with eyes open (EO) or eyes closed (EC). The COP trajectory was recorded in 21 subjects aged 42-61 standing on a firm or foam support. The improvement of postural steadiness due to vision was measured with a higher reliability (i.e. lower intra- and inter-subject variabilities) with the sway velocity V, than with the position RMS. Due to the increase of the variability of V and RMS with their own mean values, we quantified the visual contribution to posture by the stabilization ratio (SR), based on a logarithm transform of V or RMS. As compared to the Romberg quotient (EC/EO), SR improved the reliability of the measurement of the visual contribution to posture within individuals, across subjects, and even across different studies in the literature. Our method led to decrease the inter-subject coefficient of variation of this measurement to about 25%, using a foam support. It leads to a similar accuracy in binocular and monocular vision, and it also applies to the quantification of other non-visual sensory contributions to posture.

Perception and reproduction of force direction in the horizontal plane.

In this study, we evaluated the capacity of human beings to perceive and reproduce forces applied to the hand. We tested for perceptive distortions and/or privileged directions in the performance of these two tasks. Subjects resisted a reference force applied by a joystick in a given direction, with instructions to keep the hand at a constant position. In a perception task, subjects subsequently resisted a second such force, the direction of which they could adjust with a potentiometer; the task was to reorient the second force to be in the same perceived direction as the reference. In a reproduction task, subjects were instructed to push against the now elastically constrained joystick with the same force that was required to resist the initially applied reference force. Twenty-four reference force directions in the horizontal plane were tested twice each. We observed systematic distortions in the reproduction of force direction that were not present in the perception task. We further observed that the distortions could be predicted by anisotropy of limb stiffness and could be affected by manipulating the mechanical impedance of the hand-joystick interaction. We conclude that human subjects specify and store forces to be applied by the hand not in terms of a perceived force vector, but rather in terms of the motor activity required to resist or produce the force-i.e., subjects possess a multi-dimensional "sense of effort."

Contribution of extraretinal signals to the scaling of object distance during self-motion.

We investigated the role of extraretinal information in the perception of absolute distance. In a computer-simulated environment, monocular observers judged the distance of objects positioned at different locations in depth while performing frontoparallel movements of the head. The objects were spheres covered with random dots subtending three different visual angles. Observers viewed the objects ateye level, either in isolation or superimposed on a ground floor. The distance and size of the spheres were covaried to suppress relative size information. Hence, the main cues to distance were the motion parallax and the extraretinal signals. In three experiments, we found evidence that (1) perceived distance is correlated with simulated distance in terms of precision and accuracy, (2) the accuracy in the distance estimate is slightly improved by the presence of a ground-floor surface, (3) the perceived distance is not altered significantly when the visual field size increases, and (4) the absolute distance is estimated correctly during self-motion. Conversely, stationary subjects failed to report absolute distance when they passively observed a moving object producing the same retinal stimulation, unless they could rely on knowledge of the three-dimensional movements.

The stationarity hypothesis: an allocentric criterion in visual perception.

Having long considered that extraretinal information plays little or no role in spatial vision, the study of structure from motion (SfM) has confounded a moving observer perceiving a stationary object with a non-moving observer perceiving a rigid object undergoing equal and opposite motion. However, recently it has been shown that extraretinal information does play an important role in the extraction of structure from motion by enhancing motion cues for objects that are stationary in an allocentric, world-fixed reference frame (Nature 409 (2001) 85). Here, we test whether stationarity per se is a criterion in SfM by pitting it against rigidity. We have created stimuli that, for a moving observer, offer two interpretations: one that is rigid but non-stationary, another that is more stationary or less rigid. In two experiments, with subjects reporting either structure or motion, we show that stationary, non-rigid solutions are preferred over rigid, non-stationary solutions; and that when no perfectly stationary solutions is available, the visual system prefers the solution that is most stationary. These results demonstrate that allocentric criteria, derived from extra-retinal information, participate in reconstructing the visual scene.

Lines and dots: characteristics of the motion integration process.

Local motion detectors can only provide the velocity component perpendicular to a moving line that crosses their receptive field, leading to an ambiguity known as the "aperture problem". This problem is solved exactly for rigid objects translating in the screen plane via the intersection of constraints (IOC). In natural scenes, however, object motions are not restricted to fronto-parallel translations, and several objects with distinct motions may be present in the visual space. Under these conditions the usual IOC construction is no longer valid, which raises questions as its use as a basis for spatial integration and selection of motion signals in uniform and non-uniform velocity fields. The influence of the motion of random dots on the perceived direction of a horizontal line grating was measured, when dots and lines are seen through different apertures. The random dots were mapped on a plane that translates in a fronto-parallel plane (uniform 2D translation) or in depth (3D, corresponding to a non-uniform projected velocity field, either expanding or contracting). The grating was either moving rigidly with the dots or in the opposite direction. Subjects' responses show that the direction of line grating movement was reliably influenced only in conditions consistent with rigid motion; where there was a reliable influence, the perceived direction was consistent with the dot motion pattern. This finding points to the existence of a motion-based selection mechanism that operates prior to the disambiguation of the line movement direction. Disambiguation could occur for both uniform and non-uniform velocity fields, even though in the last case none of the individual dots indicated the proper direction in 2D velocity space. Finally, the capture by non-uniform motion patterns was less robust than that by uniform 2D translations, and could be disrupted by manipulations of the shape and size of the apertures.

Self-motion and the perception of stationary objects.

One of the ways that we perceive shape is through seeing motion. Visual motion may be actively generated (for example, in locomotion), or passively observed. In the study of the perception of three-dimensional structure from motion, the non-moving, passive observer in an environment of moving rigid objects has been used as a substitute for an active observer moving in an environment of stationary objects; this 'rigidity hypothesis' has played a central role in computational and experimental studies of structure from motion. Here we show that this is not an adequate substitution because active and passive observers can perceive three-dimensional structure differently, despite experiencing the same visual stimulus: active observers' perception of three-dimensional structure depends on extraretinal information about their own movements. The visual system thus treats objects that are stationary (in an allocentric, earth-fixed reference frame) differently from objects that are merely rigid. These results show that action makes an important contribution to depth perception, and argue for a revision of the rigidity hypothesis to incorporate the special case of stationary objects.

Role of lateral acceleration in curve driving: driver model and experiments on a real vehicle and a driving simulator.

Experimental studies show that automobile drivers adjust their speed in curves so that maximum vehicle lateral accelerations decrease at high speeds. This pattern of lateral accelerations is described by a new driver model, assuming drivers control a variable safety margin of perceived lateral acceleration according to their anticipated steering deviations. Compared with a minimum time-to-lane-crossing (H. Godthelp, 1986) speed modulation strategy, this model, based on nonvisual cues, predicts that extreme values of lateral acceleration in curves decrease quadratically with speed, in accordance with experimental data obtained in a vehicle driven on a test track and in a motion-based driving simulator. Variations of model parameters can characterize "normal" or "fast" driving styles on the test track. On the simulator, it was found that the upper limits of lateral acceleration decreased less steeply when the motion cuing system was deactivated, although drivers maintained a consistent driving style. This is interpreted per the model as an underestimation of curvilinear speed due to the lack of inertial stimuli. Actual or potential applications of this research include a method to assess driving simulators as well as to identify driving styles for on-board driver aid systems.

Analysis of pointing errors reveals properties of data representations and coordinate transformations within the central nervous system.

The execution of a simple pointing task invokes a chain of processing that includes visual acquisition of the target, coordination of multimodal proprioceptive signals, and ultimately the generation of a motor command that will drive the finger to the desired target location. These processes in the sensorimotor chain can be described in terms of internal representations of the target or limb positions and coordinate transformations between different internal reference frames. In this article we first describe how different types of error analysis can be used to identify properties of the internal representations and coordinate transformations within the central nervous system. We then describe a series of experiments in which subjects pointed to remembered 3D visual targets under two lighting conditions (dim light and total darkness) and after two different memory delays (0.5 and 5.0 s) and report results in terms of variable error, constant error, and local distortion. Finally, we present a set of simulations to help explain the patterns of errors produced in this pointing task. These analyses and experiments provide insight into the structure of the underlying sensorimotor processes employed by the central nervous system.

Visual perception of motion and 3-D structure from motion: an fMRI study.

Functional magnetic resonance imaging was used to study the cortical bases of 3-D structure perception from visual motion in human. Nine subjects underwent three experiments designed to locate the areas involved in (i) motion processing (random motion versus static dots), (ii) coherent motion processing (expansion/ contraction versus random motion) and (iii) 3-D shape from motion reconstruction (3-D surface oscillating in depth versus random motion). Two control experiments tested the specific influence of speed distribution and surface curvature on the activation results. All stimuli consisted of random dots so that motion parallax was the only cue available for 3-D shape perception. As expected, random motion compared with static dots induced strong activity in areas V1/V2, V5+ and the superior occipital gyrus (SOG; presumptive V3/V3A). V1/V2 and V5+ showed no activity increase when comparing coherent motion (expansion or 3-D surface) with random motion. Conversely, V3/V3A and the dorsal parieto-occipital junction were highlighted in both comparisons and showed gradually increased activity for random motion, coherent motion and a curved surface rotating in depth, which suggests their involvement in the coding of 3-D shape from motion. Also, the ventral aspect of the left occipito-temporal junction was found to be equally responsive to random and coherent motion stimuli, but showed a specific sensitivity to curved 3-D surfaces compared with plane surfaces. As this region is already known to be involved in the coding of static object shape, our results suggest that it might integrate various cues for the perception of 3-D shape.

A 6-dof device to measure head movements in active vision experiments: geometric modeling and metric accuracy.

This work describes a technique for measuring human head movements in 3D space. Rotations and translations of the head are tracked using a light helmet fastened to a multi-joint mechanical structure. This apparatus has been designed to be used in a series of psycho-physiological experiments in the field of active vision, where position and orientation of the head need to be measured in real time with high accuracy, high reliability and minimal interference with subject movements. A geometric model is developed to recover the position information and its parameters are identified through a calibration procedure. The expected accuracy, derived on the basis of the pure geometric model and the sensor resolution, is compared with the real accuracy, obtained by performing repetitive measurements on a calibration fixture. The outcome of the comparison confirms the validity of the proposed solution which turns out to be effective in providing measurement of head position with an overall accuracy of 0.6 mm and sampling frequency above 1 kHz.

Does the brain use sliding variables for the control of movements?

Delays in the transmission of sensory and motor information prevent errors from being instantaneously available to the central nervous system (CNS) and can reduce the stability of a closed-loop control strategy. On the other hand, the use of a pure feedforward control (inverse dynamics) requires a perfect knowledge of the dynamic behavior of the body and of manipulated objects. Sensory feedback is essential both to accommodate unexpected errors and events and to compensate for uncertainties about the dynamics of the body. Experimental observations concerning the control of posture, gaze and limbs have shown that the CNS certainly uses a combination of closed-loop and open-loop control. Feedforward components of movement, such as eye saccades, occur intermittently and present a stereotyped kinematic profile. In visuo-manual tracking tasks, hand movements exhibit velocity peaks that occur intermittently. When a delay or a slow dynamics are inserted in the visuo-manual control loop, intermittent step-and-hold movements appear clearly in the hand trajectory. In this study, we investigated strategies used by human subjects involved in the control of a particular dynamic system. We found strong evidence for substantial nonlinearities in the commands produced. The presence of step-and-hold movements seemed to be the major source of nonlinearities in the control loop. Furthermore, the stereotyped ballistic-like kinematics of these rapid and corrective movements suggests that they were produced in an open-loop way by the CNS. We analyzed the generation of ballistic movements in the light of sliding control theory assuming that they occurred when a sliding variable exceeded a constant threshold. In this framework, a sliding variable is defined as a composite variable (a combination of the instantaneous tracking error and its temporal derivatives) that fulfills a specific stability criterion. Based on this hypothesis and on the assumption of a constant reaction time, the tracking error and its derivatives should be correlated at a particular time lag before movement onset. A peak of correlation was found for a physiologically plausible reaction time, corresponding to a stable composite variable. The direction and amplitude of the ongoing stereotyped movements seemed also be adjusted in order to minimize this variable. These findings suggest that, during visually guided movements, human subjects attempt to minimize such a composite variable and not the instantaneous error. This minimization seems to be obtained by the execution of stereotyped corrective movements.

Measurements of human force control during a constrained arm motion using a force-actuated joystick.

1. When interacting with the environment, human arm movements may be prevented in certain directions (i.e., when sliding the hand along a surface) resulting in what is called a "constrained motion." In the directions that the movement is restricted, the subject is instead free to control the forces against the constraint. 2. Control strategies for constrained motion may be characterized by two extreme models. Under the active compliance model, an essentially feedback-based approach, measurements of contact force may be used in real time to modify the motor command and precisely control the forces generated against the constraint. Under the passive compliance model the motion would be executed in a feedforward manner, using an internal model of the constraint geometry. The feedforward model relies on the compliant behavior of the passive mechanical system to maintain contact while avoiding excessive contact forces. 3. Subjects performed a task in which they were required to slide the hand along a rigid surface. This task was performed in a virtual force environment in which contact forces were simulated by a two-dimensional force-actuated joystick. Unknown to the subject, the orientation of the surface constraint was varied from trial to trial, and contact force changes induced by these perturbations were measured. 4. Subjects showed variations in contact force correlated with the direction of the orientation perturbation. "Upward" tilts resulted in higher contact forces, whereas "downward" tilts resulted in lower contact forces. This result is consistent with a feedforward-based control of a passively compliant system. 5. Subject responses did not, however, correspond exactly to the predictions of a static analysis of a passive, feedforward-controlled system. A dynamic analysis reveals a much closer resemblance between a passive, feedforward model and the observed data. Numerical simulations demonstrate that a passive, dynamic system model of the movement captures many more of the salient features observed in the measured human data. 6. We conclude that human subjects execute surface-following motions in a largely feedforward manner, using an a priori model of the surface geometry. The evidence does not suggest that active, real time use of force feedback is used to guide the movement or to control limb impedance. We do not exclude, however, the possibility that the internal model of the constraint is updated at somewhat longer latencies on the basis of proprioceptive information.

Perception of three-dimensional shape from ego- and object-motion: comparison between small- and large-field stimuli.

We compare the performance in the detection of the shape of concave, planar and convex surfaces for small-field (8 deg) and large-field (90 deg) stimuli. Shape is perceived from head translations, object translations and object rotations. We find large differences between small-field and large-field stimulation. For small-field stimulation performance is best for object rotation, intermediate for self-motion and worst for object translation. For large-field stimulation performance is similar across conditions. Few errors on the sign of the curvature are found for self-motion for both field sizes, indicating that self-motion information disambiguates the curvature sign. For object rotation with small-field stimulation, the concave-convex ambiguity is strong with many apparent deformations. In contrast, large-field curvature signs are always accurately reported, suggesting that the weight of the rigidity hypothesis depends on field size.

The visual perception of three-dimensional shape from self-motion and object-motion.

To evaluate the influence of egomotion on the three-dimensional visual processing of structure-from-motion (SFM), we compared the visual discrimination between planar and spherical surfaces during subject-translation, object-translation, or rotation of the object in depth. Performance was the best for object-rotation, intermediate for subject-translation, and the poorest for object-translation--and thus increased with the quality of retinal image stabilization achieved in the different conditions. This suggests that the major role of self-motion information was to stabilize retinal images. In view of previous results, we propose that the interactions between self-motion information and SFM are reduced to functional complementarity, in the sense that self-motion can lift visual ambiguities but does not improve the sensitivity of SFM processes.

The influence of long-term practice on mental rotation of 3-D objects.

We evaluated the influence of long-term practice on the performance of a mental rotation task in which subjects judged whether two 3-D objects presented in different orientations were identical. Stimuli and experimental conditions were analogous to those used by Shepard and Metzler. Sixteen subjects were selected, to test the influence of aptitude for mental imagery on this learning process. Subjects participated in 12 to 15 sessions over 6 weeks. Two catalogues of different stimuli were alternatively used during three (or six) consecutive sessions to determine the influence of complexity and familiarity of figures. For all subjects, the inverse of the velocity of mental rotation along the sessions was adequately fitted by a decreasing exponential curve. However, evidence for mental rotation did not disappear, even after 15 sessions. Asymptotic variations can be attributed to differences in stimuli as well as imaging skills of subjects. Our results lead to a new interpretation of the mental rotation process.

Stereo-motion cooperation and the use of motion disparity in the visual perception of 3-D structure.

When an observer views a moving scene binocularly, both motion parallax and binocular disparity provide depth information. In Experiments 1A-1C, we measured sensitivity to surface curvature when these depth cues were available either individually or simultaneously. When the depth cues yielded comparable sensitivity to surface curvature, we found that curvature detection was easier with the cues present simultaneously, rather than individually. For 2 of the 6 subjects, this effect was stronger when the component of frontal translation of the surface was vertical, rather than horizontal. No such anisotropy was found for the 4 other subjects. If a moving object is observed binocularly, the patterns of optic flow are different on the left and right retinae. We have suggested elsewhere (Cornilleau-Pérès & Droulez, in press) that this motion disparity might be used as a visual cue for the perception of a 3-D structure. Our model consisted in deriving binocular disparity from the left and right distributions of vertical velocities, rather than from luminous intensities, as has been done in classical studies on stereoscopic vision. The model led to some predictions concerning the detection of surface curvature from motion disparity in the presence or absence of intensity-based disparity (classically termed binocular disparity). In a second set of experiments, we attempted to test these predictions, and we failed to validate our theoretical scheme from a physiological point of view.

A neural network model of sensoritopic maps with predictive short-term memory properties.

Coordinated orienting movements can be accurately performed without direct sensory control. Ocular saccades, for instance, have been shown to be reprogrammed after target disappearance when an intervening eye movement is electrically triggered before the saccade onset. Saccadic eye movements can also be executed toward memorized targets, even when the subject has been passively moved in darkness. Two hypotheses have been proposed to account for this goal-invariance property: either (i) the goal is reconstructed and memorized in the stable frame of reference linked to the environment ("allocentric, coordinates") or (ii) the goal is selected and memorized in the sensors-related maps ("egocentric coordinates") and is continuously updated by efferent copies of the motor commands. In this paper, we shall describe a formal neural network based on this second hypothesis. The results of the simulation show that target position can be memorized and accurately updated in a topologically ordered map, using a velocity-signal feedback. Moreover, this network has been submitted to a simple learning procedure by using the intermittent visual recurring afferent signal as the teaching signal. A similar mechanism could be involved in control of limb movement.