Combining sensory information: mandatory fusion within, but not between, senses.

Humans use multiple sources of sensory information to estimate environmental properties. For example, the eyes and hands both provide relevant information about an object's shape. The eyes estimate shape using binocular disparity, perspective projection, etc. The hands supply haptic shape information by means of tactile and proprioceptive cues. Combining information across cues can improve estimation of object properties but may come at a cost: loss of single-cue information. We report that single-cue information is indeed lost when cues from within the same sensory modality (disparity and texture gradients in vision) are combined, but not when different modalities (vision and haptics) are combined.

Eye position sense contributes to the judgement of slant.

We measured monocular judgements of the slant of a cube face while varying eye position in the absence of stereoscopic and external lighting cues. Errors were found to be small, only 10% on average of the cube's eccentricity. Two factors appear to have contributed approximately equally to this error: an underestimate of cube slant as seen by the eye and an underestimate of eye position. When prism adaptation altered the sensed eye position, the pattern of slant judgements changed to reflect the altered sense of eye position.

3D after-effects are due to shape and not disparity adaptation.

There are a variety of stereoscopic after-effects in which exposure to a stimulus with a particular slant or curvature affects the perceived slant or curvature of a subsequently presented stimulus. These after-effects have been explained as a consequence of fatigue (a decrease in responsiveness) among neural mechanisms that are tuned to particular disparities or patterns of disparity. In fact, a given disparity pattern is consistent with numerous slants or curvatures; to determine slant or curvature, the visual system must take the viewing distance into account. We took advantage of this property to examine whether the mechanisms underlying the stereoscopic curvature after-effect are tuned to particular disparity patterns or to some other property such as surface curvature. The results clearly support the second hypothesis. Thus, 3D after-effects appear to be caused by adaptation among mechanisms specifying surface shape rather than among mechanisms signaling the disparity pattern.

Are corresponding points fixed?

Several investigators have claimed that the retinal coordinates of corresponding points shift with vergence eye movements. Two kinds of shifts have been reported. First, global shifts that increase with retinal eccentricity; such shifts would cause a flattening of the horopter at all viewing distances and would facilitate fusion of flat surfaces. Second, local shifts that are centered on the fovea; such shifts would cause a dimple in the horopter near fixation and would facilitate fusion of points fixated at extreme viewing distances. Nearly all of the empirical evidence supporting shifts of corresponding points comes from horopter measurements and from comparisons of subjective and objective fixation disparity. In both cases, the experimenter must infer the retinal coordinates of corresponding points from external measurements. We describe four factors that could affect this inference: (1) changes in the projection from object to image points that accompany eye rotation and accommodation, (2) fixation errors during the experimental measurements, (3) non-uniform retinal stretching, and (4) changes in the perceived direction of a monocular point when presented adjacent to a binocular point. We conducted two experiments that eliminated or compensated for these potential errors. In the first experiment, observers aligned dichoptic test lines using an apparatus and procedure that eliminated all but the third error. In the second experiment, observers judged the alignment of dichoptic afterimages, and this technique eliminates all the errors. The results from both experiments show that the retinal coordinates of corresponding points do not change with vergence eye movements. We conclude that corresponding points are in fixed retinal positions for observers with normal retinal correspondence.

Perceiving slant about a horizontal axis from stereopsis.

Rotating a surface about a horizontal axis alters the retinal horizontal-shear disparities. Opposed torsional eye movements (cyclovergence) also change horizontal shear. If there were no compensation for the horizontal disparities created by cyclovergence, slant estimates would be erroneous. We asked whether compensation for cyclovergence occurs, and, if it does, whether it occurs by use of an extraretinal cyclovergence signal, by use of vertical-shear disparities, or by use of both signals. In four experiments, we found that compensation is nearly veridical when vertical-shear disparities are available and easily measured. When they are not available or easily measured, no compensation occurs. Thus, the visual system does not seem to use an extraretinal cyclovergence signal in stereoscopic slant estimation. We also looked for evidence of an extraretinal cyclovergence signal in a visual direction task and found none. We calculated the statistical reliabilities of slant-from-disparity and slant-from-texture estimates and found that the more reliable of the two means of estimation varies significantly with distance and slant. Finally, we examined how slant about a horizontal axis might be estimated when the eyes look eccentrically.

Does a front-end nonlinearity confound VEP acuity measures in human infants?

The visual evoked potential is commonly used to estimate visual acuity in infants. The stimulus used is temporally modulated in order to drive the cortical response. Here it is proposed that distortion products generated by a front-end nonlinearity may contaminate the acuity estimate. Specifically, the nonlinearity might convert temporal modulation of a high spatial frequency grating into apparent whole-field flicker. Thus, the VEP may reflect an artifactual response to a high spatial frequency that is not resolved at the cortical level. If this were the case, one could null or attenuate the flicker response by adding whole-field flicker to the grating stimulus. We looked for such nulling effects in 18 infants aged 6-17 weeks. No consistent evidence was found for the nulling effect, so it was concluded that infant VEP acuity estimates are not significantly contaminated by the hypothesized distortion product.

Extraretinal and retinal amplitude and phase errors during Filehne illusion and path perception.

Pursuit eye movements give rise to retinal motion. To judge stimulus motion relative to the head, the visual system must correct for the eye movement by using an extraretinal, eye-velocity signal. Such correction is important in a variety of motion estimation tasks including judgments of object motion relative to the head and judgments of self-motion direction from optic flow. The Filehne illusion (where a stationary object appears to move opposite to the pursuit) results from a mismatch between retinal and extraretinal speed estimates. A mismatch in timing could also exist. Speed and timing errors were investigated using sinusoidal pursuit eye movements. We describe a new illusion--the slalom illusion--in which the perceived direction of self-motion oscillates left and right when the eyes move sinusoidally. A linear model is presented that determines the gain ratio and phase difference of extraretinal and retinal signals accompanying the Filehne and slalom illusions. The speed mismatch and timing differences were measured in the Filehne and self-motion situations using a motion-nulling procedure. Timing errors were very small for the Filehne and slalom illusions. However, the ratios of extraretinal to retinal gain were consistently less than 1, so both illusions are the consequence of a mismatch between estimates of retinal and extraretinal speed. The relevance of the results for recovering the direction of self-motion during pursuit eye movements is discussed.

Touch can change visual slant perception.

The visual system uses several signals to deduce the three-dimensional structure of the environment, including binocular disparity, texture gradients, shading and motion parallax. Although each of these sources of information is independently insufficient to yield reliable three-dimensional structure from everyday scenes, the visual system combines them by weighting the available information; altering the weights would therefore change the perceived structure. We report that haptic feedback (active touch) increases the weight of a consistent surface-slant signal relative to inconsistent signals. Thus, appearance of a subsequently viewed surface is changed: the surface appears slanted in the direction specified by the haptically reinforced signal.

Horizontal and vertical disparity, eye position, and stereoscopic slant perception.

The slant of a stereoscopically defined surface cannot be determined solely from horizontal disparities or from derived quantities such as horizontal size ratio (HSR). There are four other signals that, in combination with horizontal disparity, could in principle allow an unambiguous estimate of slant: the vergence and version of the eyes, the vertical size ratio (VSR), and the horizontal gradient of VSR. Another useful signal is provided by perspective slant cues. The determination of perceived slant can be modeled as a weighted combination of three estimates based on those signals: a perspective estimate, a stereoscopic estimate based on HSR and VSR, and a stereoscopic estimate based on HSR and sensed eye position. In a series of experiments, we examined human observers' use of the two stereoscopic means of estimation. Perspective cues were rendered uninformative. We found that VSR and sensed eye position are both used to interpret the measured horizontal disparities. When the two are placed in conflict, the visual system usually gives more weight to VSR. However, when VSR is made difficult to measure by using short stimuli or stimuli composed of vertical lines, the visual system relies on sensed eye position. A model in which the observer's slant estimate is a weighted average of the slant estimate based on HSR and VSR and the one based on HSR and eye position accounted well for the data. The weights varied across viewing conditions because the informativeness of the signals they employ vary from one situation to another.

Estimator reliability and distance scaling in stereoscopic slant perception.

When a horizontal or vertical magnifier is placed before one eye, a frontoparallel surface appears slanted. It appears slanted away from the eye with horizontal magnification (geometric effect) and toward the eye with vertical magnification (induced effect). According to current theory, the apparent slant in the geometric and induced effects should increase with viewing distance. The geometric effect does scale with distance, but there are conflicting reports as to whether the induced effect does. Ogle (1938 Archives of Ophthalmology 20 604-623) reported that settings in slant-nulling tasks increase systematically with viewing distance, but Gillam et al (1988 Perception & Psychophysics 44 473-483) and Rogers et al (1995 Perception 24 Supplement, 33) reported that settings in slant-estimation tasks do not. We re-examined this apparent contradiction. First, we conducted two experiments whose results are consistent with the literature and thus replicate the apparent contradiction. Next, we analyzed the signals available for stereoscopic slant perception and developed a general model of perceived slant. The model is based on the assumption that the visual system knows the reliability of various slant-estimation methods for the viewing situation under consideration. The model's behavior explains the contradiction in the literature. The model also predicts that, by manipulating eye position, apparent slant can be made to increase with distance for vertical, but not for horizontal, magnification. This prediction was confirmed experimentally.

Use of an early nonlinearity to measure optical and receptor resolution in the human infant.

We measured the resolution of the optics and receptoral processes in human infants. To do so, we recorded visual-evoked potentials (VEPs) to sampled sinewave gratings, stimuli that generate highly visible distortion products at a nonlinearity early in the retina. We varied the spatial frequency content of the stimulus to determine the frequencies that can be transmitted through the optics and receptors and thereby generate distortion products. Data were collected from adults and 2- to 7-month-old infants. The results indicated that the resolution of the infants' optical/receptoral processes was within a factor of two of adults' even at the earliest ages tested. These first stages of processing, therefore, do not explain infants' poor performance in many visual tasks, or restrict the types of visual stimuli affecting more central mechanisms that undergo experience-dependent development.

An analysis of binocular slant contrast.

When a small frontoparallel surface (a test strip) is surrounded by a larger slanted surface (an inducer), the test strip is perceived as slanted in the direction opposite to the inducer. This has been called the depth-contrast effect, but we call it the slant-contrast effect. In nearly all demonstrations of this effect, the inducer's slant is specified by stereoscopic signals; and other signals, such as the texture gradient, specify that it is frontoparallel. We present a theory of slant estimation that determines surface slant via linear combination of various slant estimators; the weight of each estimator is proportional to its reliability. The theory explains slant contrast because the absolute slant of the inducer and the relative slant between test strip and inducer are both estimated with greater reliability than the absolute slant of the test strip. The theory predicts that slant contrast will be eliminated if the signals specifying the inducer's slant are consistent with one another. It also predicts reversed slant contrast if the inducer's slant is specified by nonstereoscopic signals rather than by stereo signals. These predictions were tested and confirmed in three experiments. The first showed that slant contrast is greatly reduced when the stereo-specified and nonstereo-specified slants of the inducer are made consistent with one another. The second showed that slant contrast is eliminated altogether when the stimulus consists of real planes rather than images on a display screen. The third showed that slant contrast is reversed when the nonstereo-specified slant of the inducer varies and the stereo-specified slant is zero. We conclude that slant contrast is a byproduct of the visual system's reconciliation of conflicting information while it attempts to determine surface slant.

Perceived visual direction near an occluder.

When an opaque object occludes a more distant object, the two eyes often see different parts of the distant object. Hering's laws of visual direction make an interesting prediction for this situation: the part seen by both eyes should be seen in a different direction than the part seen by one eye. We examined whether this prediction holds by asking observers to align a vertical monocular line segment with a nearby vertical binocular segment. We found it necessary to correct the alignment data for vergence errors, which were measured in a control experiment, and for monocular spatial distortions, which were also measured in a control experiment. Settings were reasonably consistent with Hering's laws when the monocular and binocular targets were separated by 30 arcmin or more. Observers aligned the targets as if they were viewing them from one eye only when they were separated by 2 arcmin; this behavior is consistent with an observation reported by Erkelens and colleagues. The same behavior was observed when the segments were horizontal and when no visible occluder was present. Perceived visual direction when the two eyes see different parts of a distant target is assigned in a fashion that minimizes, but does not eliminate, distortions of the shape of the occluded object.

Perceived head-centric speed is affected by both extra-retinal and retinal errors.

When we make a smooth eye movement to track a moving object, the visual system must take the eye's movement into account in order to estimate the object's velocity relative to the head. This can be done by using extra-retinal signals to estimate eye velocity and then subtracting expected from observed retinal motion. Two familiar illusions of perceived velocity--the Filehne illusion and Aubert-Fleischl phenomenon--are thought to be the consequence of the extra-retinal signal underestimating eye velocity. These explanations assume that retinal motion is encoded accurately, which is questionable because perceived retinal speed is strongly affected by several stimulus properties. We develop and test a model of head-centric velocity perception that incorporates errors in estimating eye velocity and in retinal-motion sensing. The model predicts that the magnitude and direction of the Filehne illusion and Aubert-Fleischl phenomenon depend on spatial frequency and this prediction is confirmed experimentally.

Extra-retinal and perspective cues cause the small range of the induced effect.

With a horizontal magnifier before one eye, a frontoparallel surface appears rotated about a vertical axis (geometric effect). With a vertical magnifier, apparent rotation is opposite in direction (induced effect); to restore appearance of frontoparallelism, the surface must be rotated away from the magnified eye. The induced effect is interesting because it was thought until recently that vertical disparities do not play an important role in surface perception. As with the geometric effect, the required rotation for the induced effect increases linearly to approximately equal to 4% magnification; unlike the geometric effect, it plateaus at approximately 8%. Current theory explains the linear portion: vertical size ratios (VSRs) are used to compensate for changes in horizontal size ratios (HSRs) that accompany eccentric gaze, so changes in VSR cause changes in perceived slant. The theory does not explain the plateau. We demonstrate that it results from differing slant estimates obtained by use of various retinal and extra-retinal signals. When perspective cues to slant are minimized or sensed eye position is consistent with VSR, the induced and geometric effects have similar magnitudes even at large magnifications.

Optical, receptoral, and retinal constraints on foveal and peripheral vision in the human neonate.

We examined the properties of the foveal, parafoveal, and near peripheral cone lattice in human neonates. To estimate the ability of these lattices to transmit the information used in contrast sensitivity and visual acuity tasks, we constructed ideal-observer models with the optics and photoreceptors of the neonatal eye at retinal eccentricities of 0, 5, and 10 degrees. For ideal-observer models limited by photon noise, the eye's optics, and cone properties, contrast sensitivity was higher in the parafovea and near periphery than in the fovea. However, receptor pooling probably occurs in the neonate's parafovea and near periphery as it does in mature eyes. When we add a receptor-pooling stage to the models of the parafovea and near periphery, ideal acuity is similar in the fovea, parafovea, and near periphery. Comparisons of ideal and real sensitivity indicate that optical and receptoral immaturities impose a significant constraint on neonatal contrast sensitivity and acuity, but that immaturities in later processing stages must also limit visual performance.

Visual self-motion perception during head turns.

Extra-retinal information is critical in the interpretation of visual input during self-motion. Turning our eyes and head to track objects displaces the retinal image but does not affect our ability to navigate because we use extra-retinal information to compensate for these displacements. We showed observers animated displays depicting their forward motion through a scene. They perceived the simulated self-motion accurately while smoothly shifting the gaze by turning the head, but not when the same gaze shift was simulated in the display; this indicates that the visual system also uses extra-retinal information during head turns. Additional experiments compared self-motion judgments during active and passive head turns, passive rotations of the body and rotations of the body with head fixed in space. We found that accurate perception during active head turns is mediated by contributions from three extra-retinal cues: vestibular canal stimulation, neck proprioception and an efference copy of the motor command to turn the head.

Depth information and perceived self-motion during simulated gaze rotations.

When presented with random-dot displays with little depth information, observers cannot determine their direction of self-motion accurately in the presence of rotational flow without appropriate extra-retinal information (Royden CS et al. Vis Res 1994;34:3197-214.). On theoretical grounds, one might expect improved performance when depth information is added to the display (van den Berg AV and Brenner E. Nature 1994;371:700-2). We examined this possibility by having observers indicate perceived self-motion paths when the amount of depth information was varied. When stereoscopic cues and a variety of monocular depth cues were added, observers still misperceived the depicted self-motion when the rotational flow in the display was not accompanied by an appropriate extra-retinal, eye-velocity signal. Specifically, they perceived curved self-motion paths with the curvature in the direction of the simulated eye rotation. The distance to the response marker was crucial to the objective measurement of this misperception. When the marker distance was small, the observers' settings were reasonably accurate despite the misperception of the depicted self-motion. When the marker distance was large, the settings exhibited the errors reported previously by Royden CS et al. Vis Res 1994;34-3197-3214. The path judgement errors observers make during simulated gaze rotations appear to be the result of misattributing path-independent rotation to self-motion along a circular path with path-dependent rotation. An analysis of the information an observer could use to avoid such errors reveals that the addition of depth information is of little use.

The computation of binocular visual direction: a re-examination of Mansfield and Legge (1996)

Mansfield and Legge (1996) reported recently that a target's perceived binocular direction is dependent on the ratio of contrasts presented to the two eyes. Although their main conclusion concerned the dependence of perceived direction on interocular contrast, they also argued that the change in perceived direction is due to a shift in the position of the cyclopean eye and that the relative directions of binocular targets are unaffected by eye position. We take issue with both of these arguments. With regard to the former, their task was an alignment task, not an egocenter task, so it did not provide information relevant to the position of the cyclopean eye. Indeed, their data can be explained by the conventional theory of binocular visual directions with a fixed cyclopean eye (e.g., Hering, 1879; Ono, 1981) once a simple, but important modification is added. With regard to their conclusion concerning eye position, we show that the vergence of the eyes has a clear and systematic effect on perceived relative directions in the setup used by Mansfield and Legge.