BiPMAP: a toolbox for predicting perceived motion artifacts on modern displays.

Viewers of digital displays often experience motion artifacts (e.g., flicker, judder, edge banding, motion blur, color breakup, depth distortion) when presented with dynamic scenes. We developed an interactive software tool for display designers that predicts how a viewer perceives motion artifacts for a variety of stimulus, display, and viewing parameters: the Binocular Perceived Motion Artifact Predictor (BiPMAP). The tool enables the user to specify numerous stimulus, display, and viewing parameters. It implements a model of human spatiotemporal contrast sensitivity in order to determine which artifacts will be seen by a viewer and which will not. The tool visualizes the perceptual effects of discrete space-time sampling on the display by presenting side by side the expected perception when the stimulus is continuous compared to when the same stimulus is presented with the spatial and temporal parameters of a prototype display.

Crossed-uncrossed projections from primate retina are adapted to disparities of natural scenes.

In mammals with frontal eyes, optic-nerve fibers from nasal retina project to the contralateral hemisphere of the brain, and fibers from temporal retina project ipsilaterally. The division between crossed and uncrossed projections occurs at or near the vertical meridian. If the division was precise, a problem would arise. Small objects near midline, but nearer or farther than current fixation, would produce signals that travel to opposite hemispheres, making the binocular disparity of those objects difficult to compute. However, in species that have been studied, the division is not precise. Rather, there are overlapping crossed and uncrossed projections such that some fibers from nasal retina project ipsilaterally as well as contralaterally and some from temporal retina project contralaterally as well as ipsilaterally. This increases the probability that signals from an object near vertical midline travel to the same hemisphere, thereby aiding disparity estimation. We investigated whether there is a deficit in binocular vision near the vertical meridian in humans and found no evidence for one. We also investigated the effectiveness of the observed decussation pattern, quantified from anatomical data in monkeys and humans. We used measurements of naturally occurring disparities in humans to determine disparity distributions across the visual field. We then used those distributions to calculate the probability of natural disparities transmitting to the same hemisphere, thereby aiding disparity computation. We found that the pattern of overlapping projections is quite effective. Thus, crossed and uncrossed projections from the retinas are well designed for aiding disparity estimation and stereopsis.

Optics and neural adaptation jointly limit human stereovision.

Stereovision is the ability to perceive fine depth variations from small differences in the two eyes' images. Using adaptive optics, we show that even minute optical aberrations that are not clinically correctable, and go unnoticed in everyday vision, can affect stereo acuity. Hence, the human binocular system is capable of using fine details that are not experienced in everyday vision. Interestingly, stereo acuity varied considerably across individuals even when they were provided identical perfect optics. We also found that individuals' stereo acuity is better when viewing with their habitual optics rather than someone else's (better) optics. Together, these findings suggest that the visual system compensates for habitual optical aberrations through neural adaptation and thereby optimizes stereovision uniquely for each individual. Thus, stereovision is limited by small optical aberrations and by neural adaptation to one's own optics.

The visual benefits of correcting longitudinal and transverse chromatic aberration.

We describe a system-the Binocular Varichrome and Accommodation Measurement System-that can be used to measure and correct the eye's longitudinal and transverse chromatic aberration (LCA and TCA) and to perform vision tests with custom corrections. We used the system to investigate how LCA and TCA affect visual performance. Specifically, we studied the effects of LCA and TCA on visual acuity, contrast sensitivity, and chromostereopsis. LCA exhibited inter subject variability but followed expected trends compared with previous reports. TCA at the fovea was variable between individuals but with a tendency for the shift at shorter wavelengths to be more temporalward in the visual field in each eye. We found that TCA was generally greater when LCA was corrected. For visual acuity, we found that a measurable benefit was realized only with both LCA and TCA correction unless the TCA was low. For contrast sensitivity, we found that the best sensitivity to a 10-cycle/degree polychromatic grating was attained when LCA and TCA were corrected. Finally, we found that the primary cause of chromostereopsis is the TCA of the eyes.

High-fidelity eye, head, body, and world tracking with a wearable device.

We describe the design and performance of a high-fidelity wearable head-, body-, and eye-tracking system that offers significant improvement over previous such devices. This device's sensors include a binocular eye tracker, an RGB-D scene camera, a high-frame-rate scene camera, and two visual odometry sensors, for a total of ten cameras, which we synchronize and record from with a data rate of over 700 MB/s. The sensors are operated by a mini-PC optimized for fast data collection, and powered by a small battery pack. The device records a subject's eye, head, and body positions, simultaneously with RGB and depth data from the subject's visual environment, measured with high spatial and temporal resolution. The headset weighs only 1.4 kg, and the backpack with batteries 3.9 kg. The device can be comfortably worn by the subject, allowing a high degree of mobility. Together, this system overcomes many limitations of previous such systems, allowing high-fidelity characterization of the dynamics of natural vision.

Lags and leads of accommodation in humans: Fact or fiction?

The focusing response of the human eye - accommodation - exhibits errors known as lags and leads. Lags occur when the stimulus is near and the eye appears to focus farther than the stimulus. Leads occur with far stimuli where the eye appears to focus nearer than the stimulus. We used objective and subjective measures simultaneously to determine where the eye is best focused. The objective measures were made with a wavefront sensor and an autorefractor, both of which analyze light reflected from the retina. These measures exhibited typical accommodative errors, mostly lags. The subjective measure was visual acuity, which of course depends not only on the eye's optics but also on photoreception and neural processing of the retinal image. The subjective measure revealed much smaller errors. Acuity was maximized at or very close to the distance of the accommodative stimulus. Thus, accommodation is accurate in terms of maximizing visual performance.

The blur horopter: Retinal conjugate surface in binocular viewing.

From measurements of wavefront aberrations in 16 emmetropic eyes, we calculated where objects in the world create best-focused images across the central 27\(^\circ\) (diameter) of the retina. This is the retinal conjugate surface. We calculated how the surface changes as the eye accommodates from near to far and found that it mostly maintains its shape. The conjugate surface is pitched top-back, meaning that the upper visual field is relatively hyperopic compared to the lower field. We extended the measurements of best image quality into the binocular domain by considering how the retinal conjugate surfaces for the two eyes overlap in binocular viewing. We call this binocular extension the blur horopter. We show that in combining the two images with possibly different sharpness, the visual system creates a larger depth of field of apparently sharp images than occurs with monocular viewing. We examined similarities between the blur horopter and its analog in binocular vision: the binocular horopter. We compared these horopters to the statistics of the natural visual environment. The binocular horopter and scene statistics are strikingly similar. The blur horopter and natural statistics are qualitatively, but not quantitatively, similar. Finally, we used the measurements to refine what is commonly referred to as the zone of clear single binocular vision.

Binocular Eye Movements Are Adapted to the Natural Environment.

Humans and many animals make frequent saccades requiring coordinated movements of the eyes. When landing on the new fixation point, the eyes must converge accurately or double images will be perceived. We asked whether the visual system uses statistical regularities in the natural environment to aid eye alignment at the end of saccades. We measured the distribution of naturally occurring disparities in different parts of the visual field. The central tendency of the distributions was crossed (nearer than fixation) in the lower field and uncrossed (farther) in the upper field in male and female participants. It was uncrossed in the left and right fields. We also measured horizontal vergence after completion of vertical, horizontal, and oblique saccades. When the eyes first landed near the eccentric target, vergence was quite consistent with the natural-disparity distribution. For example, when making an upward saccade, the eyes diverged to be aligned with the most probable uncrossed disparity in that part of the visual field. Likewise, when making a downward saccade, the eyes converged to enable alignment with crossed disparity in that part of the field. Our results show that rapid binocular eye movements are adapted to the statistics of the 3D environment, minimizing the need for large corrective vergence movements at the end of saccades. The results are relevant to the debate about whether eye movements are derived from separate saccadic and vergence neural commands that control both eyes or from separate monocular commands that control the eyes independently.SIGNIFICANCE STATEMENT We show that the human visual system incorporates statistical regularities in the visual environment to enable efficient binocular eye movements. We define the oculomotor horopter: the surface of 3D positions to which the eyes initially move when stimulated by eccentric targets. The observed movements maximize the probability of accurate fixation as the eyes move from one position to another. This is the first study to show quantitatively that binocular eye movements conform to 3D scene statistics, thereby enabling efficient processing. The results provide greater insight into the neural mechanisms underlying the planning and execution of saccadic eye movements.

Contributions of foveal and non-foveal retina to the human eye's focusing response.

The human eye changes focus-accommodates-to minimize blur in the retinal image. Previous work has shown that stimulation of nonfoveal retina can produce accommodative responses when no competing stimulus is presented to the fovea. In everyday situations it is very common for the fovea and other parts of the retina to be stimulated simultaneously. We examined this situation by asking how nonfoveal retina contributes to accommodation when the fovea is also stimulated. There were three experimental conditions. (a) Real change in which stimuli of different sizes, centered on the fovea, were presented at different optical distances. Accommodation was, as expected, robust because there was no conflicting stimulation of other parts of the retina. (b) Simulated change, no conflict in which stimuli of different sizes, again centered on the fovea, were presented at different simulated distances using rendered chromatic blur. Accommodation was robust in this condition because there was no conflict between the central and peripheral stimuli. (c) Simulated change, conflict in which a central disk (of different diameters) was presented along with an abutting peripheral annulus. The disk and annulus underwent opposite changes in simulated distance. Here we observed a surprisingly consistent effect of the peripheral annulus. For example, when the diameter of the central stimulus was 8° (thereby stimulating the fovea and parafovea), the abutting peripheral annulus had a significant effect on accommodation. We discuss how these results may help us understand other situations in which nonfixated targets affect the ability to focus on a fixated target. We also discuss potential implications for the development of myopia and for foveated rendering.

Creating correct blur and its effect on accommodation.

Blur occurs naturally when the eye is focused at one distance and an object is presented at another distance. Computer-graphics engineers and vision scientists often wish to create display images that reproduce such depth-dependent blur, but their methods are incorrect for that purpose. They take into account the scene geometry, pupil size, and focal distances, but do not properly take into account the optical aberrations of the human eye. We developed a method that, by incorporating the viewer's optics, yields displayed images that produce retinal images close to the ones that occur in natural viewing. We concentrated on the effects of defocus, chromatic aberration, astigmatism, and spherical aberration and evaluated their effectiveness by conducting experiments in which we attempted to drive the eye's focusing response (accommodation) through the rendering of these aberrations. We found that accommodation is not driven at all by conventional rendering methods, but that it is driven surprisingly quickly and accurately by our method with defocus and chromatic aberration incorporated. We found some effect of astigmatism but none of spherical aberration. We discuss how the rendering approach can be used in vision science experiments and in the development of ophthalmic/optometric devices and augmented- and virtual-reality displays.

Identifying Computer-Generated Portraits: The Importance of Training and Incentives.

The past two decades have seen remarkable advances in photo-realistic rendering of everything from inanimate objects to landscapes, animals, and humans. We previously showed that despite these tremendous advances, human observers remain fairly good at distinguishing computer-generated from photographic images. Building on these results, we describe a series of follow-up experiments that reveal how to improve observer performance. Of general interest to anyone performing psychophysical studies on Mechanical Turk or similar platforms, we find that observer performance can be significantly improved with the proper incentives.

Dynamic lens and monovision 3D displays to improve viewer comfort.

Stereoscopic 3D (S3D) displays provide an additional sense of depth compared to non-stereoscopic displays by sending slightly different images to the two eyes. But conventional S3D displays do not reproduce all natural depth cues. In particular, focus cues are incorrect causing mismatches between accommodation and vergence: The eyes must accommodate to the display screen to create sharp retinal images even when binocular disparity drives the eyes to converge to other distances. This mismatch causes visual discomfort and reduces visual performance. We propose and assess two new techniques that are designed to reduce the vergence-accommodation conflict and thereby decrease discomfort and increase visual performance. These techniques are much simpler to implement than previous conflict-reducing techniques. The first proposed technique uses variable-focus lenses between the display and the viewer's eyes. The power of the lenses is yoked to the expected vergence distance thereby reducing the mismatch between vergence and accommodation. The second proposed technique uses a fixed lens in front of one eye and relies on the binocularly fused percept being determined by one eye and then the other, depending on simulated distance. We conducted performance tests and discomfort assessments with both techniques and compared the results to those of a conventional S3D display. The first proposed technique, but not the second, yielded clear improvements in performance and reductions in discomfort. This dynamic-lens technique therefore offers an easily implemented technique for reducing the vergence-accommodation conflict and thereby improving viewer experience.

Blur and the perception of depth at occlusions.

The depth ordering of two surfaces, one occluding the other, can in principle be determined from the correlation between the occlusion border's blur and the blur of the two surfaces. If the border is blurred, the blurrier surface is nearer; if the border is sharp, the sharper surface is nearer. Previous research has found that observers do not use this informative cue. We reexamined this finding. Using a multiplane display, we confirmed the previous finding: Our observers did not accurately judge depth order when the blur was rendered and the stimulus presented on one plane. We then presented the same simulated scenes on multiple planes, each at a different focal distance, so the blur was created by the optics of the eye. Performance was now much better, which shows that depth order can be reliably determined from blur information but only when the optical effects are similar to those in natural viewing. We asked what the critical differences were in the single- and multiplane cases. We found that chromatic aberration provides useful information but accommodative microfluctuations do not. In addition, we examined how image formation is affected by occlusions and observed some interesting phenomena that allow the eye to see around and through occluding objects and may allow observers to estimate depth in da Vinci stereopsis, where one eye's view is blocked. Finally, we evaluated how accurately different rendering and displaying techniques reproduce the retinal images that occur in real occlusions. We discuss implications for computer graphics.

The natural statistics of blur.

Blur from defocus can be both useful and detrimental for visual perception: It can be useful as a source of depth information and detrimental because it degrades image quality. We examined these aspects of blur by measuring the natural statistics of defocus blur across the visual field. Participants wore an eye-and-scene tracker that measured gaze direction, pupil diameter, and scene distances as they performed everyday tasks. We found that blur magnitude increases with increasing eccentricity. There is a vertical gradient in the distances that generate defocus blur: Blur below the fovea is generally due to scene points nearer than fixation; blur above the fovea is mostly due to points farther than fixation. There is no systematic horizontal gradient. Large blurs are generally caused by points farther rather than nearer than fixation. Consistent with the statistics, participants in a perceptual experiment perceived vertical blur gradients as slanted top-back whereas horizontal gradients were perceived equally as left-back and right-back. The tendency for people to see sharp as near and blurred as far is also consistent with the observed statistics. We calculated how many observations will be perceived as unsharp and found that perceptible blur is rare. Finally, we found that eye shape in ground-dwelling animals conforms to that required to put likely distances in best focus.

The limits of human stereopsis in space and time.

To encode binocular disparity, the visual system determines the image patches in one eye that yield the highest correlation with patches in the other eye. The computation of interocular correlation occurs after spatiotemporal filtering of monocular signals, which leads to restrictions on disparity variations that can support depth perception. We quantified those restrictions by measuring humans' ability to see disparity variation at a wide range of spatial and temporal frequencies. Lower-disparity thresholds cut off at very low spatiotemporal frequencies, which is consistent with the behavior of V1 neurons. Those thresholds are space-time separable, suggesting that the underlying neural mechanisms are separable. We also found that upper-disparity limits were characterized by a spatiotemporal, disparity-gradient limit; to be visible, disparity variation cannot exceed a fixed amount for a given interval in space-time. Our results illustrate that the disparity variations that humans can see are very restricted compared with the corresponding luminance variations. The results also provide insight into the neural mechanisms underlying depth from disparity, such as why stimuli with long interocular delays can still yield clear depth percepts.

Stereoscopic 3D display technique using spatiotemporal interlacing has improved spatial and temporal properties.

Stereoscopic 3D (S3D) displays use spatial or temporal interlacing to send different images to the two eyes. Temporal interlacing delivers images to the left and right eyes alternately in time; it has high effective spatial resolution but is prone to temporal artifacts. Spatial interlacing delivers even pixel rows to one eye and odd rows to the other eye simultaneously; it is subject to spatial limitations such as reduced spatial resolution. We propose a spatiotemporal-interlacing protocol that interlaces the left- and right-eye views spatially, but with the rows being delivered to each eye alternating with each frame. We performed psychophysical experiments and found that flicker, motion artifacts, and depth distortion are substantially reduced relative to the temporal-interlacing protocol, and spatial resolution is better than in the spatial-interlacing protocol. Thus, the spatiotemporal-interlacing protocol retains the benefits of spatial and temporal interlacing while minimizing or even eliminating the drawbacks.

Stereoscopic 3D display with color interlacing improves perceived depth.

Temporal interlacing is a method for presenting stereoscopic 3D content whereby the two eyes' views are presented at different times and optical filtering selectively delivers the appropriate view to each eye. This approach is prone to distortions in perceived depth because the visual system can interpret the temporal delay between binocular views as spatial disparity. We propose a novel color-interlacing display protocol that reverses the order of binocular presentation for the green primary but maintains the order for the red and blue primaries: During the first sub-frame, the left eye sees the green component of the left-eye view and the right eye sees the red and blue components of the right-eye view, and vice versa during the second sub-frame. The proposed method distributes the luminance of each eye's view more evenly over time. Because disparity estimation is based primarily on luminance information, a more even distribution of luminance over time should reduce depth distortion. We conducted a psychophysical experiment to test these expectations and indeed found that less depth distortion occurs with color interlacing than temporal interlacing.

The visibility of color breakup and a means to reduce it.

Color breakup is an artifact seen on displays that present colors sequentially. When the eye tracks a moving object on such a display, different colors land on different places on the retina, and this gives rise to visible color fringes at the object's leading and trailing edges. Interestingly, color breakup is also observed when the eye is stationary and an object moves by. Using a novel psychophysical procedure, we measured breakup both when viewers tracked and did not track a moving object. Breakup was somewhat more visible in the tracking than in the non-tracking condition. The video frames contained three subframes, one each for red, green, and blue. We spatially offset the green and blue stimuli in the second and third subframes, respectively, to find the values that minimized breakup. In the tracking and non-tracking conditions, spatial offsets of Δx/3 in the second subframe (where Δx is the displacement of the object in one frame) and 2Δx/3 in the third eliminated breakup. Thus, this method offers a way to minimize or even eliminate breakup whether the viewer is tracking or not. We suggest ways to implement the method with real video content. We also developed a color-breakup model based on spatiotemporal filtering in color-opponent pathways in early vision. We found close agreement between the model's predictions and the experimental results. The model can be used to predict breakup for a wide variety of conditions.

The Role of Chromatic Aberration in Vision.

The study of biological optics would be complicated enough if light only came in a single wavelength. However, altering the wavelength (or distribution of wavelengths) of light has multiple effects on optics, including on diffraction, scattering (of various sorts), transmission through and reflection by various media, fluorescence, and waveguiding properties, among others. In this review, we consider just one wavelength-dependent optical effect: longitudinal chromatic aberration (LCA). All vertebrate eyes that have been tested have significant LCA, with shorter (bluer) wavelengths of light focusing closer to the front of the eye than longer (redder) wavelengths. We consider the role of LCA in the visual system in terms of both how it could degrade visual acuity and how biological systems make use of it.

The Statistics of Eye Movements and Binocular Disparities during VR Gaming: Implications for Headset Design.

The human visual system evolved in environments with statistical regularities. Binocular vision is adapted to these such that depth perception and eye movements are more precise, faster, and performed comfortably in environments consistent with the regularities. We measured the statistics of eye movements and binocular disparities in virtual-reality (VR) - gaming environments and found that they are quite different from those in the natural environment. Fixation distance and direction are more restricted in VR, and fixation distance is farther. The pattern of disparity across the visual field is less regular in VR and does not conform to a prominent property of naturally occurring disparities. From this we predict that double vision is more likely in VR than in the natural environment. We also determined the optimal screen distance to minimize discomfort due to the vergence-accommodation conflict, and the optimal nasal-temporal positioning of head-mounted display (HMD) screens to maximize binocular field of view. Finally, in a user study we investigated how VR content affects comfort and performance. Content that is more consistent with the statistics of the natural world yields less discomfort than content that is not. Furthermore, consistent content yields slightly better performance than inconsistent content.