Theoretical impact of chromatic aberration correction on visual acuity.

It has been known for more than 220 years that the image quality of the human eye is significantly degraded by chromatic aberrations. Recently, it was shown experimentally that correcting chromatic aberrations results in a 0.2- to 0.8-line improvement in visual acuity. Here we ask, is this expected? We developed tools that enable simulations of the optical impact of physiologically relevant amounts of chromatic aberration in real human eyes and combined these with tools that compute the visual acuity of an ideal observer. This allows us to characterize the theoretical impact of chromatic aberration correction on visual acuity. Results indicate a substantive improvement of 0.4- to 2-lines in ideal observer visual acuity with chromatic aberration correction. Ideal observer thresholds benefit significantly more from correction of longitudinal than correction of transverse chromatic aberration. Finally, improvements in ideal observer visual acuity are greater for subjects with less monochromatic aberration, such that subjects with better baseline optical quality benefit most from correction of chromatic aberrations.

Orthogonal neural representations support perceptual judgements of natural stimuli.

In natural behavior, observers must separate relevant information from a barrage of irrelevant information. Many studies have investigated the neural underpinnings of this ability using artificial stimuli presented on simple backgrounds. Natural viewing, however, carries a set of challenges that are inaccessible using artificial stimuli, including neural responses to background objects that are task-irrelevant. An emerging body of evidence suggests that the visual abilities of humans and animals can be modeled through the linear decoding of task-relevant information from visual cortex. This idea suggests the hypothesis that irrelevant features of a natural scene should impair performance on a visual task only if their neural representations intrude on the linear readout of the task relevant feature, as would occur if the representations of task-relevant and irrelevant features are not orthogonal in the underlying neural population. We tested this hypothesis using human psychophysics and monkey neurophysiology, in response to parametrically variable naturalistic stimuli. We demonstrate that 1) the neural representation of one feature (the position of a central object) in visual area V4 is orthogonal to those of several background features, 2) the ability of human observers to precisely judge object position was largely unaffected by task-irrelevant variation in those background features, and 3) many features of the object and the background are orthogonally represented by V4 neural responses. Our observations are consistent with the hypothesis that orthogonal neural representations can support stable perception of objects and features despite the tremendous richness of natural visual scenes. Significance Statement: We studied how the structure of the mid-level neural representation of multiple visual features supports robust perceptual decisions. We combined array recording with parametrically controlled naturalistic images to demonstrate that the representation of a central object's position in monkey visual area V4 is orthogonal to that of several background features. In addition, we used human psychophysics with the same stimulus set to show that observers' ability to judge a central object's position is largely unaffected by variation in the same background features. This result supports the hypothesis that orthogonal neural representations can enable stable and robust perception in naturalistic visual environments and advances our understanding of how visual processing operates in the real world.

History constrains the evolution of efficient color naming, enabling historical inference.

Color naming in natural languages is not arbitrary: It reflects efficient partitions of perceptual color space [T. Regier, P. Kay, N. Khetarpal, Proc. Natl. Acad. Sci. U.S.A. 104, 1436-1441 (2007)] modulated by the relative needs to communicate about different colors [C. Twomey, G. Roberts, D. Brainard, J. Plotkin, Proc. Natl. Acad. Sci. U.S.A. 118, e2109237118 (2021)]. These psychophysical and communicative constraints help explain why languages around the world have remarkably similar, but not identical, mappings of colors to color terms. Languages converge on a small set of efficient representations.But languages also evolve, and the number of terms in a color vocabulary may change over time. Here we show that history, i.e. the existence of an antecedent color vocabulary, acts as a nonadaptive constraint that biases the choice of efficient solution as a language transitions from a vocabulary of size [Formula: see text] to [Formula: see text] terms. Moreover, as efficient vocabularies evolve to include more terms they explore a smaller fraction of all possible efficient vocabularies compared to equally sized vocabularies constructed de novo. This path dependence of the cultural evolution of color naming presents an opportunity. Historical constraints can be used to reconstruct ancestral color vocabularies, allowing us to answer long-standing questions about the evolutionary sequences of color words, and enabling us to draw inferences from phylogenetic patterns of language change.

In vivo chromatic and spatial tuning of foveolar retinal ganglion cells in Macaca fascicularis.

The primate fovea is specialized for high acuity chromatic vision, with the highest density of cone photoreceptors and a disproportionately large representation in visual cortex. The unique visual properties conferred by the fovea are conveyed to the brain by retinal ganglion cells, the somas of which lie at the margin of the foveal pit. Microelectrode recordings of these centermost retinal ganglion cells have been challenging due to the fragility of the fovea in the excised retina. Here we overcome this challenge by combining high resolution fluorescence adaptive optics ophthalmoscopy with calcium imaging to optically record functional responses of foveal retinal ganglion cells in the living eye. We use this approach to study the chromatic responses and spatial transfer functions of retinal ganglion cells using spatially uniform fields modulated in different directions in color space and monochromatic drifting gratings. We recorded from over 350 cells across three Macaca fascicularis primates over a time period of weeks to months. We find that the majority of the L vs. M cone opponent cells serving the most central foveolar cones have spatial transfer functions that peak at high spatial frequencies (20-40 c/deg), reflecting strong surround inhibition that sacrifices sensitivity at low spatial frequencies but preserves the transmission of fine detail in the retinal image. In addition, we fit to the drifting grating data a detailed model of how ganglion cell responses draw on the cone mosaic to derive receptive field properties of L vs. M cone opponent cells at the very center of the foveola. The fits are consistent with the hypothesis that foveal midget ganglion cells are specialized to preserve information at the resolution of the cone mosaic. By characterizing the functional properties of retinal ganglion cells in vivo through adaptive optics, we characterize the response characteristics of these cells in situ.

Visual encoding: Principles and software.

For more than two centuries scientists and engineers have worked to understand and model how the eye encodes electromagnetic radiation (light). We now understand the principles of how light is transmitted through the optics of the eye and encoded by retinal photoreceptors and light-sensitive neurons. In recent years, new instrumentation has enabled scientists to measure the specific parameters of the optics and photoreceptor encoding. We implemented the principles and parameter estimates that characterize the human eye in an open-source software toolbox. This chapter describes the principles behind these tools and illustrates how to use them to compute the initial visual encoding.

Automated Assessment of Photoreceptor Visibility in Adaptive Optics Split-Detection Images Using Edge Detection.

Purpose: Adaptive optics scanning laser ophthalmoscopy (AOSLO) is a high-resolution imaging modality that allows measurements of cellular-level retinal changes in living patients. In retinal diseases, the visibility of photoreceptors in AOSLO images is affected by pathology, patient motion, and optics, which can lead to variability in analyses of the photoreceptor mosaic. Current best practice for AOSLO mosaic quantification requires manual assessment of photoreceptor visibility across overlapping images, a laborious and time-consuming task. Methods: We propose an automated measure for quantification of photoreceptor visibility in AOSLO. Our method detects salient edge features, which can represent visible photoreceptor boundaries in each image. We evaluate our measure against two human graders and two standard automated image quality assessment algorithms. Results: We evaluate the accuracy of pairwise ordering (PO) and the correlation of ordinal rankings (ORs) of photoreceptor visibility in 29 retinal regions, taken from five subjects with choroideremia. The proposed measure had high association with manual assessments (Grader 1: PO = 0.71, OR = 0.61; Grader 2: PO = 0.67, OR = 0.62), which is comparable with intergrader reliability (PO = 0.76, OR = 0.75) and outperforms the top standard approach (PO = 0.57; OR = 0.46). Conclusions: Our edge-based measure can automatically assess photoreceptor visibility and order overlapping images within AOSLO montages. This can significantly reduce the manual labor required to generate high-quality AOSLO montages and enables higher throughput for quantitative studies of photoreceptors. Translational Relevance: Automated assessment of photoreceptor visibility allows us to more rapidly quantify photoreceptor morphology in the living eye. This has applications to ophthalmic medicine by allowing detailed characterization of retinal degenerations, thus yielding potential biomarkers of treatment safety and efficacy.

Equivalent noise characterization of human lightness constancy.

A goal of visual perception is to provide stable representations of task-relevant scene properties (e.g. object reflectance) despite variation in task-irrelevant scene properties (e.g. illumination and reflectance of other nearby objects). To study such stability in the context of the perceptual representation of lightness, we introduce a threshold-based psychophysical paradigm. We measure how thresholds for discriminating the achromatic reflectance of a target object (task-relevant property) in rendered naturalistic scenes are impacted by variation in the reflectance functions of background objects (task-irrelevant property), using a two-alternative forced-choice paradigm in which the reflectance of the background objects is randomized across the two intervals of each trial. We control the amount of background reflectance variation by manipulating a statistical model of naturally occurring surface reflectances. For low background object reflectance variation, discrimination thresholds were nearly constant, indicating that observers' internal noise determines threshold in this regime. As background object reflectance variation increases, its effects start to dominate performance. A model based on signal detection theory allows us to express the effects of task-irrelevant variation in terms of the equivalent noise, that is relative to the intrinsic precision of the task-relevant perceptual representation. The results indicate that although naturally occurring background object reflectance variation does intrude on the perceptual representation of target object lightness, the effect is modest - within a factor of two of the equivalent noise level set by internal noise.

An image reconstruction framework for characterizing initial visual encoding.

We developed an image-computable observer model of the initial visual encoding that operates on natural image input, based on the framework of Bayesian image reconstruction from the excitations of the retinal cone mosaic. Our model extends previous work on ideal observer analysis and evaluation of performance beyond psychophysical discrimination, takes into account the statistical regularities of the visual environment, and provides a unifying framework for answering a wide range of questions regarding the visual front end. Using the error in the reconstructions as a metric, we analyzed variations of the number of different photoreceptor types on human retina as an optimal design problem. In addition, the reconstructions allow both visualization and quantification of information loss due to physiological optics and cone mosaic sampling, and how these vary with eccentricity. Furthermore, in simulations of color deficiencies and interferometric experiments, we found that the reconstructed images provide a reasonable proxy for modeling subjects' percepts. Lastly, we used the reconstruction-based observer for the analysis of psychophysical threshold, and found notable interactions between spatial frequency and chromatic direction in the resulting spatial contrast sensitivity function. Our method is widely applicable to experiments and applications in which the initial visual encoding plays an important role.

Melanopic stimulation does not alter psychophysical threshold sensitivity for luminance flicker.

In addition to the rod and cone photoreceptors the retina contains intrinsically photosensitive retinal ganglion cells (ipRGCs). These cells express the photopigment melanopsin and are known to be involved in reflexive visual functions such as pupil response and photo-entrainment of the circadian rhythm. It is possible that the ipRGCs contribute to conscious visual perception, either by providing an independent signal to the geniculo-striate pathway, or by interacting with and thus modifying signals arising from "classical" retinal ganglion cells that combine and contrast cone input. Here, we tested for the existence of an interaction by asking if a 350% change in melanopsin stimulation alters psychophysical sensitivity for the detection of luminance flicker. In Experiment 1, we tested for a change in the threshold for detecting luminance flicker in three participants after they adapted to backgrounds with different degrees of tonic melanopsin stimulation. In Experiments 2 and 3, this test was repeated, but now for luminance flicker presented on a transient pedestal of melanopsin stimulation. Across the three experiments, no effect of melanopsin stimulation upon threshold flicker sensitivity was found. Our results suggest that even large changes in melanopsin stimulation do not affect near-threshold, cone-mediated visual perception.

What we talk about when we talk about colors.

Names for colors vary widely across languages, but color categories are remarkably consistent. Shared mechanisms of color perception help explain consistent partitions of visible light into discrete color vocabularies. But the mappings from colors to words are not identical across languages, which may reflect communicative needs-how often speakers must refer to objects of different color. Here we quantify the communicative needs of colors in 130 different languages by developing an inference algorithm for this problem. We find that communicative needs are not uniform: Some regions of color space exhibit 30-fold greater demand for communication than other regions. The regions of greatest demand correlate with the colors of salient objects, including ripe fruits in primate diets. Our analysis also reveals a hidden diversity in the communicative needs of colors across different languages, which is partly explained by differences in geographic location and the local biogeography of linguistic communities. Accounting for language-specific, nonuniform communicative needs improves predictions for how a language maps colors to words, and how these mappings vary across languages. Our account closes an important gap in the compression theory of color naming, while opening directions to study cross-cultural variation in the need to communicate different colors and its impact on the cultural evolution of color categories.

Reflexive Eye Closure in Response to Cone and Melanopsin Stimulation: A Study of Implicit Measures of Light Sensitivity in Migraine.

BACKGROUND AND OBJECTIVES: To quantify interictal photophobia in migraine with and without aura using reflexive eye closure as an implicit measure of light sensitivity and to assess the contribution of melanopsin and cone signals to these responses. METHODS: Participants were screened to meet criteria for 1 of 3 groups: headache-free (HF) controls, migraine without aura (MO), and migraine with visual aura (MA). MO and MA participants were included if they endorsed ictal and interictal photophobia. Exclusion criteria included impaired vision, inability to collect usable pupillometry, and history of either head trauma or seizure. Participants viewed light pulses that selectively targeted melanopsin, the cones, or their combination during recording of orbicularis oculi EMG (OO-EMG) and blinking activity. RESULTS: We studied 20 participants in each group. MA and MO groups reported increased visual discomfort to light stimuli (discomfort rating, 400% contrast, MA: 4.84 [95% confidence interval 0.33, 9.35]; MO: 5.23 [0.96, 9.50]) as compared to HF controls (2.71 [0, 6.47]). Time course analysis of OO-EMG and blinking activity demonstrated that reflexive eye closure was tightly coupled to the light pulses. The MA group had greater OO-EMG and blinking activity in response to these stimuli (EMG activity, 400% contrast: 42.9%Δ [28.4, 57.4]; blink activity, 400% contrast: 11.2% [8.8, 13.6]) as compared to the MO (EMG activity, 400% contrast: 9.9%Δ [5.8, 14.0]; blink activity, 400% contrast: 4.7% [3.5, 5.9]) and HF control (EMG activity, 400% contrast: 13.2%Δ [7.1, 19.3]; blink activity, 400% contrast: 4.5% [3.1, 5.9]) groups. DISCUSSION: Our findings suggest that the intrinsically photosensitive retinal ganglion cells (ipRGCs), which integrate melanopsin and cone signals, provide the afferent input for light-induced reflexive eye closure in a photophobic state. Moreover, we find a dissociation between implicit and explicit measures of interictal photophobia depending on a history of visual aura in migraine. This implies distinct pathophysiology in forms of migraine, interacting with separate neural pathways by which the amplification of ipRGC signals elicits implicit and explicit signs of visual discomfort.

Optoretinography of individual human cone photoreceptors.

Photoreceptors mediate the first step of vision, transducing light and passing signals to retinal neurons that ultimately send signals along the optic nerve to the brain. A functional deficiency in the photoreceptors, due to either congenital or acquired disease, can significantly affect an individual's sight and quality of life. Methods for quantifying the health and function of photoreceptors are essential for understanding both the progression of disease and the efficacy of treatment. Given that emerging treatments such as gene, stem cell, and small molecule therapy are designed to operate at the cellular scale, it is desirable to monitor function at the commensurate resolution of individual photoreceptors. Previously, non-invasive imaging methods for visualizing photoreceptor mosaic structure have been used to infer photoreceptor health, but these methods do not assess function directly. Conversely, most functional techniques, such as ERG and conventional microperimetry, measure function by aggregating the effects of signals from many photoreceptors. We have previously shown that stimulus-evoked intrinsic changes in intensity can be measured reliably in populations of cone photoreceptors in the intact human eye, a measurement we refer to more generally as the cone optoretinogram. Here we report that we can resolve the intensity optoretinogram at the level of individual cones. Moreover, we show that the individual cone optoretinogram exhibits two key signatures expected of a functional measure. First, responses in individual cones increase systematically as a function of stimulus irradiance. Second, we can use the amplitude of the functional response to middle wavelength (545 nm) light to separate the population of short-wavelength-sensitive (S) cones from the population of middle- and long-wavelength-sensitive (L and M) cones. Our results demonstrate the promise of optoretinography as a direct diagnostic measure of individual cone function in the living human eye.

Selective amplification of ipRGC signals accounts for interictal photophobia in migraine.

Second only to headache, photophobia is the most debilitating symptom reported by people with migraine. While the melanopsin-containing intrinsically photosensitive retinal ganglion cells (ipRGCs) are thought to play a role, how cone and melanopsin signals are integrated in this pathway to produce visual discomfort is poorly understood. We studied 60 people: 20 without headache and 20 each with interictal photophobia from migraine with or without visual aura. Participants viewed pulses of spectral change that selectively targeted melanopsin, the cones, or both and rated the degree of visual discomfort produced by these stimuli while we recorded pupil responses. We examined the data within a model that describes how cone and melanopsin signals are weighted and combined at the level of the retina and how this combined signal is transformed into a rating of discomfort or pupil response. Our results indicate that people with migraine do not differ from headache-free controls in the manner in which melanopsin and cone signals are combined. Instead, people with migraine demonstrate an enhanced response to integrated ipRGC signals for discomfort. This effect of migraine is selective for ratings of visual discomfort, in that an enhancement of pupil responses was not seen in the migraine group, nor were group differences found in surveys of other behaviors putatively linked to ipRGC function (chronotype, seasonal sensitivity, presence of a photic sneeze reflex). By revealing a dissociation in the amplification of discomfort vs. pupil response, our findings suggest a postretinal alteration in processing of ipRGC signals for photophobia in migraine.

A computational observer model of spatial contrast sensitivity: Effects of photocurrent encoding, fixational eye movements, and inference engine.

We have recently shown that the relative spatial contrast sensitivity function (CSF) of a computational observer operating on the cone mosaic photopigment excitations of a stationary retina has the same shape as human subjects. Absolute human sensitivity, however, is 5- to 10-fold lower than the computational observer. Here we model how additional known features of early vision affect the CSF: fixational eye movements and the conversion of cone photopigment excitations to cone photocurrents (phototransduction). For a computational observer that uses a linear classifier applied to the responses of a stimulus-matched linear filter, fixational eye movements substantially change the shape of the CSF by reducing sensitivity above 10 c/deg. For a translation-invariant computational observer that operates on the squared response of a quadrature-pair of linear filters, the CSF shape is little changed by eye movements, but there is a two fold reduction in sensitivity. Phototransduction dynamics introduce an additional two fold sensitivity decrease. Hence, the combined effects of fixational eye movements and phototransduction bring the absolute CSF of the translation-invariant computational observer to within a factor of 1 to 2 of the human CSF. We note that the human CSF depends on processing of the retinal representation by many thalamo-cortical neurons, which are individually quite noisy. Our modeling suggests that the net effect of post-retinal noise on contrast-detection performance, when considered at the neural population and behavioral level, is quite small: The inference mechanisms that determine the CSF, presumably in cortex, make efficient use of the information carried by the cone photocurrents of the fixating eye.

Automatic longitudinal montaging of adaptive optics retinal images using constellation matching.

Adaptive optics (AO) scanning laser ophthalmoscopy offers a non-invasive approach for observing the retina at a cellular level. Its high resolution capabilities have direct application for monitoring and treating retinal diseases by providing quantitative assessment of cone health and density across time. However, accurate longitudinal analysis of AO images requires that AO images from different sessions be aligned, such that cell-to-cell correspondences can be established between timepoints. Such alignment is currently done manually, a time intensive task that is restrictive for large longitudinal AO studies. Automated longitudinal montaging for AO images remains a challenge because the intensity pattern of imaged cone mosaics can vary significantly, even across short timespans. This limitation prevents existing intensity-based montaging approaches from being accurately applied to longitudinal AO images. In the present work, we address this problem by presenting a constellation-based method for performing longitudinal alignment of AO images. Rather than matching intensity similarities between images, our approach finds structural patterns in the cone mosaics and leverages these to calculate the correct alignment. These structural patterns are robust to intensity variations, allowing us to make accurate longitudinal alignments. We validate our algorithm using 8 longitudinal AO datasets, each with two timepoints separated 6-12 months apart. Our results show that the proposed method can produce longitudinal AO montages with cell-to-cell correspondences across the full extent of the montage. Quantitative assessment of the alignment accuracy shows that the algorithm is able to find longitudinal alignments whose accuracy is on par with manual alignments performed by a trained rater.

Ray tracing 3D spectral scenes through human optics models.

Scientists and engineers have created computations and made measurements that characterize the first steps of seeing. ISETBio software integrates such computations and data into an open-source software package. The initial ISETBio implementations modeled image formation (physiological optics) for planar or distant scenes. The ISET3d software described here extends that implementation, simulating image formation for three-dimensional scenes. The software system relies on a quantitative computer graphics program that ray traces the scene radiance through the physiological optics to the retinal irradiance. We describe and validate the implementation for several model eyes. Then, we use the software to quantify the impact of several physiological optics parameters on three-dimensional image formation. ISET3d is integrated with ISETBio, making it straightforward to convert the retinal irradiance into cone excitations. These methods help the user compute the predictions of optics models for a wide range of spatially rich three-dimensional scenes. They can also be used to evaluate the impact of nearby visual occlusion, the information available to binocular vision, or the retinal images expected from near-field and augmented reality displays.

Computational-observer analysis of illumination discrimination.

The spectral properties of the ambient illumination provide useful information about time of day and weather. We study the perceptual representation of illumination by analyzing measurements of how well people discriminate between illuminations across scene configurations. More specifically, we compare human performance to a computational-observer analysis that evaluates the information available in the isomerizations of cone photopigment in a model human photoreceptor mosaic. The performance of such an observer is limited by the Poisson variability of the number of isomerizations in each cone. The overall level of Poisson-limited computational-observer sensitivity exceeded that of human observers. This was modeled by increasing the amount of noise in the number of isomerizations of each cone. The additional noise brought the overall level of performance of the computational observer into the same range as that of human observers, allowing us to compare the pattern of sensitivity across stimulus manipulations. Key patterns of human performance were not accounted for by the computational observer. In particular, neither the elevation of illumination-discrimination thresholds for illuminant changes in a blue color direction (when thresholds are expressed in CIELUV ΔE units), nor the effects of varying the ensemble of surfaces in the scenes being viewed, could be accounted for by variation in the information available in the cone isomerizations.