History constrains the evolution of efficient color naming, enabling historical inference.

Color naming in natural languages is not arbitrary: It reflects efficient partitions of perceptual color space [T. Regier, P. Kay, N. Khetarpal, Proc. Natl. Acad. Sci. U.S.A. 104, 1436-1441 (2007)] modulated by the relative needs to communicate about different colors [C. Twomey, G. Roberts, D. Brainard, J. Plotkin, Proc. Natl. Acad. Sci. U.S.A. 118, e2109237118 (2021)]. These psychophysical and communicative constraints help explain why languages around the world have remarkably similar, but not identical, mappings of colors to color terms. Languages converge on a small set of efficient representations.But languages also evolve, and the number of terms in a color vocabulary may change over time. Here we show that history, i.e. the existence of an antecedent color vocabulary, acts as a nonadaptive constraint that biases the choice of efficient solution as a language transitions from a vocabulary of size [Formula: see text] to [Formula: see text] terms. Moreover, as efficient vocabularies evolve to include more terms they explore a smaller fraction of all possible efficient vocabularies compared to equally sized vocabularies constructed de novo. This path dependence of the cultural evolution of color naming presents an opportunity. Historical constraints can be used to reconstruct ancestral color vocabularies, allowing us to answer long-standing questions about the evolutionary sequences of color words, and enabling us to draw inferences from phylogenetic patterns of language change.

What we talk about when we talk about colors.

Names for colors vary widely across languages, but color categories are remarkably consistent. Shared mechanisms of color perception help explain consistent partitions of visible light into discrete color vocabularies. But the mappings from colors to words are not identical across languages, which may reflect communicative needs-how often speakers must refer to objects of different color. Here we quantify the communicative needs of colors in 130 different languages by developing an inference algorithm for this problem. We find that communicative needs are not uniform: Some regions of color space exhibit 30-fold greater demand for communication than other regions. The regions of greatest demand correlate with the colors of salient objects, including ripe fruits in primate diets. Our analysis also reveals a hidden diversity in the communicative needs of colors across different languages, which is partly explained by differences in geographic location and the local biogeography of linguistic communities. Accounting for language-specific, nonuniform communicative needs improves predictions for how a language maps colors to words, and how these mappings vary across languages. Our account closes an important gap in the compression theory of color naming, while opening directions to study cross-cultural variation in the need to communicate different colors and its impact on the cultural evolution of color categories.

Selective amplification of ipRGC signals accounts for interictal photophobia in migraine.

Second only to headache, photophobia is the most debilitating symptom reported by people with migraine. While the melanopsin-containing intrinsically photosensitive retinal ganglion cells (ipRGCs) are thought to play a role, how cone and melanopsin signals are integrated in this pathway to produce visual discomfort is poorly understood. We studied 60 people: 20 without headache and 20 each with interictal photophobia from migraine with or without visual aura. Participants viewed pulses of spectral change that selectively targeted melanopsin, the cones, or both and rated the degree of visual discomfort produced by these stimuli while we recorded pupil responses. We examined the data within a model that describes how cone and melanopsin signals are weighted and combined at the level of the retina and how this combined signal is transformed into a rating of discomfort or pupil response. Our results indicate that people with migraine do not differ from headache-free controls in the manner in which melanopsin and cone signals are combined. Instead, people with migraine demonstrate an enhanced response to integrated ipRGC signals for discomfort. This effect of migraine is selective for ratings of visual discomfort, in that an enhancement of pupil responses was not seen in the migraine group, nor were group differences found in surveys of other behaviors putatively linked to ipRGC function (chronotype, seasonal sensitivity, presence of a photic sneeze reflex). By revealing a dissociation in the amplification of discomfort vs. pupil response, our findings suggest a postretinal alteration in processing of ipRGC signals for photophobia in migraine.

Equivalent noise characterization of human lightness constancy.

A goal of visual perception is to provide stable representations of task-relevant scene properties (e.g. object reflectance) despite variation in task-irrelevant scene properties (e.g. illumination and reflectance of other nearby objects). To study such stability in the context of the perceptual representation of lightness, we introduce a threshold-based psychophysical paradigm. We measure how thresholds for discriminating the achromatic reflectance of a target object (task-relevant property) in rendered naturalistic scenes are impacted by variation in the reflectance functions of background objects (task-irrelevant property), using a two-alternative forced-choice paradigm in which the reflectance of the background objects is randomized across the two intervals of each trial. We control the amount of background reflectance variation by manipulating a statistical model of naturally occurring surface reflectances. For low background object reflectance variation, discrimination thresholds were nearly constant, indicating that observers' internal noise determines threshold in this regime. As background object reflectance variation increases, its effects start to dominate performance. A model based on signal detection theory allows us to express the effects of task-irrelevant variation in terms of the equivalent noise, that is relative to the intrinsic precision of the task-relevant perceptual representation. The results indicate that although naturally occurring background object reflectance variation does intrude on the perceptual representation of target object lightness, the effect is modest - within a factor of two of the equivalent noise level set by internal noise.

Optoretinography of individual human cone photoreceptors.

Photoreceptors mediate the first step of vision, transducing light and passing signals to retinal neurons that ultimately send signals along the optic nerve to the brain. A functional deficiency in the photoreceptors, due to either congenital or acquired disease, can significantly affect an individual's sight and quality of life. Methods for quantifying the health and function of photoreceptors are essential for understanding both the progression of disease and the efficacy of treatment. Given that emerging treatments such as gene, stem cell, and small molecule therapy are designed to operate at the cellular scale, it is desirable to monitor function at the commensurate resolution of individual photoreceptors. Previously, non-invasive imaging methods for visualizing photoreceptor mosaic structure have been used to infer photoreceptor health, but these methods do not assess function directly. Conversely, most functional techniques, such as ERG and conventional microperimetry, measure function by aggregating the effects of signals from many photoreceptors. We have previously shown that stimulus-evoked intrinsic changes in intensity can be measured reliably in populations of cone photoreceptors in the intact human eye, a measurement we refer to more generally as the cone optoretinogram. Here we report that we can resolve the intensity optoretinogram at the level of individual cones. Moreover, we show that the individual cone optoretinogram exhibits two key signatures expected of a functional measure. First, responses in individual cones increase systematically as a function of stimulus irradiance. Second, we can use the amplitude of the functional response to middle wavelength (545 nm) light to separate the population of short-wavelength-sensitive (S) cones from the population of middle- and long-wavelength-sensitive (L and M) cones. Our results demonstrate the promise of optoretinography as a direct diagnostic measure of individual cone function in the living human eye.

The relative contribution of color and material in object selection.

Object perception is inherently multidimensional: information about color, material, texture and shape all guide how we interact with objects. We developed a paradigm that quantifies how two object properties (color and material) combine in object selection. On each experimental trial, observers viewed three blob-shaped objects-the target and two tests-and selected the test that was more similar to the target. Across trials, the target object was fixed, while the tests varied in color (across 7 levels) and material (also 7 levels, yielding 49 possible stimuli). We used an adaptive trial selection procedure (Quest+) to present, on each trial, the stimulus test pair that is most informative of underlying processes that drive selection. We present a novel computational model that allows us to describe observers' selection data in terms of (1) the underlying perceptual stimulus representation and (2) a color-material weight, which quantifies the relative importance of color vs. material in selection. We document large individual differences in the color-material weight across the 12 observers we tested. Furthermore, our analyses reveal limits on how precisely selection data simultaneously constrain perceptual representations and the color-material weight. These limits should guide future efforts towards understanding the multidimensional nature of object perception.

The human visual cortex response to melanopsin-directed stimulation is accompanied by a distinct perceptual experience.

The photopigment melanopsin supports reflexive visual functions in people, such as pupil constriction and circadian photoentrainment. What contribution melanopsin makes to conscious visual perception is less studied. We devised a stimulus that targeted melanopsin separately from the cones using pulsed (3-s) spectral modulations around a photopic background. Pupillometry confirmed that the melanopsin stimulus evokes a response different from that produced by cone stimulation. In each of four subjects, a functional MRI response in area V1 was found. This response scaled with melanopic contrast and was not easily explained by imprecision in the silencing of the cones. Twenty additional subjects then observed melanopsin pulses and provided a structured rating of the perceptual experience. Melanopsin stimulation was described as an unpleasant, blurry, minimal brightening that quickly faded. We conclude that isolated stimulation of melanopsin is likely associated with a response within the cortical visual pathway and with an evoked conscious percept.

A computational observer model of spatial contrast sensitivity: Effects of photocurrent encoding, fixational eye movements, and inference engine.

We have recently shown that the relative spatial contrast sensitivity function (CSF) of a computational observer operating on the cone mosaic photopigment excitations of a stationary retina has the same shape as human subjects. Absolute human sensitivity, however, is 5- to 10-fold lower than the computational observer. Here we model how additional known features of early vision affect the CSF: fixational eye movements and the conversion of cone photopigment excitations to cone photocurrents (phototransduction). For a computational observer that uses a linear classifier applied to the responses of a stimulus-matched linear filter, fixational eye movements substantially change the shape of the CSF by reducing sensitivity above 10 c/deg. For a translation-invariant computational observer that operates on the squared response of a quadrature-pair of linear filters, the CSF shape is little changed by eye movements, but there is a two fold reduction in sensitivity. Phototransduction dynamics introduce an additional two fold sensitivity decrease. Hence, the combined effects of fixational eye movements and phototransduction bring the absolute CSF of the translation-invariant computational observer to within a factor of 1 to 2 of the human CSF. We note that the human CSF depends on processing of the retinal representation by many thalamo-cortical neurons, which are individually quite noisy. Our modeling suggests that the net effect of post-retinal noise on contrast-detection performance, when considered at the neural population and behavioral level, is quite small: The inference mechanisms that determine the CSF, presumably in cortex, make efficient use of the information carried by the cone photocurrents of the fixating eye.

Illumination discrimination for chromatically biased illuminations: Implications for color constancy.

We measured discrimination thresholds for illumination changes along different chromatic directions starting from chromatically biased reference illuminations. Participants viewed a Mondrian-papered scene illuminated by LED lamps. The scene was first illuminated by a reference illumination, followed by two comparisons. One comparison matched the reference (the target); the other (the test) varied from the reference, nominally either bluer, yellower, redder, or greener. The participant's task was to correctly select the target. A staircase procedure found thresholds for discrimination of an illumination change along each axis of chromatic change. Nine participants completed the task for five different reference illumination conditions (neutral, blue, yellow, red, and green). We find that relative discrimination thresholds for different chromatic directions of illumination change vary with the reference illumination. For the neutral reference, there is a trend for thresholds to be highest in the bluer illumination-change direction, replicating our previous reports of a "blue bias" for neutral reference illuminations. For the four chromatic references (blue, yellow, red, and green), the change in illumination toward the neutral reference is less well discriminated than changes in the other directions: a "neutral bias." The results have implications for color constancy: In considering the stability of surface appearance under changes in illumination, both the starting chromaticity of the illumination and direction of change must be considered, as well as the chromatic characteristics of the surface reflectance ensemble. They also suggest it will be worthwhile to explore whether and how the human visual system has internalized the statistics of natural illumination changes.

Computational-observer analysis of illumination discrimination.

The spectral properties of the ambient illumination provide useful information about time of day and weather. We study the perceptual representation of illumination by analyzing measurements of how well people discriminate between illuminations across scene configurations. More specifically, we compare human performance to a computational-observer analysis that evaluates the information available in the isomerizations of cone photopigment in a model human photoreceptor mosaic. The performance of such an observer is limited by the Poisson variability of the number of isomerizations in each cone. The overall level of Poisson-limited computational-observer sensitivity exceeded that of human observers. This was modeled by increasing the amount of noise in the number of isomerizations of each cone. The additional noise brought the overall level of performance of the computational observer into the same range as that of human observers, allowing us to compare the pattern of sensitivity across stimulus manipulations. Key patterns of human performance were not accounted for by the computational observer. In particular, neither the elevation of illumination-discrimination thresholds for illuminant changes in a blue color direction (when thresholds are expressed in CIELUV ΔE units), nor the effects of varying the ensemble of surfaces in the scenes being viewed, could be accounted for by variation in the information available in the cone isomerizations.

A computational-observer model of spatial contrast sensitivity: Effects of wave-front-based optics, cone-mosaic structure, and inference engine.

We present a computational-observer model of the human spatial contrast-sensitivity function based on the Image Systems Engineering Toolbox for Biology (ISETBio) simulation framework. We demonstrate that ISETBio-derived contrast-sensitivity functions agree well with ones derived using traditional ideal-observer approaches, when the mosaic, optics, and inference engine are matched. Further simulations extend earlier work by considering more realistic cone mosaics, more recent measurements of human physiological optics, and the effect of varying the inference engine used to link visual representations to psychophysical performance. Relative to earlier calculations, our simulations show that the spatial structure of realistic cone mosaics reduces the upper bounds on performance at low spatial frequencies, whereas realistic optics derived from modern wave-front measurements lead to increased upper bounds at high spatial frequencies. Finally, we demonstrate that the type of inference engine used has a substantial effect on the absolute level of predicted performance. Indeed, the performance gap between an ideal observer with exact knowledge of the relevant signals and human observers is greatly reduced when the inference engine has to learn aspects of the visual task. ISETBio-derived estimates of stimulus representations at various stages along the visual pathway provide a powerful tool for computing the limits of human performance.

Ray tracing 3D spectral scenes through human optics models.

Scientists and engineers have created computations and made measurements that characterize the first steps of seeing. ISETBio software integrates such computations and data into an open-source software package. The initial ISETBio implementations modeled image formation (physiological optics) for planar or distant scenes. The ISET3d software described here extends that implementation, simulating image formation for three-dimensional scenes. The software system relies on a quantitative computer graphics program that ray traces the scene radiance through the physiological optics to the retinal irradiance. We describe and validate the implementation for several model eyes. Then, we use the software to quantify the impact of several physiological optics parameters on three-dimensional image formation. ISET3d is integrated with ISETBio, making it straightforward to convert the retinal irradiance into cone excitations. These methods help the user compute the predictions of optics models for a wide range of spatially rich three-dimensional scenes. They can also be used to evaluate the impact of nearby visual occlusion, the information available to binocular vision, or the retinal images expected from near-field and augmented reality displays.

Neuronal population mechanisms of lightness perception.

The way that humans and animals perceive the lightness of an object depends on its physical luminance as well as its surrounding context. While neuronal responses throughout the visual pathway are modulated by context, the relationship between neuronal responses and lightness perception is poorly understood. We searched for a neuronal mechanism of lightness by recording responses of neuronal populations in monkey primary visual cortex (V1) and area V4 to stimuli that produce a lightness illusion in humans, in which the lightness of a disk depends on the context in which it is embedded. We found that the way individual units encode the luminance (or equivalently for our stimuli, contrast) of the disk and its context is extremely heterogeneous. This motivated us to ask whether the population representation in either V1 or V4 satisfies three criteria: 1) disk luminance is represented with high fidelity, 2) the context surrounding the disk is also represented, and 3) the representations of disk luminance and context interact to create a representation of lightness that depends on these factors in a manner consistent with human psychophysical judgments of disk lightness. We found that populations of units in both V1 and V4 fulfill the first two criteria but that we cannot conclude that the two types of information in either area interact in a manner that clearly predicts human psychophysical measurements: the interpretation of our population measurements depends on how subsequent areas read out lightness from the population responses. NEW & NOTEWORTHY A core question in visual neuroscience is how the brain extracts stable representations of object properties from the retinal image. We searched for a neuronal mechanism of lightness perception by determining whether the responses of neuronal populations in primary visual cortex and area V4 could account for a lightness illusion measured using human psychophysics. Our results suggest that comparing psychophysics with population recordings will yield insight into neuronal mechanisms underlying a variety of perceptual phenomena.

Computational luminance constancy from naturalistic images.

The human visual system supports stable percepts of object color even though the light that reflects from object surfaces varies significantly with the scene illumination. To understand the computations that support stable color perception, we study how estimating a target object's luminous reflectance factor (LRF; a measure of the light reflected from the object under a standard illuminant) depends on variation in key properties of naturalistic scenes. Specifically, we study how variation in target object reflectance, illumination spectra, and the reflectance of background objects in a scene impact estimation of a target object's LRF. To do this, we applied supervised statistical learning methods to the simulated excitations of human cone photoreceptors, obtained from labeled naturalistic images. The naturalistic images were rendered with computer graphics. The illumination spectra of the light sources and the reflectance spectra of the surfaces in the scene were generated using statistical models of natural spectral variation. Optimally decoding target object LRF from the responses of a small learned set of task-specific linear receptive fields that operate on a contrast representation of the cone excitations yields estimates that are within 13% of the correct LRF. Our work provides a framework for evaluating how different sources of scene variability limit performance on luminance constancy.

Illumination discrimination in the absence of a fixed surface-reflectance layout.

Previous studies have shown that humans can discriminate spectral changes in illumination and that this sensitivity depends both on the chromatic direction of the illumination change and on the ensemble of surfaces in the scene. These studies, however, always used stimulus scenes with a fixed surface-reflectance layout. Here we compared illumination discrimination for scenes in which the surface reflectance layout remains fixed (fixed-surfaces condition) to those in which surface reflectances were shuffled randomly across scenes, but with the mean scene reflectance held approximately constant (shuffled-surfaces condition). Illumination discrimination thresholds in the fixed-surfaces condition were commensurate with previous reports. Thresholds in the shuffled-surfaces condition, however, were considerably elevated. Nonetheless, performance in the shuffled-surfaces condition exceeded that attainable through random guessing. Analysis of eye fixations revealed that in the fixed-surfaces condition, low illumination discrimination thresholds (across observers) were predicted by low overall fixation spread and high consistency of fixation location and fixated surface reflectances across trial intervals. Performance in the shuffled-surfaces condition was not systematically related to any of the eye-fixation characteristics we examined for that condition, but was correlated with performance in the fixed-surfaces condition.

Spatial summation in the human fovea: Do normal optical aberrations and fixational eye movements have an effect?

Psychophysical inferences about the neural mechanisms supporting spatial vision can be undermined by uncertainties introduced by optical aberrations and fixational eye movements, particularly in fovea where the neuronal grain of the visual system is fine. We examined the effect of these preneural factors on photopic spatial summation in the human fovea using a custom adaptive optics scanning light ophthalmoscope that provided control over optical aberrations and retinal stimulus motion. Consistent with previous results, Ricco's area of complete summation encompassed multiple photoreceptors when measured with ordinary amounts of ocular aberrations and retinal stimulus motion. When both factors were minimized experimentally, summation areas were essentially unchanged, suggesting that foveal spatial summation is limited by postreceptoral neural pooling. We compared our behavioral data to predictions generated with a physiologically-inspired front-end model of the visual system, and were able to capture the shape of the summation curves obtained with and without pre-retinal factors using a single postreceptoral summing filter of fixed spatial extent. Given our data and modeling, neurons in the magnocellular visual pathway, such as parasol ganglion cells, provide a candidate neural correlate of Ricco's area in the central fovea.

Human Visual Cortex Responses to Rapid Cone and Melanopsin-Directed Flicker.

Signals from cones are recombined in postreceptoral channels [luminance, L + M; red-green, L - M; blue-yellow, S - (L + M)]. The melanopsin-containing retinal ganglion cells are also active at daytime light levels and recent psychophysical results suggest that melanopsin contributes to conscious vision in humans. Here, we measured BOLD fMRI responses to spectral modulations that separately targeted the postreceptoral cone channels and melanopsin. Responses to spatially uniform (27.5° field size, central 5° obscured) flicker at 0.5, 1, 2, 4, 8, 16, 32, and 64 Hz were recorded from areas V1, V2/V3, motion-sensitive area MT, and the lateral occipital complex. In V1 and V2/V3, higher temporal sensitivity was observed to L + M + S (16 Hz) compared with L - M flicker (8 Hz), consistent with psychophysical findings. Area MT was most sensitive to rapid (32 Hz) flicker of either L + M + S or L - M. We found S cone responses only in areas V1 and V2/V3 (peak frequency: 4-8 Hz). In addition, we studied an L + M modulation and found responses that were effectively identical at all temporal frequencies to those recorded for the L + M + S modulation. Finally, we measured the cortical response to melanopsin-directed flicker and compared this response with control modulations that addressed stimulus imprecision and the possibility of stimulation of cones in the shadow of retinal blood vessels (penumbral cones). For our stimulus conditions, melanopsin flicker did not elicit a cortical response exceeding that of the control modulations. We note that failure to control for penumbral cone stimulation could be mistaken for a melanopsin response. SIGNIFICANCE STATEMENT: The retina contains cone photoreceptors and ganglion cells that contain the photopigment melanopsin. Cones provide brightness and color signals to visual cortex. Melanopsin influences circadian rhythm and the pupil, but its contribution to cortex and perception is less clear. We measured the response of human visual cortex with fMRI using spectral modulations tailored to stimulate the cones and melanopsin separately. We found that cortical responses to cone signals vary systematically across visual areas. Differences in temporal sensitivity for achromatic, red-green, and blue-yellow stimuli generally reflect the known perceptual properties of vision. We found that melanopsin signals do not produce a measurable response in visual cortex at temporal frequencies between 0.5 and 64 Hz at daytime light levels.

Opponent melanopsin and S-cone signals in the human pupillary light response.

In the human, cone photoreceptors (L, M, and S) and the melanopsin-containing, intrinsically photosensitive retinal ganglion cells (ipRGCs) are active at daytime light intensities. Signals from cones are combined both additively and in opposition to create the perception of overall light and color. Similar mechanisms seem to be at work in the control of the pupil's response to light. Uncharacterized however, is the relative contribution of melanopsin and S cones, with their overlapping, short-wavelength spectral sensitivities. We measured the response of the human pupil to the separate stimulation of the cones and melanopsin at a range of temporal frequencies under photopic conditions. The S-cone and melanopsin photoreceptor channels were found to be low-pass, in contrast to a band-pass response of the pupil to L- and M-cone signals. An examination of the phase relationships of the evoked responses revealed that melanopsin signals add with signals from L and M cones but are opposed by signals from S cones in control of the pupil. The opposition of the S cones is revealed in a seemingly paradoxical dilation of the pupil to greater S-cone photon capture. This surprising result is explained by the neurophysiological properties of ipRGCs found in animal studies.