Harmonics added to a flickering light can upset the balance between ON and OFF pathways to produce illusory colors.

The neural signals generated by the light-sensitive photoreceptors in the human eye are substantially processed and recoded in the retina before being transmitted to the brain via the optic nerve. A key aspect of this recoding is the splitting of the signals within the two major cone-driven visual pathways into distinct ON and OFF branches that transmit information about increases and decreases in the neural signal around its mean level. While this separation is clearly important physiologically, its effect on perception is unclear. We have developed a model of the ON and OFF pathways in early color processing. Using this model as a guide, we can produce imbalances in the ON and OFF pathways by changing the shapes of time-varying stimulus waveforms and thus make reliable and predictable alterations to the perceived average color of the stimulus-although the physical mean of the waveforms does not change. The key components in the model are the early half-wave rectifying synapses that split retinal photoreceptor outputs into the ON and OFF pathways and later sigmoidal nonlinearities in each pathway. The ability to systematically vary the waveforms to change a perceptual quality by changing the balance of signals between the ON and OFF visual pathways provides a powerful psychophysical tool for disentangling and investigating the neural workings of human vision.

Delayed S-cone sensitivity losses following the onset of intense yellow backgrounds linked to the lifetime of a photobleaching product?

Thirty years ago, Mollon, Stockman, & Polden (1987) reported that after the onset of intense yellow 581-nm backgrounds, S-cone threshold rose unexpectedly for several seconds before recovering to the light-adapted steady-state value-an effect they called: "transient-tritanopia of the second kind" (TT2). Given that 581-nm lights have little direct effect on S-cones, TT2 must arise indirectly from the backgrounds' effects on the L- and M-cones. We attribute the phenomenon to the action of an unknown L- and M-cone photobleaching product, X, which acts at their outputs like an "equivalent" background light that then inhibits S-cones at a cone-opponent, second-site. The time-course of TT2 is similar in form to the lifetime of X in a two-stage, first-order biochemical reaction A→X→C with successive best-fitting time-constants of 3.09 ± 0.35 and 7.73 ± 0.70 s. Alternatively, with an additional slowly recovering exponential "restoring-force" with a best-fitting time-constant 23.94 ± 1.42 s, the two-stage best-fitting time-constants become 4.15 ± 0.62 and 6.79 ± 1.00 s. Because the time-constants are roughly independent of the background illumination, and thus the rate of photoisomerization, A→X is likely to be a reaction subsidiary to the retinoid cycle, perhaps acting as a buffer when the bleaching rate is too high. X seems to be logarithmically related to S-cone threshold, which may result from the logarithmic cone-opponent, second-site response compression after multiplicative first-site adaptation. The restoring-force may be the same cone-opponent force that sets the rate of S-cone recovery following the unusual threshold increase following the offset of dimmer yellow backgrounds, an effect known as "transient-tritanopia" (TT1).

Delayed cone-opponent signals in the luminance pathway.

Cone signals in the luminance or achromatic pathway were investigated by measuring how the perceptual timing of M- or L-cone-detected flicker depended on temporal frequency and chromatic adaptation. Relative timings were measured, as a function of temporal frequency, by superimposing M- or L-cone-isolating flicker on "equichromatic" flicker (flicker of the same wavelength as the background) and asking observers to vary contrast and phase to cancel the perception of flicker. Measurements were made in four observers on up to 35 different backgrounds varying in wavelength and radiance. Observers showed substantial perceptual delays or advances of L- and M-cone flicker that varied systematically with cone class, background wavelength, and radiance. Delays were largest for M-cone-isolating flicker. Although complex, the results can be characterised by a surprisingly simple model in which the representations of L- and M-cone flicker are comprised not only of a fast copy of the flicker signal, but also of a slow copy that is delayed by roughly 30 ms and varies in strength and sign with both background wavelength and radiance. The delays, which are too large to be due to selective cone adaptation by the chromatic backgrounds, must arise postreceptorally. Clear evidence for the slow signals can also be found in physiological measurements of horizontal and magnocellular ganglion cells, thus placing the origin of the slow signals in the retina-most likely in an extended horizontal cell network. Luminance-equated stimuli chosen to isolate chromatic channels may inadvertently generate slow signals in the luminance channel.

Linear-nonlinear models of the red-green chromatic pathway.

The mean hue of flickering waveforms comprising only the first two harmonics depends on their temporal alignment. We evaluate explanatory models of this hue-shift effect using previous data obtained using L- and M-cone-isolating stimuli together with chromatic sensitivity and hue discrimination data. The key questions concerned what type of nonlinearity produced the hue shifts, and where the nonlinearities lay with respect to the early band-pass and late low-pass temporal filters in the chromatic pathways. We developed two plausible models: (a) a slew-rate limited nonlinearity that follows both early and late filters, and (b) a half-wave rectifying nonlinearity-consistent with the splitting of the visual input into ON- and OFF-channels-that lies between the early and late filters followed by a compressive nonlinearity that lies after the late filter.

Hue shifts produced by temporal asymmetries in chromatic signals depend on the alignment of the first and second harmonics.

When M- or L-cone-isolating sawtooth waveforms flicker at frequencies between 4 and 13.3 Hz, there is a mean hue shift in the direction of the shallower sawtooth slope. Here, we investigate how this shift depends on the alignment of the first and second harmonics of sawtooth-like waveforms. Below 4 Hz, observers can follow hue variations caused by both harmonics, and reliably match reddish and greenish excursions. At higher frequencies, however, the hue variations appear as chromatic flicker superimposed on a steady light, the mean hue of which varies with second-harmonic alignment. Observers can match this mean hue against a variable-duty-cycle rectangular waveform and, separately, set the alignment at which the mean hue flips between reddish and greenish. The maximum hue shifts were approximately frequency independent and occurred when the peaks or troughs of the first and second harmonics roughly aligned at the visual input-consistent with the hue shift's being caused by an early instantaneous nonlinearity that saturates larger hue excursions. These predictions, however, ignore phase delays introduced within the chromatic pathway between its input and the nonlinearity that produces the hue shifts. If the nonlinearity follows the substantial filtering implied by the chromatic temporal contrast-sensitivity function, phase delays will alter the alignment of the first and second harmonics such that at the nonlinearity, the waveforms that produce the maximum hue shifts might well be those with the largest differences in rising and falling slopes-consistent with the hue shift's being caused by a central nonlinearity that limits the rate of hue change.

Hue shifts produced by temporal asymmetries in chromatic signals.

Observers viewed M- or L-cone-isolating stimuli and compared slowly-on and slowly-off sawtooth waveforms of the same mean chromaticity and luminance. Between 6 and 13 Hz, the mean hue of slowly-on L-cone and slowly-off M-cone sawtooth flicker appeared redder, and the mean hue of slowly-off L-cone and slowly-on M-cone sawtooth stimuli appeared greener-despite all the waveforms' having the same mean, near-yellow-appearing chromaticity. We measured the effect of the modulation depth and the slope of the sawtooth on the mean hue shifts as a function of temporal frequency. The results are complex but show that discriminability depended mainly on the second harmonic of the waveforms. We considered several models with combinations of linear and nonlinear stages. First, we considered models in which a nonlinear stage limits the rate of change of hue and restricts the steep slope of the sawtooth waveform more than its shallow slope, thus shifting the mean hue in the direction of the shallower slope (such a nonlinearity is also known as a slew-rate limit). Second, we considered saturation models in which the nonlinear stage compresses hue signals and thus shifts the mean of asymmetrical waveforms with or without differentiation before the nonlinearity. Overall, our modeling and results suggest that the hue shift occurs at some nonlinear mechanism in the chromatic pathway; and that, in terms of the Fourier components of the various waveforms, the effect of the nonlinearity depends crucially on the timing of the second harmonic relative to the first.

Modeling off-frequency binaural masking for short- and long-duration signals.

Experimental binaural masking-pattern data are presented together with model simulations for 12- and 600-ms signals. The masker was a diotic 11-Hz wide noise centered on 500 Hz. The tonal signal was presented either diotically or dichotically (180° interaural phase difference) with frequencies ranging from 400 to 600 Hz. The results and the modeling agree with previous data and hypotheses; simulations with a binaural model sensitive to monaural modulation cues show that the effect of duration on off-frequency binaural masking-level differences is mainly a result of modulation cues which are only available in the monaural detection of long signals.

Spectral sensitivity measurements reveal partial success in restoring missing rod function with gene therapy.

Restored rod visual function after gene therapy can be established unequivocally by demonstrating that, after dark adaptation, spectral sensitivity has the shape characteristic of rods and that this shape collapses to a cone-like shape before rods have recovered after an intense bleach. We used these tests to assess retinal function in eight young adults and children with early-onset severe retinal dystrophy from Phase II of a clinical gene-therapy trial for RPE65 deficiency that involved the subretinal delivery of a recombinant adeno-associated viral vector carrying RPE65. We found substantial improvements in rod sensitivity in two participants: dark-adapted spectral sensitivity was rod-like after treatment and was cone-like before rods had recovered after a bleach. After 40 min of dark adaptation, one participant showed up to 1,000-fold sensitivity improvements 4 months after treatment and the second up to 100-fold improvements 6 months after treatment. The dark-adapted spectral sensitivities of the other six participants remained cone-like and showed little improvement in sensitivity.

Color and brightness encoded in a common L- and M-cone pathway with expansive and compressive nonlinearities.

Lights near 560 nm appear brighter when flickered, whereas lights near 520 or 650 nm appear yellower. Both effects are consistent with signal distortion within the visual pathway--brightness changes at an expansive nonlinearity, and hue shifts at a compressive one. We previously manipulated the distortion products generated by each nonlinearity to extract the temporal properties of stages of the L- and M-cone pathways that signal brightness and color before (early stages) and after (late stages) each nonlinearity. We find that the attenuation characteristics of the early and late stages are virtually identical in both pathways: The early temporal stage acts like a band-pass filter peaking at 10-15 Hz, while the late stage acts like low-pass filter with a cut-off frequency near 3 Hz. We propose a physiologically relevant model that accounts for the filter shapes and incorporates both nonlinearities within a common parvocellular pathway. The shape of the early band-pass filter is consistent with antagonism between center signals and more sluggish and delayed surround signals, while the late filter is consistent with a simple two-stage low-pass filter. Modeling suggests that the brightness change and hue shift are both initially caused by the half-wave rectification and partition of signals into ON and OFF components. However, the hue shift is probably caused by the additional effects of a later nonlinearity that compresses chromatic red and green signals. Plausible sites for the expansive half-wave rectifying nonlinearity are after surround antagonism, possibly from horizontal cells, but the compressive nonlinearity is likely to be after the late filter.

A novel binaural pitch elicited by phase-modulated noise: MEG and psychophysical observations.

Binaural pitches are auditory percepts that emerge from combined inputs to the ears but that cannot be heard if the stimulus is presented to either ear alone. Here, we describe a binaural pitch that is not easily accommodated within current models of binaural processing. Convergent magnetoencephalography (MEG) and psychophysical measurements were used to characterize the pitch, heard when band-limited noise had a rapidly changing interaural phase difference. Several interesting features emerged: First, the pitch was perceptually lateralized, in agreement with the lateralization of the evoked changes in MEG spectral power, and its salience depended on dichotic binaural presentation. Second, the frequency of the pure tone that matched the binaural pitch lay within a lower spectral sideband of the phase-modulated noise and followed the frequency of that sideband when the modulation frequency or center frequency and bandwidth of the noise changed. Thus, the binaural pitch depended on the processing of binaural information in that lower sideband.

The temporal characteristics of the early and late stages of the L- and M-cone pathways that signal color.

Flickering long-wavelength light appears more yellow than steady light of the same average intensity. The hue change is consistent with distortion of the visual signal at some nonlinear site (or sites) that produces temporal components not present in the original stimulus (known as distortion products). We extracted the temporal attenuation characteristics of the early (prenonlinearity) and late (post-nonlinearity) filter stages in the L- and M-cone chromatic pathway by varying the input stimulus to manipulate the distortion products and the measuring of the observers' sensitivity to them. The early, linear, filter stage acts like a band-pass filter peaking at 10-15 Hz with substantial sensitivity losses at both lower and higher frequencies. Its characteristics are consistent with nonlinearity being early in the visual pathway but following surround inhibition. The late stage, in contrast, acts like a low-pass filter with a cutoff frequency around 3 Hz. The response of the early stage speeds up with radiance, but the late stage does not. A plausible site for the nonlinearity, which modelling suggests may be smoothly compressive but with a hard limit at high input levels, is after surround inhibition from the horizontal cells.

The temporal characteristics of the early and late stages of L- and M-cone pathways that signal brightness.

Flickering 560-nm light appears brighter and less saturated than steady light of the same average intensity. The changes in appearance are consistent with the visual signal's being distorted at some nonlinear site (or sites) within the visual pathway at which new temporal components, not part of the original waveform, are produced. By varying the input stimulus to manipulate these new temporal components--called distortion products--and measuring our observers' sensitivity in detecting them, we derived the temporal attenuation characteristics of the early (prenonlinearity) and late (post-nonlinearity) stages of the L- and M-cone pathway that signals brightness. We found that the early stage acts like a band-pass filter peaking at 10-15 Hz with sensitivity losses at both lower and higher frequencies, whereas the late stage acts like a two-stage low-pass filter with a corner frequency near 3 Hz. Although brightness is often associated with the fast achromatic or luminance pathway, these filter characteristics, and particularly those of the late filter, are consistent with comparable features of the L-M chromatic pathway that produce mainly chromatic distortion products (Petrova, Henning, & Stockman, 2013). A plausible site for the nonlinearity is after surround antagonism from horizontal cells. Modeling suggested the form of the nonlinearity to be initially expansive but possibly with a hard limit at the highest input levels.

The effects of noise-bandwidth, noise-fringe duration, and temporal signal location on the binaural masking-level difference.

The effects of forward and backward noise fringes on binaural signal detectability were investigated. Masked thresholds for a 12-ms, 250-Hz, sinusoidal signal masked by Gaussian noise, centered at 250 Hz, with bandwidths from 3 to 201 Hz, were obtained in N(0)S(0) and N(0)S(π) configurations. The signal was (a) temporally centered in a 12-ms noise burst (no fringe), (b) presented at the start of a 600-ms noise burst (backward fringe), or (c) temporally centered in a 600-ms noise burst (forward-plus-backward fringe). For noise bandwidths between 3 and 75 Hz, detection in N(0)S(0) improved with the addition of a backward fringe, improving further with an additional forward fringe; there was little improvement in N(0)S(π). The binaural masking-level difference (BMLD) increased from 0 to 8 dB with a forward-plus-backward fringe as noise bandwidths increased to 100 Hz, increasing slightly to 10 dB at 201 Hz. This two-stage increase was less pronounced with a backward fringe. With no fringe, the BMLD was about 10-14 dB at all bandwidths. Performance appears to result from the interaction of across-time and across-frequency listening strategies and the possible effects of gain reduction and suppression, which combine in complex ways. Current binaural models are, as yet, unable to account fully for these effects.

A reinterpretation of critical flicker-frequency (CFF) data reveals key details about light adaptation and normal and abnormal visual processing.

Our ability to see flicker has an upper frequency limit above which flicker is invisible, known as the "critical flicker frequency" (CFF), that typically grows with light intensity (I). The relation between CFF and I, the focus of nearly 200 years of research, is roughly logarithmic, i.e., CFF âˆ log(I)-a relation called the Ferry-Porter law. However, why this law should occur, and how it relates to the underlying physiology, have never been adequately explained. Over the past two decades we have measured CFF in normal observers and in patients with retinal gene defects. Here, we reanalyse and model our data and historical CFF data. Remarkably, CFF-versus-I functions measured under a wide range of conditions in patients and in normal observers all have broadly similar shapes when plotted in double-logarithmic coordinates, i.e., log (CFF)-versus-log(I). Thus, the entire dataset can be characterised by horizontal and vertical logarithmic shifts of a fixed-shape template. Shape invariance can be predicted by a simple model of visual processing built from a sequence of low-pass filters, subtractive feedforward stages and gain adjustment (Rider, Henning & Stockman, 2019). It depends primarily on the numbers of visual processing stages that approach their power-law region at a given intensity and a frequency-independent gain reduction at higher light levels. Counter-intuitively, the CFF-versus-I relation depends primarily on the gain of the visual response rather than its speed-a conclusion that changes our understanding and interpretation of human flicker perception. The Ferry-Porter "law" is merely an approximation of the shape-invariant template.

Clinical vision and molecular loss: Integrating visual psychophysics with molecular genetics reveals key details of normal and abnormal visual processing.

Over the past two decades we have developed techniques and models to investigate the ways in which known molecular defects affect visual performance. Because molecular defects in retinal signalling invariably alter the speed of visual processing, our strategy has been to measure the resulting changes in flicker sensitivity. Flicker measurements provide not only straightforward clinical assessments of visual performance but also reveal fundamental details about the functioning of both abnormal and normal visual systems. Here, we bring together our past measurements of patients with pathogenic variants in the GNAT2, RGS9, GUCA1A, RPE65, OPA1, KCNV2 and NR2E3 genes and analyse the results using a standard model of visual processing. The model treats flicker sensitivity as the result of the actions of a sequence of simple processing steps, one or more of which is altered by the genetic defect. Our analyses show that most defects slow down the visual response directly, but some speed it up. Crucially, however, other steps in the processing sequence can make compensatory adjustments to offset the abnormality. For example, if the abnormal step slows down the visual response, another step is likely to speed up or attenuate the response to rebalance system performance. Such compensatory adjustments are probably made by steps in the sequence that usually adapt to changing light levels. Our techniques and modelling also allow us to tease apart stationary and progressive effects, and the localised molecular losses help us to unravel and characterise individual steps in the normal and abnormal processing sequences.

A neurophysiologically plausible population code model for human contrast discrimination.

The pedestal effect is the improvement in the detectability of a sinusoidal grating in the presence of another grating of the same orientation, spatial frequency, and phase-usually called the pedestal. Recent evidence has demonstrated that the pedestal effect is differently modified by spectrally flat and notch-filtered noise: The pedestal effect is reduced in flat noise but virtually disappears in the presence of notched noise (G. B. Henning & F. A. Wichmann, 2007). Here we consider a network consisting of units whose contrast response functions resemble those of the cortical cells believed to underlie human pattern vision and demonstrate that, when the outputs of multiple units are combined by simple weighted summation-a heuristic decision rule that resembles optimal information combination and produces a contrast-dependent weighting profile-the network produces contrast-discrimination data consistent with psychophysical observations: The pedestal effect is present without noise, reduced in broadband noise, but almost disappears in notched noise. These findings follow naturally from the normalization model of simple cells in primary visual cortex, followed by response-based pooling, and suggest that in processing even low-contrast sinusoidal gratings, the visual system may combine information across neurons tuned to different spatial frequencies and orientations.

The effect of notched noise on flicker detection and discrimination.

Flicker perception was investigated using two-alternative forced-choice detection and discrimination tasks with four different types of external noise: (1) broadband noise, (2) 5-Hz notched-noise--broadband noise with a 5-Hz band centered on the signal frequency removed, (3) 10-Hz notched-noise, and (4) no external noise. The signal was a burst of 10-Hz sinusoidal flicker presented in one of two observation intervals. In discrimination experiments, a pedestal--sinusoidal flicker with the same frequency, duration, and phase as the signal--was added to both observation intervals. With no noise, observers' performance first improved with increasing pedestal modulation, before deteriorating in accordance with Weber's Law, producing the typical "dipper" shaped plot of signal versus pedestal modulation. Noise affects performance, but the dipper effect persisted in each type of noise. The results exclude three models: the ideal-observer in which the pedestal improves performance by specifying the signal exactly; off-frequency-looking models in which the dipper depends on detection by channels tuned to temporal frequencies different from that of the signal; and strict energy detectors. Our data are consistent with signal processing by a single mechanism with an expansive non-linearity for near-threshold signal modulations (with an exponent of six) and a compressive "Weberian" non-linearity for high modulations.

Light adaptation controls visual sensitivity by adjusting the speed and gain of the response to light.

The range of c. 1012 ambient light levels to which we can be exposed massively exceeds the <103 response range of neurons in the visual system, but we can see well in dim starlight and bright sunlight. This remarkable ability is achieved largely by a speeding up of the visual response as light levels increase, causing characteristic changes in our sensitivity to different rates of flicker. Here, we account for over 65 years of flicker-sensitivity measurements with an elegantly-simple, physiologically-relevant model built from first-order low-pass filters and subtractive inhibition. There are only two intensity-dependent model parameters: one adjusts the speed of the visual response by shortening the time constants of some of the filters in the direct cascade as well as those in the inhibitory stages; the other parameter adjusts the overall gain at higher light levels. After reviewing the physiological literature, we associate the variable gain and three of the variable-speed filters with biochemical processes in cone photoreceptors, and a further variable-speed filter with processes in ganglion cells. The variable-speed but fixed-strength subtractive inhibition is most likely associated with lateral connections in the retina. Additional fixed-speed filters may be more central. The model can explain the important characteristics of human flicker-sensitivity including the approximate dependences of low-frequency sensitivity on contrast (Weber's law) and of high-frequency sensitivity on amplitude ("high-frequency linearity"), the exponential loss of high-frequency sensitivity with increasing frequency, and the logarithmic increase in temporal acuity with light level (Ferry-Porter law). In the time-domain, the model can account for several characteristics of flash sensitivity including changes in contrast sensitivity with light level (de Vries-Rose and Weber's laws) and changes in temporal summation (Bloch's law). The new model provides fundamental insights into the workings of the visual system and gives a simple account of many visual phenomena.

Some observations on the pedestal effect.

The pedestal or dipper effect is the large improvement in the detectability of a sinusoidal grating observed when it is added to a masking or pedestal grating of the same spatial frequency, orientation, and phase. We measured the pedestal effect in both broadband and notched noise-noise from which a 1.5-octave band centered on the signal frequency had been removed. Although the pedestal effect persists in broadband noise, it almost disappears in the notched noise. Furthermore, the pedestal effect is substantial when either high- or low-pass masking noise is used. We conclude that the pedestal effect in the absence of notched noise results principally from the use of information derived from channels with peak sensitivities at spatial frequencies different from that of the signal and the pedestal. We speculate that the spatial-frequency components of the notched noise above and below the spatial frequency of the signal and the pedestal prevent "off-frequency looking," that is, prevent the use of information about changes in contrast carried in channels tuned to spatial frequencies that are very much different from that of the signal and the pedestal. Thus, the pedestal or dipper effect measured without notched noise appears not to be a characteristic of individual spatial-frequency-tuned channels.

Psychophysical measurements in children: challenges, pitfalls, and considerations.

Measuring sensory sensitivity is important in studying development and developmental disorders. However, with children, there is a need to balance reliable but lengthy sensory tasks with the child's ability to maintain motivation and vigilance. We used simulations to explore the problems associated with shortening adaptive psychophysical procedures, and suggest how these problems might be addressed. We quantify how adaptive procedures with too few reversals can over-estimate thresholds, introduce substantial measurement error, and make estimates of individual thresholds less reliable. The associated measurement error also obscures group differences. Adaptive procedures with children should therefore use as many reversals as possible, to reduce the effects of both Type 1 and Type 2 errors. Differences in response consistency, resulting from lapses in attention, further increase the over-estimation of threshold. Comparisons between data from individuals who may differ in lapse rate are therefore problematic, but measures to estimate and account for lapse rates in analyses may mitigate this problem.