V1-based modeling of discrimination between natural scenes within the luminance and isoluminant color planes.

We have been developing a computational visual difference predictor model that can predict how human observers rate the perceived magnitude of suprathreshold differences between pairs of full-color naturalistic scenes (To, Lovell, Troscianko, & Tolhurst, 2010). The model is based closely on V1 neurophysiology and has recently been updated to more realistically implement sequential application of nonlinear inhibitions (contrast normalization followed by surround suppression; To, Chirimuuta, & Tolhurst, 2017). The model is based originally on a reliable luminance model (Watson & Solomon, 1997) which we have extended to the red/green and blue/yellow opponent planes, assuming that the three planes (luminance, red/green, and blue/yellow) can be modeled similarly to each other with narrow-band oriented filters. This paper examines whether this may be a false assumption, by decomposing our original full-color stimulus images into monochromatic and isoluminant variants, which observers rate separately and which we model separately. The ratings for the original full-color scenes correlate better with the new ratings for the monochromatic variants than for the isoluminant ones, suggesting that luminance cues carry more weight in observers' ratings to full-color images. The ratings for the original full-color stimuli can be predicted from the new monochromatic and isoluminant rating data by combining them by Minkowski summation with power m = 2.71, consistent with other studies involving feature summation. The model performed well at predicting ratings for monochromatic stimuli, but was weaker for isoluminant stimuli, indicating that mirroring the monochromatic models is not sufficient to model the color planes. We discuss several alternative strategies to improve the color modeling.

Modeling grating contrast discrimination dippers: The role of surround suppression.

We consider the role of nonlinear inhibition in physiologically realistic multineuronal models of V1 to predict the dipper functions from contrast discrimination experiments with sinusoidal gratings of different geometries. The dip in dipper functions has been attributed to an expansive transducer function, which itself is attributed to two nonlinear inhibitory mechanisms: contrast normalization and surround suppression. We ran five contrast discrimination experiments, with targets and masks of different sizes and configurations: small Gabor target/small mask, small target/large mask, large target/large mask, small target/in-phase annular mask, and small target/out-of-phase annular mask. Our V1 modeling shows that the results for small Gabor target/small mask, small target/large mask, large target/large mask configurations are easily explained only if the model includes surround suppression. This is compatible with the finding that an in-phase annular mask generates only little threshold elevation while the out-of-phase mask was more effective. Surrounding mask gratings cannot be equated with surround suppression at the receptive-field level. We examine whether normalization and surround suppression occur simultaneously (parallel model) or sequentially (a better reflection of neurophysiology). The Akaike Criterion Difference showed that the sequential model was better than the parallel, but the difference was small. The large target/large mask dipper experiment was not well fit by our models, and we suggest that this may reflect selective attention for its uniquely larger test stimulus. The best-fit model replicates some behaviors of single V1 neurons, such as the decrease in receptive-field size with increasing contrast.

Diverted by dazzle: perceived movement direction is biased by target pattern orientation.

'Motion dazzle' is the hypothesis that predators may misjudge the speed or direction of moving prey which have high-contrast patterning, such as stripes. However, there is currently little experimental evidence that such patterns cause visual illusions. Here, observers binocularly tracked a Gabor target, moving with a linear trajectory randomly chosen within 18° of the horizontal. This target then became occluded, and observers were asked to judge where they thought it would later cross a vertical line to the side. We found that internal motion of the stripes within the Gabor biased judgements as expected: Gabors with upwards internal stripe motion relative to the overall direction of motion were perceived to be crossing above Gabors with downwards internal stripe movement. However, surprisingly, we found a much stronger effect of the rigid pattern orientation. Patches with oblique stripes pointing upwards relative to the direction of motion were perceived to cross above patches with downward-pointing stripes. This effect occurred only at high speeds, suggesting that it may reflect an orientation-dependent effect in which spatial signals are used in direction judgements. These findings have implications for our understanding of motion dazzle mechanisms and how human motion and form processing interact.

Quantifying peripheral and foveal perceived differences in natural image patches to predict visual search performance.

Duncan and Humphreys (1989) identified two key factors that affected performance in a visual search task for a target among distractors. The first was the similarity of the target to distractors (TD), and the second was the similarity of distractors to each other (DD). Here we investigate if it is the perceived similarity in foveal or peripheral vision that determines performance. We studied search using stimuli made from patches cut from colored images of natural objects; differences between targets and their modified distractors were estimated using a ratings task peripherally and foveally. We used search conditions in which the targets and distractors were easy to distinguish both foveally and peripherally ("high" stimuli), in which they were difficult to distinguish both foveally and peripherally ("low"), and in which they were easy to distinguish foveally but difficult to distinguish peripherally ("metamers"). In the critical metameric condition, search slopes (change of search time with number of distractors) were similar to the "low" condition, indicating a key role for peripheral information in visual search as both conditions have low perceived similarity peripherally. Furthermore, in all conditions, search slope was well described quantitatively from peripheral TD and DD but not foveal. However, some features of search, such as error rates, do indicate roles for foveal vision too.

A commentary on 'Amplitude spectra of natural images'.

Tolhurst, Tadmor and Tang Chao (Ophthalmic Physiol Opt 1992; 12: 229-232) published their analyses of the Fourier spectra of 135 digitised monochrome photographs of everyday scenes. In 2015, this would be a trivial exercise, but in 1992, this stretched emerging computer technology to its desktop limits. The short paper made an impact because it provided a handle for extending vision science beyond laboratory stimuli such as sinewave gratings into a real world of real visual scenes and real visual tasks.

Perception of differences in naturalistic dynamic scenes, and a V1-based model.

We investigate whether a computational model of V1 can predict how observers rate perceptual differences between paired movie clips of natural scenes. Observers viewed 198 pairs of movies clips, rating how different the two clips appeared to them on a magnitude scale. Sixty-six of the movie pairs were naturalistic and those remaining were low-pass or high-pass spatially filtered versions of those originals. We examined three ways of comparing a movie pair. The Spatial Model compared corresponding frames between each movie pairwise, combining those differences using Minkowski summation. The Temporal Model compared successive frames within each movie, summed those differences for each movie, and then compared the overall differences between the paired movies. The Ordered-Temporal Model combined elements from both models, and yielded the single strongest predictions of observers' ratings. We modeled naturalistic sustained and transient impulse functions and compared frames directly with no temporal filtering. Overall, modeling naturalistic temporal filtering improved the models' performance; in particular, the predictions of the ratings for low-pass spatially filtered movies were much improved by employing a transient impulse function. The correlations between model predictions and observers' ratings rose from 0.507 without temporal filtering to 0.759 (p = 0.01%) when realistic impulses were included. The sustained impulse function and the Spatial Model carried more weight in ratings for normal and high-pass movies, whereas the transient impulse function with the Ordered-Temporal Model was most important for spatially low-pass movies. This is consistent with models in which high spatial frequency channels with sustained responses primarily code for spatial details in movies, while low spatial frequency channels with transient responses code for dynamic events.

Visual contrast sensitivity deficits in 'normal' visual field of patients with homonymous visual field defects due to stroke: a pilot study.

BACKGROUND: Homonymous visual field defects (VFD) are common following stroke, and often recover, partially or fully, by unknown mechanisms. In clinical practice, visual field recovered on perimetry is often considered perceptually normal. However, studies have shown contrast sensitivity (CS) deficits in patients with stroke and homonymous VFD. This study investigated the origin of visual CS loss in patients with VFD due to stroke. We hypothesised that CS deficits would be found in visual field areas appearing normal on perimetry, in patients with ischaemic stroke affecting the retrochiasmal visual system, and that the spatiotemporal properties of this CS loss would be consistent with those of 'blindsight', perhaps suggesting similar underlying mechanisms. METHODS: CS measurements were made in 20 healthy participants, and in 7 patients with stroke causing homonymous VFD sparing foveal vision, measured using Humphrey static perimetry (SITA-Fast 24-2 procedure). Importantly, patients with concomitant visuospatial neglect were excluded. CS measurements were made using a modification of the method of increasing contrast, corrected for reaction time. Three spatial stimuli were used, at several spatial frequencies: (1) large sinusoidal gratings; (2) foveal Gabor patches; and (3) Gabor patches presenting in the putatively recovered visual field, near VFD. Stimuli with different temporal profiles were used to selectively stimulate transient and sustained visual channels, to provide insight into mechanisms of visual loss and/or recovery. Analysis of variance (ANOVA) was used in the analysis of the measurements, allowing for correction for age and stimulus eccentricity. RESULTS: ANOVA for sustained grating stimuli showed orientation-selective (horizontal) CS loss (p = 0.025); no such loss was apparent in the central visual field (foveal Gabor stimuli). Localised CS close to VFD was reduced in stroke-affected hemifields compared with unaffected hemifields (p ≤ 0.005), though these areas appeared normal on perimetry. In these areas, CS was relatively preserved for transient compared with sustained stimuli (Wilcoxon signed rank tests). CONCLUSIONS: The finding of specific CS deficits in the normal-appearing visual field of patients with homonymous VFD due to stroke suggests that static perimetry provides an inadequate assessment of visual function in these patients, with clear implications for testing of vision in clinical practice. The results are consistent with relative sparing of the transient/magnocellular visual channel. These findings demand further investigation. If confirmed in larger, longitudinal studies, this will have important implications for the mechanisms of recovery, and may provide a target for visual rehabilitation - for example, using repeated detection practice ('perceptual learning').

Regional effects of clutter on human target detection performance.

Clutter is something that is encountered in everyday life, from a messy desk to a crowded street. Such clutter may interfere with our ability to search for objects in such environments, like our car keys or the person we are trying to meet. A number of computational models of clutter have been proposed and shown to work well for artificial and other simplified scene search tasks. In this paper, we correlate the performance of different models of visual clutter to human performance in a visual search task using natural scenes. The models we evaluate are Feature Congestion (Rosenholtz, Li, & Nakano, 2007), Sub-band Entropy (Rosenholtz et al., 2007), Segmentation (Bravo & Farid, 2008), and Edge Density (Mack & Oliva, 2004) measures. The correlations were performed across a range of target-centered subregions to produce a correlation profile, indicating the scale at which clutter was affecting search performance. Overall clutter was rather weakly correlated with performance (r ≈ 0.2). However, different measures of clutter appear to reflect different aspects of the search task: correlations with Feature Congestion are greatest for the actual target patch, whereas the Sub-band Entropy is most highly correlated in a region 12° × 12° centered on the target.

Contrast normalization contributes to a biologically-plausible model of receptive-field development in primary visual cortex (V1).

Neuronal populations in the primary visual cortex (V1) of mammals exhibit contrast normalization. Neurons that respond strongly to simple visual stimuli - such as sinusoidal gratings - respond less well to the same stimuli when they are presented as part of a more complex stimulus which also excites other, neighboring neurons. This phenomenon is generally attributed to generalized patterns of inhibitory connections between nearby V1 neurons. The Bienenstock, Cooper and Munro (BCM) rule is a neural network learning rule that, when trained on natural images, produces model neurons which, individually, have many tuning properties in common with real V1 neurons. However, when viewed as a population, a BCM network is very different from V1 - each member of the BCM population tends to respond to the same dominant features of visual input, producing an incomplete, highly redundant code for visual information. Here, we demonstrate that, by adding contrast normalization into the BCM rule, we arrive at a neurally-plausible Hebbian learning rule that can learn an efficient sparse, overcomplete representation that is a better model for stimulus selectivity in V1. This suggests that one role of contrast normalization in V1 is to guide the neonatal development of receptive fields, so that neurons respond to different features of visual input.

Perception of suprathreshold naturalistic changes in colored natural images.

Simple everyday tasks, such as visual search, require a visual system that is sensitive to differences. Here we report how observers perceive changes in natural image stimuli, and what happens if objects change color, position, or identity-i.e., when the external scene changes in a naturalistic manner. We investigated whether a V1-based difference-prediction model can predict the magnitude ratings given by observers to suprathreshold differences in numerous pairs of natural images. The model incorporated contrast normalization and surround suppression, and elongated receptive-fields. Observers' ratings were better predicted when the model included phase invariance, and even more so when the stimuli were inverted and negated to lessen their semantic impact. Some feature changes were better predicted than others: the model systematically underpredicted observers' perception of the magnitude of blur, but over-predicted their ability to report changes in textures.

Magnitude of perceived change in natural images may be linearly proportional to differences in neuronal firing rates.

We are studying how people perceive naturalistic suprathreshold changes in the colour, size, shape or location of items in images of natural scenes, using magnitude estimation ratings to characterise the sizes of the perceived changes in coloured photographs. We have implemented a computational model that tries to explain observers' ratings of these naturalistic differences between image pairs. We model the action-potential firing rates of millions of neurons, having linear and non-linear summation behaviour closely modelled on real VI neurons. The numerical parameters of the model's sigmoidal transducer function are set by optimising the same model to experiments on contrast discrimination (contrast 'dippers') on monochrome photographs of natural scenes. The model, optimised on a stimulus-intensity domain in an experiment reminiscent of the Weber-Fechner relation, then produces tolerable predictions of the ratings for most kinds of naturalistic image change. Importantly, rating rises roughly linearly with the model's numerical output, which represents differences in neuronal firing rate in response to the two images under comparison; this implies that rating is proportional to the neuronal response.

Search for gross illumination discrepancies in images of natural objects.

Shadows may be "discounted" in human visual perception because they do not provide stable, lighting-invariant, information about the properties of objects in the environment. Using visual search, R. A. Rensink and P. Cavanagh (2004) found that search for an upright discrepant shadow was less efficient than for an inverted one. Here we replicate and extend this work using photographs of real objects (pebbles) and their shadows. The orientation of the target shadows was varied between 30 and 180 degrees. Stimuli were presented upright (light from above, the usual situation in the world) or inverted (light from below, unnatural lighting). RTs for upright images were slower for shadows angled at 30 degrees, exactly as found by Rensink and Cavanagh. However, for all other shadow angles tested, the RTs were faster for upright images. This suggests, for small discrepancies in shadow orientation, a switch of processing from a relatively coarse-scaled shadow system to other general-purpose visual routines. Manipulations of the visual heterogeneity of the pebbles that cast the shadows differentially influenced performance. For inverted images, heterogeneity had the expected influence: reducing search efficiency and increasing overall search time. This effect was greatly reduced when images were presented upright, presumably when the distractors were processed as shadows. We suggest that shadows may be processed in a functionally separate, spatially coarse, mechanism. The pattern of results suggests that human vision does not use a shadow-suppressing system in search tasks.

The sparseness of neuronal responses in ferret primary visual cortex.

Various arguments suggest that neuronal coding of natural sensory stimuli should be sparse (i.e., individual neurons should respond rarely but should respond reliably). We examined sparseness of visual cortical neurons in anesthetized ferret to flashed natural scenes. Response behavior differed widely between neurons. The median firing rate of 4.1 impulses per second was slightly higher than predicted from consideration of metabolic load. Thirteen percent of neurons (12 of 89) responded to <5% of the images, but one-half responded to >25% of images. Multivariate analysis of the range of sparseness values showed that 67% of the variance was accounted for by differing response patterns to moving gratings. Repeat presentation of images showed that response variance for natural images exaggerated sparseness measures; variance was scaled with mean response, but with a lower Fano factor than for the responses to moving gratings. This response variability and the "soft" sparse responses (Rehn and Sommer, 2007) raise the question of what constitutes a reliable neuronal response and imply parallel signaling by multiple neurons. We investigated whether the temporal structure of responses might be reliable enough to give additional information about natural scenes. Poststimulus time histogram shape was similar for "strong" and "weak" stimuli, with no systematic change in first-spike latency with stimulus strength. The variance of first-spike latency for repeat presentations of the same image was greater than the latency variance between images. In general, responses to flashed natural scenes do not seem compatible with a sparse encoding in which neurons fire rarely but reliably.

Camouflage and visual perception.

How does an animal conceal itself from visual detection by other animals? This review paper seeks to identify general principles that may apply in this broad area. It considers mechanisms of visual encoding, of grouping and object encoding, and of search. In most cases, the evidence base comes from studies of humans or species whose vision approximates to that of humans. The effort is hampered by a relatively sparse literature on visual function in natural environments and with complex foraging tasks. However, some general constraints emerge as being potentially powerful principles in understanding concealment--a 'constraint' here means a set of simplifying assumptions. Strategies that disrupt the unambiguous encoding of discontinuities of intensity (edges), and of other key visual attributes, such as motion, are key here. Similar strategies may also defeat grouping and object-encoding mechanisms. Finally, the paper considers how we may understand the processes of search for complex targets in complex scenes. The aim is to provide a number of pointers towards issues, which may be of assistance in understanding camouflage and concealment, particularly with reference to how visual systems can detect the shape of complex, concealed objects.

Do we know what the early visual system does?

We can claim that we know what the visual system does once we can predict neural responses to arbitrary stimuli, including those seen in nature. In the early visual system, models based on one or more linear receptive fields hold promise to achieve this goal as long as the models include nonlinear mechanisms that control responsiveness, based on stimulus context and history, and take into account the nonlinearity of spike generation. These linear and nonlinear mechanisms might be the only essential determinants of the response, or alternatively, there may be additional fundamental determinants yet to be identified. Research is progressing with the goals of defining a single "standard model" for each stage of the visual pathway and testing the predictive power of these models on the responses to movies of natural scenes. These predictive models represent, at a given stage of the visual pathway, a compact description of visual computation. They would be an invaluable guide for understanding the underlying biophysical and anatomical mechanisms and relating neural responses to visual perception.

Independent components of color natural scenes resemble V1 neurons in their spatial and color tuning.

It has been hypothesized that mammalian sensory systems are efficient because they reduce the redundancy of natural sensory input. If correct, this theory could unify our understanding of sensory coding; here, we test its predictions for color coding in the primate primary visual cortex (V1). We apply independent component analysis (ICA) to simulated cone responses to natural scenes, obtaining a set of colored independent component (IC) filters that form a redundancy-reducing visual code. We compare IC filters with physiologically measured V1 neurons, and find great spatial similarity between IC filters and V1 simple cells. On cursory inspection, there is little chromatic similarity; however, we find that many apparent differences result from biases in the physiological measurements and ICA analysis. After correcting these biases, we find that the chromatic tuning of IC filters does indeed resemble the population of V1 neurons, supporting the redundancy-reduction hypothesis.

Organization of the visual cortex in human albinism.

In albinism there is an abnormal projection of part of the temporal retina to the visual cortex contralateral to the eye. This projection, together with the normally routed fibers from nasal retina, provides a cortical hemisphere with visual input from more than the normal hemifield of visual space. In many mammalian models of albinism, a possible sensory mismatch in the visual cortex is avoided either by reorganization of the thalamocortical connections to give the abnormal input an exclusive cortical representation, or by the abnormal input being substantially suppressed. In this study we examine, with fMRI, how the human visual cortex topographically maps its input in albinism. We find that the input from temporal retina is not substantially suppressed and forms a retinotopic mapping that is superimposed on the mapping of the nasal retina in striate and extrastriate areas. The abnormal routing of temporal fibers is not total, with the line of decussation shifting to between 6 and 14 degrees into temporal retina. Our results indicate that the abnormal input to visual cortex in human albinism does not undergo topographic reorganization between the thalamus and cortex. Furthermore, the abnormal input is not significantly suppressed in either striate or extrastriate areas. The topographic mapping that we report in human does not conform, therefore, to the commonly observed patterns in other mammals but takes the form of the "true albino" pattern that has been reported rarely in cat and in the only other individual primate studied.

Coding of the contrasts in natural images by visual cortex (V1) neurons: a Bayesian approach.

Individual V1 neurons respond dynamically over only limited ranges of stimulus contrasts, yet we can discriminate contrasts over a wide range. Different V1 neurons cover different parts of the contrast range, and the information they provide must be pooled somehow. We describe a probabilistic pooling model that shows that populations of neurons with contrast responses like those in cat and monkey V1 would most accurately code contrasts in the range actually found in natural scenes. The pooling equation is similar to Bayes's equation; however, explicit inclusion of prior probabilities in the inference increases coding accuracy only slightly.

The receptive-field organization of simple cells in primary visual cortex of ferrets under natural scene stimulation.

The responses of simple cells in primary visual cortex to sinusoidal gratings can primarily be predicted from their spatial receptive fields, as mapped using spots or bars. Although this quasilinearity is well documented, it is not clear whether it holds for complex natural stimuli. We recorded from simple cells in the primary visual cortex of anesthetized ferrets while stimulating with flashed digitized photographs of natural scenes. We applied standard reverse-correlation methods to quantify the average natural stimulus that invokes a neuronal response. Although these maps cannot be the receptive fields, we find that they still predict the preferred orientation of grating for each cell very well (r = 0.91); they do not predict the spatial-frequency tuning. Using a novel application of the linear reconstruction method called regularized pseudoinverse, we were able to recover high-resolution receptive-field maps from the responses to a relatively small number of natural scenes. These receptive-field maps not only predict the optimum orientation of each cell (r = 0.96) but also the spatial-frequency optimum (r = 0.89); the maps also predict the tuning bandwidths of many cells. Therefore, our first conclusion is that the tuning preferences of the cells are primarily linear and constant across stimulus type. However, when we used these maps to predict the actual responses of the cells to natural scenes, we did find evidence of expansive output nonlinearity and nonlinear influences from outside the classical receptive fields, orientation tuning, and spatial-frequency tuning.

Relay of visual information to the lateral geniculate nucleus and the visual cortex in albino ferrets.

The abnormal organization of the central visual pathways in the albino ferret has been characterized anatomically and physiologically. Recordings in dorsal lateral geniculate nucleus of the albino ferret show that lamina A1, which receives an aberrant projection from the contralateral eye, contains an extensive representation of the ipsilateral visual hemifield with receptive fields located up to 35 degrees from the vertical meridian. This is not the case in pigmented ferrets, for which the vast majority of units, activated through either the contralateral or ipsilateral eye, have receptive fields confined to the contralateral hemifield. The few fields found in the ipsilateral hemifield are driven through the contralateral eye and none is more than 10 degrees from the midline. Cortical topography was studied by making closely spaced electrode penetrations across the area 17/18 border. In pigmented animals, the reversal of topography at the border is characterized by units with receptive fields centered a few degrees into the ipsilateral hemifield. In 22 of 25 albinos, the "Boston" aberrant topography was found: the representation of the vertical meridian is within area 17, rather than at the area 17/18 border. Instead, at the area 17/18 border, there is a reversal in the topographic progression at up to 30 degrees into the ipsilateral hemifield. This pattern was most pronounced in the upper visual field. In agreement with the "Boston" physiology, injections of retrograde tracer made in area 17 usually label neurons in either lamina A or the part of lamina A1 that is aberrantly innervated by the contralateral eye. A column of labeled cells extending through all geniculate layers is rarely seen in albinos, although this is commonly the pattern in pigmented ferrets.