What Fechner could not do: Separating perceptual encoding and decoding with difference scaling.

A key question in perception research is how stimulus variations translate into perceptual magnitudes, that is, the perceptual encoding process. As experimenters, we cannot probe perceptual magnitudes directly, but infer the encoding process from responses obtained in a psychophysical experiment. The most prominent experimental technique to measure perceptual appearance is matching, where observers adjust a probe stimulus to match a target in its appearance along the dimension of interest. The resulting data quantify the perceived magnitude of the target in physical units of the probe, and are thus an indirect expression of the underlying encoding process. In this paper, we show analytically and in simulation that data from matching tasks do not sufficiently constrain perceptual encoding functions, because there exist an infinite number of pairs of encoding functions that generate the same matching data. We use simulation to demonstrate that maximum likelihood conjoint measurement (Ho, Landy, & Maloney, 2008; Knoblauch & Maloney, 2012) does an excellent job of recovering the shape of ground truth encoding functions from data that were generated with these very functions. Finally, we measure perceptual scales and matching data for White's effect (White, 1979) and show that the matching data can be predicted from the estimated encoding functions, down to individual differences.

Fixational eye movements enable robust edge detection.

Human vision relies on mechanisms that respond to luminance edges in space and time. Most edge models use orientation-selective mechanisms on multiple spatial scales and operate on static inputs assuming that edge processing occurs within a single fixational instance. Recent studies, however, demonstrate functionally relevant temporal modulations of the sensory input due to fixational eye movements. Here we propose a spatiotemporal model of human edge detection that combines elements of spatial and active vision. The model augments a spatial vision model by temporal filtering and shifts the input images over time, mimicking an active sampling scheme via fixational eye movements. The first model test was White's illusion, a lightness effect that has been shown to depend on edges. The model reproduced the spatial-frequency-specific interference with the edges by superimposing narrowband noise (1-5 cpd), similar to the psychophysical interference observed in White's effect. Second, we compare the model's edge detection performance in natural images in the presence and absence of Gaussian white noise with human-labeled contours for the same (noise-free) images. Notably, the model detects edges robustly against noise in both test cases without relying on orientation-selective processes. Eliminating model components, we demonstrate the relevance of multiscale spatiotemporal filtering and scale-specific normalization for edge detection. The proposed model facilitates efficient edge detection in (artificial) vision systems and challenges the notion that orientation-selective mechanisms are required for edge detection.

Conjoint measurement of perceived transparency and perceived contrast in variegated checkerboards.

One fundamental question in vision research is how the retinal input is segmented into perceptually relevant variables. A striking example of this segmentation process is transparency perception, in which luminance information in one location contributes to two perceptual variables: the properties of the transparent medium itself and of what is being seen in the background. Previous work by Robilotto et al. (2002, 2004) suggested that perceived transparency is closely related to perceived contrast, but how these two relate to retinal luminance has not been established. Here we studied the relationship between perceived transparency, perceived contrast, and image luminance using maximum likelihood conjoint measurement (MLCM). Stimuli were rendered images of variegated checkerboards that were composed of multiple reflectances and partially covered by a transparent overlay. We systematically varied the transmittance and reflectance of the transparent medium and measured perceptual scales of perceived transparency. We also measured scales of perceived contrast using cut-outs of the transparency stimuli that did not contain any geometrical cues to transparency. Perceptual scales for perceived transparency and contrast followed a remarkably similar pattern across observers. We tested the empirically observed scales against predictions from various contrast metrics and found that perceived transparency and perceived contrast were equally well predicted by a metric based on the logarithm of Michelson or Whittle contrast. We conclude that judgments of perceived transparency and perceived contrast are likely to be supported by a common mechanism, which can be computationally captured as a logarithmic contrast.

Toward reliable measurements of perceptual scales in multiple contexts.

A central question in psychophysical research is how perceptual differences between stimuli translate into physical differences and vice versa. Characterizing such a psychophysical scale would reveal how a stimulus is converted into a perceptual event, particularly under changes in viewing conditions (e.g., illumination). Various methods exist to derive perceptual scales, but in practice, scale estimation is often bypassed by assessing appearance matches. Matches, however, only reflect the underlying perceptual scales but do not reveal them directly. Two recently developed methods, MLDS (Maximum Likelihood Difference Scaling) and MLCM (Maximum Likelihood Conjoint Measurement), promise to reliably estimate perceptual scales. Here we compared both methods in their ability to estimate perceptual scales across context changes in the domain of lightness perception. In simulations, we adopted a lightness constant, a contrast, and a luminance-based observer model to generate differential patterns of perceptual scales. MLCM correctly recovered all models. MLDS correctly recovered only the lightness constant observer model. We also empirically probed both methods with two types of stimuli: (a) variegated checkerboards that support lightness constancy and (b) center-surround stimuli that do not support lightness constancy. Consistent with the simulations, MLDS and MLCM provided similar scale estimates in the first case and divergent estimates in the second. In addition, scales from MLCM-and not from MLDS-accurately predicted asymmetric matches for both types of stimuli. Taking experimental and simulation results together, MLCM seems more apt to provide a valid estimate of the perceptual scales underlying judgments of lightness across viewing conditions.

Maximum likelihood difference scales represent perceptual magnitudes and predict appearance matches.

One central problem in perception research is to understand how internal experiences are linked to physical variables. Most commonly, this relationship is measured using the method of adjustment, but this has two shortcomings: The perceptual scales that relate physical and perceptual variables are not measured directly, and the method often requires perceptual comparisons between viewing conditions. To overcome these problems, we measured perceptual scales of surface lightness using maximum likelihood difference scaling, asking observers only to compare the lightness of surfaces presented in the same context. Observers were lightness constant, and the perceptual scales qualitatively and quantitatively predicted perceptual matches obtained in a conventional adjustment experiment. Additionally, we show that a contrast-based model of lightness perception predicted 98% of the variance in the scaling and 88% in the matching data. We suggest that the predictive power was higher for scales because they are closer to the true variables of interest.

Comparing sensitivity estimates from MLDS and forced-choice methods in a slant-from-texture experiment.

Maximum likelihood difference scaling (MLDS) is a method for the estimation of perceptual scales based on the judgment of differences in stimulus appearance (Maloney & Yang, 2003). MLDS has recently also been used to estimate near-threshold discrimination performance (Devinck & Knoblauch, 2012). Using MLDS as a psychophysical method for sensitivity estimation is potentially appealing, because MLDS has been reported to need less data than forced-choice procedures, and particularly naive observers report to prefer suprathreshold comparisons to JND-style threshold tasks. Here we compare two methods, MLDS and two-interval forced-choice (2-IFC), regarding their capability to estimate sensitivity assuming an underlying signal-detection model. We first examined the theoretical equivalence between both methods using simulations. We found that they disagreed in their estimation only when sensitivity was low, or when one of the assumptions on which MLDS is based was violated. Furthermore, we found that the confidence intervals derived from MLDS had a low coverage; i.e., they were too narrow, underestimating the true variability. Subsequently we compared MLDS and 2-IFC empirically using a slant-from-texture task. The amount of agreement between sensitivity estimates from the two methods varied substantially across observers. We discuss possible reasons for the observed disagreements, most notably violations of the MLDS model assumptions. We conclude that in the present example MLDS and 2-IFC could equally be used to estimate sensitivity to differences in slant, with MLDS having the benefit of being more efficient and more pleasant, but having the disadvantage of unsatisfying coverage.

Testing the role of Michelson contrast for the perception of surface lightness.

It is still an unresolved question how the visual system perceives surface lightness given the ambiguity of the sensory input signal. We studied lightness perception using two-dimensional images of variegated checkerboards shown as perspective projections of three-dimensional objects. We manipulated the contrast of a target check relative to its surround either by rendering the image under different viewing conditions or by introducing noncoincidental changes of the reflectance of the surfaces adjacent to the target. We examined the predictive power of the normalized contrast model (Zeiner & Maertens, 2014) for the different viewing conditions (plain view vs. dark and light transparency) as well as for the noncoincidental surround changes (only high or only low reflectances in the surround). The model accounted for lightness matches across different viewing conditions but not for the surround changes. The observed simultaneous contrast effects were smaller than what would be predicted by the model. We evaluated two model extensions that-both relying on contrast-predicted the observed data well. Both model extensions point to the importance of contrast statistics across space and/or time for the computation of lightness, but it awaits future testing to evaluate whether and how the visual system could represent such statistics.

Measuring the Visual Salience of 3D Printed Objects.

To investigate human viewing behavior on physical realizations of 3D objects, the authors use an eye tracker with scene camera and fiducial markers on 3D objects to gather fixations on the presented stimuli. They use this data to validate assumptions regarding visual saliency that so far have experimentally only been analyzed for flat stimuli. They provide a way to compare fixation sequences from different subjects and developed a model for generating test sequences of fixations unrelated to the stimuli. Their results suggest that human observers agree in their fixations for the same object under similar viewing conditions. They also developed a simple procedure to validate computational models for visual saliency of 3D objects and found that popular models of mesh saliency based on center surround patterns fail to predict fixations.

Noise masking of White's illusion exposes the weakness of current spatial filtering models of lightness perception.

Spatial filtering models are currently a widely accepted mechanistic account of human lightness perception. Their popularity can be ascribed to two reasons: They correctly predict how human observers perceive a variety of lightness illusions, and the processing steps involved in the models bear an apparent resemblance with known physiological mechanisms at early stages of visual processing. Here, we tested the adequacy of these models by probing their response to stimuli that have been modified by adding narrowband noise. Psychophysically, it has been shown that noise in the range of one to five cycles per degree (cpd) can drastically reduce the strength of some lightness phenomena, while noise outside this range has little or no effect on perceived lightness. Choosing White's illusion (White, 1979) as a test case, we replicated and extended the psychophysical results, and found that none of the spatial filtering models tested was able to reproduce the spatial frequency specific effect of narrowband noise. We discuss the reasons for failure for each model individually, but we argue that the failure is indicative of the general inadequacy of this class of spatial filtering models. Given the present evidence we do not believe that spatial filtering models capture the mechanisms that are responsible for producing many of the lightness phenomena observed in human perception. Instead we think that our findings support the idea that low-level contributions to perceived lightness are primarily determined by the luminance contrast at surface boundaries.

Testing the role of luminance edges in White's illusion with contour adaptation.

White's illusion is the perceptual effect that two equiluminant gray patches superimposed on a black-and-white square-wave grating appear different in lightness: A test patch placed on a dark stripe of the grating looks lighter than one placed on a light stripe. Although the effect does not depend on the aspect ratio of the test patches, and thus on the amount of border that is shared with either the dark or the light stripe, the context of each patch must, in a yet to be specified way, influence their lightness. We employed a contour adaptation paradigm (Anstis, 2013) to test the contribution of each of the test patches' edges to the perceived lightness of the test patches. We found that adapting to the edges that are oriented parallel to the grating slightly increased the lightness illusion, whereas adapting to the orthogonal edges abolished, or for some observers even reversed, the lightness illusion. We implemented a temporal adaptation mechanism in three spatial filtering models of lightness perception, and show that the models cannot account for the observed adaptation effects. We conclude that White's illusion is largely determined by edge contrast across the edge orthogonal to the grating, whereas the parallel edge has little or no influence. We suggest mechanisms that could explain this asymmetry.

Context affects lightness at the level of surfaces.

Visual perception of object attributes such as surface lightness is crucial for successful interaction with the environment. How the visual system assigns lightness to image regions is not yet understood. It has been shown that the context in which a surface is embedded influences its perceived lightness, but whether that influence involves predominantly low-, mid-, or high-level visual mechanisms has not been resolved. To answer this question, we measured whether perceptual attributes of target image regions affected their perceived lightness when they were placed in different contexts. We varied the sharpness of the edge while keeping total target flux fixed. Targets with a sharp edge were consistent with the perceptual interpretation of a surface, and in that case, observers perceived significant brightening or darkening of the target. Targets with blurred edges rather appeared to be spotlights instead of surfaces; for targets with blurred edges, there was much less of a contextual effect on target lightness. The results indicate that the effect of context on the lightness of an image region is not fixed but is strongly affected by image manipulations that modify the perceptual attributes of the target, implying that a mid-level scene interpretation affects lightness perception.

Linking luminance and lightness by global contrast normalization.

In the present experiment we addressed the question of how the visual system determines surface lightness from luminances in the retinal image. We measured the perceived lightness of target surfaces that were embedded in custom-made checkerboards. The checkerboards consisted of 10 by 10 checks of 10 different reflectance values that were arranged randomly across the board. They were rendered under six viewing conditions including plain view, with a shadow-casting cylinder, or with one of four different transparent media covering part of the board. For each reflectance we measured its corresponding luminance in the different viewing conditions. We then assessed the lightness matches of four observers for each of the reflectances in the different viewing conditions. We derived predictions of perceived lightness based on local luminance, Michelson contrast, edge integration, anchoring theory, and a normalized Michelson contrast measure. The normalized contrast measure was the best predictor of surface lightness and was almost as good as the actual reflectance values. The normalized contrast measure combines a local computation of Michelson contrast with a region-based normalization of contrast ranges with respect to the contrast range in plain view. How the segregation of image regions is accomplished remains to be elucidated.

Linking appearance to neural activity through the study of the perception of lightness in naturalistic contexts.

The present paper deals with the classical question how a psychological experience, in this case apparent lightness, is linked by intervening neural processing to physical variables. We address two methodological issues: (a) how does one know the appropriate physical variable (what is the right x?) to look at, and (b) how can behavioral measurements be used to probe the internal transformation that leads to psychological experience. We measured so-called lightness transfer functions (LTFs), that is the functions that describe the mapping between retinal luminance and perceived lightness for naturalistic checkerboard stimuli. The LTFs were measured for different illumination situations: plain view, a cast shadow, and an intervening transparent medium. Observers adjusted the luminance of a comparison patch such that it had the same lightness as each of the test patches. When the data were plotted in luminance-luminance space, we found qualitative differences between mapping functions in different contexts. These differences were greatly diminished when the data were plotted in terms of contrast. On contrast-contrast coordinates, the data were compatible with a single linear generative model. This result is an indication that, for the naturalistic scenes used here, lightness perception depends mostly on local contrast. We further discuss that, in addition to the mean adjustments, one may find it useful to consider also the variability of an observer's adjustments in order to infer the true luminance-to-lightness mapping function.

When luminance increment thresholds depend on apparent lightness.

A fundamental question in visual perception research is whether the sensitivity to stimulus differences is limited by the sensory representation of the external stimulus, that is, the proximal stimulus, or by its perceptual representation, i.e., stimulus appearance. In the domain of lightness perception, the question translates into whether discrimination thresholds depend on the local luminance in the retinal image or on the apparent lightness of the corresponding image region. The majority of findings seem to indicate that sensitivity is limited by the sensory stimulus representation, which would imply different mechanisms for stimulus discrimination and appearance. We think this conclusion needs to be qualified. We report data suggesting that the relationship between discrimination and appearance judgments depends on how exactly they are being measured. We propose a theoretical account that provides a common mechanism for appearance and sensitivity. An interesting corollary of this model is that it also accounts for the perceptual phenomenon of assimilation.

Angle alignment evokes perceived depth and illusory surfaces.

There is a distinct visual process that triggers the perception of illusory surfaces and contours along the intersections of aligned, zigzag line patterns. Such illusory contours and surfaces are qualitatively different from illusory contours of the Kanizsa type. The illusory contours and surfaces in this case are not the product of occlusion and do not imply occlusion of one surface by another. Rather, the aligned angles in the patterns are combined by the visual system into the perception of a fold or a 3-D corner, as of stairs on a staircase or a wall ending on a floor. The depth impression is ambiguous and reversible like the Necker cube. Such patterns were used by American Indian artists of the Akimel O'odham (Pima) tribe in basketry, and also by modern European and American artists like Josef Albers, Bridget Riley, Victor Vasarely, and Frank Stella. Our research aims to find out what manipulations of the visual image affect perceived depth in such patterns in order to learn about the perceptual mechanisms. Using paired comparisons, we find that human observers perceive depth in such patterns if, and only if, lines in adjacent regions of the patterns join to form angles, and also if, and only if, the angles are aligned precisely to be consistent with a fold or 3-D corner. The amount of perceived depth is graded, depending on the steepness and the density of angles in the aligned-angle pattern. The required precision of the alignment implies that early retinotopic visual cortical areas may be involved in this perceptual behavior, but the linkage of form with perceived depth suggests involvement of higher cortical areas as well.

Retinotopic activation in response to subjective contours in primary visual cortex.

Objects in our visual environment are arranged in depth and hence there is a considerable amount of overlap and occlusion in the image they generate on the retina. In order to properly segment the image into figure and background, boundary interpolation is required even across large distances. Here we study the cortical mechanisms involved in collinear contour interpolation using fMRI. Human observers were asked to discriminate the curvature of interpolated boundaries in Kanizsa figures and in control configurations, which contained identical physical information but did not generated subjective shapes. We measured a spatially precise spin-echo BOLD signal and found stronger responses to subjective shapes than non-shapes at the subjective boundary locations, but not at the inducer locations. The responses to subjective contours within primary visual cortex were retinotopically specific and analogous to that to real contours, which is intriguing given that subjective and luminance-defined contours are physically fundamentally different. We suggest that in the absence of retinal stimulation, the observed activation changes in primary visual cortex are driven by intracortical interactions and feedback, which are revealed in the absence of a physical stimulus.

Local determinants of contour interpolation.

Objects in our visual environment are perceived as integral wholes even when their retinal images are incomplete. We ask whether the perceptual precision of subjective interpolation between isolated image parts depends on the overall proportion of visible image information or rather on its geometrical arrangement. We used Varin-type subjective shapes that provide less physical stimulus information than Kanizsa-type figures because partially occluded solid inducers are replaced by partially occluded concentric arcs. We tested whether perceptual precision varies as a function of contour support, or alternatively, depends on the number of, and the distance between, line endings within the inducers. We measured performance in a probe localization task, where a small target is presented at different distances around a subjective boundary. Sensitivity, captured by the just noticeable position difference between in- and outside probes, crucially depended on the geometric arrangement of line ends in the Varin figures. This is objective evidence that the apparent subjective contour strength does not primarily depend on contour support but is determined by the number and the separation between inducers' line endings. The results suggest that neuronal mechanisms sensitive to highly localized 2D features are crucial for determining the perceived shape of visual objects.

Illusory contours do not pass through the "blind spot".

Our visual percepts are not fully determined by the physical stimulus input. That is why we perceive crisp bounding contours even in the absence of luminance-defined borders in visual illusions such as the Kanizsa figure. It is important to understand which neural processes are involved in creating these artificial visual experiences because this might tell us how we perceive coherent objects in natural scenes, which are characterized by mutual overlap. We have already shown using functional magnetic resonance imaging [Maertens, M., & Pollmann, S. fMRI reveals a common neural substrate of illusory and real contours in v1 after perceptual learning. Journal of Cognitive Neuroscience, 17, 1553-1564, 2005] that neurons in the primary visual cortex (V1) respond to these stimuli. Here we provide support for the hypothesis that V1 is obligatory for the discrimination of the curvature of illusory contours. We presented illusory contours across the portion of the visual field corresponding to the physiological "blind spot." Four observers were extensively trained and asked to discriminate fine curvature differences in these illusory contours. A distinct performance drop (increased errors and response latencies) was observed when illusory contours traversed the blind spot compared to when they were presented in the "normal" contralateral visual field at the same eccentricity. We attribute this specific performance deficit to the failure to build up a representation of the illusory contour in the absence of a cortical representation of the "blind spot" within V1. The current results substantiate the assumption that neural activity in area V1 is closely related to our phenomenal experience of illusory contours in particular, and to the construction of our subjective percepts in general.

Perception modulates auditory cortex activation.

Event-related functional magnetic resonance imaging signal change in Heschl's gyrus and the planum temporale was found to reflect sensory decisions about target presence. In a dichotic listening task, activation was higher for target present responses, irrespective of actual target presence. In fact, activation was highest for false alarms, that is, 'present' responses in the absence of a target stimulus, and lowest for missed targets. This shows that activity at the earliest stage of cortical auditory processing reflects subjective perceptual decisions. Whether this activation is driven by bottom-up or top-down factors remains to be investigated.

Selective and interactive neural correlates of visual dimension changes and response changes.

In an event-related fMRI study, we investigated the neural correlates of visual dimension and response changes. We used a compound task, which required target selection by a singleton feature, a unique color or motion direction, before the appropriate motor response, which was determined by target orientation, could be selected. Both types of change elicited distinct patterns of activation, with dimension-change-related activation primarily in posterior visual areas and response-related activation primarily in motor-related areas of the parietal and frontal cortices. Response-change-related activation was delayed by about 1 s relative to dimension-change-related activation, suggesting that the latter is elicited by perceptual processes, whereas the former reflects response-related or post-response processes. Although dimension changes and response changes rely on different processes, they are not independent: response facilitation was observed for combined dimension and response repetitions, this facilitation, however, was disrupted by dimension changes.