The timing mega-study: comparing a range of experiment generators, both lab-based and online.

Many researchers in the behavioral sciences depend on research software that presents stimuli, and records response times, with sub-millisecond precision. There are a large number of software packages with which to conduct these behavioral experiments and measure response times and performance of participants. Very little information is available, however, on what timing performance they achieve in practice. Here we report a wide-ranging study looking at the precision and accuracy of visual and auditory stimulus timing and response times, measured with a Black Box Toolkit. We compared a range of popular packages: PsychoPy, E-Prime®, NBS Presentation®, Psychophysics Toolbox, OpenSesame, Expyriment, Gorilla, jsPsych, Lab.js and Testable. Where possible, the packages were tested on Windows, macOS, and Ubuntu, and in a range of browsers for the online studies, to try to identify common patterns in performance. Among the lab-based experiments, Psychtoolbox, PsychoPy, Presentation and E-Prime provided the best timing, all with mean precision under 1 millisecond across the visual, audio and response measures. OpenSesame had slightly less precision across the board, but most notably in audio stimuli and Expyriment had rather poor precision. Across operating systems, the pattern was that precision was generally very slightly better under Ubuntu than Windows, and that macOS was the worst, at least for visual stimuli, for all packages. Online studies did not deliver the same level of precision as lab-based systems, with slightly more variability in all measurements. That said, PsychoPy and Gorilla, broadly the best performers, were achieving very close to millisecond precision on several browser/operating system combinations. For response times (measured using a high-performance button box), most of the packages achieved precision at least under 10 ms in all browsers, with PsychoPy achieving a precision under 3.5 ms in all. There was considerable variability between OS/browser combinations, especially in audio-visual synchrony which is the least precise aspect of the browser-based experiments. Nonetheless, the data indicate that online methods can be suitable for a wide range of studies, with due thought about the sources of variability that result. The results, from over 110,000 trials, highlight the wide range of timing qualities that can occur even in these dedicated software packages for the task. We stress the importance of scientists making their own timing validation measurements for their own stimuli and computer configuration.

Measuring nonlinear signal combination using EEG.

Relatively little is known about the processes, both linear and nonlinear, by which signals are combined beyond V1. By presenting two stimulus components simultaneously, flickering at different temporal frequencies (frequency tagging) while measuring steady-state visual evoked potentials, we can assess responses to the individual components, including direct measurements of suppression on each other, and various nonlinear responses to their combination found at intermodulation frequencies. The result is a rather rich dataset of frequencies at which responses can be found. We presented pairs of sinusoidal gratings at different temporal frequencies, forming plaid patterns that were "coherent" (looking like a checkerboard) and "noncoherent" (looking like a pair of transparently overlaid gratings), and found clear intermodulation responses to compound stimuli, indicating nonlinear summation. This might have been attributed to cross-orientation suppression except that the pattern of intermodulation responses differed for coherent and noncoherent patterns, whereas the effects of suppression (measured at the component frequencies) did not. A two-stage model of nonlinear summation involving conjunction detection with a logical AND gate described the data well, capturing the difference between coherent and noncoherent plaids over a wide array of possible response frequencies. Multistimulus frequency-tagged EEG in combination with computational modeling may be a very valuable tool in studying the conjunction of these signals. In the current study the results suggest a second-order mechanism responding selectively to coherent plaid patterns.

Understanding mid-level representations in visual processing.

It is clear that early visual processing provides an image-based representation of the visual scene: Neurons in Striate cortex (V1) encode nothing about the meaning of a scene, but they do provide a great deal of information about the image features within it. The mechanisms of these "low-level" visual processes are relatively well understood. We can construct plausible models for how neurons, up to and including those in V1, build their representations from preceding inputs down to the level of photoreceptors. It is also clear that at some point we have a semantic, "high-level" representation of the visual scene because we can describe verbally the objects that we are viewing and their meaning to us. A huge number of studies are examining these "high-level" visual processes each year. Less well studied are the processes of "mid-level" vision, which presumably provide the bridge between these "low-level" representations of edges, colors, and lights and the "high-level" semantic representations of objects, faces, and scenes. This article and the special issue of papers in which it is published consider the nature of "mid-level" visual processing and some of the reasons why we might not have made as much progress in this domain as we would like.

Cue Combination of Conflicting Color and Luminance Edges.

Abrupt changes in the color or luminance of a visual image potentially indicate object boundaries. Here, we consider how these cues to the visual "edge" location are combined when they conflict. We measured the extent to which localization of a compound edge can be predicted from a simple maximum likelihood estimation model using the reliability of chromatic (L-M) and luminance signals alone. Maximum likelihood estimation accurately predicted the pattern of results across a range of contrasts. Predictions consistently overestimated the relative influence of the luminance cue; although L-M is often considered a poor cue for localization, it was used more than expected. This need not indicate that the visual system is suboptimal but that its priors about which cue is more useful are not flat. This may be because, although strong changes in chromaticity typically represent object boundaries, changes in luminance can be caused by either a boundary or a shadow.

Is it just motion that silences awareness of other visual changes?

When an array of visual elements is changing color, size, or shape incoherently, the changes are typically quite visible even when the overall color, size, or shape statistics of the field may not have changed. When the dots also move, however, the changes become much less apparent; awareness of them is "silenced" (Suchow & Alvarez, 2011). This finding might indicate that the perception of motion is of particular importance to the visual system, such that it is given priority in processing over other forms of visual change. Here we test whether that is the case by examining the converse: whether awareness of motion signals can be silenced by potent coherent changes in color or size. We find that they can, and with very similar effects, indicating that motion is not critical for silencing. Suchow and Alvarez's dots always moved in the same direction with the same speed, causing them to be grouped as a single entity. We also tested whether this coherence was a necessary component of the silencing effect. It is not; when the dot speeds are randomly selected, such that no coherent motion is present, the silencing effect remains. It is clear that neither motion nor grouping is directly responsible for the silencing effect. Silencing can be generated from any potent visual change.

Luminance cues constrain chromatic blur discrimination in natural scene stimuli.

Introducing blur into the color components of a natural scene has very little effect on its percept, whereas blur introduced into the luminance component is very noticeable. Here we quantify the dominance of luminance information in blur detection and examine a number of potential causes. We show that the interaction between chromatic and luminance information is not explained by reduced acuity or spatial resolution limitations for chromatic cues, the effective contrast of the luminance cue, or chromatic and achromatic statistical regularities in the images. Regardless of the quality of chromatic information, the visual system gives primacy to luminance signals when determining edge location. In natural viewing, luminance information appears to be specialized for detecting object boundaries while chromatic information may be used to determine surface properties.

Transfer of perceptual learning between different visual tasks.

Practice in most sensory tasks substantially improves perceptual performance. A hallmark of this 'perceptual learning' is its specificity for the basic attributes of the trained stimulus and task. Recent studies have challenged the specificity of learned improvements, although transfer between substantially different tasks has yet to be demonstrated. Here, we measure the degree of transfer between three distinct perceptual tasks. Participants trained on an orientation discrimination, a curvature discrimination, or a 'global form' task, all using stimuli comprised of multiple oriented elements. Before and after training they were tested on all three and a contrast discrimination control task. A clear transfer of learning was observed, in a pattern predicted by the relative complexity of the stimuli in the training and test tasks. Our results suggest that sensory improvements derived from perceptual learning can transfer between very different visual tasks.

The timing of binding and segregation of two compound aftereffects.

Temporal information in a scene is thought to be an important cue for visual grouping of local image features into a single object. The majority of studies on this topic have attempted to determine the conditions that facilitate segregation of a figure from a cluttered background. Here we examine the temporal characteristics of two aftereffects that appear to have roles in visual integration: the curvature aftereffect (CAE; Hancock & Peirce, 2008) and plaid-selective contrast adaptation (Peirce & Taylor, 2006). Both aftereffects used a "compound adaptation" paradigm measuring adaptation to a compound stimulus that cannot be explained by adaptation to its components presented in isolation. The temporal tuning characteristics of the two aftereffects differed in three distinct ways. First, plaid-selective adaptation was very sensitive to temporal phase asynchronies, while the CAE was not. Second, while both aftereffects showed integration of alternating components above 4 Hz, for plaids the overall magnitude of adaptation was less than to synchronous stimuli and was eliminated at the highest frequencies. Finally, plaid-selective adaptation demonstrated a low-pass dependency for temporal flicker frequency of synchronous gratings, whereas the CAE did not. Overall, these results suggest that at least two different mechanisms are involved in the binding/segregation of local signals into compound patterns: one with high temporal resolution that allows rapid parsing of plaid patterns into their components and one with a coarser temporal sensitivity that mediates the CAE.

Global shape processing: which parts form the whole?

Research suggests that detection of low-frequency radial frequency (RF) patterns involves global shape processing and that points of maximum curvature (corners) contribute more than points of minimum curvature (sides). However, this has only been tested with stimuli presented at the threshold of discriminability from a circle. We used RF pattern adaptation to (a) examine whether a supra-threshold RF pattern is processed as a global shape and (b) determine what the critical features are for representing its shape. We measured the perceived amplitude shift of an RF test pattern after prolonged exposure either to a higher amplitude pattern or to various combinations of its parts (concave maxima, convex maxima, inflections). We found greater shifts in perceived amplitude after adaptation to a "whole" pattern than after adaptation to its component parts, which alternated to produce equal net contrast. Furthermore, when adapting to specific parts of the shape in isolation, we found that each part generated a similar magnitude aftereffect. Although the whole is clearly greater than the sum of the parts, we find that concave maxima, convex maxima, and inflections contribute equally to global shape processing, a fact that is only apparent when using a supra-threshold appearance-based task.

Ameliorating the combinatorial explosion with spatial frequency-matched combinations of V1 outputs.

Little is known about the way in which the outputs of early orientation-selective neurons are combined. One particular problem is that the number of possible combinations of these outputs greatly outweighs the number of processing units available to represent them. Here we consider two of the possible ways in which the visual system might reduce the impact of this problem. First, the visual system might ameliorate the problem by collapsing across some low-level feature coded by previous processing stages, such as spatial frequency. Second, the visual system may combine only a subset of available outputs, such as those with similar receptive field characteristics. Using plaid-selective contrast adaptation and the curvature aftereffect, we found no evidence for the former solution; both aftereffects were clearly tuned to the spatial frequency of the adaptor relative to the test probe. We did, however, find evidence for the latter with both aftereffects; when the components forming our compound stimuli were dissimilar in spatial frequency, the effects of adapting to them were substantially reduced. This has important implications for mid-level visual processing, both for the combinatorial explosion and for the selective "binding" of common features that are perceived as coming from a single visual object.

The spatial characteristics of plaid-form-selective mechanisms.

Rather little is known about the mechanisms that combine the outputs of orientation- and spatial frequency-selective channels. These can be studied by measuring the selective adaptation to compound stimuli over and above that expected from adaptation to the components alone (Peirce & Taylor, 2006). Here we investigated the contrast- and spatial phase-dependency of such mechanisms. A plaid was adapted in one visual hemi-field, while its constituent gratings were simultaneously adapted in the other hemi-field. Plaid-selective adaptation was most apparent with high-contrast probes, whereas adaptation to the component grating stimuli dominated at low contrasts. The mechanisms underlying this plaid-selective adaptation also appear to be insensitive to the spatial phase of the probes relative to the adaptor, whereas we find a clear phase-dependency for suprathreshold contrast adaptation to grating stimuli. These findings suggest that the visual system is equipped with mechanisms that conduct a global analysis of the plaid pattern, which are likely derived from the non-linear outputs of V1 complex cells.

A comparison of fMRI adaptation and multivariate pattern classification analysis in visual cortex.

Functional magnetic resonance imaging (fMRI) has become a ubiquitous tool in cognitive neuroscience. The technique allows noninvasive measurements of cortical responses in the human brain, but only on the millimeter scale. Because a typical voxel contains many thousands of neurons with varied properties, establishing the selectivity of their responses directly is impossible. In recent years, two methods using fMRI aimed at studying the selectivity of neuronal populations on a 'subvoxel' scale have been heavily used. The first technique, fMRI adaptation, relies on the observation that the blood oxygen level-dependent (BOLD) response in a given voxel is reduced after prolonged presentation of a stimulus, and that this reduction is selective to the characteristics of the repeated stimuli (adapters). The second technique, multivariate pattern analysis (MVPA), makes use of multivariate statistics to recover small biases in individual voxels in their responses to different stimuli. It is thought that these biases arise due to the uneven distribution of neurons (with different properties) sampled by the many voxels in the imaged volume. These two techniques have not been compared explicitly, however, and little is known about their relative sensitivities. Here, we compared fMRI results from orientation-specific visual adaptation and orientation-classification by MVPA, using optimized experimental designs for each, and found that the multivariate pattern classification approach was more sensitive to small differences in stimulus orientation than the adaptation paradigm. Estimates of orientation selectivity obtained with the two methods were, however, very highly correlated across visual areas.

Cortical representation of color is binocular.

It is widely believed that the cortical mechanisms of color vision are monocular because stereopsis is poor for isoluminant patterns. By measuring and comparing the chromatic tuning of binocular and monocular neurons in cortical areas V1 and V2 of macaque, we show that this is not the case. Not only are many color-preferring cells in early visual cortex well-driven binocularly, but their color preferences are unusually well-matched in the two eyes. The receptive fields of these neurons are well equipped to convey information about binocular surface color, but because they are insensitive to local spatial contrast they are ill-suited to convey information about stereoscopic depth. Our observations suggest that in early cortical processing, binocular depth and binocular surface color are represented by two different groups of neurons: one that encodes binocular spatial detail at the expense of binocular chromatic detail and another that does the reverse.

Selective mechanisms for simple contours revealed by compound adaptation.

Neurons in the early stages of visual processing are often thought of as edge detectors for different orientations. Here we investigate the existence of detectors for specific combinations of edges-detectors for specific curvatures. Previous attempts to demonstrate such detectors through aftereffects have ultimately been explained by adaptation to local orientation rather than curvature per se. To control for local aftereffects, we adapted one patch of visual field to two adjacent gratings presented as an obtuse contour (compound patch), and another patch to the same component gratings presented alternately (component patch). In this way both patches are adapted equally to the local orientation components of the stimuli, but only the compound patch is adapted to the global contour. Thus any difference in adaptation between the patches must result from the presence of the contour as a global figure. We found that perceived contrast of probe stimuli was not differentially altered in the two patches. However, apparent curvature of the probes was consistently greater in the compound patch than in the component patch. This effect was considerably reduced by increasing the spatial separation of the component gratings. The results are consistent with curvature detectors involved in the perceptual grouping of edges.

Generating Stimuli for Neuroscience Using PsychoPy.

PsychoPy is a software library written in Python, using OpenGL to generate very precise visual stimuli on standard personal computers. It is designed to allow the construction of as wide a variety of neuroscience experiments as possible, with the least effort. By writing scripts in standard Python syntax users can generate an enormous variety of visual and auditory stimuli and can interact with a wide range of external hardware (enabling its use in fMRI, EEG, MEG etc.). The structure of scripts is simple and intuitive. As a result, new experiments can be written very quickly, and trying to understand a previously written script is easy, even with minimal code comments. PsychoPy can also generate movies and image sequences to be used in demos or simulated neuroscience experiments. This paper describes the range of tools and stimuli that it provides and the environment in which experiments are conducted.

The potential importance of saturating and supersaturating contrast response functions in visual cortex.

Most cortical visual neurons do not respond linearly with contrast. Generally, they show saturated responses to stimuli of high contrast, a feature often characterized by a divisive normalization function. This nonlinearity is generally thought to be useful in focusing the dynamic response range of the neuron on a particular region of contrast space, optimizing contrast gain. Some neurons not only saturate but also supersaturate; at high contrast, the response of the neuron decreases rather than plateaus. Under the contrast gain control theory, these cells would seem to reflect a nonoptimal normalization pool that provides excessive inhibition to the neurons. Since very few data on supersaturation are available, this article examines the frequency with which such neurons occur in macaque visual cortex by considering an extension of the Naka-Rushton equation with the capacity to represent nonmonotonic functions. The prevalence of gain-control theories for saturation has occluded an additional computational function for saturation, namely, in detecting the conjunction of certain features. A saturating nonlinearity is a critical part of the selective detection of compound stimuli over their components. In this role, the existence of saturating contrast response functions might be considered necessary rather than simply optimal.

Two expressions of "surround suppression" in V1 that arise independent of cortical mechanisms of suppression.

The preferred stimulus size of a V1 neuron decreases with increases in stimulus contrast. It has been supposed that stimulus contrast is the primary determinant of such spatial summation in V1 cells, though the extent to which it depends on other stimulus attributes such as orientation and spatial frequency remains untested. We investigated this by recording from single cells in V1 of anaesthetized cats and monkeys, measuring size-tuning curves for high-contrast drifting gratings of optimal spatial configuration, and comparing these curves with those obtained at lower contrast or at sub-optimal orientations or spatial frequencies. For drifting gratings of optimal spatial configuration, lower contrasts produced less surround suppression resulting in increases in preferred size. High contrast gratings of sub-optimal spatial configuration produced more surround suppression than optimal low-contrast gratings, and as much or more surround suppression than optimal high-contrast gratings. For sub-optimal spatial frequencies, preferred size was similar to that for the optimal high-contrast stimulus, whereas for sub-optimal orientations, preferred size was smaller than that for the optimal high-contrast stimulus. These results indicate that, while contrast is an important determinant of spatial summation in V1, it is not the only determinant. Simulation of these experiments on a cortical receptive field modeled as a Gabor revealed that the small preferred sizes observed for non-preferred stimuli could result simply from linear filtering by the classical receptive field. Further simulations show that surround suppression in retinal ganglion cells and LGN cells can be propagated to neurons in V1, though certain properties of the surround seen in cortex indicate that it is not solely inherited from earlier stages of processing.

PsychoPy--Psychophysics software in Python.

The vast majority of studies into visual processing are conducted using computer display technology. The current paper describes a new free suite of software tools designed to make this task easier, using the latest advances in hardware and software. PsychoPy is a platform-independent experimental control system written in the Python interpreted language using entirely free libraries. PsychoPy scripts are designed to be extremely easy to read and write, while retaining complete power for the user to customize the stimuli and environment. Tools are provided within the package to allow everything from stimulus presentation and response collection (from a wide range of devices) to simple data analysis such as psychometric function fitting. Most importantly, PsychoPy is highly extensible and the whole system can evolve via user contributions. If a user wants to add support for a particular stimulus, analysis or hardware device they can look at the code for existing examples, modify them and submit the modifications back into the package so that the whole community benefits.