The timing mega-study: comparing a range of experiment generators, both lab-based and online.

Many researchers in the behavioral sciences depend on research software that presents stimuli, and records response times, with sub-millisecond precision. There are a large number of software packages with which to conduct these behavioral experiments and measure response times and performance of participants. Very little information is available, however, on what timing performance they achieve in practice. Here we report a wide-ranging study looking at the precision and accuracy of visual and auditory stimulus timing and response times, measured with a Black Box Toolkit. We compared a range of popular packages: PsychoPy, E-Prime®, NBS Presentation®, Psychophysics Toolbox, OpenSesame, Expyriment, Gorilla, jsPsych, Lab.js and Testable. Where possible, the packages were tested on Windows, macOS, and Ubuntu, and in a range of browsers for the online studies, to try to identify common patterns in performance. Among the lab-based experiments, Psychtoolbox, PsychoPy, Presentation and E-Prime provided the best timing, all with mean precision under 1 millisecond across the visual, audio and response measures. OpenSesame had slightly less precision across the board, but most notably in audio stimuli and Expyriment had rather poor precision. Across operating systems, the pattern was that precision was generally very slightly better under Ubuntu than Windows, and that macOS was the worst, at least for visual stimuli, for all packages. Online studies did not deliver the same level of precision as lab-based systems, with slightly more variability in all measurements. That said, PsychoPy and Gorilla, broadly the best performers, were achieving very close to millisecond precision on several browser/operating system combinations. For response times (measured using a high-performance button box), most of the packages achieved precision at least under 10 ms in all browsers, with PsychoPy achieving a precision under 3.5 ms in all. There was considerable variability between OS/browser combinations, especially in audio-visual synchrony which is the least precise aspect of the browser-based experiments. Nonetheless, the data indicate that online methods can be suitable for a wide range of studies, with due thought about the sources of variability that result. The results, from over 110,000 trials, highlight the wide range of timing qualities that can occur even in these dedicated software packages for the task. We stress the importance of scientists making their own timing validation measurements for their own stimuli and computer configuration.

PsychoPy2: Experiments in behavior made easy.

PsychoPy is an application for the creation of experiments in behavioral science (psychology, neuroscience, linguistics, etc.) with precise spatial control and timing of stimuli. It now provides a choice of interface; users can write scripts in Python if they choose, while those who prefer to construct experiments graphically can use the new Builder interface. Here we describe the features that have been added over the last 10 years of its development. The most notable addition has been that Builder interface, allowing users to create studies with minimal or no programming, while also allowing the insertion of Python code for maximal flexibility. We also present some of the other new features, including further stimulus options, asynchronous time-stamped hardware polling, and better support for open science and reproducibility. Tens of thousands of users now launch PsychoPy every month, and more than 90 people have contributed to the code. We discuss the current state of the project, as well as plans for the future.

Target meta-awareness is a necessary condition for physiological responses to masked emotional faces: Evidence from combined skin conductance and heart rate assessment.

Much heated debate surrounds the extent to which we can process emotional stimuli without awareness. In particular the extent to which masked emotional faces can elicit changes in physiology measurements, such as heart rate and skin conductance responses, has produced controversial findings. In the present study, we aimed to determine whether briefly presented faces can elicit physiological changes and, specifically, whether this is due to unconscious processing. We measured and adjusted for individual differences in the detection threshold using both receiver operating characteristics and hit rates. For this we also used a strict Bayesian assessment of participant thresholds. We then measured physiological responses to threshold adjusted emotional faces and for hits, misses and post-binary subdivisions of target meta-awareness. Our findings based on receiver operating characteristics revealed that, when faces were successfully masked there were no significant physiological differences in response to stimuli with different emotional connotations. In contrast, when targets were masked based on hit rates we did find physiological responses to masked emotional faces. With further analysis we found that this effect was specific to correct detection of angry and fearful faces and that increases in experienced arousal were associated with higher confidence ratings for correct detection of these stimuli. Collectively, our results do not support the notion of unconscious processing when using markers of physiological processes. Rather they suggest that target meta-awareness is a necessary condition for - and possibly determined by - physiological changes in response to masked emotional faces.

Measuring nonlinear signal combination using EEG.

Relatively little is known about the processes, both linear and nonlinear, by which signals are combined beyond V1. By presenting two stimulus components simultaneously, flickering at different temporal frequencies (frequency tagging) while measuring steady-state visual evoked potentials, we can assess responses to the individual components, including direct measurements of suppression on each other, and various nonlinear responses to their combination found at intermodulation frequencies. The result is a rather rich dataset of frequencies at which responses can be found. We presented pairs of sinusoidal gratings at different temporal frequencies, forming plaid patterns that were "coherent" (looking like a checkerboard) and "noncoherent" (looking like a pair of transparently overlaid gratings), and found clear intermodulation responses to compound stimuli, indicating nonlinear summation. This might have been attributed to cross-orientation suppression except that the pattern of intermodulation responses differed for coherent and noncoherent patterns, whereas the effects of suppression (measured at the component frequencies) did not. A two-stage model of nonlinear summation involving conjunction detection with a logical AND gate described the data well, capturing the difference between coherent and noncoherent plaids over a wide array of possible response frequencies. Multistimulus frequency-tagged EEG in combination with computational modeling may be a very valuable tool in studying the conjunction of these signals. In the current study the results suggest a second-order mechanism responding selectively to coherent plaid patterns.

Understanding mid-level representations in visual processing.

It is clear that early visual processing provides an image-based representation of the visual scene: Neurons in Striate cortex (V1) encode nothing about the meaning of a scene, but they do provide a great deal of information about the image features within it. The mechanisms of these "low-level" visual processes are relatively well understood. We can construct plausible models for how neurons, up to and including those in V1, build their representations from preceding inputs down to the level of photoreceptors. It is also clear that at some point we have a semantic, "high-level" representation of the visual scene because we can describe verbally the objects that we are viewing and their meaning to us. A huge number of studies are examining these "high-level" visual processes each year. Less well studied are the processes of "mid-level" vision, which presumably provide the bridge between these "low-level" representations of edges, colors, and lights and the "high-level" semantic representations of objects, faces, and scenes. This article and the special issue of papers in which it is published consider the nature of "mid-level" visual processing and some of the reasons why we might not have made as much progress in this domain as we would like.

Cue Combination of Conflicting Color and Luminance Edges.

Abrupt changes in the color or luminance of a visual image potentially indicate object boundaries. Here, we consider how these cues to the visual "edge" location are combined when they conflict. We measured the extent to which localization of a compound edge can be predicted from a simple maximum likelihood estimation model using the reliability of chromatic (L-M) and luminance signals alone. Maximum likelihood estimation accurately predicted the pattern of results across a range of contrasts. Predictions consistently overestimated the relative influence of the luminance cue; although L-M is often considered a poor cue for localization, it was used more than expected. This need not indicate that the visual system is suboptimal but that its priors about which cue is more useful are not flat. This may be because, although strong changes in chromaticity typically represent object boundaries, changes in luminance can be caused by either a boundary or a shadow.

Is it just motion that silences awareness of other visual changes?

When an array of visual elements is changing color, size, or shape incoherently, the changes are typically quite visible even when the overall color, size, or shape statistics of the field may not have changed. When the dots also move, however, the changes become much less apparent; awareness of them is "silenced" (Suchow & Alvarez, 2011). This finding might indicate that the perception of motion is of particular importance to the visual system, such that it is given priority in processing over other forms of visual change. Here we test whether that is the case by examining the converse: whether awareness of motion signals can be silenced by potent coherent changes in color or size. We find that they can, and with very similar effects, indicating that motion is not critical for silencing. Suchow and Alvarez's dots always moved in the same direction with the same speed, causing them to be grouped as a single entity. We also tested whether this coherence was a necessary component of the silencing effect. It is not; when the dot speeds are randomly selected, such that no coherent motion is present, the silencing effect remains. It is clear that neither motion nor grouping is directly responsible for the silencing effect. Silencing can be generated from any potent visual change.

Luminance cues constrain chromatic blur discrimination in natural scene stimuli.

Introducing blur into the color components of a natural scene has very little effect on its percept, whereas blur introduced into the luminance component is very noticeable. Here we quantify the dominance of luminance information in blur detection and examine a number of potential causes. We show that the interaction between chromatic and luminance information is not explained by reduced acuity or spatial resolution limitations for chromatic cues, the effective contrast of the luminance cue, or chromatic and achromatic statistical regularities in the images. Regardless of the quality of chromatic information, the visual system gives primacy to luminance signals when determining edge location. In natural viewing, luminance information appears to be specialized for detecting object boundaries while chromatic information may be used to determine surface properties.

Transfer of perceptual learning between different visual tasks.

Practice in most sensory tasks substantially improves perceptual performance. A hallmark of this 'perceptual learning' is its specificity for the basic attributes of the trained stimulus and task. Recent studies have challenged the specificity of learned improvements, although transfer between substantially different tasks has yet to be demonstrated. Here, we measure the degree of transfer between three distinct perceptual tasks. Participants trained on an orientation discrimination, a curvature discrimination, or a 'global form' task, all using stimuli comprised of multiple oriented elements. Before and after training they were tested on all three and a contrast discrimination control task. A clear transfer of learning was observed, in a pattern predicted by the relative complexity of the stimuli in the training and test tasks. Our results suggest that sensory improvements derived from perceptual learning can transfer between very different visual tasks.

Using evolutionary algorithms for fitting high-dimensional models to neuronal data.

In the study of neurosciences, and of complex biological systems in general, there is frequently a need to fit mathematical models with large numbers of parameters to highly complex datasets. Here we consider algorithms of two different classes, gradient following (GF) methods and evolutionary algorithms (EA) and examine their performance in fitting a 9-parameter model of a filter-based visual neuron to real data recorded from a sample of 107 neurons in macaque primary visual cortex (V1). Although the GF method converged very rapidly on a solution, it was highly susceptible to the effects of local minima in the error surface and produced relatively poor fits unless the initial estimates of the parameters were already very good. Conversely, although the EA required many more iterations of evaluating the model neuron's response to a series of stimuli, it ultimately found better solutions in nearly all cases and its performance was independent of the starting parameters of the model. Thus, although the fitting process was lengthy in terms of processing time, the relative lack of human intervention in the evolutionary algorithm, and its ability ultimately to generate model fits that could be trusted as being close to optimal, made it far superior in this particular application than the gradient following methods. This is likely to be the case in many further complex systems, as are often found in neuroscience.

The timing of binding and segregation of two compound aftereffects.

Temporal information in a scene is thought to be an important cue for visual grouping of local image features into a single object. The majority of studies on this topic have attempted to determine the conditions that facilitate segregation of a figure from a cluttered background. Here we examine the temporal characteristics of two aftereffects that appear to have roles in visual integration: the curvature aftereffect (CAE; Hancock & Peirce, 2008) and plaid-selective contrast adaptation (Peirce & Taylor, 2006). Both aftereffects used a "compound adaptation" paradigm measuring adaptation to a compound stimulus that cannot be explained by adaptation to its components presented in isolation. The temporal tuning characteristics of the two aftereffects differed in three distinct ways. First, plaid-selective adaptation was very sensitive to temporal phase asynchronies, while the CAE was not. Second, while both aftereffects showed integration of alternating components above 4 Hz, for plaids the overall magnitude of adaptation was less than to synchronous stimuli and was eliminated at the highest frequencies. Finally, plaid-selective adaptation demonstrated a low-pass dependency for temporal flicker frequency of synchronous gratings, whereas the CAE did not. Overall, these results suggest that at least two different mechanisms are involved in the binding/segregation of local signals into compound patterns: one with high temporal resolution that allows rapid parsing of plaid patterns into their components and one with a coarser temporal sensitivity that mediates the CAE.

Global shape processing: which parts form the whole?

Research suggests that detection of low-frequency radial frequency (RF) patterns involves global shape processing and that points of maximum curvature (corners) contribute more than points of minimum curvature (sides). However, this has only been tested with stimuli presented at the threshold of discriminability from a circle. We used RF pattern adaptation to (a) examine whether a supra-threshold RF pattern is processed as a global shape and (b) determine what the critical features are for representing its shape. We measured the perceived amplitude shift of an RF test pattern after prolonged exposure either to a higher amplitude pattern or to various combinations of its parts (concave maxima, convex maxima, inflections). We found greater shifts in perceived amplitude after adaptation to a "whole" pattern than after adaptation to its component parts, which alternated to produce equal net contrast. Furthermore, when adapting to specific parts of the shape in isolation, we found that each part generated a similar magnitude aftereffect. Although the whole is clearly greater than the sum of the parts, we find that concave maxima, convex maxima, and inflections contribute equally to global shape processing, a fact that is only apparent when using a supra-threshold appearance-based task.

Ameliorating the combinatorial explosion with spatial frequency-matched combinations of V1 outputs.

Little is known about the way in which the outputs of early orientation-selective neurons are combined. One particular problem is that the number of possible combinations of these outputs greatly outweighs the number of processing units available to represent them. Here we consider two of the possible ways in which the visual system might reduce the impact of this problem. First, the visual system might ameliorate the problem by collapsing across some low-level feature coded by previous processing stages, such as spatial frequency. Second, the visual system may combine only a subset of available outputs, such as those with similar receptive field characteristics. Using plaid-selective contrast adaptation and the curvature aftereffect, we found no evidence for the former solution; both aftereffects were clearly tuned to the spatial frequency of the adaptor relative to the test probe. We did, however, find evidence for the latter with both aftereffects; when the components forming our compound stimuli were dissimilar in spatial frequency, the effects of adapting to them were substantially reduced. This has important implications for mid-level visual processing, both for the combinatorial explosion and for the selective "binding" of common features that are perceived as coming from a single visual object.

The spatial characteristics of plaid-form-selective mechanisms.

Rather little is known about the mechanisms that combine the outputs of orientation- and spatial frequency-selective channels. These can be studied by measuring the selective adaptation to compound stimuli over and above that expected from adaptation to the components alone (Peirce & Taylor, 2006). Here we investigated the contrast- and spatial phase-dependency of such mechanisms. A plaid was adapted in one visual hemi-field, while its constituent gratings were simultaneously adapted in the other hemi-field. Plaid-selective adaptation was most apparent with high-contrast probes, whereas adaptation to the component grating stimuli dominated at low contrasts. The mechanisms underlying this plaid-selective adaptation also appear to be insensitive to the spatial phase of the probes relative to the adaptor, whereas we find a clear phase-dependency for suprathreshold contrast adaptation to grating stimuli. These findings suggest that the visual system is equipped with mechanisms that conduct a global analysis of the plaid pattern, which are likely derived from the non-linear outputs of V1 complex cells.

A comparison of fMRI adaptation and multivariate pattern classification analysis in visual cortex.

Functional magnetic resonance imaging (fMRI) has become a ubiquitous tool in cognitive neuroscience. The technique allows noninvasive measurements of cortical responses in the human brain, but only on the millimeter scale. Because a typical voxel contains many thousands of neurons with varied properties, establishing the selectivity of their responses directly is impossible. In recent years, two methods using fMRI aimed at studying the selectivity of neuronal populations on a 'subvoxel' scale have been heavily used. The first technique, fMRI adaptation, relies on the observation that the blood oxygen level-dependent (BOLD) response in a given voxel is reduced after prolonged presentation of a stimulus, and that this reduction is selective to the characteristics of the repeated stimuli (adapters). The second technique, multivariate pattern analysis (MVPA), makes use of multivariate statistics to recover small biases in individual voxels in their responses to different stimuli. It is thought that these biases arise due to the uneven distribution of neurons (with different properties) sampled by the many voxels in the imaged volume. These two techniques have not been compared explicitly, however, and little is known about their relative sensitivities. Here, we compared fMRI results from orientation-specific visual adaptation and orientation-classification by MVPA, using optimized experimental designs for each, and found that the multivariate pattern classification approach was more sensitive to small differences in stimulus orientation than the adaptation paradigm. Estimates of orientation selectivity obtained with the two methods were, however, very highly correlated across visual areas.

Cortical representation of color is binocular.

It is widely believed that the cortical mechanisms of color vision are monocular because stereopsis is poor for isoluminant patterns. By measuring and comparing the chromatic tuning of binocular and monocular neurons in cortical areas V1 and V2 of macaque, we show that this is not the case. Not only are many color-preferring cells in early visual cortex well-driven binocularly, but their color preferences are unusually well-matched in the two eyes. The receptive fields of these neurons are well equipped to convey information about binocular surface color, but because they are insensitive to local spatial contrast they are ill-suited to convey information about stereoscopic depth. Our observations suggest that in early cortical processing, binocular depth and binocular surface color are represented by two different groups of neurons: one that encodes binocular spatial detail at the expense of binocular chromatic detail and another that does the reverse.