Opposing serial dependencies revealed for sequences of auditory emotional stimuli.

Our percept of the world is not solely determined by what we perceive and process at a given moment in time, but also depends on what we processed recently. In the present study, we investigate whether the perceived emotion of a spoken sentence is contingent upon the emotion of an auditory stimulus on the preceding trial (i.e., serial dependence). Thereto, participants were exposed to spoken sentences that varied in emotional affect by changing the prosody that ranged from 'happy' to 'fearful'. Participants were instructed to rate the emotion. We found a positive serial dependence for emotion processing whereby the perceived emotion was biased towards the emotion on the preceding trial. When we introduced 'no-go' trials (i.e., no rating was required), we found a negative serial dependence when participants knew in advance to withhold their response on a given trial (Experiment 2) and a positive serial dependence when participants received the information to withhold their response after the stimulus presentation (Experiment 3). We therefore established a robust serial dependence for emotion processing in speech and introduce a methodology to disentangle perceptual from post-perceptual processes. This approach can be applied to the vast majority of studies investigating sequential dependencies to separate positive from negative serial dependence.

CFS-crafter: An open-source tool for creating and analyzing images for continuous flash suppression experiments.

Continuous flash suppression (CFS) is a popular masking technique used to manipulate visual awareness. By presenting a rapidly changing stimulus to one eye (the 'mask'), a static image viewed by the other (the 'target') may remain invisible for many seconds. This effectiveness affords a means to assess unconscious visual processing, leading to the widespread use of CFS in several basic and clinical sciences. However, the lack of principled stimulus selection has impeded generalization of conclusions across studies, as the strength of interocular suppression is dependent on the spatiotemporal properties of the CFS mask and target. To address this, we created CFS-crafter, a point-and-click, open-source tool for creating carefully controlled CFS stimuli. The CFS-crafter provides a streamlined workflow to create, modify, and analyze mask and target stimuli, requiring only a rudimentary understanding of image processing that is well supported by help files in the application. Users can create CFS masks ranging from classic Mondrian patterns to those comprising objects or faces, or they can create, upload, and analyze their own images. Mask and target images can be custom-designed using image-processing operations performed in the frequency domain, including phase-scrambling and spatial/temporal/orientation filtering. By providing the means for the customization and analysis of CFS stimuli, the CFS-crafter offers controlled creation, analysis, and cross-study comparison. Thus, the CFS-crafter-with its easy-to-use image processing functionality-should facilitate the creation of visual conditions that allow a principled assessment of hypotheses about visual processing outside of awareness.

Propagation and update of auditory perceptual priors through alpha and theta rhythms.

To maintain a continuous and coherent percept over time, the brain makes use of past sensory information to anticipate forthcoming stimuli. We recently showed that auditory experience of the immediate past is propagated through ear-specific reverberations, manifested as rhythmic fluctuations of decision bias at alpha frequencies. Here, we apply the same time-resolved behavioural method to investigate how perceptual performance changes over time under conditions of stimulus expectation and to examine the effect of unexpected events on behaviour. As in our previous study, participants were required to discriminate the ear-of-origin of a brief monaural pure tone embedded in uncorrelated dichotic white noise. We manipulated stimulus expectation by increasing the target probability in one ear to 80%. Consistent with our earlier findings, performance did not remain constant across trials, but varied rhythmically with delay from noise onset. Specifically, decision bias showed a similar oscillation at ~9 Hz, which depended on ear congruency between successive targets. This suggests rhythmic communication of auditory perceptual history occurs early and is not readily influenced by top-down expectations. In addition, we report a novel observation specific to infrequent, unexpected stimuli that gave rise to oscillations in accuracy at ~7.6 Hz one trial after the target occurred in the non-anticipated ear. This new behavioural oscillation may reflect a mechanism for updating the sensory representation once a prediction error has been detected.

Separate effects of auditory and visual room size on auditorium seat preference: a virtual reality study.

Audiovisual integrations and interactions happen everywhere, including in music concerts, where combined visual and auditory perception contributes to overall enjoyment. Thirty-three participants evaluated their overall subjective preference at various seats in four virtual auditoria, which comprised congruent and incongruent auditory and visual renders of two auditoria that differ only in size. Results show no significant difference between participants who completed the experiment in a fully calibrated and standardized laboratory environment and participants who completed remotely using various VR equipment in various environments. Both visual and auditory auditorium size have significant main effects, but no interaction. The larger hall is preferred for both conditions. Audiovisual congruency does not significantly affect preference.

A shared mechanism for facial expression in human faces and face pareidolia.

Facial expressions are vital for social communication, yet the underlying mechanisms are still being discovered. Illusory faces perceived in objects (face pareidolia) are errors of face detection that share some neural mechanisms with human face processing. However, it is unknown whether expression in illusory faces engages the same mechanisms as human faces. Here, using a serial dependence paradigm, we investigated whether illusory and human faces share a common expression mechanism. First, we found that images of face pareidolia are reliably rated for expression, within and between observers, despite varying greatly in visual features. Second, they exhibit positive serial dependence for perceived facial expression, meaning an illusory face (happy or angry) is perceived as more similar in expression to the preceding one, just as seen for human faces. This suggests illusory and human faces engage similar mechanisms of temporal continuity. Third, we found robust cross-domain serial dependence of perceived expression between illusory and human faces when they were interleaved, with serial effects larger when illusory faces preceded human faces than the reverse. Together, the results support a shared mechanism for facial expression between human faces and illusory faces and suggest that expression processing is not tightly bound to human facial features.

Cost of Dual-Task Performance in Tactile Perception Is Greater for Competing Tasks of the Same Type.

This study examines dual-task performance in the tactile modality and tests whether dual-task cost depends on task type. Experiment 1 involved competing tasks of the same type, using a primary localisation task on the left hand and a secondary localisation task on the right hand. In Experiment 2, the primary task on the left hand remained the same, while an intensity discrimination task was used as the secondary task on the right hand. Subjects in both experiments completed three conditions: the primary task alone, a dual-task condition, and the primary task with the secondary stimulus present but no response required. Across both experiments, performance on the primary task was best when it was presented alone, and there was a performance decrement when the secondary stimulus was present but not responded to. Performance on the primary task was further decreased when participants had to respond to the secondary stimulus, and the decrease was larger when the secondary task was localisation rather than discrimination. This result indicates that task type in the tactile modality may modulate the attentional cost of dual-task performance and implies partially shared resources underlie localisation and intensity discrimination.

Individual difference in serial dependence results from opposite influences of perceptual choices and motor responses.

Natural image statistics exhibit temporal regularities of slow changes and short-term correlations and visual perception, too, is biased toward recently seen stimuli, i.e., a positive serial dependence. Some studies report strong individual differences in serial dependence in perceptual decision-making: some observers show positive serial effects, others repulsive effects, and some show no bias. To understand these contrasting results, this study separates the influences of physical stimuli per se, perceptual choices, and motor responses on serial dependence in perceptual decision making. In two experiments, human observers reported which orientation (45° or -45°, at threshold contrast) they perceived. Experiment 1, used a consistent mapping between stimulus and response buttons whereas in Experiment 2, observers did two tasks: one with a consistent stimulus-response mapping, the other with a random stimulus-response mapping (perceptual choice and motor response unrelated). Results show that the stimulus percept (not the physical stimulus per se) affected subsequent perceptual choices in an attractive way and that motor responses produced a repulsive serial effect. When the choice-response mapping was consistent (inseparable choice and response, typical of most experiments), individual differences in the overall serial effect was observed: some were positive, some repulsive, and some were bias-free. The multiple regression analysis revealed that observers' overall serial effects in the consistent choice-response mapping task could be predicted by their serial effects for choices and motor responses in the random mapping task. These individual differences likely reflect relative weightings of a positive choice bias and a repulsive motor bias.

Serial dependence and center bias in heading perception from optic flow.

Previous work shows that observers can use information from optic flow to perceive the direction of self-motion (i.e. heading) and that perceived heading exhibits a bias towards the center of the display (center bias). More recent work shows that the brain is sensitive to serial correlations and the perception of current stimuli can be affected by recently seen stimuli, a phenomenon known as serial dependence. In the current study, we examined whether, apart from center bias, serial dependence could be independently observed in heading judgments and how adding noise to optic flow affected center bias and serial dependence. We found a repulsive serial dependence effect in heading judgments after factoring out center bias in heading responses. The serial effect expands heading estimates away from the previously seen heading to increase overall sensitivity to changes in heading directions. Both the center bias and repulsive serial dependence effects increased with increasing noise in optic flow, and the noise-dependent changes in the serial effect were consistent with an ideal observer model. Our results suggest that the center bias effect is due to a prior of the straight-ahead direction in the Bayesian inference account for heading perception, whereas the repulsive serial dependence is an effect that reduces response errors and has the added utility of counteracting the center bias in heading judgments.

Attraction to the recent past in aesthetic judgments: A positive serial dependence for rating artwork.

Visual perception can be systematically biased towards the recent past. Many stimulus attributes-including orientation, numerosity, facial expression and attractiveness, and perceived slimness-are systematically biased towards recent past experience. This phenomenon has been termed serial dependence. In the current study, we tested whether serial dependence occurs for aesthetic ratings of artworks. A set of 100 paintings depicting scenery and still life was collected from online archives. For each participant, 40 paintings were randomly selected from the set, and presented sequentially 20 times in random order. Serial dependence was quantified for each observer by measuring how their rating response on each trial depended on the attractiveness of the previous trial. The data were pooled across participants and fitted with a Bayesian model of serial dependence. Results showed that the current painting earned significantly higher aesthetic ratings when participants viewed a more attractive painting on the previous trial, compared to when they viewed a less attractive one. The magnitude of serial dependence was greatest when the attractiveness distance between consecutive paintings was relatively close. The effect held both for 1 s exposure times, and for brief 250 ms exposures (followed by a mask). These findings show that aesthetic judgments are not sequentially independent, showing that positive serial dependencies are not limited to low-level perceptual judgments.

Positive sequential dependency for face attractiveness perception.

Recent findings from several groups have demonstrated that visual perception at a given moment can be biased toward what was recently seen. This is true both for basic visual attributes and for more complex representations, such as face identity, gender, or expression. This assimilation to the recent past is a positive serial dependency, similar to a temporal averaging process that capitalizes on short-term correlations in visual input to reduce noise and boost perceptual continuity. Here we examine serial dependencies in face perception using a simple attractiveness rating task and a rapid series of briefly presented face stimuli. In a series of three experiments, our results confirm a previous report that face attractiveness exhibits a positive serial dependency. This intertrial effect is not only determined by face attractiveness on the previous trial, but also depends on the faces shown up to five trials back. We examine the effect of stimulus presentation duration and find that stimuli as brief as 56 ms produce a significant positive dependency similar in magnitude to that produced by stimuli presented for 1,000 ms. We observed stronger positive dependencies between same-gender faces, and found a task dependency: Alternating gender discrimination trials with attractiveness rating trials produced no serial dependency. In sum, these findings show that a perception-stabilizing assimilation effect operates in face attractiveness perception that is task dependent and is acquired surprisingly quickly.

Two paradigms of bistable plaid motion reveal independent mutual inhibition processes.

Perception is sometimes bistable, switching between two possible interpretations. Levelt developed several propositions to explain bistable perception in binocular rivalry, based on a model of competing neural populations connected through reciprocal inhibition. Here we test Levelt's laws with bistable plaid motion. Plaids are typically tristable, either a coherent pattern, transparent with one component in front, or transparent with the opposite depth order. In Experiment 1, we use a large angle between component directions to prevent plaid coherence, limiting the ambiguity to alternations of grating depth order. Similar to increasing contrast in binocular rivalry, increasing component speed led to higher switch rates (analogous to Levelt's fourth proposition). In Experiment 2, we used occlusion cues to prevent one depth order and limit bistability to one transparent depth order alternating with coherence. Increasing grating speed shortened coherent motion periods but left transparent periods largely unchanged (analogous to Levelt's second proposition). Switch dynamics showed no correlation between the experiments. These data suggest that plaid component speed acts like contrast in binocular rivalry to vary switch dynamics through a mutual inhibition model. The lack of correlation between both experiments suggests reciprocal inhibition mediates bistability between a variety of neural populations across the visual system.

Time dilation effect in an active observer and virtual environment requires apparent motion: No dilation for retinal- or world-motion alone.

It is known that moving visual stimuli are perceived to last longer than stationary stimuli with the same physical duration (Kanai, Paffen, Hogendoorn, & Verstraten, 2006), and that motor actions (Tomassini & Morrone, 2016) and eye movements (Morrone, Ross, & Burr, 2005) can alter perceived duration. In the present work, we investigated the contributions of stimulus motion and self-motion to perceived duration while observers stood or walked in a virtual reality environment. Using a visual temporal reproduction task, we independently manipulated both the participants' motion (stationary or walking) and the stimulus motion (retinal stationary, real-world stationary and negative double velocity). When the observers were standing still, drifting gratings were perceived as lasting longer than duration-matched static gratings. Interestingly, we did not see any time distortion when observers were walking, neither when the gratings were kept stationary relative to the observer's point of view (i.e., no retinal motion) nor when they were stationary in the external world (i.e., producing the same retinal velocity as the walking condition with stationary grating). Self-motion caused significant dilation in perceived duration only when the gratings were moving at double speed, opposite to the observers' walking direction. Consistent with previous work (Fornaciai, Arrighi, & Burr, 2016), this suggests that the system is able to suppress self-generated motion to enhance external motion, which would have ecological benefits, for example, for threat detection while navigating through the environment.

Linear Summation of Repulsive and Attractive Serial Dependencies: Orientation and Motion Dependencies Sum in Motion Perception.

Recent work from several groups has shown that perception of various visual attributes in human observers at a given moment is biased toward what was recently seen. This positive serial dependency is a kind of temporal averaging that exploits short-term correlations in visual scenes to reduce noise and stabilize perception. To date, this stabilizing "continuity field" has been demonstrated on stable visual attributes such as orientation and face identity, yet it would be counterproductive to apply it to dynamic attributes in which change sensitivity is needed. Here, we tested this using motion direction discrimination and predict a negative perceptual dependency: a contrastive relationship that enhances sensitivity to change. Surprisingly, our data showed a cubic-like pattern of dependencies with positive and negative components. By interleaving various stimulus combinations, we separated the components and isolated a positive perceptual dependency for motion and a negative dependency for orientation. A weighted linear sum of the separate dependencies described the original cubic pattern well. The positive dependency for motion shows an integrative perceptual effect and was unexpected, although it is consistent with work on motion priming. These findings suggest that a perception-stabilizing continuity field occurs pervasively, occurring even when it obscures sensitivity to dynamic stimuli.SIGNIFICANCE STATEMENT Recent studies show that visual perception at a given moment is not entirely veridical, but rather biased toward recently seen stimuli: a positive serial dependency. This temporal smoothing process helps perceptual continuity by preserving stable aspects of the visual scene over time, yet, for dynamic stimuli, temporal smoothing would blur dynamics and reduce sensitivity to change. We tested whether this process is selective for stable attributes by examining dependencies in motion perception. We found a clear positive dependency for motion, suggesting that positive perceptual dependencies are pervasive. We also found a concurrent negative (contrastive) dependency for orientation. Both dependencies combined linearly to determine perception, showing that the brain can calculate contrastive and integrative dependencies simultaneously from recent stimulus history when making perceptual decisions.

Slow and steady, not fast and furious: Slow temporal modulation strengthens continuous flash suppression.

Continuous flash suppression (CFS) involves the presentation of a rapidly changing Mondrian sequence to one eye and a static target in the other eye. Targets presented in this manner remain suppressed for several seconds at a time, and this has seen the prevalent use of CFS in studies of unconscious visual processes. However, the mechanisms behind CFS remain unclear, complicating its use and the comprehension of results obtained with the paradigm. For example, some studies report observations indicative of faster, visual masking processes whereas others suggest slower, rivalry processes. To reconcile this discrepancy, this study investigates the effect of temporal frequency content and Mondrian pattern structure on CFS suppression. Our results show predominant influences of spatial edges and low temporal-frequency content, which are similar to binocular rivalry, affording a parsimonious alternative in unifying the two paradigms.

Strength of continuous flash suppression is optimal when target and masker modulation rates are matched.

Continuous flash suppression (CFS) is a popular technique whereby a dynamic sequence of Mondrian patterns is presented to one eye in order to suppress a static target presented to the other eye. Although the effectiveness of CFS is generally assumed to increase with the flicker rate of the Mondrian masker, a recent study has shown that suppression is optimal at very low masker rates for sustained targets, but higher rates may be necessary for transient targets. Here we vary the modulation rates of the masker and target using temporally filtered dynamic noise, which allowed us to examine the relationship between target and masker frequency and its effect on suppression strength. Using these carefully controlled, temporally narrowband stimuli, we demonstrate a pattern of results showing that suppression is greatest when target and masker modulate at similar frequencies. This finding indicates the involvement of early temporal-frequency-tuned filters underlying CFS and is consistent with many existing findings in the CFS literature. We also find that these temporally selective processes are orientation selective, which points to an early cortical substrate such as neurons in primary visual cortex. Overall, our study reveals that CFS suppression can be maximized by carefully matching the masker and target in temporal frequency and orientation. More generally, we show the importance of using carefully controlled stimuli for elucidating the underlying mechanisms of CFS. This approach is important at a theoretical level, as it will enable comparison of CFS with existing models of binocular rivalry and interocular suppression and facilitate a unified explanatory framework.

Eye gaze direction shows a positive serial dependency.

Recent work from several groups has shown that perception of various visual attributes in human observers at a given moment is biased towards what was recently seen. This positive serial dependency is a kind of temporal averaging which exploits short-term correlations in visual scenes to reduce noise and stabilize perception. Here we test for serial dependencies in perception of head and eye direction using a simple reproduction method to measure perceived head/eye gaze direction in rapid sequences of briefly presented face stimuli. In a series of three experiments, our results reveal that perceived eye gaze direction shows a positive serial dependency for changes in eye direction, along both the vertical and horizontal dimensions, although more strongly for horizontal gaze shifts. By contrast, we found no serial dependency at all for horizontal changes in head position. These findings show that a perception-stabilizing 'continuity field' operates on eye position-well known to be quite variable over short timescales-while the more inherently stable signal from head position does not.

Orientation discrimination requires coactivation of on- and off-dominated visual channels.

Orientation sensitivity depends on the cortical convergence of on- and off-center subcortical neurons. Off-center inputs are faster and stronger than their on-center counterparts: How does this asymmetry affect orientation discrimination? We tackled this question psychophysically with grating stimuli that either increased or decreased luminance. The gratings were of low contrast in order to avoid the complicating influences of nonlinearities such as response saturation, masking, and aftereffects. Gratings were presented in either of two locations, and subjects indicated the perceived location. Stimuli were randomly timed, and response correctness and reaction time were recorded. We found the following: (a) Contrast sensitivity was insignificant for a range of contrasts around zero. (b) Outside this range, contrast sensitivity for contrast decrements exceeded that for increments by an average of 15%. (c) Reaction times for contrast decrements were up to 45 ms less than for increments. (d) These findings are reproduced by a signal-detection model which incorporates recent physiological findings: Neurons in primary visual cortex are hyperpolarized at rest; these neurons respond more to darks than to lights; and off-dominated cortical neurons have shorter latencies than their on-dominated neighbors. (e) We tested orientation discrimination by splitting a grating into two components, one containing the light bars and the other the dark, and presenting the two components asynchronously. Discrimination was optimal when light bars preceded dark bars, consistent with coactivation of on- and off-center cortical inputs. We conclude that the ability to discriminate between orientations is intimately connected with the properties of subcortical channels.

Auditory and tactile signals combine to influence vision during binocular rivalry.

Resolution of perceptual ambiguity is one function of cross-modal interactions. Here we investigate whether auditory and tactile stimuli can influence binocular rivalry generated by interocular temporal conflict in human subjects. Using dichoptic visual stimuli modulating at different temporal frequencies, we added modulating sounds or vibrations congruent with one or the other visual temporal frequency. Auditory and tactile stimulation both interacted with binocular rivalry by promoting dominance of the congruent visual stimulus. This effect depended on the cross-modal modulation strength and was absent when modulation depth declined to 33%. However, when auditory and tactile stimuli that were too weak on their own to bias binocular rivalry were combined, their influence over vision was very strong, suggesting the auditory and tactile temporal signals combined to influence vision. Similarly, interleaving discrete pulses of auditory and tactile stimuli also promoted dominance of the visual stimulus congruent with the supramodal frequency. When auditory and tactile stimuli were presented at maximum strength, but in antiphase, they had no influence over vision for low temporal frequencies, a null effect again suggesting audio-tactile combination. We also found that the cross-modal interaction was frequency-sensitive at low temporal frequencies, when information about temporal phase alignment can be perceptually tracked. These results show that auditory and tactile temporal processing is functionally linked, suggesting a common neural substrate for the two sensory modalities and that at low temporal frequencies visual activity can be synchronized by a congruent cross-modal signal in a frequency-selective way, suggesting the existence of a supramodal temporal binding mechanism.

Rapid temporal recalibration is unique to audiovisual stimuli.

Following prolonged exposure to asynchronous multisensory signals, the brain adapts to reduce the perceived asynchrony. Here, in three separate experiments, participants performed a synchrony judgment task on audiovisual, audiotactile or visuotactile stimuli and we used inter-trial analyses to examine whether temporal recalibration occurs rapidly on the basis of a single asynchronous trial. Even though all combinations used the same subjects, task and design, temporal recalibration occurred for audiovisual stimuli (i.e., the point of subjective simultaneity depended on the preceding trial's modality order), but none occurred when the same auditory or visual event was combined with a tactile event. Contrary to findings from prolonged adaptation studies showing recalibration for all three combinations, we show that rapid, inter-trial recalibration is unique to audiovisual stimuli. We conclude that recalibration occurs at two different timescales for audiovisual stimuli (fast and slow), but only on a slow timescale for audiotactile and visuotactile stimuli.

Evolving the stimulus to fit the brain: a genetic algorithm reveals the brain's feature priorities in visual search.

How does the brain find objects in cluttered visual environments? For decades researchers have employed the classic visual search paradigm to answer this question using factorial designs. Although such approaches have yielded important information, they represent only a tiny fraction of the possible parametric space. Here we use a novel approach, by using a genetic algorithm (GA) to discover the way the brain solves visual search in complex environments, free from experimenter bias. Participants searched a series of complex displays, and those supporting fastest search were selected to reproduce (survival of the fittest). Their display properties (genes) were crossed and combined to create a new generation of "evolved" displays. Displays evolved quickly over generations towards a stable, efficiently searched array. Color properties evolved first, followed by orientation. The evolved displays also contained spatial patterns suggesting a coarse-to-fine search strategy. We argue that this behavioral performance-driven GA reveals the way the brain selects information during visual search in complex environments. We anticipate that our approach can be adapted to a variety of sensory and cognitive questions that have proven too intractable for factorial designs.