Age, not autism, influences multisensory integration of speech stimuli among adults in a McGurk/MacDonald paradigm.

Differences between autistic and non-autistic individuals in perception of the temporal relationships between sights and sounds are theorized to underlie difficulties in integrating relevant sensory information. These, in turn, are thought to contribute to problems with speech perception and higher level social behaviour. However, the literature establishing this connection often involves limited sample sizes and focuses almost entirely on children. To determine whether these differences persist into adulthood, we compared 496 autistic and 373 non-autistic adults (aged 17 to 75 years). Participants completed an online version of the McGurk/MacDonald paradigm, a multisensory illusion indicative of the ability to integrate audiovisual speech stimuli. Audiovisual asynchrony was manipulated, and participants responded both to the syllable they perceived (revealing their susceptibility to the illusion) and to whether or not the audio and video were synchronized (allowing insight into temporal processing). In contrast with prior research with smaller, younger samples, we detected no evidence of impaired temporal or multisensory processing in autistic adults. Instead, we found that in both groups, multisensory integration correlated strongly with age. This contradicts prior presumptions that differences in multisensory perception persist and even increase in magnitude over the lifespan of autistic individuals. It also suggests that the compensatory role multisensory integration may play as the individual senses decline with age is intact. These findings challenge existing theories and provide an optimistic perspective on autistic development. They also underline the importance of expanding autism research to better reflect the age range of the autistic population.

Opposing serial dependencies revealed for sequences of auditory emotional stimuli.

Our percept of the world is not solely determined by what we perceive and process at a given moment in time, but also depends on what we processed recently. In the present study, we investigate whether the perceived emotion of a spoken sentence is contingent upon the emotion of an auditory stimulus on the preceding trial (i.e., serial dependence). Thereto, participants were exposed to spoken sentences that varied in emotional affect by changing the prosody that ranged from 'happy' to 'fearful'. Participants were instructed to rate the emotion. We found a positive serial dependence for emotion processing whereby the perceived emotion was biased towards the emotion on the preceding trial. When we introduced 'no-go' trials (i.e., no rating was required), we found a negative serial dependence when participants knew in advance to withhold their response on a given trial (Experiment 2) and a positive serial dependence when participants received the information to withhold their response after the stimulus presentation (Experiment 3). We therefore established a robust serial dependence for emotion processing in speech and introduce a methodology to disentangle perceptual from post-perceptual processes. This approach can be applied to the vast majority of studies investigating sequential dependencies to separate positive from negative serial dependence.

Attentional Tunneling in Pilots During a Visual Tracking Task With a Head Mounted Display.

OBJECTIVE: We examined whether active head aiming with a Helmet Mounted Display (HMD) can draw the pilot's attention away from a primary flight task. Furthermore, we examined whether visual clutter increases this effect. BACKGROUND: Head up display symbology can result in attentional tunneling, and clutter makes it difficult to identify objects. METHOD: Eighteen military pilots had to simultaneously perform an attitude control task while flying in clouds and a head aiming task in a fixed-base flight simulator. The former consisted of manual compensation for roll disturbances of the aircraft, while the latter consisted of keeping a moving visual target inside a small or large head-referenced circle. A "no head aiming" condition served as a baseline. Furthermore, all conditions were performed with or without visual clutter. RESULTS: Head aiming led to deterioration of the attitude control task performance and an increase of the amount of roll-reversal errors (RREs). This was even the case when head aiming required minimal effort. Head aiming accuracy was significantly lower when the roll disturbances in the attitude control task were large compared to when they were small. Visual clutter had no effect on both tasks. CONCLUSION: We suggest that active head aiming of HMD symbology can cause attentional tunneling, as expressed by an increased number of RREs and less accuracy on a simultaneously performed attitude control task. APPLICATION: This study improves our understanding in the perceptual and cognitive effects of (military) HMDs, and has implications for operational use and possibly (re)design of HMDs.

Visual crowding: Double dissociation between orientation and brightness judgments.

Peripherally presented objects are often more difficult to identify when located in cluttered visual environments than when presented in isolation, a phenomenon known as visual crowding. Crowding tends to be stronger when target and nearby flanking elements are composed of similar sets of features. This study investigates the extent to which target-flanker orientation and/or color similarity determines luminance and orientation performance across different tasks under identical stimulus conditions. Targets were near-vertical Gabor patches defined by modulating only the green component of the RGB display. Subjects performed both target luminance and orientation discrimination tasks in separate blocks while both flanker hue (green or red flankers) and orientation (vertical or horizontal flankers) were manipulated as a function of target-flanker separation. We find strong evidence for a double dissociation between task and the specific set of features by which target-flanker similarity is defined. Whereas luminance judgments were highly contingent upon target-flanker hue similarity, orientation judgments showed the inverse pattern, largely contingent upon flanker orientation. The magnitude of this double dissociation decreased with target-flanker separation, at a rate predicted by Bouma's law. This specific pattern of performance provides strong support for the idea that crowding operates independently for the most part within orientation and color domains. That luminance judgments are constrained by target-flanker flanker hue similarity and, to a far lesser extent, by target-flanker orientation similarity suggests that the neural mechanisms responsible for mediating perceived luminance are principally linked to those mediating stimulus hue independent of those mediating stimulus orientation.

Shrinking Bouma's window: How to model crowding in dense displays.

In crowding, perception of a target deteriorates in the presence of nearby flankers. Traditionally, it is thought that visual crowding obeys Bouma's law, i.e., all elements within a certain distance interfere with the target, and that adding more elements always leads to stronger crowding. Crowding is predominantly studied using sparse displays (a target surrounded by a few flankers). However, many studies have shown that this approach leads to wrong conclusions about human vision. Van der Burg and colleagues proposed a paradigm to measure crowding in dense displays using genetic algorithms. Displays were selected and combined over several generations to maximize human performance. In contrast to Bouma's law, only the target's nearest neighbours affected performance. Here, we tested various models to explain these results. We used the same genetic algorithm, but instead of selecting displays based on human performance we selected displays based on the model's outputs. We found that all models based on the traditional feedforward pooling framework of vision were unable to reproduce human behaviour. In contrast, all models involving a dedicated grouping stage explained the results successfully. We show how traditional models can be improved by adding a grouping stage.

Helmholtz Versus Haute Couture: How Horizontal Stripes and Dark Clothes Make You Look Thinner.

In Helmholtz's illusion, a square with horizontal stripes appears taller than an identical square with vertical stripes. This effect has also been observed in experiments with human stimuli, where a human figure wearing a dress with horizontal stripes appears thinner than a drawing clad in vertical stripes. These findings do not agree with the common belief that clothes with horizontal stripes make someone appear wider, neither do they disentangle whether the horizontal or vertical stripes account for the thinning effect. In the present study, we focused on the effect of horizontal stripes in clothes comparing horizontal stripes against no-stripes (not against vertical; Experiments 1 and 2), using photos of a real-life female model, and controlling for the average luminance of the stripes (Experiment 2). Results showed that horizontal stripes and lower luminance have-independently-a small-to-moderate thinning effect on the perceived size of the body, and the effect is larger when the two variables are combined. In Experiment 3, we further show that the thinning effect due to the luminance of the dress is enhanced when the general background gets darker.

Positive sequential dependency for face attractiveness perception.

Recent findings from several groups have demonstrated that visual perception at a given moment can be biased toward what was recently seen. This is true both for basic visual attributes and for more complex representations, such as face identity, gender, or expression. This assimilation to the recent past is a positive serial dependency, similar to a temporal averaging process that capitalizes on short-term correlations in visual input to reduce noise and boost perceptual continuity. Here we examine serial dependencies in face perception using a simple attractiveness rating task and a rapid series of briefly presented face stimuli. In a series of three experiments, our results confirm a previous report that face attractiveness exhibits a positive serial dependency. This intertrial effect is not only determined by face attractiveness on the previous trial, but also depends on the faces shown up to five trials back. We examine the effect of stimulus presentation duration and find that stimuli as brief as 56 ms produce a significant positive dependency similar in magnitude to that produced by stimuli presented for 1,000 ms. We observed stronger positive dependencies between same-gender faces, and found a task dependency: Alternating gender discrimination trials with attractiveness rating trials produced no serial dependency. In sum, these findings show that a perception-stabilizing assimilation effect operates in face attractiveness perception that is task dependent and is acquired surprisingly quickly.

Linear Summation of Repulsive and Attractive Serial Dependencies: Orientation and Motion Dependencies Sum in Motion Perception.

Recent work from several groups has shown that perception of various visual attributes in human observers at a given moment is biased toward what was recently seen. This positive serial dependency is a kind of temporal averaging that exploits short-term correlations in visual scenes to reduce noise and stabilize perception. To date, this stabilizing "continuity field" has been demonstrated on stable visual attributes such as orientation and face identity, yet it would be counterproductive to apply it to dynamic attributes in which change sensitivity is needed. Here, we tested this using motion direction discrimination and predict a negative perceptual dependency: a contrastive relationship that enhances sensitivity to change. Surprisingly, our data showed a cubic-like pattern of dependencies with positive and negative components. By interleaving various stimulus combinations, we separated the components and isolated a positive perceptual dependency for motion and a negative dependency for orientation. A weighted linear sum of the separate dependencies described the original cubic pattern well. The positive dependency for motion shows an integrative perceptual effect and was unexpected, although it is consistent with work on motion priming. These findings suggest that a perception-stabilizing continuity field occurs pervasively, occurring even when it obscures sensitivity to dynamic stimuli.SIGNIFICANCE STATEMENT Recent studies show that visual perception at a given moment is not entirely veridical, but rather biased toward recently seen stimuli: a positive serial dependency. This temporal smoothing process helps perceptual continuity by preserving stable aspects of the visual scene over time, yet, for dynamic stimuli, temporal smoothing would blur dynamics and reduce sensitivity to change. We tested whether this process is selective for stable attributes by examining dependencies in motion perception. We found a clear positive dependency for motion, suggesting that positive perceptual dependencies are pervasive. We also found a concurrent negative (contrastive) dependency for orientation. Both dependencies combined linearly to determine perception, showing that the brain can calculate contrastive and integrative dependencies simultaneously from recent stimulus history when making perceptual decisions.

Rapid, generalized adaptation to asynchronous audiovisual speech.

The brain is adaptive. The speed of propagation through air, and of low-level sensory processing, differs markedly between auditory and visual stimuli; yet the brain can adapt to compensate for the resulting cross-modal delays. Studies investigating temporal recalibration to audiovisual speech have used prolonged adaptation procedures, suggesting that adaptation is sluggish. Here, we show that adaptation to asynchronous audiovisual speech occurs rapidly. Participants viewed a brief clip of an actor pronouncing a single syllable. The voice was either advanced or delayed relative to the corresponding lip movements, and participants were asked to make a synchrony judgement. Although we did not use an explicit adaptation procedure, we demonstrate rapid recalibration based on a single audiovisual event. We find that the point of subjective simultaneity on each trial is highly contingent upon the modality order of the preceding trial. We find compelling evidence that rapid recalibration generalizes across different stimuli, and different actors. Finally, we demonstrate that rapid recalibration occurs even when auditory and visual events clearly belong to different actors. These results suggest that rapid temporal recalibration to audiovisual speech is primarily mediated by basic temporal factors, rather than higher-order factors such as perceived simultaneity and source identity.

Rapid temporal recalibration is unique to audiovisual stimuli.

Following prolonged exposure to asynchronous multisensory signals, the brain adapts to reduce the perceived asynchrony. Here, in three separate experiments, participants performed a synchrony judgment task on audiovisual, audiotactile or visuotactile stimuli and we used inter-trial analyses to examine whether temporal recalibration occurs rapidly on the basis of a single asynchronous trial. Even though all combinations used the same subjects, task and design, temporal recalibration occurred for audiovisual stimuli (i.e., the point of subjective simultaneity depended on the preceding trial's modality order), but none occurred when the same auditory or visual event was combined with a tactile event. Contrary to findings from prolonged adaptation studies showing recalibration for all three combinations, we show that rapid, inter-trial recalibration is unique to audiovisual stimuli. We conclude that recalibration occurs at two different timescales for audiovisual stimuli (fast and slow), but only on a slow timescale for audiotactile and visuotactile stimuli.

Stretching time: Relativistic lag-induced shifts in perceived audiovisual synchrony using cluttered displays.

When presented with temporally displaced audiovisual events, observers shift their point of subjective simultaneity (PSS) in the direction of this prior lag. This effect, known as temporal recalibration (TR), has been inferred previously using single audiovisual events. Here we investigate TR using an audiovisual synchrony search paradigm employing spatiotemporally cluttered visual displays. By manipulating the relative modulation frequency of the adaptor (0.72 Hz) and test (0.36, 0.72 Hz) we find that following lag-adaptation, PSS shifts preserve the relative phase-not the latency-of the adapted lag. Applying this cross-frequency design to a classic simultaneity discrimination task, we find TR is unaffected by the relative frequency of adaptor and test in terms of latency rather than phase. This dissociation implies that under conditions of low spatial certainty TR obeys a relativistic (phase-conserving) temporal scaling law, whereas high spatial certainty affords PSS shifts, which operate in absolute (latency-conserving) temporal coordinates.

Evolving the stimulus to fit the brain: a genetic algorithm reveals the brain's feature priorities in visual search.

How does the brain find objects in cluttered visual environments? For decades researchers have employed the classic visual search paradigm to answer this question using factorial designs. Although such approaches have yielded important information, they represent only a tiny fraction of the possible parametric space. Here we use a novel approach, by using a genetic algorithm (GA) to discover the way the brain solves visual search in complex environments, free from experimenter bias. Participants searched a series of complex displays, and those supporting fastest search were selected to reproduce (survival of the fittest). Their display properties (genes) were crossed and combined to create a new generation of "evolved" displays. Displays evolved quickly over generations towards a stable, efficiently searched array. Color properties evolved first, followed by orientation. The evolved displays also contained spatial patterns suggesting a coarse-to-fine search strategy. We argue that this behavioral performance-driven GA reveals the way the brain selects information during visual search in complex environments. We anticipate that our approach can be adapted to a variety of sensory and cognitive questions that have proven too intractable for factorial designs.

Rapid recalibration to audiovisual asynchrony.

To combine information from different sensory modalities, the brain must deal with considerable temporal uncertainty. In natural environments, an external event may produce simultaneous auditory and visual signals yet they will invariably activate the brain asynchronously due to different propagation speeds for light and sound, and different neural response latencies once the signals reach the receptors. One strategy the brain uses to deal with audiovisual timing variation is to adapt to a prevailing asynchrony to help realign the signals. Here, using psychophysical methods in human subjects, we investigate audiovisual recalibration and show that it takes place extremely rapidly without explicit periods of adaptation. Our results demonstrate that exposure to a single, brief asynchrony is sufficient to produce strong recalibration effects. Recalibration occurs regardless of whether the preceding trial was perceived as synchronous, and regardless of whether a response was required. We propose that this rapid recalibration is a fast-acting sensory effect, rather than a higher-level cognitive process. An account in terms of response bias is unlikely due to a strong asymmetry whereby stimuli with vision leading produce bigger recalibrations than audition leading. A fast-acting recalibration mechanism provides a means for overcoming inevitable audiovisual timing variation and serves to rapidly realign signals at onset to maximize the perceptual benefits of audiovisual integration.

A CODE model bridging crowding in sparse and dense displays.

Visual crowding is arguably the strongest limitation imposed on extrafoveal vision, and is a relatively well-understood phenomenon. However, most investigations and theories are based on sparse displays consisting of a target and at most a handful of flanker objects. Recent findings suggest that the laws thought to govern crowding may not hold for densely cluttered displays, and that grouping and nearest neighbour effects may be more important. Here we present a computational model that accounts for crowding effects in both sparse and dense displays. The model is an adaptation and extension of an earlier model that has previously successfully accounted for spatial clustering, numerosity and object-based attention phenomena. Our model combines grouping by proximity and similarity with a nearest neighbour rule, and defines crowding as the extent to which target and flankers fail to segment. We show that when the model is optimized for explaining crowding phenomena in classic, sparse displays, it also does a good job in capturing novel crowding patterns in dense displays, in both existing and new data sets. The model thus ties together different principles governing crowding, specifically Bouma's law, grouping, and nearest neighbour similarity effects.

Perception of temporal synchrony not a prerequisite for multisensory integration.

Temporal alignment is often viewed as the most essential cue the brain can use to integrate information from across sensory modalities. However, the importance of conscious perception of synchrony to multisensory integration is a controversial topic. Conversely, the influence of cross-modal incongruence of higher level stimulus features such as phonetics on temporal processing is poorly understood. To explore the nuances of this relationship between temporal processing and multisensory integration, we presented 101 participants (ranging from 19 to 73 years of age) with stimuli designed to elicit the McGurk/MacDonald illusion (either matched or mismatched pairs of phonemes and visemes) with varying degrees of stimulus onset asynchrony between the visual and auditory streams. We asked them to indicate which syllable they perceived and whether the video and audio were synchronized on each trial. We found that participants often experienced the illusion despite not perceiving the stimuli as synchronous, and the same phonetic incongruence that produced the illusion also led to significant interference in simultaneity judgments. These findings challenge the longstanding assumption that perception of synchrony is a prerequisite to multisensory integration, support a more flexible view of multisensory integration, and suggest a complex, reciprocal relationship between temporal and multisensory processing.

Helicopter Pilot Performance and Workload in a Following Task in a Degraded Visual Environment.

BACKGROUND: In this study, we investigated the impact of a loss of horizon due to atmospheric conditions on flight performance and workload of helicopter pilots during a low-altitude, dynamic flight task in windy conditions at sea. We also examined the potential benefits of a helmet-mounted display (HMD) for this specific task.METHODS: In a fixed-based helicopter simulator, 16 military helicopter pilots were asked to follow a maneuvering go-fast vessel in a good visual environment (GVE) and in a degraded visual environment (DVE). DVE was simulated by fog, obscuring the horizon and reducing contrast. Both visual conditions were performed once with and once without an HMD, which was simulated by projecting head-slaved symbology in the outside visuals. Objective measures included flight performance, control inputs, gaze direction, and relative positioning. Subjective measures included self-ratings on performance, situation awareness, and workload.RESULTS: The results showed that in DVE the pilots perceived higher workload and were flying closer to the go-fast vessel than in GVE. Consequently, they responded with larger control inputs to maneuvers of the vessel. The availability of an HMD hardly improved flight performance but did allow the pilots to focus their attention more outside, significantly improving their situation awareness and reducing workload. These benefits were found in DVE as well as GVE conditions.DISCUSSION: DVE negatively affects workload and flight performance of helicopter pilots in a dynamic, low-altitude following task. An HMD can help improve situation awareness and lower the workload during such a task, irrespective of the visual conditions.Ledegang WD, van der Burg E, Valk PJL, Houben MMJ, Groen EL. Helicopter pilot performance and workload in a following task in a degraded visual environment. Aerosp Med Hum Perform. 2024; 95(1):16-24.

The capacity of audiovisual integration is limited to one item.

The human visual attention system is geared toward detecting the most salient and relevant events in an overwhelming stream of information. There has been great interest in measuring how many visual events can be processed at a time, and most of the work has suggested that the limit is three to four. However, attention to a visual stimulus can also be driven by a synchronous auditory event. The present work indicates that a fundamentally different limit applies to audiovisual processing, such that at most only a single audiovisual event can be processed at a time. This limited capacity is not due to a limitation in visual selection; participants were able to process about four visual objects simultaneously. Instead, we propose that audiovisual orienting is subject to a fundamentally different capacity limit than pure visual selection is.

Priming makes a stimulus more salient.

The present study used visual prior entry to determine which of two stimuli received attention first. Observers were asked to judge whether two test stimuli across a range of stimulus onset asynchronies (SOAs) were synchronized or not (simultaneity judgment task; SJ), or to report the temporal order of the two test stimuli (temporal order judgment task; TOJ). Before the presentation of the two test stimuli, a single noninformative stimulus that matched the color of one of the test stimuli was presented in the center of the display. The results showed that, in both the TOJ and SJ tasks, the noninformative stimulus caused a shift in the psychometric function such that the test stimulus that had the same color as the preceding noninformative stimulus was seen earlier in time than the test stimulus that had a color that did not match. In other words, the mere processing of the color of a noninformative stimulus, rendered the stimulus having that same color more salient, an effect that we attributed to priming. Because priming made one of the stimuli more salient, it received attention first and accelerated its processing, causing prior entry into awareness. Importantly, when the noninformative stimulus was a color word, no such priming effect was observed. We conclude that a primed test stimulus has the ability to capture attention in an automatic way.

Amplitude-modulated auditory stimuli influence selection of visual spatial frequencies.

A recent study by Guzman-Martinez, Ortega, Grabowecky, Mossbridge, and Suzuki (2012) showed that participants match the frequency of an amplitude-modulated auditory stimulus to visual spatial frequency with a linear relationship and suggested this crossmodal matching guided attention to specific spatial frequencies. Here, we replicated this matching relationship and used the visual search paradigm to investigate whether auditory signals guide attention to matched visual spatial frequencies. Participants were presented with a search display of Gabors, all with different spatial frequencies. When the auditory signal was informative, improved search efficiency occurred for some spatial frequencies. However, when uninformative, a matched auditory signal produced no effect on visual search performance whatsoever. Moreover, search benefits were also observed when the auditory signal was informative, but did not match the spatial frequency. Together, these findings suggest that an amplitude-modulated auditory signal can influence visual selection of a matched spatial frequency, but the effect is due to top-down knowledge rather than resulting from automatic attentional capture derived from low-level mapping.

No differences between adults with and without autism in audiovisual synchrony perception.

LAY ABSTRACT: It has been known for a long time that individuals diagnosed with autism spectrum disorder perceive the world differently. In this study, we investigated how people with or without autism perceive visual and auditory information. We know that an auditory and a visual stimulus do not have to be perfectly synchronous for us to perceive them as synchronous: first, when the two are within a certain time window (temporal binding window), the brain will tell us that they are synchronous. Second, the brain can also adapt quickly to audiovisual asynchronies (rapid recalibration). Although previous studies have shown that people with autism spectrum disorder have different temporal binding windows, and less rapid recalibration, we did not find these differences in our study. However, we did find that both processes develop over age, and since previous studies tested only young people (children, adolescents, and young adults), and we tested adults from 18 to 55 years, this might explain the different findings. In the end, there might be quite a complex story, where people with and without autism spectrum disorder perceive the world differently, even dependent on how old they are.