Formal models and quantitative measures of multisensory integration: a selective overview.

Multisensory integration (MI) is defined as the neural process by which unisensory signals are combined to form a new product that is significantly different from the responses evoked by the modality-specific component stimuli. In recent years, MI research has seen exponential growth in the number of empirical and theoretical studies. This study presents a selective overview of formal modeling approaches to MI. Emphasis is on models and measures for behavioral paradigms, such as localization, judgment of temporal order or simultaneity, and reaction times, but some concepts for the modeling of single-cell spike rates are treated as well. We identify a number of essential concepts underlying most model classes, such as Bayesian causal inference, probability summation, coactivation, and time window of integration. Quantitative indexes for measuring and comparing the strength of MI across different paradigms are also discussed. Whereas progress over the last years is remarkable, we point out some strengths and weaknesses of the modeling approaches and discuss some obstacles toward a unified theory of MI.

Multisensory Integration and Exogenous Spatial Attention: A Time-window-of-integration Analysis.

Although it is well documented that occurrence of an irrelevant and nonpredictive sound facilitates motor responses to a subsequent target light appearing nearby, the cause of this "exogenous spatial cuing effect" has been under discussion. On the one hand, it has been postulated to be the result of a shift of visual spatial attention possibly triggered by parietal and/or cortical supramodal "attention" structures. On the other hand, the effect has been considered to be due to multisensory integration based on the activation of multisensory convergence structures in the brain. Recent RT experiments have suggested that multisensory integration and exogenous spatial cuing differ in their temporal profiles of facilitation: When the nontarget occurs 100-200 msec before the target, facilitation is likely driven by crossmodal exogenous spatial attention, whereas multisensory integration effects are still seen when target and nontarget are presented nearly simultaneously. Here, we develop an extension of the time-window-of-integration model that combines both mechanisms within the same formal framework. The model is illustrated by fitting it to data from a focused attention task with a visual target and an auditory nontarget presented at horizontally or vertically varying positions. Results show that both spatial cuing and multisensory integration may coexist in a single trial in bringing about the crossmodal facilitation of RT effects. Moreover, the formal analysis via time window of integration allows to predict and quantify the contribution of either mechanism as they occur across different spatiotemporal conditions.

Dependency in multisensory integration: a copula-based analysis.

The notion of copula has attracted attention from the field of contextuality and probability. A copula is a function that joins a multivariate distribution to its one-dimensional marginal distributions. Thereby, it allows characterizing the multivariate dependency separately from the specific choice of margins. Here, we demonstrate the use of copulas by investigating the structure of dependency between processing stages in a stochastic model of multisensory integration, which describes the effect of stimulation by several sensory modalities on human reaction times. We derive explicit terms for the covariance and Kendall's tau between the processing stages and point out the specific role played by two stochastic order relations, the usual stochastic order and the likelihood ratio order, in determining the sign of dependency. This article is part of the theme issue 'Contextuality and probability in quantum mechanics and beyond'.

A new nonparametric test for the race model inequality.

The race model inequality (RMI), as first introduced by Miller (Cognitive Psychology, 14, 247-279, 1982), entails an upper bound on the amount of statistical facilitation for reaction times (RTs) attainable by a race model within the redundant-signals paradigm. A violation of RMI may be considered as empirical evidence for a coactivation model rather than a race model. Here, we introduce a novel nonparametric procedure for evaluating the RMI for single participant analysis. The statistical procedure is based on a new probabilistic representation that highlights some neglected, but important distributional features of the RMI. In particular, we show how the reconstructed distribution function under maximal statistical facilitation for a race model is characterized by a specific truncated-type property. The results of two Monte Carlo simulation studies suggest that our procedure efficiently controls for type I error with reasonable power. Finally, unlike previous proposals for single participant analysis (e.g., Maris and Maris (Journal of Mathematical Psychology 47, 507-514, 2003)), our approach is also more consistent with the typical way to collect RT data in experimental works. R script functions for running the statistical analysis on single participant data are made freely available to the readers on a dedicated web server.

Does hearing aid use affect audiovisual integration in mild hearing impairment?

There is converging evidence for altered audiovisual integration abilities in hearing-impaired individuals and those with profound hearing loss who are provided with cochlear implants, compared to normal-hearing adults. Still, little is known on the effects of hearing aid use on audiovisual integration in mild hearing loss, although this constitutes one of the most prevalent conditions in the elderly and, yet, often remains untreated in its early stages. This study investigated differences in the strength of audiovisual integration between elderly hearing aid users and those with the same degree of mild hearing loss who were not using hearing aids, the non-users, by measuring their susceptibility to the sound-induced flash illusion. We also explored the corresponding window of integration by varying the stimulus onset asynchronies. To examine general group differences that are not attributable to specific hearing aid settings but rather reflect overall changes associated with habitual hearing aid use, the group of hearing aid users was tested unaided while individually controlling for audibility. We found greater audiovisual integration together with a wider window of integration in hearing aid users compared to their age-matched untreated peers. Signal detection analyses indicate that a change in perceptual sensitivity as well as in bias may underlie the observed effects. Our results and comparisons with other studies in normal-hearing older adults suggest that both mild hearing impairment and hearing aid use seem to affect audiovisual integration, possibly in the sense that hearing aid use may reverse the effects of hearing loss on audiovisual integration. We suggest that these findings may be particularly important for auditory rehabilitation and call for a longitudinal study.

Prior knowledge of spatiotemporal configuration facilitates crossmodal saccadic response : A TWIN analysis.

Saccadic reaction times from a focused-attention task with a visual target and an acoustic nontarget support the hypothesis that the amount of saccadic facilitation in the presence of a nontarget increases with the prior knowledge of alignment with the target across different blocks of trials. The time-window-of-integration model can account for the size of the effect by having window size depend on the prior knowledge of alignment. Some efforts to identify the neural correlates of the effect are discussed.

Predicting auditory space calibration from recent multisensory experience.

Multisensory experience can lead to auditory space recalibration. After exposure to discrepant audiovisual stimulation, sound percepts are displaced in space, in the direction of the previous visual stimulation. This study focuses on identifying the factors in recent sensory experience leading to such auditory space shifts. Sequences of five audiovisual pairs were presented, each randomly congruent or discrepant in space. Each sequence was followed by a single auditory trial and two visual trials. In each trial, participants had to identify the perceived stimuli positions. We found that auditory localization is shifted during audiovisual discrepant trials and during subsequent auditory trials, suggesting a recalibration effect. Time did not lead to greater recalibration effects. The last audiovisual trial affects the subsequent auditory shift the most. The number of discrepant trials in a sequence, and the number of consecutive trials in sequence, also correlated with the subsequent auditory shift. To estimate the individual contribution of previously presented trials to the recalibration effect, a best-fitting model was developed to predict the shift in a linear weighted combination of stimulus features: (1) whether matching or discrepant trials occurred in the sequence, (2) total number of discrepant trials, and (3) maximum number of consecutive discrepant trials, (4) whether the last trial was discrepant or not. The selected model consists of a function including as properties the type of stimulus of the last audiovisual sequence trial and the overall probability of mismatching trials in sequence.

Behavioral measures of multisensory integration: bounds on bimodal detection probability.

One way to test and quantify multisensory integration in a behavioral paradigm is to compare bimodal detection probability with bounds defined by some combination of the unimodal detection probabilities. Here we (1) improve on an upper bound recently suggested by Stevenson et al. (Brain Topogr 27(6):707-730, 2014), (2) present a lower bound, (3) interpret the bounds in terms of stochastic dependency between the detection probabilities, (4) discuss some additional assumptions required for the validity of any such bound, (5) suggest some potential applications to neurophysiologic measures, and point out some parallels to the 'race model inequality' for reaction times.

Parameter recovery for the time-window-of-integration (TWIN) model of multisensory integration in focused attention.

In multisensory settings such as the focused attention paradigm (FAP), subjects are instructed to respond to stimuli of the target modality only, yet reaction times tend to be shorter if an unattended stimulus is presented within a certain spatiotemporal vicinity of the target. The time window of integration (TWIN) model predicts successfully these observed cross-modal reaction time effects. It proposes that all the initially unimodal information must arrive at a point of integration within a certain time window in order to be integrated and thus to initiate response enhancements like the observed reaction time reductions. Here we conducted a parameter recovery study of the TWIN model for focused attention tasks, with five parameters (the durations of the visual and auditory unimodal and the integrated second stage, the width of the time window, and the effect size). Results show that parameter estimates were highly accurate (unbiased, constant error less than 5 ms) and precise (variable error less than 8 ms) throughout, speaking to a high reliability and criterion validity of the process. Further analyses ensured that the estimation procedure is consistent and sufficiently robust against contamination (faulty integration). It can thus be used to estimate reliably the point of integration and the width of the time window.

Terms of debate: Consensus definitions to guide the scientific discourse on visual distraction.

Hypothesis-driven research rests on clearly articulated scientific theories. The building blocks for communicating these theories are scientific terms. Obviously, communication - and thus, scientific progress - is hampered if the meaning of these terms varies idiosyncratically across (sub)fields and even across individual researchers within the same subfield. We have formed an international group of experts representing various theoretical stances with the goal to homogenize the use of the terms that are most relevant to fundamental research on visual distraction in visual search. Our discussions revealed striking heterogeneity and we had to invest much time and effort to increase our mutual understanding of each other's use of central terms, which turned out to be strongly related to our respective theoretical positions. We present the outcomes of these discussions in a glossary and provide some context in several essays. Specifically, we explicate how central terms are used in the distraction literature and consensually sharpen their definitions in order to enable communication across theoretical standpoints. Where applicable, we also explain how the respective constructs can be measured. We believe that this novel type of adversarial collaboration can serve as a model for other fields of psychological research that strive to build a solid groundwork for theorizing and communicating by establishing a common language. For the field of visual distraction, the present paper should facilitate communication across theoretical standpoints and may serve as an introduction and reference text for newcomers.

Computing an optimal time window of audiovisual integration in focused attention tasks: illustrated by studies on effect of age and prior knowledge.

The concept of a "time window of integration" holds that information from different sensory modalities must not be perceived too far apart in time in order to be integrated into a multisensory perceptual event. Empirical estimates of window width differ widely, however, ranging from 40 to 600 ms depending on context and experimental paradigm. Searching for theoretical derivation of window width, Colonius and Diederich (Front Integr Neurosci 2010) developed a decision-theoretic framework using a decision rule that is based on the prior probability of a common source, the likelihood of temporal disparities between the unimodal signals, and the payoff for making right or wrong decisions. Here, this framework is extended to the focused attention task where subjects are asked to respond to signals from a target modality only. Evoking the framework of the time-window-of-integration (TWIN) model, an explicit expression for optimal window width is obtained. The approach is probed on two published focused attention studies. The first is a saccadic reaction time study assessing the efficiency with which multisensory integration varies as a function of aging. Although the window widths for young and older adults differ by nearly 200 ms, presumably due to their different peripheral processing speeds, neither of them deviates significantly from the optimal values. In the second study, head saccadic reactions times to a perfectly aligned audiovisual stimulus pair had been shown to depend on the prior probability of spatial alignment. Intriguingly, they reflected the magnitude of the time-window widths predicted by our decision-theoretic framework, i.e., a larger time window is associated with a higher prior probability.

On quantifying multisensory interaction effects in reaction time and detection rate.

Both mean reaction time (RT) and detection rate (DR) are important measures for assessing the amount of multisensory interaction occurring in crossmodal experiments, but they are often applied separately. Here we demonstrate that measuring multisensory performance using either RT or DR alone misses out on important information. We suggest an integration of RT and DR into a single measure of multisensory performance: the first index (MRE*) is based on an arithmetic combination of RT and DR, the second (MPE) is constructed from parameters derived from fitting a sequential sampling model to RT and DR data simultaneously. Our approach is illustrated by data from two audio-visual experiments. In the first, a redundant targets detection experiment using stimuli of different intensity, both measures yield similar pattern of results supporting the "principle of inverse effectiveness". The second experiment, introducing stimulus onset asynchrony and differing instructions (focused attention vs. redundant targets task) further supports the usefulness of both indices. Statistical properties of both measures are investigated via bootstrapping procedures.

The Fechnerian idea.

From the principle that subjective dissimilarity between 2 stimuli is determined by their ratio, Fechner derives his logarithmic law in 2 ways. In one derivation, ignored and forgotten in modern accounts of Fechner's theory, he formulates the principle in question as a functional equation and reduces it to one with a known solution. In the other derivation, well known and often criticized, he solves the same functional equation by differentiation. Both derivations are mathematically valid (the much-derided "expedient principle" mentioned by Fechner can be viewed as merely an inept way of pointing at a certain property of the differentiation he uses). Neither derivation uses the notion of just-noticeable differences. But if Weber's law is accepted in addition to the principle in question, then the dissimilarity between 2 stimuli is approximately proportional to the number of just-noticeable differences that fit between these stimuli: The smaller Weber's fraction the better the approximation, and Weber's fraction can always be made arbitrarily small by an appropriate convention. We argue, however, that neither the 2 derivations of Fechner's law nor the relation of this law to thresholds constitutes the essence of Fechner's approach. We see this essence in the idea of additive cumulation of sensitivity values. Fechner's work contains a surprisingly modern definition of sensitivity at a given stimulus: the rate of growth of the probability-of-greater function, with this stimulus serving as a standard. The idea of additive cumulation of sensitivity values lends itself to sweeping generalizations of Fechnerian scaling.

A two-stage diffusion modeling approach to the compelled-response task.

The issue of how perception and motor planning interact to generate a given choice between actions is a fundamental question in both psychology and neuroscience. Salinas and colleagues have developed a behavioral paradigm, the compelled-response task, where the signal that instructs the subject to make an eye movement is given before the cue that indicates which of two possible target choices is the correct one. When the cue is given rather late, the participant must guess and make an uninformed random choice. Perceptual performance can be tracked as a function of the amount of time during which sensory information is available. In Salinas' accelerated race-to-threshold model, two variables race against each other to a threshold, at which a saccade is initiated. The source of random variability is in the initial state of information buildup across trials. This implies that incorrect decisions are due to the inertia of the racing variables that have, at the start, sampled a constant buildup in the "wrong" direction. Here we suggest an alternative, non-time-homogeneous two-stage-diffusion model that is able to predict both response time distributions and choice probabilities with a few easy-to-interpret parameters and without assuming cross-trial parameter variability. It is falsifiable already at the level of qualitative features, for example, predicting bimodal reaction time (RT) distributions for particular gap times. It connects the compelled-response paradigm with an approach to decision making that has been uniquely successful in describing both behavioral and neural data in a variety of experimental settings for the last 40 years. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Treatment of Age-Related Hearing Loss Alters Audiovisual Integration and Resting-State Functional Connectivity: A Randomized Controlled Pilot Trial.

Untreated age-related hearing loss increases audiovisual integration and impacts resting state functional brain connectivity. Further, there is a relation between crossmodal plasticity and audiovisual integration strength in cochlear implant patients. However, it is currently unclear whether amplification of the auditory input by hearing aids influences audiovisual integration and resting state functional brain connectivity. We conducted a randomized controlled pilot study to investigate how the McGurk illusion, a common measure for audiovisual integration, and resting state functional brain connectivity of the auditory cortex are altered by six-month hearing aid use. Thirty-two older participants with slight-to-moderate, symmetric, age-related hearing loss were allocated to a treatment or waiting control group and measured one week before and six months after hearing aid fitting with functional magnetic resonance imaging. Our results showed a statistical trend for an increased McGurk illusion after six months of hearing aid use. We further demonstrated that an increase in McGurk susceptibility is related to a decreased hearing aid benefit for auditory speech intelligibility in noise. No significant interaction between group and time point was obtained in the whole-brain resting state analysis. However, a region of interest (ROI)-to-ROI analysis indicated that hearing aid use of six months was associated with a decrease in resting state functional connectivity between the auditory cortex and the fusiform gyrus and that this decrease was related to an increase of perceived McGurk illusions. Our study, therefore, suggests that even short-term hearing aid use alters audiovisual integration and functional brain connectivity between auditory and visual cortices.

Reduced resting state functional connectivity with increasing age-related hearing loss and McGurk susceptibility.

Age-related hearing loss has been related to a compensatory increase in audio-visual integration and neural reorganization including alterations in functional resting state connectivity. How these two changes are linked in elderly listeners is unclear. The current study explored modulatory effects of hearing thresholds and audio-visual integration on resting state functional connectivity. We analysed a large set of resting state data of 65 elderly participants with a widely varying degree of untreated hearing loss. Audio-visual integration, as gauged with the McGurk effect, increased with progressing hearing thresholds. On the neural level, McGurk illusions were negatively related to functional coupling between motor and auditory regions. Similarly, connectivity of the dorsal attention network to sensorimotor and primary motor cortices was reduced with increasing hearing loss. The same effect was obtained for connectivity between the salience network and visual cortex. Our findings suggest that with progressing untreated age-related hearing loss, functional coupling at rest declines, affecting connectivity of brain networks and areas associated with attentional, visual, sensorimotor and motor processes. Especially connectivity reductions between auditory and motor areas were related to stronger audio-visual integration found with increasing hearing loss.

Time-window-of-integration (TWIN) model for saccadic reaction time: effect of auditory masker level on visual-auditory spatial interaction in elevation.

Saccadic reaction time (SRT) to a visual target tends to be shorter when auditory stimuli are presented in close temporal and spatial proximity, even when subjects are instructed to ignore the auditory non-target (focused attention paradigm). Previous studies using pairs of visual and auditory stimuli differing in both azimuth and vertical position suggest that the amount of SRT facilitation decreases not with the physical but with the perceivable distance between visual target and auditory non-target. Steenken et al. (Brain Res 1220:150-156, 2008) presented an additional white-noise masker background of three seconds duration. Increasing the masker level had a diametrical effect on SRTs in spatially coincident versus disparate stimulus configurations: saccadic responses to coincident visual-auditory stimuli are slowed down, whereas saccadic responses to disparate stimuli are speeded up. Here we show that the time-window-of-integration model accounts for this observation by variation of a perceivable-distance parameter in the second stage of the model whose value does not depend on stimulus onset asynchrony between target and non-target.

A consensus guide to capturing the ability to inhibit actions and impulsive behaviors in the stop-signal task.

Response inhibition is essential for navigating everyday life. Its derailment is considered integral to numerous neurological and psychiatric disorders, and more generally, to a wide range of behavioral and health problems. Response-inhibition efficiency furthermore correlates with treatment outcome in some of these conditions. The stop-signal task is an essential tool to determine how quickly response inhibition is implemented. Despite its apparent simplicity, there are many features (ranging from task design to data analysis) that vary across studies in ways that can easily compromise the validity of the obtained results. Our goal is to facilitate a more accurate use of the stop-signal task. To this end, we provide 12 easy-to-implement consensus recommendations and point out the problems that can arise when they are not followed. Furthermore, we provide user-friendly open-source resources intended to inform statistical-power considerations, facilitate the correct implementation of the task, and assist in proper data analysis.

Assessing age-related multisensory enhancement with the time-window-of-integration model.

Although from multisensory research a great deal is known about how the different senses interact, there is little knowledge as to the impact of aging on these multisensory processes. In this study, we measured saccadic reaction time (SRT) of aged and young individuals to the onset of a visual target stimulus with and without an accessory auditory stimulus occurring (focused attention task). The response time pattern for both groups was similar: mean SRT to bimodal stimuli was generally shorter than to unimodal stimuli, and mean bimodal SRT was shorter when the auditory accessory was presented ipsilaterally rather than contralaterally to the target. The elderly participants were considerably slower than the younger participants under all conditions but showed a greater multisensory enhancement, that is, they seem to benefit more from bimodal stimulus presentation. In an attempt to weigh the contributions of peripheral sensory processes relative to more central cognitive processes possibly responsible for the difference in the younger and older adults, the time-window-of-integration (TWIN) model for crossmodal interaction in saccadic eye movements developed by the authors was fitted to the data from both groups. The model parameters suggest that (i) there is a slowing of the peripheral sensory processing in the elderly, (ii) as a result of this slowing, the probability of integration is smaller in the elderly even with a wider time-window-of-integration, and (iii) multisensory integration, if it occurs, manifests itself in larger neural enhancement in the elderly; however, because of (ii), on average the integration effect is not large enough to compensate for the peripheral slowing in the elderly.