Formal models and quantitative measures of multisensory integration: a selective overview.

Multisensory integration (MI) is defined as the neural process by which unisensory signals are combined to form a new product that is significantly different from the responses evoked by the modality-specific component stimuli. In recent years, MI research has seen exponential growth in the number of empirical and theoretical studies. This study presents a selective overview of formal modeling approaches to MI. Emphasis is on models and measures for behavioral paradigms, such as localization, judgment of temporal order or simultaneity, and reaction times, but some concepts for the modeling of single-cell spike rates are treated as well. We identify a number of essential concepts underlying most model classes, such as Bayesian causal inference, probability summation, coactivation, and time window of integration. Quantitative indexes for measuring and comparing the strength of MI across different paradigms are also discussed. Whereas progress over the last years is remarkable, we point out some strengths and weaknesses of the modeling approaches and discuss some obstacles toward a unified theory of MI.

Multisensory Integration and Exogenous Spatial Attention: A Time-window-of-integration Analysis.

Although it is well documented that occurrence of an irrelevant and nonpredictive sound facilitates motor responses to a subsequent target light appearing nearby, the cause of this "exogenous spatial cuing effect" has been under discussion. On the one hand, it has been postulated to be the result of a shift of visual spatial attention possibly triggered by parietal and/or cortical supramodal "attention" structures. On the other hand, the effect has been considered to be due to multisensory integration based on the activation of multisensory convergence structures in the brain. Recent RT experiments have suggested that multisensory integration and exogenous spatial cuing differ in their temporal profiles of facilitation: When the nontarget occurs 100-200 msec before the target, facilitation is likely driven by crossmodal exogenous spatial attention, whereas multisensory integration effects are still seen when target and nontarget are presented nearly simultaneously. Here, we develop an extension of the time-window-of-integration model that combines both mechanisms within the same formal framework. The model is illustrated by fitting it to data from a focused attention task with a visual target and an auditory nontarget presented at horizontally or vertically varying positions. Results show that both spatial cuing and multisensory integration may coexist in a single trial in bringing about the crossmodal facilitation of RT effects. Moreover, the formal analysis via time window of integration allows to predict and quantify the contribution of either mechanism as they occur across different spatiotemporal conditions.

Dependency in multisensory integration: a copula-based analysis.

The notion of copula has attracted attention from the field of contextuality and probability. A copula is a function that joins a multivariate distribution to its one-dimensional marginal distributions. Thereby, it allows characterizing the multivariate dependency separately from the specific choice of margins. Here, we demonstrate the use of copulas by investigating the structure of dependency between processing stages in a stochastic model of multisensory integration, which describes the effect of stimulation by several sensory modalities on human reaction times. We derive explicit terms for the covariance and Kendall's tau between the processing stages and point out the specific role played by two stochastic order relations, the usual stochastic order and the likelihood ratio order, in determining the sign of dependency. This article is part of the theme issue 'Contextuality and probability in quantum mechanics and beyond'.

Thinking Fast Increases Framing Effects in Risky Decision Making.

Every day, people face snap decisions when time is a limiting factor. In addition, the way a problem is presented can influence people's choices, which creates what are known as framing effects. In this research, we explored how time pressure interacts with framing effects in risky decision making. Specifically, does time pressure strengthen or weaken framing effects? On one hand, research has suggested that framing effects evolve through the deliberation process, growing larger with time. On the other hand, dual-process theory attributes framing effects to an intuitive, emotional system that responds automatically to stimuli. In our experiments, participants made decisions about gambles framed in terms of either gains or losses, and time pressure was manipulated across blocks. Results showed increased framing effects under time pressure in both hypothetical and incentivized choices, which supports the dual-process hypothesis that these effects arise from a fast, intuitive system.

Prior knowledge of spatiotemporal configuration facilitates crossmodal saccadic response : A TWIN analysis.

Saccadic reaction times from a focused-attention task with a visual target and an acoustic nontarget support the hypothesis that the amount of saccadic facilitation in the presence of a nontarget increases with the prior knowledge of alignment with the target across different blocks of trials. The time-window-of-integration model can account for the size of the effect by having window size depend on the prior knowledge of alignment. Some efforts to identify the neural correlates of the effect are discussed.

Parameter recovery for the time-window-of-integration (TWIN) model of multisensory integration in focused attention.

In multisensory settings such as the focused attention paradigm (FAP), subjects are instructed to respond to stimuli of the target modality only, yet reaction times tend to be shorter if an unattended stimulus is presented within a certain spatiotemporal vicinity of the target. The time window of integration (TWIN) model predicts successfully these observed cross-modal reaction time effects. It proposes that all the initially unimodal information must arrive at a point of integration within a certain time window in order to be integrated and thus to initiate response enhancements like the observed reaction time reductions. Here we conducted a parameter recovery study of the TWIN model for focused attention tasks, with five parameters (the durations of the visual and auditory unimodal and the integrated second stage, the width of the time window, and the effect size). Results show that parameter estimates were highly accurate (unbiased, constant error less than 5 ms) and precise (variable error less than 8 ms) throughout, speaking to a high reliability and criterion validity of the process. Further analyses ensured that the estimation procedure is consistent and sufficiently robust against contamination (faulty integration). It can thus be used to estimate reliably the point of integration and the width of the time window.

Individual differences moderate effects in an Unusual Disease paradigm: A psychophysical data collection lab approach and an online experiment.

We report two studies investigating individual intuitive-deliberative cognitive-styles and risk-styles as moderators of the framing effect in Tversky and Kahneman's famous Unusual Disease problem setting. We examined framing effects in two ways: counting the number of frame-inconsistent choices and comparing the proportions of risky choices depending on gain-loss framing. Moreover, in addition to gain-loss frames, we systematically varied the number of affected people, probabilities of surviving/dying, type of disease, and response deadlines. Study 1 used a psychophysical data collection approach and a sample of 43 undergraduate students, each performing 480 trials. Study 2 was an online study incorporating psychophysical elements in a social science approach using a larger and more heterogeneous sample, i.e., 262 participants performed 80 trials each. In both studies, the effect of framing on risky choice proportions was moderated by risk-styles. Cognitive-styles measured on different scales moderated the framing effect only in study 2. The effects of disease type, probability of surviving/dying, and number of affected people on risky choice frequencies were also affected by cognitive-styles and risk-styles but different for both studies and to different extents. We found no relationship between the number of frame-inconsistent choices and cognitive-styles or risk-styles, respectively.

Keep your budget together! Investigating determinants on risky decision-making about losses.

The present study investigates the influence of framing, different amounts to lose, and probabilities of a risky and sure choice option, time limits, and need on risky decision-making. For a given block of trials, participants were equipped with a personal budget (number of points). On each trial within a block, a specific initial amount is possibly taken from the budget by the outcome of a gamble or the choice of a sure loss option. The goal was to avoid losing points from the budget for not falling below a predefined need threshold. Three different levels of induced need were included. Employing a psychophysical experimental approach, we furthermore tested a sequential component of human risk behavior towards a need threshold inspired by research on animal foraging behavior. Risk-sensitivity models and the Stone-Geary framework serve as generating hypotheses on need thresholds. We found that framing, need, and probabilities influenced risky choices. Time limits and initial amounts moderated the framing effect. No sequential component was observed.

Computing an optimal time window of audiovisual integration in focused attention tasks: illustrated by studies on effect of age and prior knowledge.

The concept of a "time window of integration" holds that information from different sensory modalities must not be perceived too far apart in time in order to be integrated into a multisensory perceptual event. Empirical estimates of window width differ widely, however, ranging from 40 to 600 ms depending on context and experimental paradigm. Searching for theoretical derivation of window width, Colonius and Diederich (Front Integr Neurosci 2010) developed a decision-theoretic framework using a decision rule that is based on the prior probability of a common source, the likelihood of temporal disparities between the unimodal signals, and the payoff for making right or wrong decisions. Here, this framework is extended to the focused attention task where subjects are asked to respond to signals from a target modality only. Evoking the framework of the time-window-of-integration (TWIN) model, an explicit expression for optimal window width is obtained. The approach is probed on two published focused attention studies. The first is a saccadic reaction time study assessing the efficiency with which multisensory integration varies as a function of aging. Although the window widths for young and older adults differ by nearly 200 ms, presumably due to their different peripheral processing speeds, neither of them deviates significantly from the optimal values. In the second study, head saccadic reactions times to a perfectly aligned audiovisual stimulus pair had been shown to depend on the prior probability of spatial alignment. Intriguingly, they reflected the magnitude of the time-window widths predicted by our decision-theoretic framework, i.e., a larger time window is associated with a higher prior probability.

On quantifying multisensory interaction effects in reaction time and detection rate.

Both mean reaction time (RT) and detection rate (DR) are important measures for assessing the amount of multisensory interaction occurring in crossmodal experiments, but they are often applied separately. Here we demonstrate that measuring multisensory performance using either RT or DR alone misses out on important information. We suggest an integration of RT and DR into a single measure of multisensory performance: the first index (MRE*) is based on an arithmetic combination of RT and DR, the second (MPE) is constructed from parameters derived from fitting a sequential sampling model to RT and DR data simultaneously. Our approach is illustrated by data from two audio-visual experiments. In the first, a redundant targets detection experiment using stimuli of different intensity, both measures yield similar pattern of results supporting the "principle of inverse effectiveness". The second experiment, introducing stimulus onset asynchrony and differing instructions (focused attention vs. redundant targets task) further supports the usefulness of both indices. Statistical properties of both measures are investigated via bootstrapping procedures.

A two-stage diffusion modeling approach to the compelled-response task.

The issue of how perception and motor planning interact to generate a given choice between actions is a fundamental question in both psychology and neuroscience. Salinas and colleagues have developed a behavioral paradigm, the compelled-response task, where the signal that instructs the subject to make an eye movement is given before the cue that indicates which of two possible target choices is the correct one. When the cue is given rather late, the participant must guess and make an uninformed random choice. Perceptual performance can be tracked as a function of the amount of time during which sensory information is available. In Salinas' accelerated race-to-threshold model, two variables race against each other to a threshold, at which a saccade is initiated. The source of random variability is in the initial state of information buildup across trials. This implies that incorrect decisions are due to the inertia of the racing variables that have, at the start, sampled a constant buildup in the "wrong" direction. Here we suggest an alternative, non-time-homogeneous two-stage-diffusion model that is able to predict both response time distributions and choice probabilities with a few easy-to-interpret parameters and without assuming cross-trial parameter variability. It is falsifiable already at the level of qualitative features, for example, predicting bimodal reaction time (RT) distributions for particular gap times. It connects the compelled-response paradigm with an approach to decision making that has been uniquely successful in describing both behavioral and neural data in a variety of experimental settings for the last 40 years. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Gamble for the needy! Does identifiability enhances donation?

To investigate how neediness and identifiability of a recipient influence the willingness of a donor to invest resources in charity-like lotteries we propose a new game, called 'need game'. Similar to the dictator game, the need game includes two players, one active player (the donor or dictator) and one passive player (the recipient). Both players require a minimum need (ND and NR), expressed in terms of points. The donor is endowed with KD points and must retain at least ND points, i.e., the need, with ND < KD, at the end of the game with n rounds. The recipient starts with KR points and must end the game with at least NR points, i.e., the need, with KR < NR < KD. The donor is asked to choose one of three different amounts from KD to place a bet on a lottery. If won, the gain is added to the endowment. If lost, the recipient receives the points. The recipient is paid only when his/her need threshold is obtained; likewise the donor gets paid only when his/her need threshold is maintained. The main focus here is on need of both players (ND = NR = 2, 200, and ND = NR = 0 serving as baseline control) and the identifiability of the recipient (no information, described by text and picture, and physical presence). We probe whether the amount invested by the donor depends on need and identifiability of the recipient. In addition, we include the framing of the game as gain or loss, different probabilities to win/lose, and different time limits as covariates. We found that each of these factors can play a role when investing in charity-like lotteries.

Time-window-of-integration (TWIN) model for saccadic reaction time: effect of auditory masker level on visual-auditory spatial interaction in elevation.

Saccadic reaction time (SRT) to a visual target tends to be shorter when auditory stimuli are presented in close temporal and spatial proximity, even when subjects are instructed to ignore the auditory non-target (focused attention paradigm). Previous studies using pairs of visual and auditory stimuli differing in both azimuth and vertical position suggest that the amount of SRT facilitation decreases not with the physical but with the perceivable distance between visual target and auditory non-target. Steenken et al. (Brain Res 1220:150-156, 2008) presented an additional white-noise masker background of three seconds duration. Increasing the masker level had a diametrical effect on SRTs in spatially coincident versus disparate stimulus configurations: saccadic responses to coincident visual-auditory stimuli are slowed down, whereas saccadic responses to disparate stimuli are speeded up. Here we show that the time-window-of-integration model accounts for this observation by variation of a perceivable-distance parameter in the second stage of the model whose value does not depend on stimulus onset asynchrony between target and non-target.

A Dynamic Dual Process Model of Intertemporal Choice.

Dual process theories of decision making describe choice as the result of an automatic System 1, which is quick to activate but behaves impulsively, and a deliberative System 2, which is slower to activate but makes decisions in a rational and controlled manner. However, most existent dual process theories are verbal descriptions and do not generate testable qualitative and quantitative predictions. In this paper, we describe a formalized dynamic dual process model framework of intertemporal choice that allows for precise, experimentally testable predictions regarding choice probability and response time distributions. The framework is based on two-stage stochastic process models to account for the two postulated systems and to capture the dynamics and uncertainty involved in decision making. Using quasi closed form solutions, we illustrate how different factors (timing of System 1, time constraint, and preferences in both systems), which are reflected in the model parameters, influence qualitative and quantitative model predictions. Furthermore, we show how an existing static-deterministic model on intertemporal choice can be implemented in the framework allowing for testable predictions. The proposed framework can bring novel insights into the processes underlying intertemporal choices.

Automatic biases in intertemporal choice.

Dual process theories of intertemporal decision making propose that decision makers automatically favor immediate rewards. In this paper, we use a drift diffusion model to implement these theories, and empirically investigate the role of their proposed automatic biases. Our model permits automatic biases in the response process, in the form of a shifted starting point, as well as automatic biases in the evaluation process, in the form of an additive drift rate intercept. We fit our model to individual-level choice and response time data, and find that automatic biases (as measured though the starting point and drift rate intercept in our model) are prevalent in intertemporal choice, but that the type, magnitude, and direction of these biases vary greatly across individuals. Our results pose new challenges for theories of intertemporal choice behavior.

Assessing age-related multisensory enhancement with the time-window-of-integration model.

Although from multisensory research a great deal is known about how the different senses interact, there is little knowledge as to the impact of aging on these multisensory processes. In this study, we measured saccadic reaction time (SRT) of aged and young individuals to the onset of a visual target stimulus with and without an accessory auditory stimulus occurring (focused attention task). The response time pattern for both groups was similar: mean SRT to bimodal stimuli was generally shorter than to unimodal stimuli, and mean bimodal SRT was shorter when the auditory accessory was presented ipsilaterally rather than contralaterally to the target. The elderly participants were considerably slower than the younger participants under all conditions but showed a greater multisensory enhancement, that is, they seem to benefit more from bimodal stimulus presentation. In an attempt to weigh the contributions of peripheral sensory processes relative to more central cognitive processes possibly responsible for the difference in the younger and older adults, the time-window-of-integration (TWIN) model for crossmodal interaction in saccadic eye movements developed by the authors was fitted to the data from both groups. The model parameters suggest that (i) there is a slowing of the peripheral sensory processing in the elderly, (ii) as a result of this slowing, the probability of integration is smaller in the elderly even with a wider time-window-of-integration, and (iii) multisensory integration, if it occurs, manifests itself in larger neural enhancement in the elderly; however, because of (ii), on average the integration effect is not large enough to compensate for the peripheral slowing in the elderly.

Visual-auditory interaction in saccadic reaction time: effects of auditory masker level.

Saccadic reaction time (SRT) to a visual target tends to be shorter when auditory stimuli are presented in close temporal and spatial proximity, even when subjects are instructed to ignore the auditory non-target (focused attention paradigm). Observed SRT reductions typically range between 10 and 50 ms and decrease as spatial disparity between the stimuli increases. Previous studies using pairs of visual and auditory stimuli differing in both azimuth and vertical position suggest that the amount of SRT facilitation decreases not with the physical but with the perceivable distance between visual target and auditory accessory. Here we probe this hypothesis by presenting an additional white-noise masker background of 3 s duration. Increasing the masker level had a diametrical effect on SRTs in spatially coincident vs. disparate stimulus configurations: saccadic responses to coincident visual-auditory stimuli are slowed down, whereas saccadic responses to disparate stimuli are speeded up. As verified in a separate auditory localization task, localizability of the auditory accessory decreases with masker level. The SRT results are accounted for by a conceptual model positing that increasing masker level enlarges the area of possible auditory stimulus locations: it implies that perceivable distances decrease for disparate stimulus configurations and increase for coincident stimulus pairs.

Time course of auditory masker effects: tapping the locus of audiovisual integration?

In a focused attention paradigm, saccadic reaction time (SRT) to a visual target tends to be shorter when an auditory accessory stimulus is presented in close temporal and spatial proximity. Observed SRT reductions typically diminish as spatial disparity between the stimuli increases. Here a visual target LED (500 ms duration) was presented above or below the fixation point and a simultaneously presented auditory accessory (2 ms duration) could appear at the same or the opposite vertical position. SRT enhancement was about 35 ms in the coincident and 10 ms in the disparate condition. In order to further probe the audiovisual integration mechanism, in addition to the auditory non-target an auditory masker (200 ms duration) was presented before, simultaneous to, or after the accessory stimulus. In all interstimulus interval (ISI) conditions, SRT enhancement went down both in the coincident and disparate configuration, but this decrement was fairly stable across the ISI values. If multisensory integration solely relied on a feed-forward process, one would expect a monotonic decrease of the masker effect with increasing ISI in the backward masking condition. It is therefore conceivable that the relatively high-energetic masker causes a broad excitatory response of SC neurons. During this state, the spatial audio-visual information from multisensory association areas is fed back and merged with the spatially unspecific excitation pattern induced by the masker. Assuming that a certain threshold of activation has to be achieved in order to generate a saccade in the correct direction, the blurred joint output of noise and spatial audio-visual information needs more time to reach this threshold prolonging SRT to an audio-visual object.

Crossmodal interaction in saccadic reaction time: separating multisensory from warning effects in the time window of integration model.

In a focused attention task saccadic reaction time (SRT) to a visual target stimulus (LED) was measured with an auditory (white noise burst) or tactile (vibration applied to palm) non-target presented in ipsi- or contralateral position to the target. Crossmodal facilitation of SRT was observed under all configurations and stimulus onset asynchrony (SOA) values ranging from -500 (non-target prior to target) to 0 ms, but the effect was larger for ipsi- than for contralateral presentation within an SOA range from -200 ms to 0. The time-window-of-integration (TWIN) model (Colonius and Diederich in J Cogn Neurosci 16:1000, 2004) is extended here to separate the effect of a spatially unspecific warning effect of the non-target from a spatially specific and genuine multisensory integration effect.

Paradox resolved: Stop signal race model with negative dependence.

The ability to inhibit our responses voluntarily is an important case of cognitive control. The stop-signal paradigm is a popular tool to study response inhibition. Participants perform a response time task (go task), and occasionally, the go stimulus is followed by a stop signal after a variable delay, indicating subjects to withhold their response (stop task). The main interest of modeling is in estimating the unobservable stop-signal processing time, that is, the covert latency of the stopping process as a characterization of the response inhibition mechanism. In the independent race model, the stop-signal task is represented as a race between stochastically independent go and stop processes. Without making any specific distributional assumptions about the processing times, the model allows estimating the mean time to cancel a response. Neurophysiological studies on countermanding saccadic eye movements, however, have shown that the neural correlates of go and stop processes consist of networks of mutually interacting gaze-shifting and gaze-holding neurons. This poses a major challenge in formulating linking propositions between the behavioral and neural findings. Here we propose a dependent race model that postulates perfect negative stochastic dependence between go and stop activations. The model is consistent with the concept of interacting processes while retaining the simplicity and elegance of the distribution-free independent race model. For mean data, the dependent model's predictions remain identical to those of the independent model. The resolution of this apparent paradox advances the understanding of mechanisms of response inhibition and paves the way for modeling more complex situations. (PsycINFO Database Record (c) 2018 APA, all rights reserved).