Modal and amodal cognition: an overarching principle in various domains of psychology.

Accounting for how the human mind represents the internal and external world is a crucial feature of many theories of human cognition. Central to this question is the distinction between modal as opposed to amodal representational formats. It has often been assumed that one but not both of these two types of representations underlie processing in specific domains of cognition (e.g., perception, mental imagery, and language). However, in this paper, we suggest that both formats play a major role in most cognitive domains. We believe that a comprehensive theory of cognition requires a solid understanding of these representational formats and their functional roles within and across different domains of cognition, the developmental trajectory of these representational formats, and their role in dysfunctional behavior. Here we sketch such an overarching perspective that brings together research from diverse subdisciplines of psychology on modal and amodal representational formats so as to unravel their functional principles and their interactions.

On the time-course of automatic response activation in the Simon task.

The Simon effect (prolonged RT when the task-irrelevant stimulus location is incongruent with the response side) has been reported to decrease at longer RTs, which is reflected in negative-going delta functions. This finding has been attributed to gradual dissipation of the response automatically activated by the task-irrelevant location information. The Diffusion Model for Conflict Tasks (DMC, Ulrich, Schröter, Leuthold, & Birngruber, Cognitive Psychology 78:148-174, 2015) formally specifies the time-course of this automatic activation process as a pulse-like function. In contrast to alternative views, DMC is consistent with the notion that this time-course is unaffected by the presentation duration of the target stimulus. Therefore, we expected that delta functions are invariant against changes of stimulus duration. This prediction was verified in two Simon task experiments. Consistent with this general result, DMC's parameter τ which defines the time-course of the automatic response activation was estimated to not meaningfully differ between short and long durations. We argue that our results are coherent with processing architectures that assume a transient automatic process that is virtually unaffected by stimulus duration.

Order effects in stimulus discrimination challenge established models of comparative judgement: A meta-analytic review of the Type B effect.

This paper provides a comprehensive review of the Type B effect (TBE), a phenomenon reflected in the observation that discrimination sensitivity varies with the order of stimuli in comparative judgment tasks, such as the two-alternative forced-choice (2AFC) paradigm. Specifically, when the difference threshold is lower (higher) with the constant standard preceding rather than following the variable comparison, one speaks of a negative (positive) TBE. Importantly, prominent psychophysical difference models such as signal detection theory (Green & Swets, 1966) cannot easily account for the TBE, and are hence challenged by it. The present meta-analysis provides substantial evidence for the TBE across various stimulus attributes, suggesting that the TBE is a general feature of discrimination experiments when standard and comparison are presented successively. Thus, inconsistent with psychophysical difference models, subjective differences between stimuli are not merely a function of their physical differences but rather also depend on their temporal order. From the literature, we identify four classes of potential candidate theories explaining the origin of the TBE, namely (1) differential weighting of the stimulus magnitudes at the two positions (e.g., Hellström, Psychological Research, 39, 345-388 1977), (2) internal reference formation (e.g., Dyjas, Bausenhart, & Ulrich, Attention, Perception, & Psychophysics, 74, 1819-1841 2012), (3) Bayesian updating (e.g., de Jong, Akyürek, & van Rijn, Psychonomic Bulletin and Review, 28, 1183-1190 2021), and (4) biased threshold estimation (García-Pérez & Alcalá-Quintana, Attention, Perception & Psychophysics, 72, 1155-1178 2010). As these models, to some extent, make differential predictions about the direction of the TBE, investigating the respective boundary conditions of positive and negative TBEs might be a valuable perspective for diagnostic future research.

Association between abstraction level and time: Are future and past more abstract than the present?

Construal level theory suggests that objects or events are represented differently depending on their psychological distance from ourselves. Specifically, objects and events should be represented more abstractly the farther they are removed from direct experience through distance in the spatial, temporal, social, or hypotheticality domains. Bar-Anan et al. reported a key finding supporting this assumed association of the various distance dimensions and abstraction level. In their study, participants responded faster in an Implicit Association Task when temporally near and concrete concepts, as well as temporally far and abstract concepts, were mapped to the same rather than different response keys. In this study, we conceptually replicated this basic finding when employing temporal adverbs relating to present versus future time, and nouns referring to concrete versus abstract concepts (Experiment 1). Evidence for such an association, however, was largely absent (and significantly weaker than in Experiment 1) when temporal adverbs relating to the past were employed as instances of the large temporal distance category (Experiment 2). We propose that the uncertainty associated with the future, as opposed to the past, might play an important role in this temporal asymmetry by increasing psychological distance.

Estimating discrimination performance in two-alternative forced choice tasks: routines for MATLAB and R.

Ulrich and Vorberg (Attention, Perception, & Psychophysics 71: 1219-1227, 2009) introduced a novel approach for estimating discrimination performance in two-alternative forced choice (2AFC) tasks. This approach avoids pitfalls that are inherent when the order of the standard and the comparison is neglected in estimating the difference limen (DL), as in traditional approaches. The present article provides MATLAB and R routines that implement this novel procedure for estimating DLs. These routines also allow to account for processing failures such as lapses or finger errors and can be applied to experimental designs in which the standard and comparison differ only along the task-relevant dimension, as well as to designs in which the stimuli differ in more than one dimension. In addition, Monte Carlo simulations were conducted to check the quality of our routines.

Effects of conflict trial proportion: A comparison of the Eriksen and Simon tasks.

Two experiments examined global and local behavioral adaptation effects within and across the Eriksen task, where conflict is based on stimulus letter identities, and the Simon task, where conflict is based on stimulus and response locations. Trials of the two tasks were randomly intermixed, and the list-wide proportion of congruent trials was varied in both tasks (Experiment 1) or in just one task (Experiment 2). The global adaptation effect of list-wide congruency proportion (LWPC effect) was at least as large in the Simon task as in the Eriksen task. Likewise, the local adaptation effect of previous-trial congruency (Gratton effect) was at least as large in the Simon task as in the Eriksen task. In contrast to prior studies investigating transfer across Stroop and Simon tasks, there was no dissociation between global and local adaptation effects regarding their transfer across the different conflict tasks. In fact, both local and global adaptation effects appeared largely task-specific, because there was no or only little transfer of either Gratton effects or LWPC effects from the Eriksen to the Simon task or vice versa. On the whole, the results suggest that behavioral adaptation observed in the present design does not carry over from one of these tasks to the other, suggesting no involvement of a higher-order, task-general mechanism of cognitive control.

Decay of internal reference information in duration discrimination: Intertrial interval modulates the Type B effect.

Psychophysical evidence suggests that human perception of a stimulus is assimilated towards previous stimuli. The internal reference model (IRM) explains such assimilation through an internal reference (IR), which integrates past and present stimulus representations and thus might be conceived as a form of perceptual memory. In this study, we investigated whether the IR decays with time, as previously shown for perceptual memory representations in general. One specific prediction of IRM is higher discrimination sensitivity when a constant standard precedes rather than follows a variable comparison in a two-alternative forced-choice (2AFC) discrimination task. Furthermore, the magnitude of this so-called negative Type B effect should decrease with decreasing weighting of past stimulus information in the integration process. Therefore, decay of the IR should result in a reduced Type B effect. To examine this prediction, we carried out a 2AFC duration discrimination experiment with a short (1,600 ms) and a long (3,200 ms) intertrial interval (ITI). As expected, a reduced negative Type B effect was observed at the long compared with the short ITI, consistent with the idea that humans rely on the immediate past when evaluating current sensory input, however, less so when the IR incorporating the perceptual short-term memory representation of these past stimuli has already decayed.

Effects of stimulus order on comparative judgments across stimulus attributes and sensory modalities.

In psychophysical experiments, participants are often asked to compare the magnitude of a constant standard against the magnitude of a variable comparison. According to prominent models of stimulus discrimination, discrimination sensitivity should depend only on the physical magnitude difference between these two stimuli but not on the order of their presentation. However, previous experiments on auditory duration discrimination have shown that discrimination sensitivity is higher when the standard precedes rather than follows the comparison. It is presently unclear whether this Type B effect emerges only for duration discrimination or generalizes across modalities and stimulus attributes. Therefore, we conducted a study in which participants performed several discrimination tasks for various stimulus attributes (i.e., duration, frequency, intensity, and numerosity), each in the visual and in the auditory modality. In all cases, discrimination sensitivity was higher when the standard preceded rather than followed the comparison. This result indicates that the Type B effect is not restricted to the domain of temporal cognition but rather reflects a general phenomenon across a range of domains and modalities. The outcome of the present experiment is consistent with the internal reference model according to which the Type B effect is a consequence of a dynamically updated internal reference, which is used in the comparison process. Alternatively, a weighted difference model with a larger weight for the second stimulus position than for the first stimulus position can also account for this result. (PsycINFO Database Record

On the interplay of visuospatial and audiotemporal dominance: Evidence from a multimodal kappa effect.

When participants judge multimodal audiovisual stimuli, the auditory information strongly dominates temporal judgments, whereas the visual information dominates spatial judgments. However, temporal judgments are not independent of spatial features. For example, in the kappa effect, the time interval between two marker stimuli appears longer when they originate from spatially distant sources rather than from the same source. We investigated the kappa effect for auditory markers presented with accompanying irrelevant visual stimuli. The spatial sources of the markers were varied such that they were either congruent or incongruent across modalities. In two experiments, we demonstrated that the spatial layout of the visual stimuli affected perceived auditory interval duration. This effect occurred although the visual stimuli were designated to be task-irrelevant for the duration reproduction task in Experiment 1, and even when the visual stimuli did not contain sufficient temporal information to perform a two-interval comparison task in Experiment 2. We conclude that the visual and auditory marker stimuli were integrated into a combined multisensory percept containing temporal as well as task-irrelevant spatial aspects of the stimulation. Through this multisensory integration process, visuospatial information affected even temporal judgments, which are typically dominated by the auditory modality.

Enhancement of letter identification by concurrent auditory stimuli of varying duration.

Previously it has been shown that the concurrent presentation of a sound can improve processing of visual information at higher perceptual levels, for example, in letter identification tasks. Moreover, increasing the duration of the concurrent sounds can enhance performance in low-level tasks as contrast detection, which has been attributed to a sustained visual activation corresponding to the duration of the sound. Yet, the role of sound duration has so far not been investigated in higher-level visual processing. In a series of five Experiments, we again demonstrated that the mere presence of a concurrent sound can enhance the identification of a masked, centrally presented letter compared to unimodal presentation, even though this benefit was absent in one experiment for high-contrast letters yielding an especially high level of task-performance. In general, however, the sound-induced benefit was not modulated by a variation of target contrast or by the duration of the target-to-mask interstimulus interval. Taking individual performance differences into account, a further analysis suggested that the sound-induced facilitation effect may nevertheless be most pronounced at specific performance levels. Beyond this general sound-induced facilitation, letter identification performance was not further affected by the duration of the concurrent sounds, even though in a control experiment it could be established that letter identification performance improved with increasing letter duration, and perceived letter duration was prolonged with increasing auditory duration. The results and their interpretation with respect to the large observed interindividual performance differences are discussed in terms of potential underlying mechanisms of multisensory facilitation, as preparedness enhancement, signal enhancement, and object enhancement.

Still no Evidence for Sustained Effects of Multisensory Integration of Duration.

In studies on temporal order perception, immediate as well as sustained effects of multisensory integration have been demonstrated repeatedly. Regarding duration perception, the corresponding literature reports clear immediate effects of multisensory integration, but evidence on sustained effects of multisensory duration integration is scarce. In fact, a single study [Heron, J. et al. (2013). A neural hierarchy for illusions of time: Duration adaptation precedes multisensory integration, J. Vis. 13, 1-12.] investigated adaptation to multisensory conflicting intervals, and found no sustained effects of the audiovisual conflict on perceived duration of subsequently presented unimodal visual intervals. In two experiments, we provide independent evidence in support of this finding. In Experiment 1, we demonstrate that adaptation to audiovisual conflict does not alter perceived duration of subsequently presented visual test intervals. Thus, replicating the results of Heron et al. (2013), we observed no sustained effect of multisensory duration integration. However, one might argue that the prolonged exposure to consistent multisensory conflict might have prevented or hampered multisensory integration per se. In Experiment 2, we rule out this alternative explanation by showing that multisensory integration of audiovisual conflicting intervals is still effective after exposure to audiovisual conflict. This further strengthens the conclusion that multisensory integration of interval duration affects perception in an immediate, but not in a sustained manner.

Attention delays perceived stimulus offset.

Yeshurun and Levy (2003) [Transient spatial attention degrades temporal resolution. Psychological Science, 14, 225 -231.] have suggested that visual attention enhances the activation of the parvocellular system and thus delays the perceived offset of a stimulus. We tested this assumption in a spatial cueing paradigm in which participants responded to stimulus offset. Consistent with this assumption, offset reaction time (RT) was prolonged for attended compared to unattended stimuli. For onset RT, however, we confirmed the well-known spatial cueing effect that attention speeds up the detection of stimulus onset. The results provide direct evidence for the proposal made by Yeshurun and Levy.

The locus of temporal preparation effects: evidence from the psychological refractory period paradigm.

In reaction time (RT) tasks, responses are especially fast when participants can anticipate the onset of an imperative response signal. Although this RT facilitation is commonly attributed to temporal preparation, it is unclear whether this preparation shortens the duration of early or late processes. We used the effect propagation property of the psychological refractory period paradigm to localize the effect of temporal preparation. Manipulation of temporal uncertainty affected the RT of Task 1, regardless of the level of stimulus onset asynchrony (SOA). Consistent with the prediction of an early locus of temporal preparation, this effect propagated completely to the RT of Task 2 at short SOAs, but propagation diminished virtually to zero at long SOAs.

Representations of temporal information in short-term memory: Are they modality-specific?

Rattat and Picard (2012) reported that the coding of temporal information in short-term memory is modality-specific, that is, temporal information received via the visual (auditory) modality is stored as a visual (auditory) code. This conclusion was supported by modality-specific interference effects on visual and auditory duration discrimination, which were induced by secondary tasks (visual tracking or articulatory suppression), presented during a retention interval. The present study assessed the stability of these modality-specific interference effects. Our study did not replicate the selective interference pattern but rather indicated that articulatory suppression not only impairs short-term memory for auditory but also for visual durations. This result pattern supports a crossmodal or an abstract view of temporal encoding.

Effects of stimulus order on discrimination sensitivity for short and long durations.

Previous studies have shown that discrimination sensitivity in 2AFC tasks depends on the presentation order of the standard and comparison stimulus. The present study examined whether this so-called Type B effect generalizes across different standard magnitudes. Therefore, Experiment 1 employed an auditory duration discrimination task with short (100 ms) and long (1,000 ms) standard durations and a constant interstimulus interval (ISI) of 1,000 ms. For both standard durations, a clear Type B effect emerged. In Experiment 2, discrimination sensitivity was assessed for short (300 ms) and long (1,000 ms) ISIs and a constant standard duration of 100 ms, in order to examine whether the Type B effect diminishes or even reverses when both stimuli are presented in rapid succession, as was suggested by previous studies. In the short, but not the long ISI condition, the Type B effect was virtually eliminated. Taken together, the present experiments suggest that the Type B effect is robust across standard magnitude, but diminishes when the time interval between both stimuli is reduced. This result pattern is discussed within the framework of the Internal Reference Model and the Sensation Weighting Model. It is also demonstrated that both models provide a quantitative account of the present results.

Effects of stimulus order on duration discrimination sensitivity are under attentional control.

When participants are asked to discriminate between a fixed-magnitude standard stimulus and a variable comparison stimulus, discrimination performance is typically better when the standard precedes rather than follows the comparison. To date it is unclear whether this Type B effect is the sign of an automatic process or whether it is under participants' control. In a series of three experiments, participants compared the duration of two successively presented intervals. At the beginning of each trial, a symbolic cue either indicated the temporal position of the comparison validly or it was neutral with respect to comparison position. With neutral cues, a strong Type B effect was observed in all experiments. With valid cues, however, this effect was reduced. This pattern of results suggests that the Type B effect is, at least in part, under participants' control. Recently, we have shown that the Type B effect might be due to the formation of an internal reference that is dynamically updated from trial to trial. Within this framework, the attention-based reduction of the Type B effect might be explained by a flexible weighting mechanism that regulates the level of integration of remote stimulus information.

Temporal reproductions are influenced by an internal reference: explaining the Vierordt effect.

Several findings from duration perception literature suggest that when making decisions about time, participants rely on an internal reference memory for time rather than merely on the current physical stimuli. According to a recent account, such an internal reference is formed by a continuous dynamic updating process that integrates duration information from previous trials and the current trial. In the present work, we show how such a dynamic mechanism can account for the classical yet unresolved Vierordt effect, which refers to the overestimation of relatively short and the underestimation of relatively long temporal intervals. We conducted an experiment to examine this and related predictions by means of a temporal reproduction task. Specifically, participants were presented with two successive time intervals - a standard s with constant duration and a comparison c with variable duration. Instead of performing a comparison judgment, however, the participants were subsequently cued to reproduce one of the two presented stimuli. Reproductions were affected not only by the temporal position of the to-be-reproduced stimulus, but also by the stimuli presented on earlier trials. These results support the notion of a dynamically updated internal reference underlying our judgments about the time elapsed, which might also be the basis of the Vierordt effect.

Multimodal integration of time.

Recent studies suggest that the accuracy of duration discrimination for visually presented intervals is strongly impaired by concurrently presented auditory intervals of different duration, but not vice versa. Because these studies rely mostly on accuracy measures, it remains unclear whether this impairment results from changes in perceived duration or rather from a decrease in perceptual sensitivity. We therefore assessed complete psychometric functions in a duration discrimination task to disentangle effects on perceived duration and sensitivity. Specifically, participants compared two empty intervals marked by either visual or auditory pulses. These pulses were either presented unimodally, or accompanied by task-irrelevant pulses in the respective other modality, which defined conflicting intervals of identical, shorter, or longer duration. Participants were instructed to base their temporal judgments solely on the task-relevant modality. Despite this instruction, perceived duration was clearly biased toward the duration of the intervals marked in the task-irrelevant modality. This was not only found for the discrimination of visual intervals, but also, to a lesser extent, for the discrimination of auditory intervals. Discrimination sensitivity, however, was similar between all multimodal conditions, and only improved compared to the presentation of unimodal visual intervals. In a second experiment, evidence for multisensory integration was even found when the task-irrelevant modality did not contain any duration information, thus excluding noncompliant attention allocation as a basis of our results. Our results thus suggest that audiovisual integration of temporally discrepant signals does not impair discrimination sensitivity but rather alters perceived duration, presumably by means of a temporal ventriloquism effect.

Trial-by-trial updating of an internal reference in discrimination tasks: evidence from effects of stimulus order and trial sequence.

In psychophysics, participants are often asked to discriminate between a constant standard and a variable comparison. Previous studies have shown that discrimination performance is better when the comparison follows, rather than precedes, the standard. Prominent difference models of psychophysics and decision making cannot easily explain this order effect. However, a simple extension of this model class involving dynamical updating of an internal reference accounts for this order effect. In addition, this Internal Reference Model (IRM) predicts sequential response effects. We examined the predictions of IRM in two duration discrimination experiments. The obtained results are in agreement with the predictions of IRM, suggesting that participants update their internal reference on every trial. Additional simulations show that IRM also accounts for the negative sequential effects observed in single-stimulus paradigms.

Does temporal preparation increase the rate of sensory information accumulation?

Recent studies showed that temporal preparation, i.e., the ability to prepare for an upcoming stimulus, improves perceptual processing. The mechanisms underlying this benefit are still controversial. Based upon the theoretical framework of accumulation models, it has been proposed that the accumulation of sensory stimulus information begins earlier when participants are temporally prepared than when they are unprepared. Alternatively, however, temporal preparation might also affect the accumulation rate of sensory information. In the present study, we examined these possibilities. Specifically, in three experiments, we manipulated participants' decision criterion. This manipulation should interact with any experimental manipulation affecting the rate of information processing, but produce additive effects with any manipulation affecting the onset of information accumulation rather than its rate. We obtained additive effects on RT, irrespective of whether the decision criterion was manipulated by increasing catch trial proportion or nogo trial proportion. These results suggest that temporal preparation improves perceptual processing by operating on the onset of sensory information accumulation rather than the rate of sensory information accumulation.