Trial-by-trial predictions of subjective time from human brain activity.

Human experience of time exhibits systematic, context-dependent deviations from clock time; for example, time is experienced differently at work than on holiday. Here we test the proposal that differences from clock time in subjective experience of time arise because time estimates are constructed by accumulating the same quantity that guides perception: salient events. Healthy human participants watched naturalistic, silent videos of up to 24 seconds in duration and estimated their duration while fMRI was acquired. We were able to reconstruct trial-by-trial biases in participants' duration reports, which reflect subjective experience of duration, purely from salient events in their visual cortex BOLD activity. By contrast, salient events in neither of two control regions-auditory and somatosensory cortex-were predictive of duration biases. These results held despite being able to (trivially) predict clock time from all three brain areas. Our results reveal that the information arising during perceptual processing of a dynamic environment provides a sufficient basis for reconstructing human subjective time duration.

The Neural Representation of Events Is Dominated by Elements that Are Most Reliably Present.

An episodic memory is specific to an event that occurred at a particular time and place. However, the elements that constitute the event-the location, the people present, and their actions and goals-might be shared with numerous other similar events. Does the brain preferentially represent certain elements of a remembered event? If so, which elements dominate its neural representation: those that are shared across similar events, or the novel elements that define a specific event? We addressed these questions by using a novel experimental paradigm combined with fMRI. Multiple events were created involving conversations between two individuals using the format of a television chat show. Chat show "hosts" occurred repeatedly across multiple events, whereas the "guests" were unique to only one event. Before learning the conversations, participants were scanned while viewing images or names of the (famous) individuals to be used in the study to obtain person-specific activity patterns. After learning all the conversations over a week, participants were scanned for a second time while they recalled each event multiple times. We found that during recall, person-specific activity patterns within the posterior midline network were reinstated for the hosts of the shows but not the guests, and that reinstatement of the hosts was significantly stronger than the reinstatement of the guests. These findings demonstrate that it is the more generic, familiar, and predictable elements of an event that dominate its neural representation compared with the more idiosyncratic, event-defining, elements.

A Predictive Processing Model of Episodic Memory and Time Perception.

Human perception and experience of time are strongly influenced by ongoing stimulation, memory of past experiences, and required task context. When paying attention to time, time experience seems to expand; when distracted, it seems to contract. When considering time based on memory, the experience may be different than what is in the moment, exemplified by sayings like "time flies when you're having fun." Experience of time also depends on the content of perceptual experience-rapidly changing or complex perceptual scenes seem longer in duration than less dynamic ones. The complexity of interactions among attention, memory, and perceptual stimulation is a likely reason that an overarching theory of time perception has been difficult to achieve. Here, we introduce a model of perceptual processing and episodic memory that makes use of hierarchical predictive coding, short-term plasticity, spatiotemporal attention, and episodic memory formation and recall, and apply this model to the problem of human time perception. In an experiment with approximately 13,000 human participants, we investigated the effects of memory, cognitive load, and stimulus content on duration reports of dynamic natural scenes up to about 1 minute long. Using our model to generate duration estimates, we compared human and model performance. Model-based estimates replicated key qualitative biases, including differences by cognitive load (attention), scene type (stimulation), and whether the judgment was made based on current or remembered experience (memory). Our work provides a comprehensive model of human time perception and a foundation for exploring the computational basis of episodic memory within a hierarchical predictive coding framework.

Intentional Binding Without Intentional Action.

The experience of authorship over one's actions and their consequences-sense of agency-is a fundamental aspect of conscious experience. In recent years, it has become common to use intentional binding as an implicit measure of the sense of agency. However, it remains contentious whether reported intentional-binding effects indicate the role of intention-related information in perception or merely represent a strong case of multisensory causal binding. Here, we used a novel virtual-reality setup to demonstrate identical magnitude-binding effects in both the presence and complete absence of intentional action, when perceptual stimuli were matched for temporal and spatial information. Our results demonstrate that intentional-binding-like effects are most simply accounted for by multisensory causal binding without necessarily being related to intention or agency. Future studies that relate binding effects to agency must provide evidence for effects beyond that expected for multisensory causal binding by itself.

Serial dependence in the perception of visual variance.

The recent history of perceptual experience has been shown to influence subsequent perception. Classically, this dependence on perceptual history has been examined in sensory-adaptation paradigms, wherein prolonged exposure to a particular stimulus (e.g., a vertically oriented grating) produces changes in perception of subsequently presented stimuli (e.g., the tilt aftereffect). More recently, several studies have investigated the influence of shorter perceptual exposure with effects, referred to as serial dependence, being described for a variety of low- and high-level perceptual dimensions. In this study, we examined serial dependence in the processing of dispersion statistics, namely variance-a key descriptor of the environment and indicative of the precision and reliability of ensemble representations. We found two opposite serial dependences operating at different timescales, and likely originating at different processing levels: A positive, Bayesian-like bias was driven by the most recent exposures, dependent on feature-specific decision making and appearing only when high confidence was placed in that decision; and a longer lasting negative bias-akin to an adaptation aftereffect-becoming manifest as the positive bias declined. Both effects were independent of spatial presentation location and the similarity of other close traits, such as mean direction of the visual variance stimulus. These findings suggest that visual variance processing occurs in high-level areas but is also subject to a combination of multilevel mechanisms balancing perceptual stability and sensitivity, as with many different perceptual dimensions.

Sensory adaptation for timing perception.

Recent sensory experience modifies subjective timing perception. For example, when visual events repeatedly lead auditory events, such as when the sound and video tracks of a movie are out of sync, subsequent vision-leads-audio presentations are reported as more simultaneous. This phenomenon could provide insights into the fundamental problem of how timing is represented in the brain, but the underlying mechanisms are poorly understood. Here, we show that the effect of recent experience on timing perception is not just subjective; recent sensory experience also modifies relative timing discrimination. This result indicates that recent sensory history alters the encoding of relative timing in sensory areas, excluding explanations of the subjective phenomenon based only on decision-level changes. The pattern of changes in timing discrimination suggests the existence of two sensory components, similar to those previously reported for visual spatial attributes: a lateral shift in the nonlinear transducer that maps relative timing into perceptual relative timing and an increase in transducer slope around the exposed timing. The existence of these components would suggest that previous explanations of how recent experience may change the sensory encoding of timing, such as changes in sensory latencies or simple implementations of neural population codes, cannot account for the effect of sensory adaptation on timing perception.

The sense of agency is action-effect causality perception based on cross-modal grouping.

Sense of agency, the experience of controlling external events through one's actions, stems from contiguity between action- and effect-related signals. Here we show that human observers link their action- and effect-related signals using a computational principle common to cross-modal sensory grouping. We first report that the detection of a delay between tactile and visual stimuli is enhanced when both stimuli are synchronized with separate auditory stimuli (experiment 1). This occurs because the synchronized auditory stimuli hinder the potential grouping between tactile and visual stimuli. We subsequently demonstrate an analogous effect on observers' key press as an action and a sensory event. This change is associated with a modulation in sense of agency; namely, sense of agency, as evaluated by apparent compressions of action-effect intervals (intentional binding) or subjective causality ratings, is impaired when both participant's action and its putative visual effect events are synchronized with auditory tones (experiments 2 and 3). Moreover, a similar role of action-effect grouping in determining sense of agency is demonstrated when the additional signal is presented in the modality identical to an effect event (experiment 4). These results are consistent with the view that sense of agency is the result of general processes of causal perception and that cross-modal grouping plays a central role in these processes.

Direction of visual apparent motion driven by perceptual organization of cross-modal signals.

A critical function of the human brain is to determine the relationship between sensory signals. In the case of signals originating from different sensory modalities, such as audition and vision, several processes have been proposed that may facilitate perception of correspondence between two signals despite any temporal discrepancies in physical or neural transmission. One proposal, temporal ventriloquism, suggests that audio-visual temporal discrepancies can be resolved with a capture of visual event timing by that of nearby auditory events. Such an account implies a fundamental change in the timing representations of the involved events. Here we examine if such changes are necessary to account for a recently demonstrated effect, the modulation of visual apparent motion direction by audition. By contrast, we propose that the effect is driven by segmentation of the visual sequence on the basis of perceptual organization in the cross-modal sequence. Using different sequences of cross-modal (auditory and tactile) events, we found that the direction of visual apparent motion was not consistent with a temporal capture explanation. Rather, reports of visual apparent motion direction were dictated by perceptual organization within cross-modal sequences, determined on the basis of apparent relatedness. This result adds to the growing literature indicating the importance of apparent relatedness and sequence segmentation in apparent timing. Moreover, it demonstrates that, contrary to previous findings, cross-modal interaction can play a critical role in determining organization of signals within a single sensory modality.

The best fitting of three contemporary observer models reveals how participants' strategy influences the window of subjective synchrony.

When experimenters vary the timing between two intersensory events, and participants judge their simultaneity, an inverse-U-shaped psychometric function is obtained. Typically, this simultaneity function is first fitted with a model for each participant separately, before best-fitting parameters are utilized (e.g., compared across conditions) in the second stage of a two-step inferential procedure. Often, simultaneity-function width is interpreted as representing sensitivity to asynchrony, and/or ascribed theoretical equivalence to a window of multisensory temporal binding. Here, we instead fit a single (principled) multilevel model to data from the entire group and across several conditions at once. By asking 20 participants to sometimes be more conservative in their judgments, we demonstrate how the width of the simultaneity function is prone to strategic change and thus questionable as a measure of either sensitivity to asynchrony or multisensory binding. By repeating our analysis with three different models (two implying a decision based directly on subjective asynchrony, and a third deriving this decision from the correlation between filtered responses to sensory inputs) we find that the first model, which hypothesizes, in particular, Gaussian latency noise and difficulty maintaining the stability of decision criteria across trials, is most plausible for these data. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Twice upon a time: multiple concurrent temporal recalibrations of audiovisual speech.

Audiovisual timing perception can recalibrate following prolonged exposure to asynchronous auditory and visual inputs. It has been suggested that this might contribute to achieving perceptual synchrony for auditory and visual signals despite differences in physical and neural signal times for sight and sound. However, given that people can be concurrently exposed to multiple audiovisual stimuli with variable neural signal times, a mechanism that recalibrates all audiovisual timing percepts to a single timing relationship could be dysfunctional. In the experiments reported here, we showed that audiovisual temporal recalibration can be specific for particular audiovisual pairings. Participants were shown alternating movies of male and female actors containing positive and negative temporal asynchronies between the auditory and visual streams. We found that audiovisual synchrony estimates for each actor were shifted toward the preceding audiovisual timing relationship for that actor and that such temporal recalibrations occurred in positive and negative directions concurrently. Our results show that humans can form multiple concurrent estimates of appropriate timing for audiovisual synchrony.

Spatiotemporal rivalry: a perceptual conflict involving illusory moving and static forms.

In human vision, mechanisms specialized for encoding static form can signal the presence of blurred forms trailing behind moving objects. People are typically unaware of these motion-blur signals because other mechanisms signal sharply defined moving forms. When active, these mechanisms can suppress awareness of motion blur. Thus, although discrepant form signals can be produced, human vision usually settles on a single coherent perceptual outcome. Here we report a dramatic exception. We found that, in some circumstances, static motion-blur form signals and moving-form signals can engage in a dynamic competition for perceptual dominance. We refer to the phenomenon as spatiotemporal rivalry (STR). Our data confirm that moving- and static-form mechanisms can generate independent signals, each of which can intermittently dominate perception. STR could therefore be exploited to investigate how these mechanisms contribute to determining the content of visual awareness.

Binocular rivalry: spreading dominance through complex images.

When different images are presented to the two eyes, each can intermittently disappear, leaving the other to dominate perception. This is called binocular rivalry. When using radial gratings, focal contrast increments can trigger a traveling wave of perceptual dominance change, originating at the locus of the contrast increment and circling the stimulus. This has been linked to a sweep of activity through V1 that can be traced via fMRI. The dominance of more complex images, like human faces, has been linked to higher level processing structures characterized by more holistic object centered properties. We therefore decided to assess how dominance would spread through more complex images. Using Kanisza squares and human faces we found that dominance tended to spread gradually away from the locus of the contrast increment, often along real or illusory contours. We also found that perceptual dominance was slow to spread between facial regions encoded by different monocular channels. These data are consistent with low-level monocular mechanisms, like those found in V1, playing a determinant role in the spread of perceptual dominance through complex images during binocular rivalry.

Simple differential latencies modulate, but do not cause the flash-lag effect.

When a brief static flash is presented in physical alignment with a moving target, the position of the flash can seem to lag behind that of the moving target. Various explanations exist. One of the most popular is the differential latency hypothesis. This assumes that neural latencies for moving stimuli are shorter than those for static stimuli. Accordingly, the flash lag would occur because perception is temporally fragmented, with moving stimuli seen before static. To test this, observers were asked to make different judgments concerning the same stimulus. Observers either judged if color changes in moving and static stimulus sections were synchronous, or whether the different stimulus sections were aligned at the time of a color change. If the flash-lag were driven by a simple differential latency, we would expect both judgments to be marked by a temporal advantage for moving stimuli. Our results suggest the contrary, as only the later judgment was marked by a flash-lag effect. The apparent timing of moving and static color changes was veridical. However, when we introduced a systematic differential latency, by modulating image contrast, both judgments were affected. Our data therefore suggest that a simple differential can modulate flash-lag type effects, but they do not cause the phenomenon.

The sliding window of audio-visual simultaneity.

Humans exist in an environment wherein many unrelated events occur in close spatial and temporal proximity. Audio-visual timing experiments, however, have often examined only isolated pairs of sensory events. We therefore decided to assess how audio-visual timing perception would be shaped by the presence of an additional audio or visual event. We found that the point of subjective synchrony for a sensory event can be shifted away from the presence of other temporally proximate events. These interactions made audio-visual pairs seem unrelated, or asynchronous, at timings at which they had seemed synchronous when presented in isolation. This shows that the interval across which humans are insensitive to audio-visual asynchrony is not fixed, but dynamic, shaped by interactions between multiple sensory events. Importantly, we establish that these interactions can enhance the sensitivity of timing judgments. These interactions could therefore help to segregate unrelated sensory events across time. Such effects are likely to be common in the cluttered environments in which humans exist.

Serial dependence in timing perception.

Recent sensory history affects subsequent experience. Behavioral results have demonstrated this effect in two forms: repeated exposure to the same sensory input produces negative aftereffects wherein sensory stimuli like those previously experienced are judged as less like the exposed stimulation, while singular exposures can produce positive aftereffects wherein judgments are more like previously experienced stimulation. For timing perception, there is controversy regarding the influence of recent exposure-both singular and repeated exposure produce apparently negative aftereffects-often referred to as temporal recalibration and rapid temporal recalibration, respectively. While negative aftereffects have been found following repeated exposure for all timing tasks, following a single exposure, they have only been demonstrated using synchrony judgments (SJs). Here, we examine the influence of a single presentation-serial dependence for timing-for standard timing tasks: SJ, temporal order judgments, and magnitude estimation judgments. We found that serial dependence produced apparently negative aftereffects in SJ, but positive aftereffects in temporal order judgment and magnitude estimation judgment. We propose that these findings, and those following repeated exposure, can be reconciled within a framework wherein negative aftereffects occur at sensory layers, consistent with classical depictions of sensory adaptation, and Bayesian-like positive aftereffects operate across different, higher, decision levels. These findings are consistent with the aftereffects known from other perceptual dimensions and provide a general framework for interpreting positive (serial dependence) and negative (sensory adaptation) aftereffects across different tasks. (PsycINFO Database Record (c) 2018 APA, all rights reserved).

Thermal-Tactile Integration in Object Temperature Perception.

The brain consistently faces a challenge of whether and how to combine the available information sources to estimate the properties of an object explored by hand. While object perception is an inference process involving multisensory inputs, thermal referral (TR) is an illusion demonstrating how the interaction between thermal and tactile systems can lead to deviations from physical reality-when observers touch three stimulators simultaneously with the middle three fingers of one hand but only the outer two stimulators are heated (or cooled), thermal uniformity is perceived across three fingers. Here, we used TR of warmth to examine the thermal-tactile interaction in object temperature perception. We show that TR is consistent with precision-weighted averaging of thermal sensation across tactile locations. Furthermore, we show that prolonged contact with TR stimulation results in adaptation to the local variations of veridical temperatures instead of the thermal uniformity perceived across three fingers. Our results illuminate the flexibility of processing that underlies thermal-tactile interactions and serve as a basis for thermal display design.

Activity in perceptual classification networks as a basis for human subjective time perception.

Despite being a fundamental dimension of experience, how the human brain generates the perception of time remains unknown. Here, we provide a novel explanation for how human time perception might be accomplished, based on non-temporal perceptual classification processes. To demonstrate this proposal, we build an artificial neural system centred on a feed-forward image classification network, functionally similar to human visual processing. In this system, input videos of natural scenes drive changes in network activation, and accumulation of salient changes in activation are used to estimate duration. Estimates produced by this system match human reports made about the same videos, replicating key qualitative biases, including differentiating between scenes of walking around a busy city or sitting in a cafe or office. Our approach provides a working model of duration perception from stimulus to estimation and presents a new direction for examining the foundations of this central aspect of human experience.

Intentional binding as Bayesian cue combination: Testing predictions with trait individual differences.

We investigated differences in intentional binding in high and low hypnotizable groups to explore two questions relating to (a) trait differences in the availability of motor intentions to metacognitive processes and (b) a proposed cue combination model of binding. An experience of involuntariness is central to hypnotic responding and may arise from strategically being unaware of one's intentions. Trait differences in the ability to respond to hypnotic suggestion may reflect differing levels of access to motor intentions. Intentional binding refers to the subjective compression of the time between an action and its outcome, indicated by a forward shift in the judged time of an action toward its outcome (action binding) and the backward shift of an outcome toward a causal action (outcome binding). Intentional binding is sensitive to intentional action without requiring explicit reflection upon agency. One way of explaining the sensitivity of intentional binding is to see it as a simple case of multisensory cue combination in which awareness of intentions increases knowledge of the timing of actions. Here we present results consistent with such a mechanism. In a contingent presentation of action and outcome events, low hypnotizable had more precise timing judgments of actions and also showed weaker action binding than highs. These results support the theory that trait hypnotizability is related to access to information related to motor intentions, and that intentional binding reflects the Bayesian combination of cross-modal cues. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

The Illusion of Uniformity Does Not Depend on the Primary Visual Cortex: Evidence From Sensory Adaptation.

Visual experience appears richly detailed despite the poor resolution of the majority of the visual field, thanks to foveal-peripheral integration. The recently described uniformity illusion (UI), wherein peripheral elements of a pattern take on the appearance of foveal elements, may shed light on this integration. We examined the basis of UI by generating adaptation to a pattern of Gabors suitable for producing UI on orientation. After removing the pattern, participants reported the tilt of a single peripheral Gabor. The tilt aftereffect followed the physical adapting orientation rather than the global orientation perceived under UI, even when the illusion had been reported for a long time. Conversely, a control experiment replacing illusory uniformity with a physically uniform Gabor pattern for the same durations did produce an aftereffect to the global orientation. Results indicate that UI is not associated with changes in sensory encoding at V1 but likely depends on higher level processes.

Grouping by feature of cross-modal flankers in temporal ventriloquism.

Signals in one sensory modality can influence perception of another, for example the bias of visual timing by audition: temporal ventriloquism. Strong accounts of temporal ventriloquism hold that the sensory representation of visual signal timing changes to that of the nearby sound. Alternatively, underlying sensory representations do not change. Rather, perceptual grouping processes based on spatial, temporal, and featural information produce best-estimates of global event properties. In support of this interpretation, when feature-based perceptual grouping conflicts with temporal information-based in scenarios that reveal temporal ventriloquism, the effect is abolished. However, previous demonstrations of this disruption used long-range visual apparent-motion stimuli. We investigated whether similar manipulations of feature grouping could also disrupt the classical temporal ventriloquism demonstration, which occurs over a short temporal range. We estimated the precision of participants' reports of which of two visual bars occurred first. The bars were accompanied by different cross-modal signals that onset synchronously or asynchronously with each bar. Participants' performance improved with asynchronous presentation relative to synchronous - temporal ventriloquism - however, unlike the long-range apparent motion paradigm, this was unaffected by different combinations of cross-modal feature, suggesting that featural similarity of cross-modal signals may not modulate cross-modal temporal influences in short time scales.