No effect of spatial congruence on rapid temporal recalibration to audiovisual asynchrony.

The brain integrates multisensory information to construct coherent perceptual representations based on spatial and temporal congruence. Intriguingly, multisensory timing perception can be flexibly calibrated. Repeated exposure to audiovisual asynchrony induces shifts in subjective simultaneity (temporal recalibration). Spatial congruence is known to serve as a grouping cue for recalibration when the audiovisual temporal relationship is ambiguous during exposure. A single exposure to audiovisual asynchrony can also trigger temporal recalibration (rapid recalibration). However, it has been suggested that the underlying mechanisms of these temporal recalibrations differ. Here, we examined whether spatial congruence can be a grouping cue for rapid recalibration when audiovisual pairs are not defined by temporal relationships. Participants made a simultaneity judgment for a pair of audiovisual stimuli after adapting three consecutive stimuli once in a "light-sound-light" or "sound-light-sound" order with an equal temporal interval. The spatial positions of the adapting stimuli were manipulated as an audiovisual pair from the same position (e.g., left) and the remaining stimulus from another position (e.g., right). In three experiments, the spatial congruence of the audiovisual adapting stimuli did not show a modulatory effect, while we replicated the rapid recalibration effects. Rather, rapid recalibration occurred according to the temporal order of the first light and sound. Our findings suggest that, in contrast to temporal recalibration with repeated exposure, the perceptual systems underlying rapid recalibration simply combine individual visual and auditory inputs based on the order in which they arrive.

Does cross-modal correspondence modulate modality-specific perceptual processing? Study using timing judgment tasks.

Cross-modal correspondences refer to associations between stimulus features across sensory modalities. Previous studies have shown that cross-modal correspondences modulate reaction times for detecting and identifying stimuli in one modality when uninformative stimuli from another modality are present. However, it is unclear whether such modulation reflects changes in modality-specific perceptual processing. We used two psychophysical timing judgment tasks to examine the effects of audiovisual correspondences on visual perceptual processing. In Experiment 1, we conducted a temporal order judgment (TOJ) task that asked participants to judge which of two visual stimuli presented with various stimulus onset asynchronies (SOAs) appeared first. In Experiment 2, we conducted a simultaneous judgment (SJ) task that asked participants to report whether the two visual stimuli were simultaneous or successive. We also presented an unrelated auditory stimulus, simultaneously or preceding the first visual stimulus, and manipulated the congruency between audiovisual stimuli. Experiment 1 indicated that the points of subjective simultaneity (PSSs) between the two visual stimuli estimated in the TOJ task shifted according to the audiovisual correspondence between the auditory pitch and visual features of vertical location and size. However, these audiovisual correspondences did not affect PSS estimated using the SJ task in Experiment 2. The different results of the two tasks can be explained through the response bias triggered by audiovisual correspondence that only the TOJ task included. We concluded that audiovisual correspondence would not modulate visual perceptual timing and that changes in modality-specific perceptual processing might not trigger the congruency effects reported in previous studies.

Cross-modal correspondence between auditory pitch and visual elevation modulates audiovisual temporal recalibration.

Cross-modal correspondences refer to associations between feature dimensions of stimuli across sensory modalities. Research has indicated that correspondence between audiovisual stimuli influences whether these stimuli are integrated or segregated. On the other hand, the audiovisual integration process plastically changes to compensate for continuously observed spatiotemporal conflicts between sensory modalities. If and how cross-modal correspondence modulates the "recalibration" of integration is unclear. We investigated whether cross-modal correspondence between auditory pitch and visual elevation affected audiovisual temporal recalibration. Participants judged the simultaneity of a pair of audiovisual stimuli after an adaptation phase in which alternating auditory and visual stimuli equally spaced in time were presented. In the adaptation phase, auditory pitch and visual elevation were manipulated to fix the order within each pairing of audiovisual stimuli congruent with pitch-elevation correspondence (visual leading or auditory leading). We found a shift in the point of subjective simultaneity (PSS) between congruent audiovisual stimuli as a function of the adaptation conditions (Experiment 1, 2), but this shift in the PSS was not observed within incongruent pairs (Experiment 2). These results indicate that asynchronies between audiovisual signals congruent with cross-modal correspondence are selectively recalibrated.

Pitch-elevation and pitch-size cross-modal correspondences do not affect temporal ventriloquism.

Temporal ventriloquism refers to the shift in the perceived timing of a visual stimulus towards a transient auditory stimulus presented close in time. This effect is demonstrated by greater sensitivity of temporal order judgments of two visual stimuli when a sound is presented before the first visual stimulus and after the second visual stimulus. Recent studies suggest that temporal ventriloquism is affected by cross-modal correspondence between auditory pitch and visual elevation but not by correspondence between pitch and visual size. Here we examined the possibility that these results do not reflect a difference in the effects of different types of cross-modal correspondences on temporal ventriloquism but are rather mediated by shifts in visual-spatial attention triggered by preceding auditory stimuli. In Experiment 1, we replicated the results of previous studies that varied with the type of correspondence. Experiment 2 investigated the effects of the second audiovisual stimuli's asynchrony while the first audiovisual stimuli were synchronized. The results, unlike in Experiment 1, revealed that the magnitude of the temporal ventriloquism effect did not change with the congruency of pitch-elevation correspondence. Experiment 3 also indicated that the asynchrony of the first audiovisual stimuli modulated visual discrimination sensitivity irrespective of temporal ventriloquism. These results suggest that cross-modal correspondences do not affect temporal ventriloquism. Greater visual sensitivity when audiovisual stimuli are congruent with pitch-elevation correspondence may be attributable to shifts in visual attention caused by pitches of the preceding auditory stimulus, which speeds up detection of the first visual stimulus.

Consistency of synesthetic association varies with grapheme familiarity: A longitudinal study of grapheme-color synesthesia.

Grapheme-color synesthesia is a condition in which the visual perception of letters or numbers induces a specific color sensation. The consistency of grapheme-color association has been considered as a fundamental characteristic of synesthesia. However, recent studies have indicated that this association can change across the adult lifespan, and it has become necessary to investigate the factors behind the changes within each synesthete. We conducted a longitudinal study of Japanese adult synesthetes to investigate long-term (5-8 years) changes in color responses to 300 graphemes (alphanumeric and Japanese characters). Graphemes with lower long-term consistency of synesthetic association also tended to have lower short-term consistency, indicating that grapheme-color association's consistency is determined for each grapheme. Further, less familiar graphemes had less consistent associations with their synesthetic colors. These findings suggest that a stronger grapheme-color association is formed for more familiar graphemes, leading to the consolidation of synesthetic color for such graphemes.

Apparent physical brightness of graphemes is altered by their synaesthetic colour in grapheme-colour synaesthetes.

Grapheme-colour synaesthesia is a condition in which the visual perception of letters or numbers induces a specific colour sensation. In this study, we demonstrated that the apparent physical brightness of graphemes is modulated by the synaesthetic colours elicited by them. Synaesthetes first selected a synaesthetic colour corresponding to each capital letter and digit. Then, we selected a grapheme stimulus with a bright synaesthetic colour and one with a dark colour for each synaesthete. Finally, synaesthetes and non-synaesthete controls participated in a brightness judgment task, in which each participant judged the real brightness of each of the two stimuli compared to a standard stimulus. Compared to non-synaesthetes, synaesthetes judged a grapheme with a bright synaesthetic colour to be brighter than one with a dark synaesthetic colour, suggesting that the synaesthetic colour experience of synaesthetes alters their brightness perception. Such alteration in real brightness perception was observed both in those who experienced synaesthetic colours in external space (projector-type synaesthetes) and in those who experienced such colours 'in the mind's eye' (associator-type synaesthetes). These results support the view that early visual processing is modulated by feedback transmitted from the V4 colour area, the neural activation of which accompanies synaesthetic colour experience.

Grapheme-color associations can transfer to novel graphemes when synesthetic colors function as grapheme "discriminating markers".

Synesthesia is a condition in which the perception of a stimulus in one modality automatically triggers a secondary sensation in another modality or processing stream. Our study focused on grapheme-color synesthesia, in which the visual perception of letters or numbers (graphemes) induces a specific color sensation (the synesthetic color). Grapheme-color synesthetes do not typically experience colors for novel graphemes. However, synesthetic colors associated with familiar graphemes can be transferred to graphemes learned later, even in adulthood. A previous study has shown that such a transfer can take place after only a 10-min writing exercise. In this study, we found that this immediate transfer occurs only when the synesthetic colors for familiar graphemes contribute to the discrimination of the graphemes to be learned. Synesthetes learned six novel graphemes, each of which was arbitrarily associated with one of six familiar graphemes. Half of the synesthetes were assigned to the heterogeneous condition, in which the synesthetic colors of one group of familiar graphemes were different from one another. The other half of the synesthetes were assigned to the homogeneous condition, in which the various colors of a whole group of familiar graphemes were categorically the same. After this learning session, less transfer of synesthetic colors to novel graphemes from the corresponding familiar graphemes occurred in the homogeneous condition than in the heterogeneous condition. These results support the view that synesthetic colors for graphemes may function as a grapheme acquisition aid.