Does cross-modal correspondence modulate modality-specific perceptual processing? Study using timing judgment tasks.

Cross-modal correspondences refer to associations between stimulus features across sensory modalities. Previous studies have shown that cross-modal correspondences modulate reaction times for detecting and identifying stimuli in one modality when uninformative stimuli from another modality are present. However, it is unclear whether such modulation reflects changes in modality-specific perceptual processing. We used two psychophysical timing judgment tasks to examine the effects of audiovisual correspondences on visual perceptual processing. In Experiment 1, we conducted a temporal order judgment (TOJ) task that asked participants to judge which of two visual stimuli presented with various stimulus onset asynchronies (SOAs) appeared first. In Experiment 2, we conducted a simultaneous judgment (SJ) task that asked participants to report whether the two visual stimuli were simultaneous or successive. We also presented an unrelated auditory stimulus, simultaneously or preceding the first visual stimulus, and manipulated the congruency between audiovisual stimuli. Experiment 1 indicated that the points of subjective simultaneity (PSSs) between the two visual stimuli estimated in the TOJ task shifted according to the audiovisual correspondence between the auditory pitch and visual features of vertical location and size. However, these audiovisual correspondences did not affect PSS estimated using the SJ task in Experiment 2. The different results of the two tasks can be explained through the response bias triggered by audiovisual correspondence that only the TOJ task included. We concluded that audiovisual correspondence would not modulate visual perceptual timing and that changes in modality-specific perceptual processing might not trigger the congruency effects reported in previous studies.

Auditory scene context facilitates visual recognition of objects in consistent visual scenes.

Visual object recognition is facilitated by contextually consistent scenes in which the object is embedded. Scene gist representations extracted from the scenery backgrounds yield this scene consistency effect. Here we examined whether the scene consistency effect is specific to the visual domain or if it is crossmodal. Through four experiments, the accuracy of the naming of briefly presented visual objects was assessed. In each trial, a 4-s sound clip was presented and a visual scene containing the target object was briefly shown at the end of the sound clip. In a consistent sound condition, an environmental sound associated with the scene in which the target object typically appears was presented (e.g., forest noise for a bear target object). In an inconsistent sound condition, a sound clip contextually inconsistent with the target object was presented (e.g., city noise for a bear). In a control sound condition, a nonsensical sound (sawtooth wave) was presented. When target objects were embedded in contextually consistent visual scenes (Experiment 1: a bear in a forest background), consistent sounds increased object-naming accuracy. In contrast, sound conditions did not show a significant effect when target objects were embedded in contextually inconsistent visual scenes (Experiment 2: a bear in a pedestrian crossing background) or in a blank background (Experiments 3 and 4). These results suggested that auditory scene context has weak or no direct influence on visual object recognition. It seems likely that consistent auditory scenes indirectly facilitate visual object recognition by promoting visual scene processing.

Attentional allocation and the pan-field color illusion.

Humans subjectively experience a scene as rendered in color across the entire visual field, a visual phenomenon called "pan-field color" (Balas & Sinha, 2007). This experience is inconsistent with the limited color sensitivity in the peripheral visual field. We investigated the effects of visual attention allocated to the peripheral visual field on the pan-field color illusion. Using "chimera" stimuli in which color was restricted to a circular central area, we assessed observers' tendency to perceive color throughout images with achromatized peripheral regions. We separately analyzed sensitivity and response bias in judging the color content of the scene image as full-color, chimera, or gray. Using a dual-task paradigm, we manipulated observers' attentional allocation by controlling the stimulus presentation time of the central task, making the foveal attentional load change. The slope of the foveal load-sensitivity function suggests that attention was modulated by foveal load even in the peripheral visual field. Bias was affected by the size of the central colored area, such that the tendency to answer "full-color" to the chimera image increased with eccentricity. Based on these effects of attention on sensitivity and bias, we suggest that the pan-field color illusion cannot be fully explained by the decrease of sensitivity that is modulated by attentional allocation in the periphery. Our results rather indicate that the pan-field color illusion at least partly reflects a liberal bias in peripheral vision.

Cross-modal correspondence between auditory pitch and visual elevation modulates audiovisual temporal recalibration.

Cross-modal correspondences refer to associations between feature dimensions of stimuli across sensory modalities. Research has indicated that correspondence between audiovisual stimuli influences whether these stimuli are integrated or segregated. On the other hand, the audiovisual integration process plastically changes to compensate for continuously observed spatiotemporal conflicts between sensory modalities. If and how cross-modal correspondence modulates the "recalibration" of integration is unclear. We investigated whether cross-modal correspondence between auditory pitch and visual elevation affected audiovisual temporal recalibration. Participants judged the simultaneity of a pair of audiovisual stimuli after an adaptation phase in which alternating auditory and visual stimuli equally spaced in time were presented. In the adaptation phase, auditory pitch and visual elevation were manipulated to fix the order within each pairing of audiovisual stimuli congruent with pitch-elevation correspondence (visual leading or auditory leading). We found a shift in the point of subjective simultaneity (PSS) between congruent audiovisual stimuli as a function of the adaptation conditions (Experiment 1, 2), but this shift in the PSS was not observed within incongruent pairs (Experiment 2). These results indicate that asynchronies between audiovisual signals congruent with cross-modal correspondence are selectively recalibrated.

Pitch-elevation and pitch-size cross-modal correspondences do not affect temporal ventriloquism.

Temporal ventriloquism refers to the shift in the perceived timing of a visual stimulus towards a transient auditory stimulus presented close in time. This effect is demonstrated by greater sensitivity of temporal order judgments of two visual stimuli when a sound is presented before the first visual stimulus and after the second visual stimulus. Recent studies suggest that temporal ventriloquism is affected by cross-modal correspondence between auditory pitch and visual elevation but not by correspondence between pitch and visual size. Here we examined the possibility that these results do not reflect a difference in the effects of different types of cross-modal correspondences on temporal ventriloquism but are rather mediated by shifts in visual-spatial attention triggered by preceding auditory stimuli. In Experiment 1, we replicated the results of previous studies that varied with the type of correspondence. Experiment 2 investigated the effects of the second audiovisual stimuli's asynchrony while the first audiovisual stimuli were synchronized. The results, unlike in Experiment 1, revealed that the magnitude of the temporal ventriloquism effect did not change with the congruency of pitch-elevation correspondence. Experiment 3 also indicated that the asynchrony of the first audiovisual stimuli modulated visual discrimination sensitivity irrespective of temporal ventriloquism. These results suggest that cross-modal correspondences do not affect temporal ventriloquism. Greater visual sensitivity when audiovisual stimuli are congruent with pitch-elevation correspondence may be attributable to shifts in visual attention caused by pitches of the preceding auditory stimulus, which speeds up detection of the first visual stimulus.

Adaptation to delayed visual feedback of the body movement extends multisensory peripersonal space.

The space surrounding the body in which individuals interact with the environment is known as the peripersonal space (PPS). Previous studies have reported that PPS has multisensory nature. However, the relationship between the multisensory nature of PPS and an individuals' defensive actions has not been fully clarified to date. We investigated this relationship by examining the multisensory representation of PPS under situations in which visual feedback of body movements was delayed by using a virtual reality system. The results indicated that body-movement delays extended the multisensory PPS, suggesting that body-movement delays increased the potential threat of distant objects because it was necessary to prepare defensive actions sooner. The previous findings can be interpreted that PPS is modulated by the spatio-temporal relationship between people and external stimuli. This view may provide evidence of interactions between defensive and nondefensive functions of the multisensory PPS.

Do the colors of your letters depend on your language? Language-dependent and universal influences on grapheme-color synesthesia in seven languages.

Grapheme-color synesthetes experience graphemes as having a consistent color (e.g., "N is turquoise"). Synesthetes' specific associations (which letter is which color) are often influenced by linguistic properties such as phonetic similarity, color terms ("Y is yellow"), and semantic associations ("D is for dog and dogs are brown"). However, most studies of synesthesia use only English-speaking synesthetes. Here, we measure the effect of color terms, semantic associations, and non-linguistic shape-color associations on synesthetic associations in Dutch, English, Greek, Japanese, Korean, Russian, and Spanish. The effect size of linguistic influences (color terms, semantic associations) differed significantly between languages. In contrast, the effect size of non-linguistic influences (shape-color associations), which we predicted to be universal, indeed did not differ between languages. We conclude that language matters (outcomes are influenced by the synesthete's language) and that synesthesia offers an exceptional opportunity to study influences on letter representations in different languages.

Does response facilitation to visuo-tactile stimuli around a remote-controlled hand avatar reflect peripersonal space or attentional bias?

People react faster to visuo-tactile stimuli presented near the body (i.e., in peripersonal space) than to tactile stimuli presented alone. This multi-sensory facilitation effect has been used as a measurement of peripersonal space. Previous research has reported that peripersonal space representations can be modulated by actively using hand-held tools or disconnected hand avatars. However, previous research has ignored the possibility that the attentional effect of active tool use could affect multi-sensory facilitation. In the present study, we delivered tactile stimuli to participants' left or right hand concurrently with visual stimuli presented near a virtual hand avatar operated by the movements of participants' left or right hand, which was shown far in a virtual environment and disconnected from the body. Participants reacted to tactile stimuli while ignoring the visual stimuli. The results indicated a multi-sensory facilitation effect when tactile stimuli were delivered to the hand used to operate the hand avatar. In contrast, the facilitation was not observed when the tactile stimuli were delivered to the hand that is not operating the hand avatar. These results suggest that the strength of the multi-sensory facilitation effect differed across conditions, even though the visual attention captured around the hand avatar was controlled across conditions. We concluded that the modulation of peripersonal space resulting from using tools or avatars is nearly independent of visual attention captured around tools or avatars.

Remote hand: Hand-centered peripersonal space transfers to a disconnected hand avatar.

The space surrounding our body is called peripersonal space (PPS). It has been reported that visuo-tactile facilitation occurs more strongly within PPS than outside PPS. Furthermore, previous research has revealed several methods by which PPS can be extended. The present study provides the first behavioral evidence of the transfer of PPS in a virtual environment by a novel technique. PPS representation was investigated using a remote-controlled hand avatar presented far from the body in a virtual environment. Participants showed strongest visuo-tactile facilitation at the far space around the remote hand and no facilitation at the near space around the real hand, suggesting that PPS transfers from near the body to the space around the hand avatar. The present results extend previous findings of the plasticity of PPS and demonstrate flexibility of PPS representation beyond the physical and anatomical limits of body representation.

Discounting mechanism underlies extinction illusion.

Humans can perceive a coherent visual scene despite a low spatial resolution in peripheral vision. How does the visual system determine whether an object exists in the periphery? We addressed this question by focusing on the extinction illusion in which a disk becomes subjectively invisible when presented at the intersection of grids. We hypothesized that the disk would go unnoticed when the stimuli with and without the disk produced the same strength of visual signals. The visual system would miss the disk by confounding the target signals with the intersection signals that should be discounted. Computational analysis revealed that the energy ratio between the stimuli with and without the disk decreased with stimulus eccentricity and such energy ratio could successfully explain the observer's d' to detect the disk. These results indicate that the discounting mechanism relying on stimulus energy determines the awareness toward a peripheral object.

Consistency of synesthetic association varies with grapheme familiarity: A longitudinal study of grapheme-color synesthesia.

Grapheme-color synesthesia is a condition in which the visual perception of letters or numbers induces a specific color sensation. The consistency of grapheme-color association has been considered as a fundamental characteristic of synesthesia. However, recent studies have indicated that this association can change across the adult lifespan, and it has become necessary to investigate the factors behind the changes within each synesthete. We conducted a longitudinal study of Japanese adult synesthetes to investigate long-term (5-8 years) changes in color responses to 300 graphemes (alphanumeric and Japanese characters). Graphemes with lower long-term consistency of synesthetic association also tended to have lower short-term consistency, indicating that grapheme-color association's consistency is determined for each grapheme. Further, less familiar graphemes had less consistent associations with their synesthetic colors. These findings suggest that a stronger grapheme-color association is formed for more familiar graphemes, leading to the consolidation of synesthetic color for such graphemes.

Disconnected hand avatar can be integrated into the peripersonal space.

Several studies have shown that space immediately surrounding the body, or the peripersonal space is represented differently in the brain from the more distant extra-personal space. Moreover, the boundary of peripersonal space can be extended to space surrounding the tip of a tool held by the hand. However, it is not known if tools need to be connected to the body to modulate the peripersonal space. We used a line bisection task to investigate whether peripersonal space representation surrounds a virtual hand avatar that is disconnected from the body. Healthy participants conducted a line bisection task by responding with either a virtual hand avatar or a laser pointer. The to-be-bisected lines were presented either in peripersonal or extra-personal space. When the lines were placed in extra-personal space, the virtual hand avatar was presented near the line such that the hand avatar was far from participants and disconnected from their bodies. Results indicated a shift in the line bisection bias from the left to the right as the line presentation distance increased when using the laser pointer, whereas no shift in bias was observed when using the virtual hand avatar. This result indicates that objects resembling human hands presented even at a distance and disconnected from the body can be integrated into the peripersonal space, which suggests that peripersonal space representation is more flexible than previously reported.

Apparent physical brightness of graphemes is altered by their synaesthetic colour in grapheme-colour synaesthetes.

Grapheme-colour synaesthesia is a condition in which the visual perception of letters or numbers induces a specific colour sensation. In this study, we demonstrated that the apparent physical brightness of graphemes is modulated by the synaesthetic colours elicited by them. Synaesthetes first selected a synaesthetic colour corresponding to each capital letter and digit. Then, we selected a grapheme stimulus with a bright synaesthetic colour and one with a dark colour for each synaesthete. Finally, synaesthetes and non-synaesthete controls participated in a brightness judgment task, in which each participant judged the real brightness of each of the two stimuli compared to a standard stimulus. Compared to non-synaesthetes, synaesthetes judged a grapheme with a bright synaesthetic colour to be brighter than one with a dark synaesthetic colour, suggesting that the synaesthetic colour experience of synaesthetes alters their brightness perception. Such alteration in real brightness perception was observed both in those who experienced synaesthetic colours in external space (projector-type synaesthetes) and in those who experienced such colours 'in the mind's eye' (associator-type synaesthetes). These results support the view that early visual processing is modulated by feedback transmitted from the V4 colour area, the neural activation of which accompanies synaesthetic colour experience.

Grapheme-color associations can transfer to novel graphemes when synesthetic colors function as grapheme "discriminating markers".

Synesthesia is a condition in which the perception of a stimulus in one modality automatically triggers a secondary sensation in another modality or processing stream. Our study focused on grapheme-color synesthesia, in which the visual perception of letters or numbers (graphemes) induces a specific color sensation (the synesthetic color). Grapheme-color synesthetes do not typically experience colors for novel graphemes. However, synesthetic colors associated with familiar graphemes can be transferred to graphemes learned later, even in adulthood. A previous study has shown that such a transfer can take place after only a 10-min writing exercise. In this study, we found that this immediate transfer occurs only when the synesthetic colors for familiar graphemes contribute to the discrimination of the graphemes to be learned. Synesthetes learned six novel graphemes, each of which was arbitrarily associated with one of six familiar graphemes. Half of the synesthetes were assigned to the heterogeneous condition, in which the synesthetic colors of one group of familiar graphemes were different from one another. The other half of the synesthetes were assigned to the homogeneous condition, in which the various colors of a whole group of familiar graphemes were categorically the same. After this learning session, less transfer of synesthetic colors to novel graphemes from the corresponding familiar graphemes occurred in the homogeneous condition than in the heterogeneous condition. These results support the view that synesthetic colors for graphemes may function as a grapheme acquisition aid.

The relationship between the body and the environment in the virtual world: The interpupillary distance affects the body size perception.

Previous research suggests that the size of one's body is used as a metric to scale the external world. On the other hand, the influence of information from the external world on the perception of body size is unclear. It has been suggested that increased inter-pupillary distance (IPD) leads people to perceive the external world as smaller than it actually is. The present study investigated the effect of the IPD on body size perception, and the relationship between the perceived scale of the body and the external world when the IPD is manipulated. To this end, in a virtual environment, we manipulated the IPD as well as the size and presence of participants' hands, while participant's eye height was increased vertically. Results showed that, when participants' eye height was increased and their hands were enlarged, people with a fixed IPD perceived the size of their body to be large (like a giant) while the external world was perceived to be changed minimally. Alternatively, people with increased IPD perceived that the external world as having shrank, whereas their perception of their body size changed little. However, when a viewers' virtual hands were not shown, the IPD did not affect the individual's percept of body size, although the IPD did affect one's perception of the external world. These results suggest that, when the ratio of the size between one's body and the external world are explicit, the perceived size of one's body is affected by the IPD or perceived scale of the external world that is affected by the IPD.

Synaesthetic colour associations for Japanese Kanji characters: from the perspective of grapheme learning.

One of the fundamental questions about grapheme-colour synaesthesia is how specific associations between the graphemes and colours are formed. We addressed this question by focusing on the determinants of synaesthetic colours for Japanese Kanji characters (logographic characters) using a psycholinguistic approach. Study 1 explored the influence meaning has on synaesthetic colours for Kanji characters representing abstract meanings by examining synaesthetic colours for antonym pairs (i.e. characters with meanings opposed to each other) in Japanese synaesthetes. Results showed that semantic relations influenced the grapheme-colour associations for characters representing abstract meanings in the early stages of learning abstract Kanji, while the influence was reduced in the grapheme-colour associations for those learned later. Study 2 examined the effect that learning new sounds or meanings of graphemes has on synaesthetic colours for those graphemes. Japanese synaesthetes were taught new sounds or new meanings for familiar Kanji characters. Results indicated that acquiring new information for graphemes slightly but significantly reduced the test-retest grapheme-colour association consistency, suggesting that synaesthetic colours can be modulated to reflect the synaesthete's latest knowledge about graphemes. Implications of these findings are discussed from the perspective of the relationship between synaesthesia and grapheme learning. This article is part of a discussion meeting issue 'Bridging senses: novel insights from synaesthesia'.

Developmental Changes in Number Personification by Elementary School Children.

Children often personify non-living objects, such as puppets and stars. This attribution is considered a healthy phenomenon, which can simulate social exchange and enhance children's understanding of social relationships. In this study, we considered that the tendency of children to engage in personification could potentially be observed in abstract entities, such as numbers. We hypothesized that children tend to attribute personalities to numbers, which diminishes during the course of development. By consulting the methodology to measure ordinal linguistic personification (OLP), which is a type of synesthesia, we quantified the frequency with which child and adult populations engage in number personification. Questionnaires were completed by 151 non-synesthetic children (9-12 years old) and 55 non-synesthetic adults. Children showed a higher tendency than adults to engage in number personification, with respect to temporal consistency and the frequency of choosing meaningful answers. Additionally, children tended to assign unique and exclusive descriptions to each number from zero to nine. By synthesizing the series of analyses, we revealed the process in which number personification diminishes throughout development. In the discussion, we examined the possibility that number personification serves as a discrimination clue to aid children's comprehension of the relationships between numbers.

Task-irrelevant spatial dividers facilitate counting and numerosity estimation.

Counting is characterized as a slow and error-prone action relying heavily on serial allocation of focused attention. However, quick and accurate counting is required for many real-world tasks (e.g., counting heads to ensure everyone is evacuated to a safe place in an emergency). Previous research suggests that task-irrelevant spatial dividers, which segment visual displays into small areas, facilitate focused attention and improve serial search. The present study investigated whether counting, which is also closely related to focused attention, can be facilitated by spatial dividers. Furthermore, the effect of spatial dividers on numerosity estimation, putatively dependent upon distributed attention, was also examined to provide insights into different types of number systems and different modes of visual attention. The results showed profound performance improvement by task-irrelevant spatial dividers in both counting and numerosity estimation tasks, indicating that spatial dividers may activate interaction between number and visual attention systems. Our findings provide the first evidence that task-irrelevant spatial dividers can be used to facilitate various types of numerical cognition.

Why is the synesthete's "A" red? Using a five-language dataset to disentangle the effects of shape, sound, semantics, and ordinality on inducer-concurrent relationships in grapheme-color synesthesia.

Grapheme-color synesthesia is a neurological phenomenon in which viewing a grapheme elicits an additional, automatic, and consistent sensation of color. Color-to-letter associations in synesthesia are interesting in their own right, but also offer an opportunity to examine relationships between visual, acoustic, and semantic aspects of language. Research using large populations of synesthetes has indeed found that grapheme-color pairings can be influenced by numerous properties of graphemes, but the contributions made by each of these explanatory factors are often confounded in a monolingual dataset (i.e., only English-speaking synesthetes). Here, we report the first demonstration of how a multilingual dataset can reveal potentially-universal influences on synesthetic associations, and disentangle previously-confounded hypotheses about the relationship between properties of synesthetic color and properties of the grapheme that induces it. Numerous studies have reported that for English-speaking synesthetes, "A" tends to be colored red more often than predicted by chance, and several explanatory factors have been proposed that could explain this association. Using a five-language dataset (native English, Dutch, Spanish, Japanese, and Korean speakers), we compare the predictions made by each explanatory factor, and show that only an ordinal explanation makes consistent predictions across all five languages, suggesting that the English "A" is red because the first grapheme of a synesthete's alphabet or syllabary tends to be associated with red. We propose that the relationship between the first grapheme and the color red is an association between an unusually-distinct ordinal position ("first") and an unusually-distinct color (red). We test the predictions made by this theory, and demonstrate that the first grapheme is unusually distinct (has a color that is distant in color space from the other letters' colors). Our results demonstrate the importance of considering cross-linguistic similarities and differences in synesthesia, and suggest that some influences on grapheme-color associations in synesthesia might be universal.

Temporal Characteristics of Radiologists' and Novices' Lesion Detection in Viewing Medical Images Presented Rapidly and Sequentially.

Although viewing multiple stacks of medical images presented on a display is a relatively new but useful medical task, little is known about this task. Particularly, it is unclear how radiologists search for lesions in this type of image reading. When viewing cluttered and dynamic displays, continuous motion itself does not capture attention. Thus, it is effective for the target detection that observers' attention is captured by the onset signal of a suddenly appearing target among the continuously moving distractors (i.e., a passive viewing strategy). This can be applied to stack viewing tasks, because lesions often show up as transient signals in medical images which are sequentially presented simulating a dynamic and smoothly transforming image progression of organs. However, it is unclear whether observers can detect a target when the target appears at the beginning of a sequential presentation where the global apparent motion onset signal (i.e., signal of the initiation of the apparent motion by sequential presentation) occurs. We investigated the ability of radiologists to detect lesions during such tasks by comparing the performances of radiologists and novices. Results show that overall performance of radiologists is better than novices. Furthermore, the temporal locations of lesions in CT image sequences, i.e., when a lesion appears in an image sequence, does not affect the performance of radiologists, whereas it does affect the performance of novices. Results indicate that novices have greater difficulty in detecting a lesion appearing early than late in the image sequence. We suggest that radiologists have other mechanisms to detect lesions in medical images with little attention which novices do not have. This ability is critically important when viewing rapid sequential presentations of multiple CT images, such as stack viewing tasks.