Neural basis of sound-symbolic pseudoword-shape correspondences.

Non-arbitrary mapping between the sound of a word and its meaning, termed sound symbolism, is commonly studied through crossmodal correspondences between sounds and visual shapes, e.g., auditory pseudowords, like 'mohloh' and 'kehteh', are matched to rounded and pointed visual shapes, respectively. Here, we used functional magnetic resonance imaging (fMRI) during a crossmodal matching task to investigate the hypotheses that sound symbolism (1) involves language processing; (2) depends on multisensory integration; (3) reflects embodiment of speech in hand movements. These hypotheses lead to corresponding neuroanatomical predictions of crossmodal congruency effects in (1) the language network; (2) areas mediating multisensory processing, including visual and auditory cortex; (3) regions responsible for sensorimotor control of the hand and mouth. Right-handed participants (n = 22) encountered audiovisual stimuli comprising a simultaneously presented visual shape (rounded or pointed) and an auditory pseudoword ('mohloh' or 'kehteh') and indicated via a right-hand keypress whether the stimuli matched or not. Reaction times were faster for congruent than incongruent stimuli. Univariate analysis showed that activity was greater for the congruent compared to the incongruent condition in the left primary and association auditory cortex, and left anterior fusiform/parahippocampal gyri. Multivoxel pattern analysis revealed higher classification accuracy for the audiovisual stimuli when congruent than when incongruent, in the pars opercularis of the left inferior frontal (Broca's area), the left supramarginal, and the right mid-occipital gyri. These findings, considered in relation to the neuroanatomical predictions, support the first two hypotheses and suggest that sound symbolism involves both language processing and multisensory integration.

Crossmodal Visuospatial Effects on Auditory Perception of Musical Contour.

The crossmodal correspondence between auditory pitch and visuospatial elevation (in which high- and low-pitched tones are associated with high and low spatial elevation respectively) has been proposed as the basis for Western musical notation. One implication of this is that music perception engages visuospatial processes and may not be exclusively auditory. Here, we investigated how music perception is influenced by concurrent visual stimuli. Participants listened to unfamiliar five-note musical phrases with four kinds of pitch contour (rising, falling, rising-falling, or falling-rising), accompanied by incidental visual contours that were either congruent (e.g., auditory rising/visual rising) or incongruent (e.g., auditory rising/visual falling) and judged whether the final note of the musical phrase was higher or lower in pitch than the first. Response times for the auditory judgment were significantly slower for incongruent compared to congruent trials, i.e., there was a congruency effect, even though the visual contours were incidental to the auditory task. These results suggest that music perception, although generally regarded as an auditory experience, may actually be multisensory in nature.

Neural basis of the crossmodal correspondence between auditory pitch and visuospatial elevation.

Crossmodal correspondences refer to associations between otherwise unrelated stimulus features in different sensory modalities. For example, high and low auditory pitches are associated with high and low visuospatial elevation, respectively. The neural mechanisms underlying crossmodal correspondences are currently unknown. Here, we used functional magnetic resonance imaging (fMRI) to investigate the neural basis of the pitch-elevation correspondence. Pitch-elevation congruency effects were observed bilaterally in the inferior frontal and insular cortex, the right frontal eye field and right inferior parietal cortex. Independent functional localizers failed to provide strong evidence for any of three proposed mechanisms for crossmodal correspondences: semantic mediation, magnitude estimation, and multisensory integration. Instead, pitch-elevation congruency effects overlapped with areas selective for visually presented non-word strings relative to sentences, and with regions sensitive to audiovisual asynchrony. Taken together with the prior literature, the observed congruency effects are most consistent with mediation by multisensory attention.

Enhanced verbal abilities in the congenitally blind.

Numerous studies have found that congenitally blind individuals have better verbal memory than their normally sighted counterparts. However, it is not known whether this reflects superiority of verbal or memory abilities. In order to distinguish between these possibilities, we tested congenitally blind participants and normally sighted control participants, matched for age and education, on a range of verbal and spatial tasks. Congenitally blind participants were significantly better than sighted controls on all the verbal tasks but the groups did not differ significantly on the spatial tasks. Thus, the congenitally blind appear to have superior verbal, but not spatial, abilities. This may reflect greater reliance on verbal information and the involvement of visual cortex in language processing in the congenitally blind.

Loss of form vision impairs spatial imagery.

Previous studies have reported inconsistent results when comparing spatial imagery performance in the blind and the sighted, with some, but not all, studies demonstrating deficits in the blind. Here, we investigated the effect of visual status and individual preferences ("cognitive style") on performance of a spatial imagery task. Participants with blindness resulting in the loss of form vision at or after age 6, and age- and gender-matched sighted participants, performed a spatial imagery task requiring memorization of a 4 × 4 lettered matrix and subsequent mental construction of shapes within the matrix from four-letter auditory cues. They also completed the Santa Barbara Sense of Direction Scale (SBSoDS) and a self-evaluation of cognitive style. The sighted participants also completed the Object-Spatial Imagery and Verbal Questionnaire (OSIVQ). Visual status affected performance on the spatial imagery task: the blind performed significantly worse than the sighted, independently of the age at which form vision was completely lost. Visual status did not affect the distribution of preferences based on self-reported cognitive style. Across all participants, self-reported verbalizer scores were significantly negatively correlated with accuracy on the spatial imagery task. There was a positive correlation between the SBSoDS score and accuracy on the spatial imagery task, across all participants, indicating that a better sense of direction is related to a more proficient spatial representation and that the imagery task indexes ecologically relevant spatial abilities. Moreover, the older the participants were, the worse their performance was, indicating a detrimental effect of age on spatial imagery performance. Thus, spatial skills represent an important target for rehabilitative approaches to visual impairment, and individual differences, which can modulate performance, should be taken into account in such approaches.

Multisensory object representation: insights from studies of vision and touch.

Behavioral studies show that the unisensory representations underlying within-modal visual and haptic object recognition are strikingly similar in terms of view- and size-sensitivity, and integration of structural and surface properties. However, the basis for these attributes differs in each modality, indicating that while these representations are functionally similar, they are not identical. Imaging studies reveal bisensory, visuo-haptic object selectivity, notably in the lateral occipital complex and the intraparietal sulcus, that suggests a shared representation of objects. Such a multisensory representation could underlie visuo-haptic cross-modal object recognition. In this chapter, we compare visual and haptic within-modal object recognition and trace a progression from functionally similar but separate unisensory representations to a shared multisensory representation underlying cross-modal object recognition as well as view-independence, regardless of modality. We outline, and provide evidence for, a model of multisensory object recognition in which representations are flexibly accessible via top-down or bottom-up processing, the choice of route being influenced by object familiarity and individual preference along the object-spatial continuum of mental imagery.

Vision and touch: multiple or multisensory representations of objects?

The relationship between visually and haptically derived representations of objects is an important question in multisensory processing and, increasingly, in mental representation. We review evidence for the format and properties of these representations, and address possible theoretical models. We explore the relevance of visual imagery processes and highlight areas for further research, including the neglected question of asymmetric performance in the visuo-haptic cross-modal memory paradigm. We conclude that the weight of evidence suggests the existence of a multisensory representation, spatial in format, and flexibly accessible by both bottom-up and top-down inputs, although efficient comparison between modality-specific representations cannot entirely be ruled out.

Multisensory cortical processing of object shape and its relation to mental imagery.

Here, we used functional magnetic resonance imaging to investigate the multisensory processing of object shape in the human cerebral cortex and explored the role of mental imagery in such processing. Regions active bilaterally during both visual and haptic shape perception, relative to texture perception in the respective modality, included parts of the superior parietal gyrus, the anterior intraparietal sulcus, and the lateral occipital complex. Of these bimodal regions, the lateral occipital complexes preferred visual over haptic stimuli, whereas the parietal areas preferred haptic over visual stimuli. Whereas most subjects reported little haptic imagery during visual shape perception, experiences of visual imagery during haptic shape perception were common. Across subjects, ratings of the vividness of visual imagery strongly predicted the amount of haptic shape-selective activity in the right, but not in the left, lateral occipital complex. Thus, visual imagery appears to contribute to activation of some, but not all, visual cortical areas during haptic perception.

Task-specific recruitment of dorsal and ventral visual areas during tactile perception.

Many studies have found that visual cortical areas are active during tactile perception. Here we used positron emission tomographic (PET) scanning in normally sighted humans to show that extrastriate cortical regions are recruited in a task-specific manner during perceptual processing of tactile stimuli varying in two dimensions. Mental rotation of tactile Forms activated a focus around the anterior part of the left intraparietal sulcus. Since prior studies have reported activity nearby during mental rotation of visual stimuli, this focus appears to be associated with the dorsal visual (visuospatial) pathway. Discrimination between tactile Forms activated the right lateral occipital complex, an object-selective region in the ventral visual (visual Form) pathway. Thus, tactile tasks appear to recruit cortical regions that are active during corresponding visual tasks. Activation of these areas in both visual and tactile tasks could reflect visual imagery during tactile perception, activity in multisensory representations, or both.

Temporal cues contribute to tactile perception of roughness.

Optimal perception of surface roughness requires lateral movement between skin and surface, suggesting the importance of temporal cues. The roughness of periodic gratings is affected by changing either inter-element spacing (groove width, G) or element width (ridge width, R). Peripheral neural responses to gratings depend quantitatively on a spatial variable, G, and a temporal variable, grating temporal frequency (F(t)), with changes in R acting indirectly through concomitant changes in F(t). We investigated, psychophysically, the contribution of temporal cues to human tactile perception of roughness, using gratings varying in either R or G. Gratings were scanned across the immobile fingerpad with controlled movement speed (S) and contact force. In one experiment, we found that roughness magnitude estimates depended on both G and F(t). In a second experiment, discrimination of the roughness of gratings varying in either R or G was affected by manipulating F(t). Overall, the effect of G on roughness judgments was much stronger than that of F(t), probably explaining why many previous studies using surfaces that varied only in inter-element spacing led to the conclusion that temporal factors play no role in roughness perception. However, the perceived roughness of R-varying gratings was determined by F(t) and not spatial variables. Roughness judgments were influenced by G and F(t) in a manner entirely consistent with predicted afferent response rates. Thus perceived roughness, like peripheral afferent responses, depends in part on temporal variables.

Feeling with the mind's eye: the role of visual imagery in tactile perception.

Cross-modal interactions are characteristic of normal perception. In this article, we discuss our work on cross-modal interactions between touch and vision in normally sighted humans. A region of extrastriate visual cortex, near the parieto-occipital fissure, is not only active during but also necessary for tactile discrimination of grating orientation (but not spatial frequency). This is consistent with a role for visual imagery in certain aspects of tactile perception. These findings have implications for the interpretation of visual cortical involvement in Braille reading by the blind.

Feeling with the mind's eye.

Mental imagery is thought to play a key role in certain aspects of visual perception and to depend on neural activity in visual cortex. We asked whether tactile discrimination of grating orientation, which appears to involve visual mental imagery, recruits visual cortical areas. H215O positron emission tomography was performed in humans during presentation of gratings to the right index fingerpad. Selective attention to grating orientation significantly increased regional cerebral blood flow, relative to a control task involving selective attention to grating dimensions, in a region located in left parieto-occipital cortex. We propose that this activation reflects the use of imagery-related visuo-spatial processes to enable the tactile discrimination of orientation.