Steady state visual evoked potentials reveal a signature of the pitch-size crossmodal association in visual cortex.

Crossmodal correspondences describe our tendency to associate sensory features from different modalities with each other, such as the pitch of a sound with the size of a visual object. While such crossmodal correspondences (or associations) are described in many behavioural studies their neurophysiological correlates remain unclear. Under the current working model of multisensory perception both a low- and a high-level account seem plausible. That is, the neurophysiological processes shaping these associations could commence in low-level sensory regions, or may predominantly emerge in high-level association regions of semantic and object identification networks. We exploited steady-state visual evoked potentials (SSVEP) to directly probe this question, focusing on the associations between pitch and the visual features of size, hue or chromatic saturation. We found that SSVEPs over occipital regions are sensitive to the congruency between pitch and size, and a source analysis pointed to an origin around primary visual cortices. We speculate that this signature of the pitch-size association in low-level visual cortices reflects the successful pairing of congruent visual and acoustic object properties and may contribute to establishing causal relations between multisensory objects. Besides this, our study also provides a paradigm can be exploited to study other crossmodal associations involving visual stimuli in the future.

Feature Correspondences Increase and Hybrid Terms Optimization Warp for Image Stitching.

Feature detection and correct matching are the basis of the image stitching process. Whether the matching is correct and the number of matches directly affect the quality of the final stitching results. At present, almost all image stitching methods use SIFT+RANSAC pattern to extract and match feature points. However, it is difficult to obtain sufficient correct matching points in low-textured or repetitively-textured regions, resulting in insufficient matching points in the overlapping region, and this further leads to the warping model being estimated erroneously. In this paper, we propose a novel and flexible approach by increasing feature correspondences and optimizing hybrid terms. It can obtain sufficient correct feature correspondences in the overlapping region with low-textured or repetitively-textured areas to eliminate misalignment. When a weak texture and large parallax coexist in the overlapping region, the alignment and distortion often restrict each other and are difficult to balance. Accurate alignment is often accompanied by projection distortion and perspective distortion. Regarding this, we propose hybrid terms optimization warp, which combines global similarity transformations on the basis of initial global homography and estimates the optimal warping by adjusting various term parameters. By doing this, we can mitigate projection distortion and perspective distortion, while effectively balancing alignment and distortion. The experimental results demonstrate that the proposed method outperforms the state-of-the-art in accurate alignment on images with low-textured areas in the overlapping region, and the stitching results have less perspective and projection distortion.

Fast and Accurate Pose Estimation with Unknown Focal Length Using Line Correspondences.

Estimating camera pose is one of the key steps in computer vison, photogrammetry and SLAM (Simultaneous Localization and Mapping). It is mainly calculated based on the 2D-3D correspondences of features, including 2D-3D point and line correspondences. If a zoom lens is equipped, the focal length needs to be estimated simultaneously. In this paper, a new method of fast and accurate pose estimation with unknown focal length using two 2D-3D line correspondences and the camera position is proposed. Our core contribution is to convert the PnL (perspective-n-line) problem with 2D-3D line correspondences into an estimation problem with 3D-3D point correspondences. One 3D line and the camera position in the world frame can define a plane, the 2D line projection of the 3D line and the camera position in the camera frame can define another plane, and actually the two planes are the same plane, which is the key geometric characteristic in this paper's estimation of focal length and pose. We establish the transform between the normal vectors of the two planes with this characteristic, and this transform can be regarded as the camera projection of a 3D point. Then, the pose estimation using 2D-3D line correspondences is converted into pose estimation using 3D-3D point correspondences in intermediate frames, and, lastly, pose estimation can be finished quickly. In addition, using the property whereby the angle between two planes is invariant in both the camera frame and world frame, we can estimate the camera focal length quickly and accurately. Experimental results show that our proposed method has good performance in numerical stability, noise sensitivity and computational speed with synthetic data and real scenarios, and has strong robustness to camera position noise.

Exhaustive Search of Correspondences between Multimodal Remote Sensing Images Using Convolutional Neural Network.

Finding putative correspondences between a pair of images is an important prerequisite for image registration. In complex cases such as multimodal registration, a true match could be less plausible than a false match within a search zone. Under these conditions, it is important to detect all plausible matches. This could be achieved by an exhaustive search using a handcrafted similarity measure (SM, e.g., mutual information). It is promising to replace handcrafted SMs with deep learning ones that offer better performance. However, the latter are not designed for an exhaustive search of all matches but for finding the most plausible one. In this paper, we propose a deep-learning-based solution for exhaustive multiple match search between two images within a predefined search area. We design a computationally efficient convolutional neural network (CNN) that takes as input a template fragment from one image, a search fragment from another image and produces an SM map covering the entire search area in spatial dimensions. This SM map finds multiple plausible matches, locates each match with subpixel accuracy and provides a covariance matrix of localization errors for each match. The proposed CNN is trained with a specially designed loss function that enforces the translation and rotation invariance of the SM map and enables the detection of matches that have no associated ground truth data (e.g., multiple matches for repetitive textures). We validate the approach on multimodal remote sensing images and show that the proposed "area" SM performs better than "point" SM.

Using information-theoretic measures to characterize the structure of the writing system: the case of orthographic-phonological regularities in English.

It is generally well accepted that proficient reading requires the assimilation of myriad statistical regularities present in the writing system, including in particular the correspondences between words' orthographic and phonological forms. There is considerably less agreement, however, as to how to quantify these regularities. Here we present a comprehensive approach for this quantification using tools from Information Theory. We start by providing a glossary of the relevant information-theoretic metrics, with simplified examples showing their potential in assessing orthographic-phonological regularities. We specifically highlight the flexibility of our approach in quantifying information under different contexts (i.e., context-independent and dependent readings) and in different types of mappings (e.g., orthography-to-phonology and phonology-to-orthography). Then, we use these information-theoretic measures to assess real-world orthographic-phonological regularities of 10,093 mono-syllabic English words and examine whether these measures predict inter-item variability in accuracy and response times using available large-scale datasets of naming and lexical decision tasks. Together, the analyses demonstrate how information-theoretical measures can be used to quantify orthographical-phonological correspondences, and show that they capture variance in reading performance that is not accounted for by existing measures. We discuss the similarities and differences between the current framework and previous approaches as well as future directions towards understanding how the statistical regularities embedded in a writing system impact reading and reading acquisition.

Letter Teaching in Parent-Child Conversations.

Talk about letters is an important part of the home literacy environment. Such talk has been studied primarily through questionnaires, but these are limited in the amount of information they provide. Here we analyzed conversations between 55 U.S. children and their parents who were visited in their homes every 4 months when the child was between 1.2 and 4.8 years old. We examined the aspects of alphabet knowledge that parents and children discussed, the materials they used, and how these varied with the age of the child and the socioeconomic status of the family. Children primarily focused on identifying letters, while parents also emphasized letter writing and spelling. Talk about the associations between letters and sounds, which is critical in learning to read and write, was less common than anticipated based on the results of questionnaire studies. Teachers should thus not overestimate the knowledge of letter sounds that children acquire at home.

Audio-visual crossmodal correspondences in domestic dogs (Canis familiaris).

Crossmodal correspondences are intuitively held relationships between non-redundant features of a stimulus, such as auditory pitch and visual illumination. While a number of correspondences have been identified in humans to date (e.g. high pitch is intuitively felt to be luminant, angular and elevated in space), their evolutionary and developmental origins remain unclear. Here, we investigated the existence of audio-visual crossmodal correspondences in domestic dogs, and specifically, the known human correspondence in which high auditory pitch is associated with elevated spatial position. In an audio-visual attention task, we found that dogs engaged more with audio-visual stimuli that were congruent with human intuitions (high auditory pitch paired with a spatially elevated visual stimulus) compared to incongruent (low pitch paired with elevated visual stimulus). This result suggests that crossmodal correspondences are not a uniquely human or primate phenomenon and they cannot easily be dismissed as merely lexical conventions (i.e. matching 'high' pitch with 'high' elevation).

Development of context-sensitive pronunciation in reading: The case of and .

Writing systems sometimes deviate from one-to-one associations between letters and phonemes, but the deviations are often predictable from sublexical context. For initial and in English, deviations from the typical /k/ and /g/ pronunciations are influenced by adjacent context (the following vowel, as in center vs. canter) and nonadjacent context (the presence of a Latinate vs. basic suffix, as in gigantic vs. giggling). We conducted two experiments with participants ranging in reading level from early elementary school to university to study the development of context use. Experiment 1 focused on adjacent context, and Experiment 2 also examined nonadjacent context. Use of context developed slowly, and readers at all levels were not as influenced by it as would be expected given the contextual effects in the English vocabulary. We discuss possible reasons for these phenomena and the need to teach children to use context more effectively.

Aircraft Pose Estimation Based on Geometry Structure Features and Line Correspondences.

A robust and accurate aircraft pose estimation method is proposed in this paper. The aircraft pose reflects the flight status of the aircraft and accurate pose measurement is of great importance in many aerospace applications. This work aims to establish a universal framework to estimate the aircraft pose based on generic geometry structure features. In our method, line features are extracted to describe the structure of an aircraft in single images and the generic geometry features are exploited to form line groups for aircraft structure recognition. Parallel line clustering is utilized to detect the fuselage reference line and bilateral symmetry property of aircraft provides an important constraint for the extraction of wing edge lines under weak perspective projection. After identifying the main structure of the aircraft, a planes intersection method is used to obtain the 3D pose parameters based on the established line correspondences. Our proposed method can increase the measuring range of binocular vision sensors and has the advantage of not relying on 3D models, cooperative marks or other feature datasets. Experimental results show that our method can obtain reliable and accurate pose information of different types of aircraft.

Cross-modal associations and synesthesia: Categorical perception and structure in vowel-color mappings in a large online sample.

We report associations between vowel sounds, graphemes, and colors collected online from over 1,000 Dutch speakers. We also provide open materials, including a Python implementation of the structure measure and code for a single-page web application to run simple cross-modal tasks. We also provide a full dataset of color-vowel associations from 1,164 participants, including over 200 synesthetes identified using consistency measures. Our analysis reveals salient patterns in the cross-modal associations and introduces a novel measure of isomorphism in cross-modal mappings. We found that, while the acoustic features of vowels significantly predict certain mappings (replicating prior work), both vowel phoneme category and grapheme category are even better predictors of color choice. Phoneme category is the best predictor of color choice overall, pointing to the importance of phonological representations in addition to acoustic cues. Generally, high/front vowels are lighter, more green, and more yellow than low/back vowels. Synesthetes respond more strongly on some dimensions, choosing lighter and more yellow colors for high and mid front vowels than do nonsynesthetes. We also present a novel measure of cross-modal mappings adapted from ecology, which uses a simulated distribution of mappings to measure the extent to which participants' actual mappings are structured isomorphically across modalities. Synesthetes have mappings that tend to be more structured than nonsynesthetes', and more consistent color choices across trials correlate with higher structure scores. Nevertheless, the large majority (~ 70%) of participants produce structured mappings, indicating that the capacity to make isomorphically structured mappings across distinct modalities is shared to a large extent, even if the exact nature of the mappings varies across individuals. Overall, this novel structure measure suggests a distribution of structured cross-modal association in the population, with synesthetes at one extreme and participants with unstructured associations at the other.

Newborns are sensitive to the correspondence between auditory pitch and visuospatial elevation.

Amodal (redundant) and arbitrary cross-sensory feature associations involve the context-insensitive mapping of absolute feature values across sensory domains. Cross-sensory associations of a different kind, known as correspondences, involve the context-sensitive mapping of relative feature values. Are such correspondences in place at birth (like amodal associations), or are they learned from subsequently experiencing relevant feature co-occurrences in the world (like arbitrary associations)? To decide between these two possibilities, human newborns (median age = 44 hr) watched animations in which two balls alternately rose and fell together in space. The pitch of an accompanying sound rose and fell either congruently with this visual change (pitch rising and falling as the balls moved up and down), or incongruently (pitch rising and falling as the balls moved down and up). Newborns' looking behavior was sensitive to this congruence, providing the strongest indication to date that cross-sensory correspondences can be in place at birth.

"A sweet smile": the modulatory role of emotion in how extrinsic factors influence taste evaluation.

It has recently been demonstrated that the reported tastes/flavours of food/beverages can be modulated by means of external visual and auditory stimuli such as typeface, shapes, and music. The present study was designed to assess the role of the emotional valence of the product-extrinsic stimuli in such crossmodal modulations of taste. Participants evaluated samples of mixed fruit juice whilst simultaneously being presented with auditory or visual stimuli having either positive or negative valence. The soundtracks had either been harmonised with consonant (positive valence) or dissonant (negative valence) musical intervals. The visual stimuli consisted of images of emotional faces from the International Affective Picture System (IAPS) with valence ratings matched to the soundtracks. Each juice sample was rated on two computer-based scales: One anchored with the words sour and sweet, while the other scale required hedonic ratings. Those participants who tasted the juice sample while presented with the positively-valenced stimuli rated the juice as tasting sweeter compared to negatively-valenced stimuli, regardless of whether the stimuli were visual or auditory. These results suggest that the emotional valence of food-extrinsic stimuli can play a role in shaping food flavour evaluation and liking.

Cross-Sensory Correspondences: Heaviness is Dark and Low-Pitched.

Everyday language reveals how stimuli encoded in one sensory feature domain can possess qualities normally associated with a different domain (e.g., higher pitch sounds are bright, light in weight, sharp, and thin). Such cross-sensory associations appear to reflect crosstalk among aligned (corresponding) feature dimensions, including brightness, heaviness, and sharpness. Evidence for heaviness being one such dimension is very limited, with heaviness appearing primarily as a verbal associate of other feature contrasts (e.g., darker objects and lower pitch sounds are heavier than their opposites). Given the presumed bidirectionality of the crosstalk between corresponding dimensions, heaviness should itself induce the cross-sensory associations observed elsewhere, including with brightness and pitch. Taking care to dissociate effects arising from the size and mass of an object, this is confirmed. When hidden objects varying independently in size and mass are lifted, objects that feel heavier are judged to be darker and to make lower pitch sounds than objects feeling less heavy. These judgements track the changes in perceived heaviness induced by the size-weight illusion. The potential involvement of language, natural scene statistics, and Bayesian processes in correspondences, and the effects they induce, is considered.

"Smooth operator": Music modulates the perceived creaminess, sweetness, and bitterness of chocolate.

There has been a recent growth of interest in determining whether sound (specifically music and soundscapes) can enhance not only the basic taste attributes associated with food and beverage items (such as sweetness, bitterness, sourness, etc.), but also other important components of the tasting experience, such as, for instance, crunchiness, creaminess, and/or carbonation. In the present study, participants evaluated the perceived creaminess of chocolate. Two contrasting soundtracks were produced with such texture-correspondences in mind, and validated by means of a pre-test. The participants tasted the same chocolate twice (without knowing that the chocolates were identical), each time listening to one of the soundtracks. The 'creamy' soundtrack enhanced the perceived creaminess and sweetness of the chocolates, as compared to the ratings given while listening to the 'rough' soundtrack. Moreover, while the participants preferred the creamy soundtrack, this difference did not appear to affect their overall enjoyment of the chocolates. Interestingly, and in contrast with previous similar studies, these results demonstrate that in certain cases, sounds can have a perceptual effect on gustatory food attributes without necessarily altering the hedonic experience.

"Turn Up the Taste": Assessing the Role of Taste Intensity and Emotion in Mediating Crossmodal Correspondences between Basic Tastes and Pitch.

People intuitively match basic tastes to sounds of different pitches, and the matches that they make tend to be consistent across individuals. It is, though, not altogether clear what governs such crossmodal mappings between taste and auditory pitch. Here, we assess whether variations in taste intensity influence the matching of taste to pitch as well as the role of emotion in mediating such crossmodal correspondences. Participants were presented with 5 basic tastants at 3 concentrations. In Experiment 1, the participants rated the tastants in terms of their emotional arousal and valence/pleasantness, and selected a musical note (from 19 possible pitches ranging from C2 to C8) and loudness that best matched each tastant. In Experiment 2, the participants made emotion ratings and note matches in separate blocks of trials, then made emotion ratings for all 19 notes. Overall, the results of the 2 experiments revealed that both taste quality and concentration exerted a significant effect on participants' loudness selection, taste intensity rating, and valence and arousal ratings. Taste quality, not concentration levels, had a significant effect on participants' choice of pitch, but a significant positive correlation was observed between individual perceived taste intensity and pitch choice. A significant and strong correlation was also demonstrated between participants' valence assessments of tastants and their valence assessments of the best-matching musical notes. These results therefore provide evidence that: 1) pitch-taste correspondences are primarily influenced by taste quality, and to a lesser extent, by perceived intensity; and 2) such correspondences may be mediated by valence/pleasantness.

Drawing sounds: representing tones and chords spatially.

Research on the crossmodal correspondences has revealed that seemingly unrelated perceptual information can be matched across the senses in a manner that is consistent across individuals. An interesting extension of this line of research is to study how sensory information biases action. In the present study, we investigated whether different sounds (i.e. tones and piano chords) would bias participants' hand movements in a free movement task. Right-handed participants were instructed to move a computer mouse in order to represent three tones and two chords. They also had to rate each sound in terms of three visual analogue scales (slow-fast, unpleasant-pleasant, and weak-strong). The results demonstrate that tones and chords influence hand movements, with higher-(lower-)pitched sounds giving rise to a significant bias towards upper (lower) locations in space. These results are discussed in terms of the literature on forward models, embodied cognition, crossmodal correspondences, and mental imagery. Potential applications sports and rehabilitation are discussed briefly.

A Probabilistic Feature Map-Based Localization System Using a Monocular Camera.

Image-based localization is one of the most widely researched localization techniques in the robotics and computer vision communities. As enormous image data sets are provided through the Internet, many studies on estimating a location with a pre-built image-based 3D map have been conducted. Most research groups use numerous image data sets that contain sufficient features. In contrast, this paper focuses on image-based localization in the case of insufficient images and features. A more accurate localization method is proposed based on a probabilistic map using 3D-to-2D matching correspondences between a map and a query image. The probabilistic feature map is generated in advance by probabilistic modeling of the sensor system as well as the uncertainties of camera poses. Using the conventional PnP algorithm, an initial camera pose is estimated on the probabilistic feature map. The proposed algorithm is optimized from the initial pose by minimizing Mahalanobis distance errors between features from the query image and the map to improve accuracy. To verify that the localization accuracy is improved, the proposed algorithm is compared with the conventional algorithm in a simulation and realenvironments.

Sound symbolism in synesthesia: evidence from a lexical-gustatory synesthete.

Synesthesia is a condition in which perceptual or cognitive stimuli (e.g., a written letter) trigger atypical additional percepts (e.g., the color yellow). Although these cross-modal pairings appear idiosyncratic in that they superficially differ from synesthete to synesthete, underlying patterns do exist and these can, in some circumstances, reflect the cross-modal intuitions of nonsynesthetes (e.g., higher pitch sounds tend to be "seen" in lighter colors by synesthetes and are also paired to lighter colors by nonsynesthetes in cross-modal matching tasks). We recently showed that grapheme-color synesthetes are more sensitive to sound symbolism (i.e., cross-modal sound-meaning correspondences) in natural language compared to nonsynesthetes. Accordingly, we hypothesize that sound symbolism may be a guiding force in synesthesia to dictate what types of synesthetic experiences are triggered by words. We tested this hypothesis by examining the cross-modal mappings of lexical-gustatory synesthete, JIW, for whom words trigger flavor experiences. We show that certain phonological features (e.g., front vowels) systematically trigger particular categories of taste (e.g., bitter) in his synesthesia. Some of these associations agree with sound symbolic patterns in natural language. This supports the view that synesthesia may be an exaggeration of cross-modal associations found in the general population and that sound symbolic properties of language may arise from similar mechanisms as those found in synesthesia.

Colour/shape-taste correspondences across three languages in ChatGPT.

Crossmodal correspondences, the tendency for a sensory feature / attribute in one sensory modality (either physically present or merely imagined), to be associated with a sensory feature in another sensory modality, have been studied extensively, revealing consistent patterns, such as sweet tastes being associated with pink colours and round shapes across languages. The present research explores whether such correspondences are captured by ChatGPT, a large language model developed by OpenAI. Across twelve studies, this research investigates colour/shapes-taste crossmodal correspondences in ChatGPT-3.5 and -4o, focusing on associations between shapes/colours and the five basic tastes across three languages (English, Japanese, and Spanish). Studies 1A-F examined taste-shape associations, using prompts in three languages to assess ChatGPT's association of round and angular shapes with the five basic tastes. The results indicated significant, consistent, associations between shape and taste, with, for example, round shapes strongly associated with sweet/umami tastes and angular shapes with bitter/salty/sour tastes. The magnitude of shape-taste matching appears to be greater in ChatGPT-4o than in ChatGPT-3.5, and ChatGPT prompted in English and Spanish than ChatGPT prompted in Japanese. Studies 2A-F focused on colour-taste correspondences, using ChatGPT to assess associations between eleven colours and the five basic tastes. The results indicated that ChatGPT-4o, but not ChatGPT-3.5, generally replicates the patterns of colour-taste correspondences that have previously been observed in human participants. Specifically, ChatGPT-4o associates sweet tastes with pink, sour with yellow, salty with white/blue, bitter with black, and umami with red across languages. However, the magnitude/similarity of shape/colour-taste matching observed in ChatGPT-4o appears to be more pronounced (i.e., having little variance, large mean difference), which does not adequately reflect the subtle nuances typically seen in human shape/colour-taste correspondences. These findings suggest that ChatGPT captures colour/shapes-taste correspondences, with language- and GPT version-specific variations, albeit with some differences when compared to previous studies involving human participants. These findings contribute valuable knowledge to the field of crossmodal correspondences, explore the possibility of generative AI that resembles human perceptual systems and cognition across languages, and provide insight into the development and evolution of generative AI systems that capture human crossmodal correspondences.

Hearing temperatures: employing machine learning for elucidating the cross-modal perception of thermal properties through audition.

People can use their sense of hearing for discerning thermal properties, though they are for the most part unaware that they can do so. While people unequivocally claim that they cannot perceive the temperature of pouring water through the auditory properties of hearing it being poured, our research further strengthens the understanding that they can. This multimodal ability is implicitly acquired in humans, likely through perceptual learning over the lifetime of exposure to the differences in the physical attributes of pouring water. In this study, we explore people's perception of this intriguing cross modal correspondence, and investigate the psychophysical foundations of this complex ecological mapping by employing machine learning. Our results show that not only can the auditory properties of pouring water be classified by humans in practice, the physical characteristics underlying this phenomenon can also be classified by a pre-trained deep neural network.