Prior conscious experience modulates the impact of audiovisual temporal correspondence on unconscious visual processing.

Conscious visual experiences are enriched by concurrent auditory information, implying audiovisual interactions. In the present study, we investigated how prior conscious experience of auditory and visual information influences the subsequent audiovisual temporal integration under the surface of awareness. We used continuous flash suppression (CFS) to render perceptually invisible a ball-shaped object constantly moving and bouncing inside a square frame window. To examine whether audiovisual temporal correspondence facilitates the ball stimulus to enter awareness, the visual motion was accompanied by click sounds temporally congruent or incongruent with the bounces of the ball. In Experiment 1, where no prior experience of the audiovisual events was given, we found no significant impact of audiovisual correspondence on visual detection time. However, when the temporally congruent or incongruent bounce-sound relations were consciously experienced prior to CFS in Experiment 2, congruent sounds yielded faster detection time compared to incongruent sounds during CFS. In addition, in Experiment 3, explicit processing of the incongruent bounce-sound relation prior to CFS slowed down detection time when the ball bounces became later congruent with sounds during CFS. These findings suggest that audiovisual temporal integration may take place outside of visual awareness though its potency is modulated by previous conscious experiences of the audiovisual events. The results are discussed in light of the framework of multisensory causal inference.

Audiovisual interactions outside of visual awareness during motion adaptation.

Motion aftereffects (MAEs), illusory motion experienced in a direction opposed to real motion experienced during prior adaptation, have been used to assess audiovisual interactions. In a previous study from our laboratory, we demonstrated that a congruent direction of auditory motion presented concurrently with visual motion during adaptation strengthened the consequent visual MAE, compared to when auditory motion was incongruent in direction. Those judgments of MAE strength, however, could have been influenced by expectations or response bias from mere knowledge of the state of audiovisual congruity during adaptation. To prevent such knowledge, we now employed continuous flash suppression to render visual motion perceptually invisible during adaptation, ensuring that observers were completely unaware of visual adapting motion and only aware of the motion direction of the sound they were hearing. We found a small but statistically significant congruence effect of sound on adaptation strength produced by invisible adaptation motion. After considering alternative explanations for this finding, we conclude that auditory motion can impact the strength of visual processing produced by translational visual motion even when that motion transpires outside of awareness.

Complex Shapes Are Bluish, Darker, and More Saturated; Shape-Color Correspondence in 3D Object Perception.

It has been shown that there is a non-random association between shape and color. However, the results of previous studies on the shape-color correspondence did not converge. To address the issue, we focused on shape complexity among a number of shape properties, particularly in terms of 3D shape, and parametrically manipulated the shape complexity and all three components of color. With two experiments, the current study aimed to closely examine the correspondence between shape complexity of 3D shape and color in terms of hue (Experiment 1), luminance, and saturation (Experiment 2). Participants were presented with the 3D shapes in either visual or visuo-haptic modes of exploration. Subsequently, they had to pick from a color palette the color best matching each shape of the object. In Experiment 1, we found that as shapes became more complex, the best associated hue changed from those with long wavelengths to ones with short wavelengths. Results of Experiment 2 demonstrated that as the shapes grew more complex, the associated luminance decreased, and saturation increased. Additionally, adding haptic exploration to visual exploration strengthened the association - for saturation in particular - with the pattern of shape-color correspondence maintained. Taken together, we demonstrated that complex shapes are associated with bluish, darker and more saturated colors, suggesting that shape complexity has a systematic relationship with color including hue, luminance, and saturation.

Do the colors of your letters depend on your language? Language-dependent and universal influences on grapheme-color synesthesia in seven languages.

Grapheme-color synesthetes experience graphemes as having a consistent color (e.g., "N is turquoise"). Synesthetes' specific associations (which letter is which color) are often influenced by linguistic properties such as phonetic similarity, color terms ("Y is yellow"), and semantic associations ("D is for dog and dogs are brown"). However, most studies of synesthesia use only English-speaking synesthetes. Here, we measure the effect of color terms, semantic associations, and non-linguistic shape-color associations on synesthetic associations in Dutch, English, Greek, Japanese, Korean, Russian, and Spanish. The effect size of linguistic influences (color terms, semantic associations) differed significantly between languages. In contrast, the effect size of non-linguistic influences (shape-color associations), which we predicted to be universal, indeed did not differ between languages. We conclude that language matters (outcomes are influenced by the synesthete's language) and that synesthesia offers an exceptional opportunity to study influences on letter representations in different languages.

Familiarity and Novelty in Aesthetic Preference: The Effects of the Properties of the Artwork and the Beholder.

Familiarity and novelty are fundamental yet competing factors influencing aesthetic preference. However, whether people prefer familiar paintings or novel paintings has not been clear. Using both behavioral and eye-tracking measures, the present study aimed to investigate whether the effect of familiarity-novelty on aesthetic preference is independent or dependent on artwork properties (painting content, visual complexity) and viewer characteristics (experience in art). Participants were presented with two images of paintings, one of which was repeatedly presented but was always paired with a new painting in a randomized lateral arrangement. They were asked to indicate which of the two images they preferred with the degree of their preference. Behavioral results demonstrated an interactive influence of painting content and complexity on familiarity-novelty preference, especially alongside the distinction between representational and abstract paintings. Also, the familiarity-novelty preference was modulated by the degree of art experience, for abstract paintings in particular. Gaze results showed the differential effects of painting content, complexity, and art experience echoing the behavioral results. Taken together, the convergent results derived from behavioral and eye-tracking measures imply that novelty is an important feature of aesthetic appreciation, but its influence is modulated by properties of both the artwork and the beholder.

Development and validation of the food involvement inventory (FII) featuring the attitudinal constructs.

Food involvement is one of the personality traits that influences the food cognition and behavior of consumers. Given the irrelevance of previous food involvement scales such as measuring lifestyle diligence and a food culture, this study aimed to develop the food involvement inventory (FII) designed specifically for foods and current consumers. This study focused on identifying the factors/constructs of the FII that would best represent and explain food involvement through item generation, administration, and validation. The collected items from involvement-related scales and expert discussions were administrated based on the evidence of construct and content. Validation of the FII based on the exploratory factor analysis identified four attitudinal components as constructs (affective, cognitive, behavioral-purchase, and behavioral-cooking) within the final 25 items. These four constructs constituted structural models for the FII, and the models were shown to be acceptable in the confirmatory factor analysis.

Cross-Modal Correspondence Between Speech Sound and Visual Shape Influencing Perceptual Representation of Shape: the Role of Articulation and Pitch.

Cross-modal correspondence is the tendency to systematically map stimulus features across sensory modalities. The current study explored cross-modal correspondence between speech sound and shape (Experiment 1), and whether such association can influence shape representation (Experiment 2). For the purpose of closely examining the role of the two factors - articulation and pitch - combined in speech acoustics, we generated two sets of 25 vowel stimuli - pitch-varying and pitch-constant sets. Both sets were generated by manipulating articulation - frontness and height of the tongue body's positions - but differed in terms of whether pitch varied among the sounds within the same set. In Experiment 1, participants made a forced choice between a round and a spiky shape to indicate the shape better associated with each sound. Results showed that shape choice was modulated according to both articulation and pitch, and we therefore concluded that both factors play significant roles in sound-shape correspondence. In Experiment 2, participants reported their subjective experience of shape accompanied by vowel sounds by adjusting an ambiguous shape in the response display. We found that sound-shape correspondence exerts an effect on shape representation by modulating audiovisual interaction, but only in the case of pitch-varying sounds. Therefore, pitch information within vowel acoustics plays the leading role in sound-shape correspondence influencing shape representation. Taken together, our results suggest the importance of teasing apart the roles of articulation and pitch for understanding sound-shape correspondence.

Congruent audio-visual stimulation during adaptation modulates the subsequently experienced visual motion aftereffect.

Sensory information registered in one modality can influence perception associated with sensory information registered in another modality. The current work focuses on one particularly salient form of such multisensory interaction: audio-visual motion perception. Previous studies have shown that watching visual motion and listening to auditory motion influence each other, but results from those studies are mixed with regard to the nature of the interactions promoting that influence and where within the sequence of information processing those interactions transpire. To address these issues, we investigated whether (i) concurrent audio-visual motion stimulation during an adaptation phase impacts the strength of the visual motion aftereffect (MAE) during a subsequent test phase, and (ii) whether the magnitude of that impact was dependent on the congruence between auditory and visual motion experienced during adaptation. Results show that congruent direction of audio-visual motion during adaptation induced a stronger initial impression and a slower decay of the MAE than did the incongruent direction, which is not attributable to differential patterns of eye movements during adaptation. The audio-visual congruency effects measured here imply that visual motion perception emerges from integration of audio-visual motion information at a sensory neural stage of processing.

Why is the synesthete's "A" red? Using a five-language dataset to disentangle the effects of shape, sound, semantics, and ordinality on inducer-concurrent relationships in grapheme-color synesthesia.

Grapheme-color synesthesia is a neurological phenomenon in which viewing a grapheme elicits an additional, automatic, and consistent sensation of color. Color-to-letter associations in synesthesia are interesting in their own right, but also offer an opportunity to examine relationships between visual, acoustic, and semantic aspects of language. Research using large populations of synesthetes has indeed found that grapheme-color pairings can be influenced by numerous properties of graphemes, but the contributions made by each of these explanatory factors are often confounded in a monolingual dataset (i.e., only English-speaking synesthetes). Here, we report the first demonstration of how a multilingual dataset can reveal potentially-universal influences on synesthetic associations, and disentangle previously-confounded hypotheses about the relationship between properties of synesthetic color and properties of the grapheme that induces it. Numerous studies have reported that for English-speaking synesthetes, "A" tends to be colored red more often than predicted by chance, and several explanatory factors have been proposed that could explain this association. Using a five-language dataset (native English, Dutch, Spanish, Japanese, and Korean speakers), we compare the predictions made by each explanatory factor, and show that only an ordinal explanation makes consistent predictions across all five languages, suggesting that the English "A" is red because the first grapheme of a synesthete's alphabet or syllabary tends to be associated with red. We propose that the relationship between the first grapheme and the color red is an association between an unusually-distinct ordinal position ("first") and an unusually-distinct color (red). We test the predictions made by this theory, and demonstrate that the first grapheme is unusually distinct (has a color that is distant in color space from the other letters' colors). Our results demonstrate the importance of considering cross-linguistic similarities and differences in synesthesia, and suggest that some influences on grapheme-color associations in synesthesia might be universal.

[i] is Lighter and More Greenish Than [o]: Intrinsic Association Between Vowel Sounds and Colors.

It has recently been reported in the synesthesia literature that graphemes sharing the same phonetic feature tend to induce similar synesthetic colors. In the present study, we investigated whether phonetic properties are associated with colors in a specific manner among the general population, even when other visual and linguistic features of graphemes are removed. To test this hypothesis, we presented vowel sounds synthesized by systematically manipulating the position of the tongue body's center. Participants were asked to choose a color after hearing each sound. Results from the main experiment showed that lightness and chromaticity of matched colors exhibited systematic variations along the two axes of the position of the tongue body's center. Some non-random associations between vowel sounds and colors remained effective with pitch and intensity of the sounds equalized in the control experiment, which suggests that other acoustic factors such as inherent pitch of vowels cannot solely account for the current results. Taken together, these results imply that the association between phonetic features and colors is not random, and this synesthesia-like association is shared by people in the general population.

Sequential effects in preference decision: Prior preference assimilates current preference.

An important factor affecting preference formation is the context in which that preference decision takes place. The current research examined whether one's preference formed for a previously presented stimulus influences the processing of a subsequent preference decision, henceforth referred to as the preference sequence effect. Using a novel sequential rating/judgment paradigm, the present study demonstrated the presence of a preference sequence effect using artistic photographs and face stimuli: A neutral stimulus was preferred more following a preferable stimulus than a less preferable stimulus. Furthermore, a similar trend was found even when the potential influence of response bias was controlled. These results suggest that an assimilative sequential effect exists even when sequential judgments are made solely based on one's subjective feeling; preference formed for a preceding stimulus modulates preference for a subsequent stimulus. This implies the need for a consideration of trial sequence as a factor creating a psychological context affecting the subsequent preference decisions.

Audio-visual interactions uniquely contribute to resolution of visual conflict in people possessing absolute pitch.

Individuals possessing absolute pitch (AP) are able to identify a given musical tone or to reproduce it without reference to another tone. The present study sought to learn whether this exceptional auditory ability impacts visual perception under stimulus conditions that provoke visual competition in the form of binocular rivalry. Nineteen adult participants with 3-19 years of musical training were divided into two groups according to their performance on a task involving identification of the specific note associated with hearing a given musical pitch. During test trials lasting just over half a minute, participants dichoptically viewed a scrolling musical score presented to one eye and a drifting sinusoidal grating presented to the other eye; throughout the trial they pressed buttons to track the alternations in visual awareness produced by these dissimilar monocular stimuli. On "pitch-congruent" trials, participants heard an auditory melody that was congruent in pitch with the visual score, on "pitch-incongruent" trials they heard a transposed auditory melody that was congruent with the score in melody but not in pitch, and on "melody-incongruent" trials they heard an auditory melody completely different from the visual score. For both groups, the visual musical scores predominated over the gratings when the auditory melody was congruent compared to when it was incongruent. Moreover, the AP participants experienced greater predominance of the visual score when it was accompanied by the pitch-congruent melody compared to the same melody transposed in pitch; for non-AP musicians, pitch-congruent and pitch-incongruent trials yielded equivalent predominance. Analysis of individual durations of dominance revealed differential effects on dominance and suppression durations for AP and non-AP participants. These results reveal that AP is accompanied by a robust form of bisensory interaction between tonal frequencies and musical notation that boosts the salience of a visual score.

Monocular Perceptual Deprivation from Interocular Suppression Temporarily Imbalances Ocular Dominance.

Early visual experience sculpts neural mechanisms that regulate the balance of influence exerted by the two eyes on cortical mechanisms underlying binocular vision [1, 2], and experience's impact on this neural balancing act continues into adulthood [3-5]. One recently described, compelling example of adult neural plasticity is the effect of patching one eye for a relatively short period of time: contrary to intuition, monocular visual deprivation actually improves the deprived eye's competitive advantage during a subsequent period of binocular rivalry [6-8], the robust form of visual competition prompted by dissimilar stimulation of the two eyes [9, 10]. Neural concomitants of this improvement in monocular dominance are reflected in measurements of brain responsiveness following eye patching [11, 12]. Here we report that patching an eye is unnecessary for producing this paradoxical deprivation effect: interocular suppression of an ordinarily visible stimulus being viewed by one eye is sufficient to produce shifts in subsequent predominance of that eye to an extent comparable to that produced by patching the eye. Moreover, this imbalance in eye dominance can also be induced by prior, extended viewing of two monocular images differing only in contrast. Regardless of how shifts in eye dominance are induced, the effect decays once the two eyes view stimuli equal in strength. These novel findings implicate the operation of interocular neural gain control that dynamically adjusts the relative balance of activity between the two eyes [13, 14].

Graphemes Sharing Phonetic Features Tend to Induce Similar Synesthetic Colors.

Individuals with grapheme-color synesthesia experience idiosyncratic colors when viewing achromatic letters or digits. Despite large individual differences in grapheme-color association, synesthetes tend to associate graphemes sharing a perceptual feature with similar synesthetic colors. Sound has been suggested as one such feature. In the present study, we investigated whether graphemes of which representative phonemes have similar phonetic features tend to be associated with analogous synesthetic colors. We tested five Korean multilingual synesthetes on a color-matching task using graphemes from Korean, English, and Japanese orthography. We then compared the similarity of synesthetic colors induced by those characters sharing a phonetic feature. Results showed that graphemes associated with the same phonetic feature tend to induce synesthetic color in both within- and cross-script analyses. Moreover, this tendency was consistent for graphemes that are not transliterable into each other as well as graphemes that are. These results suggest that it is the perceptual-i.e., phonetic-properties associated with graphemes, not just conceptual associations such as transliteration, that determine synesthetic color.

Gender Differences in Neural Responses to Perceptually Invisible Fearful Face-An ERP Study.

Women tend to respond to emotional stimuli differently from men. This study aimed at investigating whether neural responses to perceptually "invisible" emotional stimuli differ between men and women by exploiting event-related potential (ERP). Forty healthy participants (21 women) were recruited for the main experiment. A control experiment was conducted by excluding nine (7 women) participants from the main experiment and replacing them with additional ten (6 women) participants (total 41 participants) where Beck's Anxiety Inventory (BAI) and Beck's Depression Inventory (BDI) scores were controlled. Using the visual backward masking paradigm, either a fearful or a neutral face stimulus was presented in varied durations (subthreshold, near-threshold, or suprathreshold) followed by a mask. Participants performed a two-alternative forced choice (2-AFC) emotion discrimination task on each face. Behavioral analysis showed that participants were unaware of masked stimuli of which duration was the shortest and, therefore, processed at subthreshold. Nevertheless, women showed significantly larger response in P100 amplitude to subthreshold fearful faces than men. This result remained consistent in the control experiment. Our findings indicate gender-differences in neural response to subthreshold emotional face, which is reflected in the early processing stage.

Attentional processing of emotional faces in schizophrenia: Evidence from eye tracking.

Severe emotional disturbances such as anxiety and depression have been closely related to aberrant attentional processing of emotional stimuli. However, this has been little studied in schizophrenia, which is also characterized by marked emotional impairments such as heightened negative affect and anhedonia. In the current study, we investigated temporal dynamics of motivated attention to emotional stimuli in schizophrenia. For this purpose, we tracked eye movements of 22 individuals with schizophrenia or schizoaffective disorder (ISZs) and 19 healthy controls (HCs) to emotional (i.e., happy, sad, angry) and neutral face pairs presented either for 500 ms or 1,500 ms. Initial fixation direction and viewing time at 3 successive intervals (0-500, 500-1,000, 1,000-1,500 ms) were calculated. The results showed that both ISZs and HCs were more likely to orient initial fixations and exhibited longer viewing times to emotional than neutral faces. However, compared with HCs, ISZs allocated less attention to overall faces during the late stage (1,000-1,500 ms) when one of the paired faces displayed negative emotions. Furthermore, positive symptoms were highly associated with initial fixation avoidance to angry faces while depressive symptoms were related to later avoidance of angry faces. Both social amotivation and poor interpersonal functioning were closely related to diminished sustained attention to happy faces. This suggests that early attentional capture of emotional salience may be relatively preserved in schizophrenia, but the people with this disorder display an atypical late attentional process characterized by generalized attentional avoidance of negative stimuli. Of note, aberrant attentional processes of social threat and reward were closely associated with major symptoms and functioning in this disorder. (PsycINFO Database Record

Non-Conscious Perception of Emotions in Psychiatric Disorders: The Unsolved Puzzle of Psychopathology.

Psychophysiological and functional neuroimaging studies have frequently and consistently shown that emotional information can be processed outside of the conscious awareness. Non-conscious processing comprises automatic, uncontrolled, and fast processing that occurs without subjective awareness. However, how such non-conscious emotional processing occurs in patients with various psychiatric disorders requires further examination. In this article, we reviewed and discussed previous studies on the non-conscious emotional processing in patients diagnosed with anxiety disorder, schizophrenia, bipolar disorder, and depression, to further understand how non-conscious emotional processing varies across these psychiatric disorders. Although the symptom profile of each disorder does not often overlap with one another, these patients commonly show abnormal emotional processing based on the pathology of their mood and cognitive function. This indicates that the observed abnormalities of emotional processing in certain social interactions may derive from a biased mood or cognition process that precedes consciously controlled and voluntary processes. Since preconscious forms of emotional processing appear to have a major effect on behaviour and cognition in patients with these disorders, further investigation is required to understand these processes and their impact on patient pathology.

Melodic sound enhances visual awareness of congruent musical notes, but only if you can read music.

Predictive influences of auditory information on resolution of visual competition were investigated using music, whose visual symbolic notation is familiar only to those with musical training. Results from two experiments using different experimental paradigms revealed that melodic congruence between what is seen and what is heard impacts perceptual dynamics during binocular rivalry. This bisensory interaction was observed only when the musical score was perceptually dominant, not when it was suppressed from awareness, and it was observed only in people who could read music. Results from two ancillary experiments showed that this effect of congruence cannot be explained by differential patterns of eye movements or by differential response sluggishness associated with congruent score/melody combinations. Taken together, these results demonstrate robust audiovisual interaction based on high-level, symbolic representations and its predictive influence on perceptual dynamics during binocular rivalry.

Both "나" and "な" are yellow: cross-linguistic investigation in search of the determinants of synesthetic color.

Individuals with grapheme-color synesthesia experience "colors" when viewing achromatic letters and digits. Despite the large individual difference in synesthetic association between inducing graphemes and induced colors, the search for the determinants of synesthetic experience has begun. So far, however, research has drawn an inconsistent picture; some studies have shown that graphemes of similar visual shape tend to induce similar synesthetic colors, while others suggested sound as an important factor. Moreover, meaning seems to affect synesthetic color. In the present work, we sought to investigate the determinants of synesthetic color by testing four multilingual grapheme-color synesthetes who experience "colors" upon viewing Korean (hangul), Japanese (katakana and hiragana), and English (Latin alphabet) characters on a standardized color-matching procedure. Results showed that pairs of characters of matched sound tended to induce similar synesthetic colors. This was the case not only between two scripts within the same language (Japanese hiragana and katakana) but also between two different languages (Japanese and Korean). In addition, pairs of characters with similar initial phonemes tended to induce similar colors; this was general across multiple languages. Results also showed that pairs of sequential words in Korean, Japanese, English, and Chinese that have the same meaning tended to elicit similar synesthetic colors. When those pairs of words shared not only meaning but also sound, the similarity of the induced synesthetic colors was even greater. Our work is one of the few initial attempts to examine the influence of visual shape, sound, meaning, and their interaction on synesthetic color induced by characters across multiple languages.

Is "Σ" purple or green? Bistable grapheme-color synesthesia induced by ambiguous characters.

People with grapheme-color synesthesia perceive specific colors when viewing different letters or numbers. Previous studies have suggested that synesthetic color experience can be bistable when induced by an ambiguous character. However, the exact relationship between processes underlying the identity of an alphanumeric character and the experience of the induced synesthetic color has not been examined. In the present study, we explored this by focusing on the temporal relation of inducer identification and color emergence using inducers whose identity could be rendered ambiguous upon rotation of the characters. Specifically, achromatic alphabetic letters (W/M) and digits (6/9) were presented at varying angles to 9 grapheme-color synesthetes. Results showed that grapheme identification and synesthetically perceived grapheme color covary with the orientation of the test stimulus and that synesthetes were slower naming the experienced color than identifying the character, particularly at intermediate angles where ambiguity was greatest.