Advances in Understanding the Phenomena and Processing in Audiovisual Speech Perception.

The Special Issue entitled "Advances in Understanding the Phenomena and Processing in Audiovisual Speech Perception" attracted a variety of articles written by prominent authors in the field [...].

Investigation of Cross-Language and Stimulus-Dependent Effects on the McGurk Effect with Finnish and Japanese Speakers and Listeners.

In the McGurk effect, perception of a spoken consonant is altered when an auditory (A) syllable is presented with an incongruent visual (V) syllable (e.g., A/pa/V/ka/ is often heard as /ka/ or /ta/). The McGurk effect provides a measure for visual influence on speech perception, becoming stronger the lower the proportion of auditory correct responses. Cross-language effects are studied to understand processing differences between one's own and foreign languages. Regarding the McGurk effect, it has sometimes been found to be stronger with foreign speakers. However, other studies have shown the opposite, or no difference between languages. Most studies have compared English with other languages. We investigated cross-language effects with native Finnish and Japanese speakers and listeners. Both groups of listeners had 49 participants. The stimuli (/ka/, /pa/, /ta/) were uttered by two female and male Finnish and Japanese speakers and presented in A, V and AV modality, including a McGurk stimulus A/pa/V/ka/. The McGurk effect was stronger with Japanese stimuli in both groups. Differences in speech perception were prominent between individual speakers but less so between native languages. Unisensory perception correlated with McGurk perception. These findings suggest that stimulus-dependent features contribute to the McGurk effect. This may have a stronger influence on syllable perception than cross-language factors.

Exploring body consciousness of dancers, athletes, and lightly physically active adults.

Body consciousness is associated with kinetic skills and various aspects of wellbeing. Physical activities have been shown to contribute to the development of body consciousness. Methodological studies are needed in improving the assessment of body consciousness in adults with distinct physical activity backgrounds. This study (1) examined whether dancers, athletes, and lightly physically active individuals differed regarding the level of their body consciousness, and (2) evaluated the usability of different methods in assessing body consciousness. Fifty-seven healthy adults (aged 20-37) were included in the study. Three experimental methods (aperture task, endpoint matching, and posture copying) and two self-report questionnaires (the Private Body Consciousness Scale, PBCS, and the Body Awareness Questionnaire, BAQ) were used in assessing body consciousness. Athletes outperformed the lightly physically active participants in the posture copying task with the aid of vision when copying leg postures. Dancers performed better than the athletes without the aid of vision when their back and upper body were involved, and better than the lightly active participants when copying leg postures. Dancers and athletes had higher self-reported cognitive and perceptual knowledge of their body than lightly physically active participants. To examine the role of different physical activities in developing body consciousness, experimental methods involving the use of the whole body might be most suitable. Subjective measures may provide complementary evidence for experimental testing.

The Paradox of Fiction Revisited-Improvised Fictional and Real-Life Social Rejections Evoke Associated and Relatively Similar Psychophysiological Responses.

Theatre-based practices, such as improvisation, are frequently applied to simulate everyday social interactions. Although the improvisational context is acknowledged as fictional, realistic emotions may emerge, a phenomenon labelled the 'paradox of fiction'. This study investigated how manipulating the context (real-life versus fictional) modulates psychophysiological reactivity to social rejection during dyadic interactions. We measured psychophysiological responses elicited during real-life (interview) and fictional (improvisation exercises) social rejections. We analysed the heart rate (HR), skin conductance, facial muscle activity, and electrocortical activity (electroencephalographic (EEG) alpha asymmetry) of student teachers (N = 39) during various social rejections (devaluing, interrupting, nonverbal rejection). All social rejections evoked negative EEG alpha asymmetry, a measure reflecting behavioural withdrawal motivation. Psychophysiological responses during real-life and fictional rejections correlated, and rejection type modified the responses. When comparing responses across all rejection types, facial muscle activity and EEG alpha asymmetry did not differ between real-life and fictional rejections, whereas HR decelerated and skin conductance increased during fictional rejections. These findings demonstrate that regardless of cognitive awareness of fictionality, relatively subtle social rejections elicited psychophysiological reactivity indicating emotional arousal and negative valence. These findings provide novel, biological evidence for the application of theatre-based improvisation to studying experientially everyday social encounters.

Haptic recognition memory and lateralisation for verbal and nonverbal shapes.

Laterality effects generally refer to an advantage for verbal processing in the left hemisphere and for non-verbal processing in the right hemisphere, and are often demonstrated in memory tasks in vision and audition. In contrast, their role in haptic memory is less understood. In this study, we examined haptic recognition memory and laterality for letters and nonsense shapes. We used both upper and lower case letters, with the latter designed as more complex in shape. Participants performed a recognition memory task with the left and right hand separately. Recognition memory performance (capacity and bias-free d') was higher and response times were faster for upper case letters than for lower case letters and nonsense shapes. The right hand performed best for upper case letters when it performed the task after the left hand. This right hand/left hemisphere advantage appeared for upper case letters, but not lower case letters, which also had a lower memory capacity, probably due to their more complex spatial shape. These findings suggest that verbal laterality effects in haptic memory are not very prominent, which may be due to the haptic verbal stimuli being processed mainly as spatial objects without reaching robust verbal coding into memory.

Annoyance, perception, and physiological effects of wind turbine infrasound.

Even though some individuals subjectively associate various symptoms with infrasound, there are very few systematic studies on the contribution of infrasound to the perception, annoyance, and physiological reactions elicited by wind turbine sound. In this study, sound samples were selected among long-term measurement data from wind power plant and residential areas, both indoors and outdoors, and used in laboratory experiments. In the experiments, the detectability and annoyance of both inaudible and audible characteristics of wind turbine noise were determined, as well as autonomic nervous system responses: heart rate, heart rate variability, and skin conductance response. The participants were divided into two groups based on whether they reported experiencing wind turbine infrasound related symptoms or not. The participants did not detect infrasonic contents of wind turbine noise. The presence of infrasound had no influence on the reported annoyance nor the measured autonomic nervous system responses. No differences were observed between the two groups. These findings suggest that the levels of infrasound in the current study did not affect perception and annoyance or autonomic nervous system responses, even though the experimental conditions corresponded acoustically to real wind power plant areas.

Laterality effects in the haptic discrimination of verbal and non-verbal shapes.

The left hemisphere is known to be generally predominant in verbal processing and the right hemisphere in non-verbal processing. We studied whether verbal and non-verbal lateralization is present in haptics by comparing discrimination performance between letters and nonsense shapes. We addressed stimulus complexity by introducing lower case letters, which are verbally identical with upper case letters but have a more complex shape. The participants performed a same-different haptic discrimination task for upper and lower case letters and nonsense shapes with the left and right hand separately. We used signal detection theory to determine discriminability (d'), criterion (c) and we measured reaction times. Discrimination was better for the left hand for nonsense shapes, close to significantly better for the right hand for upper case letters and with no difference between the hands for lower case letters. For lower case letters, right hand showed a strong bias to respond "different", while the left hand showed faster reaction times. Our results are in agreement with the right lateralization for non-verbal material. Complexity of the verbal shape is important in haptics as the lower case letters seem to be processed as less verbal and more as spatial shapes than the upper case letters.

Connecting directional limb movements to vowel fronting and backing.

It has been shown recently that when participants are required to pronounce a vowel at the same time with the hand movement, the vocal and manual responses are facilitated when a front vowel is produced with forward-directed hand movements and a back vowel is produced with backward-directed hand movements. This finding suggests a coupling between spatial programing of articulatory tongue movements and hand movements. The present study revealed that the same effect can be also observed in relation to directional leg movements. The study suggests that the effect operates within the common directional processes of movement planning including at least tongue, hand and leg movements, and these processes might contribute sound-to-meaning mappings to the semantic concepts of 'forward' and 'backward'.

Reaching and grasping with the tongue: Shared motor planning between hand actions and articulatory gestures.

Research has shown connections between articulatory mouth actions and manual actions. This study investigates whether forward-backward hand movements could be associated with vowel production processes that programme tongue fronting/backing, lip rounding/spreading (Experiment 1), and/or consonant production processes that programme tongue tip and tongue dorsum actions (Experiment 2). The participants had to perform either forward or backward hand movement and simultaneously pronounce different vowels or consonants. The results revealed a response benefit, measured in vocal and manual reaction times, when the responses consisted of front vowels and forward hand movements. Conversely, back vowels were associated with backward hand movements. Articulation of rounded versus unrounded vowels or coronal versus dorsal consonants did not produce the effect. In contrast, when the manual responses of forward-backward hand movements were replaced by precision and power grip responses, the coronal consonants [t] and [r] were associated with the precision grip, whereas the dorsal consonant [k] was associated with the power grip. We propose that the movements of the tongue body, operating mainly for vowel production, share the directional action planning processes with the hand movements. Conversely, the tongue articulators related to tongue tip and dorsum movements, operating mainly for consonant production, share the action planning processes with the precision and power grip, respectively.

Exploring laterality and memory effects in the haptic discrimination of verbal and non-verbal shapes.

The brain's left hemisphere often displays advantages in processing verbal information, while the right hemisphere favours processing non-verbal information. In the haptic domain due to contra-lateral innervations, this functional lateralization is reflected in a hand advantage during certain functions. Findings regarding the hand-hemisphere advantage for haptic information remain contradictory, however. This study addressed these laterality effects and their interaction with memory retention times in the haptic modality. Participants performed haptic discrimination of letters, geometric shapes and nonsense shapes at memory retention times of 5, 15 and 30 s with the left and right hand separately, and we measured the discriminability index d'. The d' values were significantly higher for letters and geometric shapes than for nonsense shapes. This might result from dual coding (naming + spatial) or/and from a low stimulus complexity. There was no stimulus-specific laterality effect. However, we found a time-dependent laterality effect, which revealed that the performance of the left hand-right hemisphere was sustained up to 15 s, while the performance of the right-hand-left hemisphere decreased progressively throughout all retention times. This suggests that haptic memory traces are more robust to decay when they are processed by the left hand-right hemisphere.

Stimulus duration has little effect on auditory, visual and audiovisual temporal order judgement.

Some classical studies on temporal order judgments (TOJ) suggested a single central process comparing stimulus onsets across modalities. The prevalent current view suggests that there is modality-specific timing estimation followed by a cross-modal stage. If the latter view is correct, TOJ's may vary depending on stimulus modality. Further, if TOJ is based only on onsets, stimulus duration should be irrelevant. To address these issues, we used both unisensory and multisensory stimuli to test whether unisensory duration processing influences cross-modal TOJ's. The stimuli were auditory noise bursts, visual squares, and their cross-modal combinations presented at 10, 40 and 500 ms durations, and various stimulus onset asynchronies. Psychometric functions were measured with an identical task in all conditions: On each trial, two stimuli were presented, one to the left, the other to the right of fixation. The participants judged which one started first. TOJ's were little affected by stimulus duration, implying that they are mainly determined by stimulus onsets. Throughout, the cross-modal just noticeable differences were larger than the unisensory ones. In accordance with the current view, our results suggest that cross-modal TOJ's require a comparison of timing after modality-specific estimations.

Semantically Congruent Visual Information Can Improve Auditory Recognition Memory in Older Adults.

In the course of normal aging, memory functions show signs of impairment. Studies of memory in the elderly have previously focused on a single sensory modality, although multisensory encoding has been shown to improve memory performance in children and young adults. In this study, we investigated how audiovisual encoding affects auditory recognition memory in older (mean age 71 years) and younger (mean age 23 years) adults. Participants memorized auditory stimuli (sounds, spoken words) presented either alone or with semantically congruent visual stimuli (pictures, text) during encoding. Subsequent recognition memory performance of auditory stimuli was better for stimuli initially presented together with visual stimuli than for auditory stimuli presented alone during encoding. This facilitation was observed both in older and younger participants, while the overall memory performance was poorer in older participants. However, the pattern of facilitation was influenced by age. When encoding spoken words, the gain was greater for older adults. When encoding sounds, the gain was greater for younger adults. These findings show that semantically congruent audiovisual encoding improves memory performance in late adulthood, particularly for auditory verbal material.

Mismatch negativity (MMN) to speech sounds is modulated systematically by manual grip execution.

Manual actions and speech are connected: for example, grip execution can influence simultaneous vocalizations and vice versa. Our previous studies show that the consonant [k] is associated with the power grip and the consonant [t] with the precision grip. Here we studied whether the interaction between speech sounds and grips could operate already at a pre-attentive stage of auditory processing, reflected by the mismatch-negativity (MMN) component of the event-related potential (ERP). Participants executed power and precision grips according to visual cues while listening to syllable sequences consisting of [ke] and [te] utterances. The grips modulated the MMN amplitudes to these syllables in a systematic manner so that when the deviant was [ke], the MMN response was larger with a precision grip than with a power grip. There was a converse trend when the deviant was [te]. These results suggest that manual gestures and speech can interact already at a pre-attentive processing level of auditory perception, and show, for the first time that manual actions can systematically modulate the MMN.

Connections of Grasping and Horizontal Hand Movements with Articulation in Czech Speakers.

We have recently shown in Finnish speakers that articulation of certain vowels and consonants has a systematic influence on simultaneous grasp actions as well as on forward and backward hand movements. Here we studied whether these effects generalize to another language, namely Czech. We reasoned that if the results generalized to another language environment, it would suggest that the effects arise through other processes than language-dependent semantic associations. Rather, the effects would be likely to arise through language-independent interactions between processes that plan articulatory gestures and hand movements. Participants were presented with visual stimuli specifying articulations to be uttered (e.g., A or I), and they were required to produce a manual response concurrently with the articulation. In Experiment 1 they responded with a precision or a power grip, whereas in Experiment 2 they responded with a forward or a backward hand movement. The grip congruency effect was fully replicated: the consonant [k] and the vowel [α] were associated with power grip responses, while the consonant [t] and the vowel [i] were associated with precision grip responses. The forward/backward congruency effect was replicated with vowels [α], [o], which were associated with backward movement and with [i], which was associated with forward movement, but not with consonants [k] and [t]. These findings suggest that the congruency effects mostly reflect interaction between processes that plan articulatory gestures and hand movements with an exception that the forward/backward congruency effect might only work with vowel articulation.

Sensory and Emotional Perception of Wooden Surfaces through Fingertip Touch.

Previous studies on tactile experiences have investigated a wide range of material surfaces across various skin sites of the human body in self-touch or other touch modes. Here, we investigate whether the sensory and emotional aspects of touch are related when evaluating wooden surfaces using fingertips in the absence of other sensory modalities. Twenty participants evaluated eight different pine and oak wood surfaces, using sensory and emotional touch descriptors, through the lateral motion of active fingertip exploration. The data showed that natural and smooth wood surfaces were perceived more positively in emotional touch than coated surfaces. We highlight the importance of preserving the naturalness of the surface texture in the process of wood-surface treatment so as to improve positive touch experiences, as well as avoid negative ones. We argue that the results may offer possibilities in the design of wood-based interior products with a view to improving consumer touch experiences.

Lipreading Ability and Its Cognitive Correlates in Typically Developing Children and Children With Specific Language Impairment.

Purpose: Lipreading and its cognitive correlates were studied in school-age children with typical language development and delayed language development due to specific language impairment (SLI). Method: Forty-two children with typical language development and 20 children with SLI were tested by using a word-level lipreading test and an extensive battery of standardized cognitive and linguistic tests. Results: Children with SLI were poorer lipreaders than their typically developing peers. Good phonological skills were associated with skilled lipreading in both typically developing children and in children with SLI. Lipreading was also found to correlate with several cognitive skills, for example, short-term memory capacity and verbal motor skills. Conclusions: Speech processing deficits in SLI extend also to the perception of visual speech. Lipreading performance was associated with phonological skills. Poor lipreading in children with SLI may be, thus, related to problems in phonological processing.

Preparation and execution of teeth clenching and foot muscle contraction influence on corticospinal hand-muscle excitability.

Contraction of a muscle modulates not only the corticospinal excitability (CSE) of the contracting muscle but also that of different muscles. We investigated to what extent the CSE of a hand muscle is modulated during preparation and execution of teeth clenching and ipsilateral foot dorsiflexion either separately or in combination. Hand-muscle CSE was estimated based on motor evoked potentials (MEPs) elicited by transcranial magnetic stimulation (TMS) and recorded from the first dorsal interosseous (FDI) muscle. We found higher excitability during both preparation and execution of all the motor tasks than during mere observation of a fixation cross. As expected, the excitability was greater during the execution phase than the preparation one. Furthermore, both execution and preparation of combined motor tasks led to higher excitability than individual tasks. These results extend our current understanding of the neural interactions underlying simultaneous contraction of muscles in different body parts.

Systematic Influence of Perceived Grasp Shape on Speech Production.

Previous research has shown that precision and power grip performance is consistently influenced by simultaneous articulation. For example, power grip responses are performed relatively fast with the open-back vowel [a], whereas precision grip responses are performed relatively fast with the close-front vowel [i]. In the present study, the participants were presented with a picture of a hand shaped to the precision or power grip. They were required to pronounce speech sounds according to the front/above perspective of the hand. The results showed that not only the grip performance is affected by simultaneously pronouncing the speech sound but also the production of speech sound can be affected by viewing an image of a grip. The precision grip stimulus triggered relatively rapid production of the front-close vowel [i]. In contrast, the effect related to the power grip stimulus was mostly linked to the vertical dimension of the pronounced vowel since this stimulus triggered relatively rapid production of the back-open vowel [a] and back-mid-open vowel [o] while production of the back-close vowel [u] was not influenced by it. The fact that production of the dorsal consonant [k] or coronal consonant [t] were not influenced by these stimuli suggests that the effect was not associated with a relative front-back tongue shape of the articulation in the absence of changes in any vertical articulatory components. These findings provide evidence for an intimate interaction between certain articulatory gestures and grip types, suggesting that an overlapping visuomotor network operates for planning articulatory gestures and grasp actions.

Interaction in planning vocalizations and grasping.

Previous studies have shown a congruency effect between manual grasping and syllable articulation. For instance, a power grip is associated with syllables whose articulation involves the tongue body and/or large mouth aperture ([kɑ]) whereas a precision grip is associated with articulations that involve the tongue tip and/or small mouth aperture ([ti]). Previously, this effect has been observed in manual reaction times. The primary aim of the current study was to investigate whether this congruency effect also takes place in vocal responses and to investigate involvement of action selection processes in the effect. The congruency effect was found in vocal and manual responses regardless of whether or not the syllable or grip was known a priori, suggesting that the effect operates with minimal or absent action selection processes. In addition, the effect was observed in vocal responses even when the grip was only prepared but not performed, suggesting that merely planning a grip response primes the corresponding articulatory response. These results support the view that articulation and grasping are processed in a partially overlapping network.

Excitability of hand motor areas during articulation of syllables.

It is known that articulating different syllables is linked to different grasp actions, e.g. [ti] is linked to precision grip, and [kɑ] to power grip. The aim of the present study was to test whether articulating or hearing these syllables would result in an increased activity in the representation of hand muscles involved in these two actions in a muscle-specific manner. To this end, we used transcranial magnetic stimulation (TMS) to investigate changes in the excitability of the left primary motor cortex (M1) innervating hand muscles while participants articulated or listened to meaningless syllables, listened to a metronome, or observed a fixation cross. The motor-evoked potentials of two hand muscles associated with either a precision or power grip exhibited significantly greater amplitudes during articulation than in passive listening, metronome, and fixation cross conditions. Moreover, these muscles exhibited similar patterns of excitability during articulation regardless of which syllable was articulated. The increased excitability of the left M1 hand area during articulation, but not during perception of the syllables, might be due to the cortico-cortical interaction between the motor representations of oral organs with the hand area.