Multisensory integration of speech and gestures in a naturalistic paradigm.

Speech comprehension is crucial for human social interaction, relying on the integration of auditory and visual cues across various levels of representation. While research has extensively studied multisensory integration (MSI) using idealised, well-controlled stimuli, there is a need to understand this process in response to complex, naturalistic stimuli encountered in everyday life. This study investigated behavioural and neural MSI in neurotypical adults experiencing audio-visual speech within a naturalistic, social context. Our novel paradigm incorporated a broader social situational context, complete words, and speech-supporting iconic gestures, allowing for context-based pragmatics and semantic priors. We investigated MSI in the presence of unimodal (auditory or visual) or complementary, bimodal speech signals. During audio-visual speech trials, compared to unimodal trials, participants more accurately recognised spoken words and showed a more pronounced suppression of alpha power-an indicator of heightened integration load. Importantly, on the neural level, these effects surpassed mere summation of unimodal responses, suggesting non-linear MSI mechanisms. Overall, our findings demonstrate that typically developing adults integrate audio-visual speech and gesture information to facilitate speech comprehension in noisy environments, highlighting the importance of studying MSI in ecologically valid contexts.

Do 14-17-month-old infants use iconic speech and gesture cues to interpret word meanings?a).

This experimental study investigated whether infants use iconicity in speech and gesture cues to interpret word meanings. Specifically, we tested infants' sensitivity to size sound symbolism and iconic gesture cues and asked whether combining these cues in a multimodal fashion would enhance infants' sensitivity in a superadditive manner. Thirty-six 14-17-month-old infants participated in a preferential looking task in which they heard a spoken nonword (e.g., "zudzud") while observing a small and large object (e.g., a small and large square). All infants were presented with an iconic cue for object size (small or large) (1) in the pitch of the spoken non-word (high vs low), (2) in gesture (small or large), or (3) congruently in pitch and gesture (e.g., a high pitch and small gesture indicating a small square). Infants did not show a preference for congruently sized objects in any iconic cue condition. Bayes factor analyses showed moderate to strong support for the null hypotheses. In conclusion, 14-17-month-old infants did not use iconic pitch cues, iconic gesture cues, or iconic multimodal cues (pitch and gesture) to associate speech sounds with their referents. These findings challenge theories that emphasize the role of iconicity in early language development.

Encouraging pointing with the right hand, but not the left hand, gives right-handed 3-year-olds a linguistic advantage.

Previous research has shown a strong positive association between right-handed gesturing and vocabulary development. However, the causal nature of this relationship remains unclear. In the current study, we tested whether gesturing with the right hand enhances linguistic processing in the left hemisphere, which is contralateral to the right hand. We manipulated the gesture hand children used in pointing tasks to test whether it would affect their performance. In either a linguistic task (verb learning) or a non-linguistic control task (memory), 131 typically developing right-handed 3-year-olds were encouraged to use either their right hand or left hand to respond. While encouraging children to use a specific hand to indicate their responses had no effect on memory performance, encouraging children to use the right hand to respond, compared to the left hand, significantly improved their verb learning performance. This study is the first to show that manipulating the hand with which children are encouraged to gesture gives them a linguistic advantage. Language lateralization in healthy right-handed children typically involves a dominant left hemisphere. Producing right-handed gestures may therefore lead to increased activation in the left hemisphere which may, in turn, facilitate forming and accessing lexical representations. It is important to note that this study manipulated gesture handedness among right-handers and does therefore not support the practice of encouraging children to become right-handed in manual activities. RESEARCH HIGHLIGHTS: Right-handed 3-year-olds were instructed to point to indicate their answers exclusively with their right or left hand in either a memory or verb learning task. Right-handed pointing was associated with improved verb generalization performance, but not improved memory performance. Thus, gesturing with the right hand, compared to the left hand, gives right-handed 3-year-olds an advantage in a linguistic but not a non-linguistic task. Right-handed pointing might lead to increased activation in the left hemisphere and facilitate forming and accessing lexical representations.

Beyond the Shape of Things: Infants Can Be Taught to Generalize Nouns by Objects' Functions.

Two-year-olds typically extend labels of novel objects by the objects' shape (shape bias), whereas adults do so by the objects' function. Is this because shape is conceptually easier to comprehend than function? To test whether the conceptual complexity of function prevents infants from developing a function bias, we trained twelve 17-month-olds (function-training group) to focus on objects' functions when labeling the objects over a period of 7 weeks. Our training was similar to previously used methods in which 17-month-olds were successfully taught to focus on the shape of objects, resulting in a precocious shape bias. We exposed another 12 infants (control group) to the same objects over 7 weeks but without labeling the items or demonstrating their functions. Only the infants in the function-training group developed a function bias. Thus, the conceptual complexity of function was not a barrier for developing a function bias, which suggests that the shape bias emerges naturally because shape is perceptually more accessible than function.

Seeing Iconic Gesture Promotes First- and Second-Order Verb Generalization in Preschoolers.

This study investigated whether seeing iconic gestures depicting verb referents promotes two types of generalization. We taught 3- to 4-year-olds novel locomotion verbs. Children who saw iconic manner gestures during training generalized more verbs to novel events (first-order generalization) than children who saw interactive gestures (Experiment 1, N = 48; Experiment 2, N = 48) and path-tracing gestures (Experiment 3, N = 48). Furthermore, immediately (Experiments 1 and 3) and after 1 week (Experiment 2), the iconic manner gesture group outperformed the control groups in subsequent generalization trials with different novel verbs (second-order generalization), although all groups saw interactive gestures. Thus, seeing iconic gestures that depict verb referents helps children (a) generalize individual verb meanings to novel events and (b) learn more verbs from the same subcategory.

Stopping at nothing: Two-year-olds differentiate between interrupted and abandoned goals.

Previous research has established that goal tracking emerges early in the first year of life and rapidly becomes increasingly sophisticated. However, it has not yet been shown whether young children continue to update their representations of others' goals over time. The current study investigated this by probing young children's (24- to 30-month-olds; N = 24) ability to differentiate between goal-directed actions that have been halted because the goal was interrupted and those that have been halted because the goal was abandoned. To test whether children are sensitive to this distinction, we manipulated the experimenter's reason for not completing a goal-directed action; his initial goal was either interrupted by an obstacle or abandoned in favor of an alternative. We measured whether children's helping behavior was sensitive to the experimenter's reason for not completing his goal-directed action by recording whether children completed the experimenter's initial goal or the alternative goal. The results showed that children helped to complete the experimenter's initial goal significantly more often after this goal had been interrupted than after it had been abandoned. These results support the hypothesis that children continue to update their representations of others' goals over time by 2 years of age and specifically that they differentiate between abandoned and interrupted goals.

Seeing Iconic Gestures While Encoding Events Facilitates Children's Memory of These Events.

An experiment with 72 three-year-olds investigated whether encoding events while seeing iconic gestures boosts children's memory representation of these events. The events, shown in videos of actors moving in an unusual manner, were presented with either iconic gestures depicting how the actors performed these actions, interactive gestures, or no gesture. In a recognition memory task, children in the iconic gesture condition remembered actors and actions better than children in the control conditions. Iconic gestures were categorized based on how much of the actors was represented by the hands (feet, legs, or body). Only iconic hand-as-body gestures boosted actor memory. Thus, seeing iconic gestures while encoding events facilitates children's memory of those aspects of events that are schematically highlighted by gesture.

GestuRe and ACtion Exemplar (GRACE) video database: stimuli for research on manners of human locomotion and iconic gestures.

Human locomotion is a fundamental class of events, and manners of locomotion (e.g., how the limbs are used to achieve a change of location) are commonly encoded in language and gesture. To our knowledge, there is no openly accessible database containing normed human locomotion stimuli. Therefore, we introduce the GestuRe and ACtion Exemplar (GRACE) video database, which contains 676 videos of actors performing novel manners of human locomotion (i.e., moving from one location to another in an unusual manner) and videos of a female actor producing iconic gestures that represent these actions. The usefulness of the database was demonstrated across four norming experiments. First, our database contains clear matches and mismatches between iconic gesture videos and action videos. Second, the male actors and female actors whose action videos matched the gestures in the best possible way, perform the same actions in very similar manners and different actions in highly distinct manners. Third, all the actions in the database are distinct from each other. Fourth, adult native English speakers were unable to describe the 26 different actions concisely, indicating that the actions are unusual. This normed stimuli set is useful for experimental psychologists working in the language, gesture, visual perception, categorization, memory, and other related domains.

Individual Differences in Children's Corepresentation of Self and Other in Joint Action.

Previous research has shown that children aged 4-5 years, but not 2-3 years, show adult-like interference from a partner when performing a joint task (Milward, Kita, & Apperly, 2014). This raises questions about the cognitive skills involved in the development of such "corepresentation (CR)" of a partner (Sebanz, Knoblich, & Prinz, 2003). Here, individual differences data from one hundred and thirteen 4- to 5-year-olds showed theory of mind (ToM) and inhibitory control (IC) as predictors of ability to avoid CR interference, suggesting that children with better ToM abilities are more likely to succeed in decoupling self and other representations in a joint task, while better IC is likely to help children avoid interference from a partner's response when selecting their own response on the task.

Semantics is crucial for the right-hemisphere involvement in metaphor processing: evidence from mouth asymmetry during speaking.

Research on the neural basis of metaphor provides contradicting evidence about the role of right and left hemispheres. We used the mouth-opening asymmetry technique to investigate the relative involvement of the two hemispheres whilst right-handed healthy male participants explained the meaning of English phrases. This technique is based on the contralateral cortical control of the facial musculature and reflects the relative hemispheric involvement during different cognitive tasks. In particular, right-handers show a right-sided mouth asymmetry (right side of the mouth opens wider than the left) during linguistic tasks, thus reflecting the left-hemisphere specialization for language. In the current study, we compared the right-sided mouth asymmetry during metaphor explanation (e.g., explain the meaning of the phrase "to spin a yarn") and concrete explanation (e.g., explain the meaning of the phrase "to spin a golf ball") and during the production of content and function words. The expected right-sided mouth asymmetry reduced during metaphorical compared to concrete explanations suggesting the relative right-hemispheric involvement for metaphor processing. Crucially, this right-sided mouth asymmetry reduction was particularly pronounced for the production of content words. Thus, we concluded that semantics is crucial to the right-hemispheric involvement for metaphorical speech production.

Young children make their gestural communication systems more language-like: segmentation and linearization of semantic elements in motion events.

Research on Nicaraguan Sign Language, created by deaf children, has suggested that young children use gestures to segment the semantic elements of events and linearize them in ways similar to those used in signed and spoken languages. However, it is unclear whether this is due to children's learning processes or to a more general effect of iterative learning. We investigated whether typically developing children, without iterative learning, segment and linearize information. Gestures produced in the absence of speech to express a motion event were examined in 4-year-olds, 12-year-olds, and adults (all native English speakers). We compared the proportions of gestural expressions that segmented semantic elements into linear sequences and that encoded them simultaneously. Compared with adolescents and adults, children reshaped the holistic stimuli by segmenting and recombining their semantic features into linearized sequences. A control task on recognition memory ruled out the possibility that this was due to different event perception or memory. Young children spontaneously bring fundamental properties of language into their communication system.

Children use gesture to interpret novel verb meanings.

Children often find it difficult to map verbs to specific referents within complex scenes, often believing that additional features are part of the referents. This study investigated whether 3-year-olds could use iconic gestures to map novel verbs to specific referents. One hundred and twenty children were taught verbs that could be interpreted as change-of-state or manner verbs while presented with manner, end-state, or no iconic gestures. Children were then presented with a choice that forced them to generalize either on the basis of manner or end state. Results showed that children who saw manner gestures showed a stronger manner bias compared to the other groups. Thus, the specific feature of an event encoded in gestures guides children's interpretations of novel words.

When do speakers use gestures to specify who does what to whom? The role of language proficiency and type of gestures in narratives.

Previous research has found that iconic gestures (i.e., gestures that depict the actions, motions or shapes of entities) identify referents that are also lexically specified in the co-occurring speech produced by proficient speakers. This study examines whether concrete deictic gestures (i.e., gestures that point to physical entities) bear a different kind of relation to speech, and whether this relation is influenced by the language proficiency of the speakers. Two groups of speakers who had different levels of English proficiency were asked to retell a story in English. Their speech and gestures were transcribed and coded. Our findings showed that proficient speakers produced concrete deictic gestures for referents that were not specified in speech, and iconic gestures for referents that were specified in speech, suggesting that these two types of gestures bear different kinds of semantic relations with speech. In contrast, less proficient speakers produced concrete deictic gestures and iconic gestures whether or not referents were lexically specified in speech. Thus, both type of gesture and proficiency of speaker need to be considered when accounting for how gesture and speech are used in a narrative context.

Japanese two-year-olds use morphosyntax to learn novel verb meanings.

Previous research has found that children who are acquiring argument-drop languages such as Turkish and Chinese make use of syntactic frames to extend familiar verb meanings (Göksun, Küntay & Naigles, 2008; Lee & Naigles, 2008). This article investigates whether two-year-olds learning Japanese, another argument-drop language, make use of argument number and case markings in learning novel verbs. Children watched videos of novel causative and non-causative actions via Intermodal Preferential Looking. The novel verbs were presented in transitive or intransitive frames; the NPs in the transitive frames appeared 'bare' or with case markers. Consistent with previous findings of Morphosyntactic Bootstrapping, children who heard the novel verbs in the transitive frame with case markers reliably assigned those verbs to the novel causative actions.

Prior experience with unlabeled actions promotes 3-year-old children's verb learning.

[Correction Notice: An Erratum for this article was reported online in Journal of Experimental Psychology: General on Jan 6 2022 (see record 2022-20753-001). In the original article, acknowledgment of and formatting for Economic and Social Research Council funding was omitted. The author note and copyright line now reflect the standard acknowledgment of and formatting for the funding received for this article. All versions of this article have been corrected.] This study investigated what type of prior experience with unlabeled actions promotes 3-year-old children's verb learning. We designed a novel verb learning task in which we manipulated prior experience with unlabeled actions and the gesture type children saw with this prior experience. Experiment 1 showed that children (N = 96) successfully generalized more novel verbs when they had prior experience with unlabeled exemplars of the referent actions ("relevant exemplars"), but only if the referent actions were highlighted with iconic gestures during prior experience. Experiment 2 showed that children (N = 48) successfully generalized more novel verbs when they had prior experience with one relevant exemplar and an iconic gesture than with two relevant exemplars (i.e., the same referent action performed by different actors) shown simultaneously. However, children also successfully generalized verbs above chance in the two-relevant-exemplars condition (without the help of iconic gesture). Overall, these findings suggest that prior experience with unlabeled actions is an important first step in children's verb learning process, provided that children get a cue for focusing on the relevant information (i.e., actions) during prior experience so that they can create stable memory representations of the actions. Such stable action memory representations promote verb learning because they make the actions stand out when children later encounter labeled exemplars of the same actions. Adults can provide top-down cues (e.g., iconic gestures) and bottom-up cues (e.g., simultaneous exemplars) to focus children's attention on actions; however, iconic gesture is more beneficial for successful verb learning than simultaneous exemplars. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

"Prior experience with unlabeled actions promotes 3-year-old children's verb learning": Correction.

Reports an error in "Prior experience with unlabeled actions promotes 3-year-old children's verb learning" by Suzanne Aussems, Katherine H. Mumford and Sotaro Kita (Journal of Experimental Psychology: General, Advanced Online Publication, Jul 15, 2021, np). In the original article, acknowledgment of and formatting for Economic and Social Research Council funding was omitted. The author note and copyright line now reflect the standard acknowledgment of and formatting for the funding received for this article. All versions of this article have been corrected. (The following abstract of the original article appeared in record 2021-63321-001). This study investigated what type of prior experience with unlabeled actions promotes 3-year-old children's verb learning. We designed a novel verb learning task in which we manipulated prior experience with unlabeled actions and the gesture type children saw with this prior experience. Experiment 1 showed that children (N = 96) successfully generalized more novel verbs when they had prior experience with unlabeled exemplars of the referent actions ("relevant exemplars"), but only if the referent actions were highlighted with iconic gestures during prior experience. Experiment 2 showed that children (N = 48) successfully generalized more novel verbs when they had prior experience with one relevant exemplar and an iconic gesture than with two relevant exemplars (i.e., the same referent action performed by different actors) shown simultaneously. However, children also successfully generalized verbs above chance in the two-relevant-exemplars condition (without the help of iconic gesture). Overall, these findings suggest that prior experience with unlabeled actions is an important first step in children's verb learning process, provided that children get a cue for focusing on the relevant information (i.e., actions) during prior experience so that they can create stable memory representations of the actions. Such stable action memory representations promote verb learning because they make the actions stand out when children later encounter labeled exemplars of the same actions. Adults can provide top-down cues (e.g., iconic gestures) and bottom-up cues (e.g., simultaneous exemplars) to focus children's attention on actions; however, iconic gesture is more beneficial for successful verb learning than simultaneous exemplars. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Finish What you Started: 2-Year-Olds Motivated by a Preference for Completing Others' Unfinished Actions in Instrumental Helping Contexts.

A considerable body of research has documented the emergence of what appears to be instrumental helping behavior in early childhood. The current study tested the hypothesis that one basic psychological mechanism motivating this behavior is a preference for completing unfinished actions. To test this, a paradigm was implemented in which 2-year-olds (n = 34, 16 females/18 males, mostly White middle-class children) could continue an adult's action when the adult no longer wanted to complete the action. The results showed that children continued the adult's actions more often when the goal had been abandoned than when it had been reached (OR = 2.37). This supports the hypothesis that apparent helping behavior in 2-year-olds is motivated by a preference for completing unfinished actions.

The role of synchrony and ambiguity in speech-gesture integration during comprehension.

During face-to-face communication, one does not only hear speech but also see a speaker's communicative hand movements. It has been shown that such hand gestures play an important role in communication where the two modalities influence each other's interpretation. A gesture typically temporally overlaps with coexpressive speech, but the gesture is often initiated before (but not after) the coexpressive speech. The present ERP study investigated what degree of asynchrony in the speech and gesture onsets are optimal for semantic integration of the concurrent gesture and speech. Videos of a person gesturing were combined with speech segments that were either semantically congruent or incongruent with the gesture. Although gesture and speech always overlapped in time, gesture and speech were presented with three different degrees of asynchrony. In the SOA 0 condition, the gesture onset and the speech onset were simultaneous. In the SOA 160 and 360 conditions, speech was delayed by 160 and 360 msec, respectively. ERPs time locked to speech onset showed a significant difference between semantically congruent versus incongruent gesture-speech combinations on the N400 for the SOA 0 and 160 conditions. No significant difference was found for the SOA 360 condition. These results imply that speech and gesture are integrated most efficiently when the differences in onsets do not exceed a certain time span because of the fact that iconic gestures need speech to be disambiguated in a way relevant to the speech context.

Spontaneous gestures influence strategy choices in problem solving.

Do gestures merely reflect problem-solving processes, or do they play a functional role in problem solving? We hypothesized that gestures highlight and structure perceptual-motor information, and thereby make such information more likely to be used in problem solving. Participants in two experiments solved problems requiring the prediction of gear movement, either with gesture allowed or with gesture prohibited. Such problems can be correctly solved using either a perceptual-motor strategy (simulation of gear movements) or an abstract strategy (the parity strategy). Participants in the gesture-allowed condition were more likely to use perceptual-motor strategies than were participants in the gesture-prohibited condition. Gesture promoted use of perceptual-motor strategies both for participants who talked aloud while solving the problems (Experiment 1) and for participants who solved the problems silently (Experiment 2). Thus, spontaneous gestures influence strategy choices in problem solving.

Gesture and speech integration: an exploratory study of a man with aphasia.

BACKGROUND: In order to comprehend fully a speaker's intention in everyday communication, information is integrated from multiple sources, including gesture and speech. There are no published studies that have explored the impact of aphasia on iconic co-speech gesture and speech integration. AIMS: To explore the impact of aphasia on co-speech gesture and speech integration in one participant with aphasia and 20 age-matched control participants. METHODS & PROCEDURES: The participant with aphasia and 20 control participants watched video vignettes of people producing 21 verb phrases in three different conditions, verbal only (V), gesture only (G), and verbal gesture combined (VG). Participants were required to select a corresponding picture from one of four alternatives: integration target, a verbal-only match, a gesture-only match, and an unrelated foil. The probability of choosing the integration target in the VG that goes beyond what is expected from the probabilities of choosing the integration target in V and G was referred to as multi-modal gain (MMG). OUTCOMES & RESULTS: The participant with aphasia obtained a significantly lower multi-modal gain score than the control participants (p<0.05). Error analysis indicated that in speech and gesture integration tasks, the participant with aphasia relied on gesture in order to decode the message, whereas the control participants relied on speech in order to decode the message. Further analysis of the speech-only and gesture-only tasks indicated that the participant with aphasia had intact gesture comprehension but impaired spoken word comprehension. CONCLUSIONS & IMPLICATIONS: The results confirm findings by Records ( 1994 ) that reported that impaired verbal comprehension leads to a greater reliance on gesture to decode messages. Moreover, multi-modal integration of information from speech and iconic gesture can be impaired in aphasia. The findings highlight the need for further exploration of the impact of aphasia on gesture and speech integration.