Expanding the scope: multimodal dimensions in aphasia discourse analysis-preliminary findings.

Background: Aphasia, resulting from acquired brain injury, disrupts language processing and usage, significantly impacting individuals' social communication and life participation. Given the limitations of traditional assessments in capturing the nuanced challenges faced by individuals with aphasia, this study seeks to explore the potential benefits of integrating multimodal communication elements into discourse analysis to better capture narrative proficiency in this population. Objective: This study examined how incorporating multimodal communication elements (e.g., physical gestures, writing, drawing) into discourse analysis may affect the narrative outcomes of persons with aphasia compared to those observed using methods that exclude multimodal considerations. Methods: Participants included individuals with chronic aphasia and age-and education-matched healthy controls who completed a storytelling task-the Bear and the Fly story. Macrolinguistic scores were obtained using verbal-only and multimodal scoring approaches. Additionally, the frequency and type of multimodal communication use during storytelling were examined in relation to aphasia characteristics. Statistical analyses included both within-group and between-group comparisons as well as correlational analyses. Results: Individuals with aphasia scored significantly higher in terms of their macrolinguistic abilities when multimodal scoring was considered compared to verbal-only scoring. Within the aphasia group, there were prominent differences noted in macrolinguistic scores for both fluent and nonfluent aphasia. Specifically, both groups scored higher on Main Concepts when multimodal scoring was considered, with the nonfluent group demonstrating significantly higher Main Concept and total macrolinguistic rubric scores in multimodal scoring compared to verbal scoring on the storytelling task. Additionally, aphasia severity showed moderate positive correlations with total macrolinguistic scores, indicating that individuals with less severe aphasia tended to produce higher quality narratives. Lastly, although persons with aphasia used different types of nonverbal modalities (i.e., drawing, writing), the use of meaning-laden gestures was most predominant during storytelling, emphasizing the importance of multimodal elements in communication for individuals with aphasia. Conclusion: Our preliminary study findings underscore the importance of considering multimodal communication in assessing discourse performance among individuals with aphasia. Tailoring assessment approaches based on aphasia subtypes can provide valuable insights into linguistic abilities and inform targeted intervention strategies for improving communication outcomes.

A cross-species framework for classifying sound-movement couplings.

Sound and movement are entangled in animal communication. This is obviously true in the case of sound-constituting vibratory movements of biological structures which generate acoustic waves. A little less obvious is that other moving structures produce the energy required to sustain these vibrations. In many species, the respiratory system moves to generate the expiratory flow which powers the sound-constituting movements (sound-powering movements). The sound may acquire additional structure via upper tract movements, such as articulatory movements or head raising (sound-filtering movements). Some movements are not necessary for sound production, but when produced, impinge on the sound-producing process due to weak biomechanical coupling with body parts (e.g., respiratory system) that are necessary for sound production (sound-impinging movements). Animals also produce sounds contingent with movement, requiring neuro-physiological control regimes allowing to flexibly couple movements to a produced sound, or coupling movements to a perceived external sound (sound-contingent movement). Here, we compare and classify the variety of ways sound and movements are coupled in animal communication; our proposed framework should help structure previous and future studies on this topic.

Toward a semiotics of midwifery: Multimodal communication's effects on accessibility, equity, and power dynamics.

According to semiotics, we live in a world of signs, where almost anything can act as a signifier and convey meaning. But what of the semiotic landscape of midwifery? What signs are present within a client's multi-sensory experience of their midwifery care? How are these signs functioning to increase equity and accessibility? Or worse, how might certain aspects of the client's experience communicate unjust power dynamics? Semiotics allows us to examine a wide communicative and educational environment. By paying particular attention to the multivalent meanings of different signs-be they written, visual, oral, or even physical-we can start to see how multimodal communication plays a vital role in a client's perception of equity and power. One way to improve client experience is by approaching education and semiotic experience from the same place as trauma-informed care. A more health-literate sensitive approach viewed through the lens of semiotics assumes all clients have little previous knowledge or comfort within a care setting. This hyperawareness and criticality of the semiotic environment would allow midwives to acknowledge various sensory and communicative biases and intentionally redesign the entire client experience. The semiotic landscape is then curated to meet the needs of the most important audience-those marginalized and discriminated against whether that is because of education, finances, race, gender, or any other intersectional identity. We must acknowledge the fact that all sign systems can either reinforce abusive power relations or work to improve them. For what is at stake here is not just a client's overall comfort, but their full understanding of the care they are receiving, the options they have, and their autonomy within their entire perinatal experience.

Grammatical structures of emoji in Japanese-language text conversations.

Emojis have become a ubiquitous part of everyday text communication worldwide. Cohn et al. (Cognit Res Princ Implic 4(1):1-18, 2019) studied the grammatical structure of emoji usage among English speakers and found a correlation between the sequence of emojis used and English word order, tending towards an subject-verb-object (SVO) sequence. However, it remains unclear whether emoji usage follows a universal grammar or whether it is influenced by native language grammar. Therefore, this study explored the potential influence of Japanese grammar on emoji usage by Japanese speakers. Twenty adults, all native Japanese speakers, participated in pairs. In Experiment 1, participants engaged in conversations through Google Hangouts on iPads. The experiment consisted of four conversation rounds of approximately 8 min each. The first two rounds involved one participant using only written Japanese and the other using only emojis and punctuation, with roles reversed in the second round. The third round required both participants to use only emojis and punctuation. The results indicated that participants preferred subject-object-verb (SOV) or object-verb (OV) sequences, with OV patterns being more common. This pattern reflects a distinctive attribute of Japanese grammatical structure, marked by the frequent omission of the subject. Experiment 2 substituted emojis for words, showing nouns were more commonly replaced than verbs due to the difficulty in conveying complex meanings. Reduced subject replacements again emphasised Japanese grammatical structure. In essence, emoji usage reflects native language structures, but complexities are challenging to convey, resulting in simplified sequences. This study offers insights for enhancing emoji-based communication and interface design, with implications for translation and broader communication.

Developmental cascades of vocal turn-taking connect prelinguistic vocalizing with early language.

Conversational turn-taking is ubiquitously found in caregiver-infant interactions, and robustly predictive of infant communicative development. Over the first year, infants take quick adult-like vocal turns with caregivers. Many studies have documented the consistency of caregiver responsiveness and its influence on infant rapid language growth. However, few have examined how caregiver responsiveness facilitates extended vocal turn-taking in real-time with infants over the first year. The influence of prelinguistic vocal turn-taking on the emergence of language has also been under-investigated. We analyzed free-play sessions of 30 caregivers and their infants at both 5 and 10 months, and obtained infant language outcomes at 18 months. We examined the developmental consistency (group-level continuity and dyad-order stability) and change of infant volubility, caregiver responses to babbling in vocal, non-vocal and multimodal modalities, and the influence of modality on caregiver-infant vocal turn-taking. Caregiver contingent responsiveness to infant babbling at 5 months predicted vocal turn-taking at 10 months. Developmental increases in prelinguistic vocalizing and vocal turn-taking from 5 to 10 months predicted infant language outcomes at 18 months. At both 5 and 10 months, caregiver vocal responses were more effective in extending turn-taking than non-vocal or multimodal responses. In summary, prelinguistic vocal turn-taking, facilitated by caregiver vocal responsiveness, is positively related to the emergence of early language.

Infant-directed communication: Examining the many dimensions of everyday caregiver-infant interactions.

Everyday caregiver-infant interactions are dynamic and multidimensional. However, existing research underestimates the dimensionality of infants' experiences, often focusing on one or two communicative signals (e.g., speech alone, or speech and gesture together). Here, we introduce "infant-directed communication" (IDC): the suite of communicative signals from caregivers to infants including speech, action, gesture, emotion, and touch. We recorded 10 min of at-home play between 44 caregivers and their 18- to 24-month-old infants from predominantly white, middle-class, English-speaking families in the United States. Interactions were coded for five dimensions of IDC as well as infants' gestures and vocalizations. Most caregivers used all five dimensions of IDC throughout the interaction, and these dimensions frequently overlapped. For example, over 60% of the speech that infants heard was accompanied by one or more non-verbal communicative cues. However, we saw marked variation across caregivers in their use of IDC, likely reflecting tailored communication to the behaviors and abilities of their infant. Moreover, caregivers systematically increased the dimensionality of IDC, using more overlapping cues in response to infant gestures and vocalizations, and more IDC with infants who had smaller vocabularies. Understanding how and when caregivers use all five signals-together and separately-in interactions with infants has the potential to redefine how developmental scientists conceive of infants' communicative environments, and enhance our understanding of the relations between caregiver input and early learning. RESEARCH HIGHLIGHTS: Infants' everyday interactions with caregivers are dynamic and multimodal, but existing research has underestimated the multidimensionality (i.e., the diversity of simultaneously occurring communicative cues) inherent in infant-directed communication. Over 60% of the speech that infants encounter during at-home, free play interactions overlap with one or more of a variety of non-speech communicative cues. The multidimensionality of caregivers' communicative cues increases in response to infants' gestures and vocalizations, providing new information about how infants' own behaviors shape their input. These findings emphasize the importance of understanding how caregivers use a diverse set of communicative behaviors-both separately and together-during everyday interactions with infants.

Hand Gestures Have Predictive Potential During Conversation: An Investigation of the Timing of Gestures in Relation to Speech.

During face-to-face conversation, transitions between speaker turns are incredibly fast. These fast turn exchanges seem to involve next speakers predicting upcoming semantic information, such that next turn planning can begin before a current turn is complete. Given that face-to-face conversation also involves the use of communicative bodily signals, an important question is how bodily signals such as co-speech hand gestures play into these processes of prediction and fast responding. In this corpus study, we found that hand gestures that depict or refer to semantic information started before the corresponding information in speech, which held both for the onset of the gesture as a whole, as well as the onset of the stroke (the most meaningful part of the gesture). This early timing potentially allows listeners to use the gestural information to predict the corresponding semantic information to be conveyed in speech. Moreover, we provided further evidence that questions with gestures got faster responses than questions without gestures. However, we found no evidence for the idea that how much a gesture precedes its lexical affiliate (i.e., its predictive potential) relates to how fast responses were given. The findings presented here highlight the importance of the temporal relation between speech and gesture and help to illuminate the potential mechanisms underpinning multimodal language processing during face-to-face conversation.

Attention Drives Visual Processing and Audiovisual Integration During Multimodal Communication.

During communication in real-life settings, our brain often needs to integrate auditory and visual information and at the same time actively focus on the relevant sources of information, while ignoring interference from irrelevant events. The interaction between integration and attention processes remains poorly understood. Here, we use rapid invisible frequency tagging and magnetoencephalography to investigate how attention affects auditory and visual information processing and integration, during multimodal communication. We presented human participants (male and female) with videos of an actress uttering action verbs (auditory; tagged at 58 Hz) accompanied by two movie clips of hand gestures on both sides of fixation (attended stimulus tagged at 65 Hz; unattended stimulus tagged at 63 Hz). Integration difficulty was manipulated by a lower-order auditory factor (clear/degraded speech) and a higher-order visual semantic factor (matching/mismatching gesture). We observed an enhanced neural response to the attended visual information during degraded speech compared to clear speech. For the unattended information, the neural response to mismatching gestures was enhanced compared to matching gestures. Furthermore, signal power at the intermodulation frequencies of the frequency tags, indexing nonlinear signal interactions, was enhanced in the left frontotemporal and frontal regions. Focusing on the left inferior frontal gyrus, this enhancement was specific for the attended information, for those trials that benefitted from integration with a matching gesture. Together, our results suggest that attention modulates audiovisual processing and interaction, depending on the congruence and quality of the sensory input.

Species- and sex-specific chemical composition from an internal gland-like tissue of an African frog family.

Intraspecific chemical communication in frogs is understudied and the few published cases are limited to externally visible and male-specific breeding glands. Frogs of the family Odontobatrachidae, a West African endemic complex of five morphologically cryptic species, have large, fatty gland-like strands along their lower mandible. We investigated the general anatomy of this gland-like strand and analysed its chemical composition. We found the strand to be present in males and females of all species. The strand varies in markedness, with well-developed strands usually found in reproductively active individuals. The strands are situated under particularly thin skin sections, the vocal sac in male frogs and a respective area in females. Gas-chromatography/mass spectrometry and multivariate analysis revealed that the strands contain sex- and species-specific chemical profiles, which are consistent across geographically distant populations. The profiles varied between reproductive and non-reproductive individuals. These results indicate that the mandibular strands in the Odontobatrachidae comprise a so far overlooked structure (potentially a gland) that most likely plays a role in the mating and/or breeding behaviour of the five Odontobatrachus species. Our results highlight the relevance of multimodal signalling in anurans, and indicate that chemical communication in frogs may not be restricted to sexually dimorphic, apparent skin glands.

The analysis of gestural and verbal pragmatic markers produced by Mild Cognitive Impaired participants during longitudinal and autobiographical interviews.

CONTEXT: This corpus-based study presents a multimodal analysis of verbal pragmatic markers and non-verbal pragmatic markers in elderly people with Mild Cognitive Impairment aged over 75 years. METHODS: The corpus collection and analysis methodology has been described in the Belgian CorpAGEst transversal study and the French VintAGE longitudinal and transversal oriented pilot studies. The protocols are available online in both English and French. RESULTS & CONCLUSION: Our general findings indicate that with ageing, verbal pragmatic markers acquire an interactive function that allows people with MCI to maintain intersubjective relationships with their interlocutor. Furthermore, at the non-verbal level, gestural manifestations are increasingly used over time with a preference for non-verbal pragmatic markers with a referential function and an adaptive function. We aim to show the benefits of linguistic and interactional scientific investigation methods through cognitive impaired ageing for clinicians and family caregivers.

Beyond the tried and true: How virtual reality, dialog setups, and a focus on multimodality can take bilingual language production research forward.

Bilinguals possess the ability of expressing themselves in more than one language, and typically do so in contextually rich and dynamic settings. Theories and models have indeed long considered context factors to affect bilingual language production in many ways. However, most experimental studies in this domain have failed to fully incorporate linguistic, social, or physical context aspects, let alone combine them in the same study. Indeed, most experimental psycholinguistic research has taken place in isolated and constrained lab settings with carefully selected words or sentences, rather than under rich and naturalistic conditions. We argue that the most influential experimental paradigms in the psycholinguistic study of bilingual language production fall short of capturing the effects of context on language processing and control presupposed by prominent models. This paper therefore aims to enrich the methodological basis for investigating context aspects in current experimental paradigms and thereby move the field of bilingual language production research forward theoretically. After considering extensions of existing paradigms proposed to address context effects, we present three far-ranging innovative proposals, focusing on virtual reality, dialog situations, and multimodality in the context of bilingual language production.

Common ground in AAC: how children who use AAC and teaching staff shape interaction in the multimodal classroom.

Children who use augmentative and alternative communication (AAC) are multimodal communicators. However, in classroom interactions involving children and staff, achieving mutual understanding and accomplishing task-oriented goals by attending to the child's unaided AAC can be challenging. This study draws on excerpts of video recordings of interactions in a classroom for 6-9-year-old children who used AAC to explore how three child participants used the range of multimodal resources available to them - vocal, movement-based, and gestural, technological, temporal - to shape (and to some degree, co-control) classroom interactions. Our research was concerned with examining achievements and problems in establishing a sense of common ground and the realization of child agency. Through detailed multimodal analysis, this paper renders visible different types of practices rejecting a request for clarification, drawing new parties into a conversation, disrupting whole-class teacher talk-through which the children in the study voiced themselves in persuasive ways. It concludes by suggesting that multimodal accounts paint a more nuanced picture of children's resourcefulness and conversational asymmetry that highlights children's agency amidst material, semiotic, and institutional constraints.

Response to comment on 'Parasite defensive limb movements enhance acoustic signal attraction in male little torrent frogs'.

Recently we showed that limb movements associated with anti-parasite defenses can enhance acoustic signal attraction in male little torrent frogs (Amolops torrentis), which suggests a potential pathway for physical movements to become co-opted into mating displays (Zhao et al., 2022). Anderson et al. argue for alternative explanations of our results and provide a reanalysis of part of our data (Anderson et al., 2023). We acknowledge some of the points raised and provide an additional analysis in support of our hypothesis.

A test of multimodal communication in humans using 881 judgements of men and women's physical, vocal, and olfactory attractiveness.

Human mate value is assessed on numerous variables including, reproductive potential and disease resistance. Many of these variables have been correlated with judgments of physical, vocal, and odor attractiveness. While some researchers posit that attractiveness judgments made across different sensory modalities reflect the same underlying variable(s) (i.e., the information is redundant), others suggest that judgments made in different modalities reflect different variables. Previous studies of human attractiveness indicate that attractiveness judgments of others' faces, bodies, and voices are intercorrelated, which is suggested to support the redundancy hypothesis. Less is known about body odor attractiveness. Only one study has simultaneously investigated the relationships between judgments of body odor, face, and voice attractiveness finding weak positive associations, but small effect sizes. In this study, we empirically investigate the correlation between different modalities of attractiveness in men and women in the largest sample to date (N = 881 ratings). For men, we find no correlations between modalities of attractiveness. However, for women we find odor, face, and voice attractiveness are weakly correlated. Moreover, a general attractiveness factor (i.e., a common underlying variable) modestly contributed to the observed correlations between modality-specific attractiveness judgments, providing some evidence for the redundancy hypothesis.

The Role of Representational Gestures and Speech Synchronicity in Auditory Input by L2 and L1 Speakers.

Speech and gesture are two integrated and temporally coordinated systems. Manual gestures can help second language (L2) speakers with vocabulary learning and word retrieval. However, it is still under-investigated whether the synchronisation of speech and gesture has a role in helping listeners compensate for the difficulties in processing L2 aural information. In this paper, we tested, in two behavioural experiments, how L2 speakers process speech and gesture asynchronies in comparison to native speakers (L1). L2 speakers responded significantly faster when gestures and the semantic relevant speech were synchronous than asynchronous. They responded significantly slower than L1 speakers regardless of speech/gesture synchronisation. On the other hand, L1 speakers did not show a significant difference between asynchronous and synchronous integration of gestures and speech. We conclude that gesture-speech asynchrony affects L2 speakers more than L1 speakers.

Multimodal Communication in the Human-Cat Relationship: A Pilot Study.

Across all species, communication implies that an emitter sends signals to a receiver, through one or more channels. Cats can integrate visual and auditory signals sent by humans and modulate their behaviour according to the valence of the emotion perceived. However, the specific patterns and channels governing cat-to-human communication are poorly understood. This study addresses whether, in an extraspecific interaction, cats are sensitive to the communication channel used by their human interlocutor. We examined three types of interactions-vocal, visual, and bimodal-by coding video clips of 12 cats living in cat cafés. In a fourth (control) condition, the human interlocutor refrained from emitting any communication signal. We found that the modality of communication had a significant effect on the latency in the time taken for cats to approach the human experimenter. Cats interacted significantly faster to visual and bimodal communication compared to the "no communication" pattern, as well as to vocal communication. In addition, communication modality had a significant effect on tail-wagging behaviour. Cats displayed significantly more tail wagging when the experimenter engaged in no communication (control condition) compared to visual and bimodal communication modes, indicating that they were less comfortable in this control condition. Cats also displayed more tail wagging in response to vocal communication compared to the bimodal communication. Overall, our data suggest that cats display a marked preference for both visual and bimodal cues addressed by non-familiar humans compared to vocal cues only. Results arising from the present study may serve as a basis for practical recommendations to navigate the codes of human-cat interactions.

Electrophysiological evidence for the enhancement of gesture-speech integration by linguistic predictability during multimodal discourse comprehension.

In face-to-face discourse, listeners exploit cues in the input to generate predictions about upcoming words. Moreover, in addition to speech, speakers produce a multitude of visual signals, such as iconic gestures, which listeners readily integrate with incoming words. Previous studies have shown that processing of target words is facilitated when these are embedded in predictable compared to non-predictable discourses and when accompanied by iconic compared to meaningless gestures. In the present study, we investigated the interaction of both factors. We recorded electroencephalogram from 60 Dutch adults while they were watching videos of an actress producing short discourses. The stimuli consisted of an introductory and a target sentence; the latter contained a target noun. Depending on the preceding discourse, the target noun was either predictable or not. Each target noun was paired with an iconic gesture and a gesture that did not convey meaning. In both conditions, gesture presentation in the video was timed such that the gesture stroke slightly preceded the onset of the spoken target by 130 ms. Our ERP analyses revealed independent facilitatory effects for predictable discourses and iconic gestures. However, the interactive effect of both factors demonstrated that target processing (i.e., gesture-speech integration) was facilitated most when targets were part of predictable discourses and accompanied by an iconic gesture. Our results thus suggest a strong intertwinement of linguistic predictability and non-verbal gesture processing where listeners exploit predictive discourse cues to pre-activate verbal and non-verbal representations of upcoming target words.

Six-month-old infants' communication in a comparative perspective: Do maternal attention and interaction matter?

Developmental precursors of the prelinguistic transition from gestures to word use can be found in the early pragmatic usage of auditory and visual signals across contexts. This study examined whether 6-month-old infants are capable of attention-sensitive communication with their mother, that is, adjusting the sensory modality of their communicative signals to their mother's attention. Proxies of maternal attention implemented in experimental conditions were the mother's visual attention (attentive/inattentive), interaction directed at the infant (interactive/non-interactive), and distance (far/close). The infants' signals were coded as either visual or auditory, following an ethological coding. Infants adjusted the sensory modality of their communicative signals mostly to maternal interaction. More auditory signals were produced when the mother was non-interactive than when she was interactive. Interactive conditions were characterized by higher rates of visual signaling and of gaze-coordinated non-vocal oral sounds. The more time infants spent looking at their attentive mother, the more they produced auditory signals, specifically non-vocal oral sounds. These findings are discussed within the articulated frameworks of evolutionary developmental psychology and early pragmatics.

Multimodal processing in face-to-face interactions: A bridging link between psycholinguistics and sensory neuroscience.

In face-to-face communication, humans are faced with multiple layers of discontinuous multimodal signals, such as head, face, hand gestures, speech and non-speech sounds, which need to be interpreted as coherent and unified communicative actions. This implies a fundamental computational challenge: optimally binding only signals belonging to the same communicative action while segregating signals that are not connected by the communicative content. How do we achieve such an extraordinary feat, reliably, and efficiently? To address this question, we need to further move the study of human communication beyond speech-centred perspectives and promote a multimodal approach combined with interdisciplinary cooperation. Accordingly, we seek to reconcile two explanatory frameworks recently proposed in psycholinguistics and sensory neuroscience into a neurocognitive model of multimodal face-to-face communication. First, we introduce a psycholinguistic framework that characterises face-to-face communication at three parallel processing levels: multiplex signals, multimodal gestalts and multilevel predictions. Second, we consider the recent proposal of a lateral neural visual pathway specifically dedicated to the dynamic aspects of social perception and reconceive it from a multimodal perspective ("lateral processing pathway"). Third, we reconcile the two frameworks into a neurocognitive model that proposes how multiplex signals, multimodal gestalts, and multilevel predictions may be implemented along the lateral processing pathway. Finally, we advocate a multimodal and multidisciplinary research approach, combining state-of-the-art imaging techniques, computational modelling and artificial intelligence for future empirical testing of our model.