Uses of Linguistic Context in Speech Listening: Does Acquired Hearing Loss Lead to Reduced Engagement of Prediction?

Research investigating the complex interplay of cognitive mechanisms involved in speech listening for people with hearing loss has been gaining prominence. In particular, linguistic context allows the use of several cognitive mechanisms that are not well distinguished in hearing science, namely those relating to "postdiction", "integration", and "prediction". We offer the perspective that an unacknowledged impact of hearing loss is the differential use of predictive mechanisms relative to age-matched individuals with normal hearing. As evidence, we first review how degraded auditory input leads to reduced prediction in people with normal hearing, then consider the literature exploring context use in people with acquired postlingual hearing loss. We argue that no research on hearing loss has directly assessed prediction. Because current interventions for hearing do not fully alleviate difficulty in conversation, and avoidance of spoken social interaction may be a mediator between hearing loss and cognitive decline, this perspective could lead to greater understanding of cognitive effects of hearing loss and provide insight regarding new targets for intervention.

A common neural code for meaning in discourse production and comprehension.

How does the brain code the meanings conveyed by language? Neuroimaging studies have investigated this by linking neural activity patterns during discourse comprehension to semantic models of language content. Here, we applied this approach to the production of discourse for the first time. Participants underwent fMRI while producing and listening to discourse on a range of topics. We used a distributional semantic model to quantify the similarity between different speech passages and identified where similarity in neural activity was predicted by semantic similarity. When people produced discourse, speech on similar topics elicited similar activation patterns in a widely distributed and bilateral brain network. This network was overlapping with, but more extensive than, the regions that showed similarity effects during comprehension. Critically, cross-task neural similarities between comprehension and production were also predicted by similarities in semantic content. This result suggests that discourse semantics engages a common neural code that is shared between comprehension and production. Effects of semantic similarity were bilateral in all three RSA analyses, even while univariate activation contrasts in the same data indicated left-lateralised BOLD responses. This indicates that right-hemisphere regions encode semantic properties even when they are not activated above baseline. We suggest that right-hemisphere regions play a supporting role in processing the meaning of discourse during both comprehension and production.

Similar neural networks respond to coherence during comprehension and production of discourse.

When comprehending discourse, listeners engage default-mode regions associated with integrative semantic processing to construct a situation model of its content. We investigated how similar networks are engaged when we produce, as well as comprehend, discourse. During functional magnetic resonance imaging, participants spoke about a series of specific topics and listened to discourse on other topics. We tested how activation was predicted by natural fluctuations in the global coherence of the discourse, that is, the degree to which utterances conformed to the expected topic. The neural correlates of coherence were similar across speaking and listening, particularly in default-mode regions. This network showed greater activation when less coherent speech was heard or produced, reflecting updating of mental representations when discourse did not conform to the expected topic. In contrast, regions that exert control over semantic activation showed task-specific effects, correlating negatively with coherence during listening but not during production. Participants who showed greater activation in left inferior prefrontal cortex also produced more coherent discourse, suggesting a specific role for this region in goal-directed regulation of speech content. Results suggest strong correspondence of discourse representations during speaking and listening. However, they indicate that the semantic control network plays different roles in comprehension and production.

Ideography in interaction.

The standardization account predicts short message service (SMS) interactions, allowed by current technology, will support the use and conventionalization of ideographs. Relying on psycholinguistic theories of dialogue, we argue that ideographs (such as emoji) can be used by interlocutors in SMS interactions, so that the main contributor can use them to accompany language and the addressee can use them as stand-alone feedback.

The Relation Between Preschoolers' Vocabulary Development and Their Ability to Predict and Recognize Words.

By age 2, children are developing foundational language processing skills, such as quickly recognizing words and predicting words before they occur. How do these skills relate to children's structural knowledge of vocabulary? Multiple aspects of language processing were simultaneously measured in a sample of 2-to-5-year-olds (N = 215): While older children were more fluent at recognizing words, at predicting words in a graded fashion, and at revising incorrect predictions, only revision was associated with concurrent vocabulary knowledge once age was accounted for. However, an exploratory longitudinal follow-up (N = 55) then found that word recognition and prediction skills were associated with rate of subsequent vocabulary development, but revision skills were not. We argue that prediction skills may facilitate language learning through enhancing processing speed.

Shared neural representations of syntax during online dyadic communication.

When people communicate, they come to see the world in a similar way to each other by aligning their mental representations at such levels as syntax. Syntax is an essential feature of human language that distinguishes humans from other non-human animals. However, whether and how communicators share neural representations of syntax is not well understood. Here we addressed this issue by measuring the brain activity of both communicators in a series of dyadic communication contexts, by using functional near-infrared spectroscopy (fNIRS)-based hyperscanning. Two communicators alternatively spoke sentences either with the same or with different syntactic structures. Results showed a significantly higher-level increase of interpersonal neural synchronization (INS) at right posterior superior temporal cortex when communicators produced the same syntactic structures as each other compared to when they produced different syntactic structures. These increases of INS correlated significantly with communication quality. Our findings provide initial evidence for shared neural representations of syntax between communicators.

The development of linguistic prediction: Predictions of sound and meaning in 2- to 5-year-olds.

Language processing in adults is facilitated by an expert ability to generate detailed predictions about upcoming words. This may seem like an acquired skill, but some models of language acquisition assume that the ability to predict is a prerequisite for learning. This raises a question: Do children learn to predict, or do they predict to learn? We tested whether children, like adults, can generate expectations about not just the meanings of upcoming words but also their sounds, which would be critical for using prediction to learn about language. In two looking-while-listening experiments, we show that 2-year-olds can generate expectations about meaning based on a determiner (Can you see one…ball/two…ice creams?) but that even children as old as 5 years do not show an adult-like ability to predict the phonology of upcoming words based on a determiner (Can you see a…ball/an…ice cream?). Our results, therefore, suggest that the ability to generate detailed predictions is a late-acquired skill. We argue that prediction might not be the key mechanism driving children's learning, but that the ability to generate accurate semantic predictions may nevertheless have facilitative effects on language development.

Structural priming and the representation of language.

Structural priming offers a powerful method for experimentally investigating the mental representation of linguistic structure. We clarify the nature of our proposal, justify the versatility of priming, consider alternative approaches, and discuss how our specific account can be extended to new questions as part of an interdisciplinary programme integrating linguistics and psychology as part of the cognitive sciences of language.

An experimental approach to linguistic representation.

Within the cognitive sciences, most researchers assume that it is the job of linguists to investigate how language is represented, and that they do so largely by building theories based on explicit judgments about patterns of acceptability - whereas it is the task of psychologists to determine how language is processed, and that in doing so, they do not typically question the linguists' representational assumptions. We challenge this division of labor by arguing that structural priming provides an implicit method of investigating linguistic representations that should end the current reliance on acceptability judgments. Moreover, structural priming has now reached sufficient methodological maturity to provide substantial evidence about such representations. We argue that evidence from speakers' tendency to repeat their own and others' structural choices supports a linguistic architecture involving a single shallow level of syntax connected to a semantic level containing information about quantification, thematic relations, and information structure, as well as to a phonological level. Many of the linguistic distinctions often used to support complex (or multilevel) syntactic structure are instead captured by semantics; however, the syntactic level includes some specification of "missing" elements that are not realized at the phonological level. We also show that structural priming provides evidence about the consistency of representations across languages and about language development. In sum, we propose that structural priming provides a new basis for understanding the nature of language.

Causal Role of Motor Simulation in Turn-Taking Behavior.

Overlap between sensory and motor representations has been documented for a range of human actions, from grasping (Rizzolatti et al., 1996b) to playing a musical instrument (Novembre and Keller, 2014). Such overlap suggests that individuals use motor simulation to predict the outcome of observed actions (Wolpert, 1997). Here we investigate motor simulation as a basis of human communication. Using a musical turn-taking task, we show that pianists call on motor representations of their partner's part to predict when to come in for their own turn. Pianists played alternating solos with a videoed partner, and double-pulse transcranial magnetic stimulation was applied around the turn-switch to temporarily disrupt processing in two cortical regions implicated previously in different forms of motor simulation: (1) the dorsal premotor cortex (dPMC), associated with automatic motor resonance during passive observation of hand actions, especially when the actions are familiar (Lahav et al., 2007); and (2) the supplementary motor area (SMA), involved in active motor imagery, especially when the actions are familiar (Baumann et al., 2007). Stimulation of the right dPMC decreased the temporal accuracy of pianists' (right-hand) entries relative to sham when the partner's (left-hand) part had been rehearsed previously. This effect did not occur for dPMC stimulation without rehearsal or for SMA stimulation. These findings support the role of the dPMC in predicting the time course of observed actions via resonance-based motor simulation during turn-taking. Because turn-taking spans multiple modes of human interaction, we suggest that simulation is a foundational mechanism underlying the temporal dynamics of joint action. SIGNIFICANCE STATEMENT: Even during passive observation, seeing or hearing somebody execute an action from within our repertoire activates motor cortices of our brain. But what is the functional relevance of such "motor simulation"? By combining a musical duet task with a real-time repetitive transcranial magnetic stimulation protocol, we provide evidence indicating that the dorsal premotor cortex plays a causal role in accurate turn-taking coordination between a pianist and their observed interaction partner. Given that turn-taking behavior is a fundamental feature of human communication, we suggest that simulation is a foundational mechanism underlying the temporal dynamics of communicative joint action.

The effect of noun phrase length on the form of referring expressions.

The length of a noun phrase has been shown to influence choices such as syntactic role assignment (e.g., whether the noun phrase is realized as the subject or the object). But does length also affect the choice between different forms of referring expressions? Three experiments investigated the effect of antecedent length on the choice between pronouns (e.g., he) and repeated nouns (e.g., the actor) using a sentence-continuation paradigm. Experiments 1 and 2 found an effect of antecedent length on written continuations: Participants used more pronouns (relative to repeated nouns) when the antecedent was longer than when it was shorter. Experiment 3 used a spoken continuation task and replicated the effect of antecedent length on the choice of referring expressions. Taken together, the results suggest that longer antecedents increase the likelihood of pronominal reference. The results support theories arguing that length enhances the accessibility of the associated entity through richer semantic encoding.

Authors' response: forward models and their implications for production, comprehension, and dialogue.

Our target article proposed that language production and comprehension are interwoven, with speakers making predictions of their own utterances and comprehenders making predictions of other people's utterances at different linguistic levels. Here, we respond to comments about such issues as cognitive architecture and its neural basis, learning and development, monitoring, the nature of forward models, communicative intentions, and dialogue.

An integrated theory of language production and comprehension.

Currently, production and comprehension are regarded as quite distinct in accounts of language processing. In rejecting this dichotomy, we instead assert that producing and understanding are interwoven, and that this interweaving is what enables people to predict themselves and each other. We start by noting that production and comprehension are forms of action and action perception. We then consider the evidence for interweaving in action, action perception, and joint action, and explain such evidence in terms of prediction. Specifically, we assume that actors construct forward models of their actions before they execute those actions, and that perceivers of others' actions covertly imitate those actions, then construct forward models of those actions. We use these accounts of action, action perception, and joint action to develop accounts of production, comprehension, and interactive language. Importantly, they incorporate well-defined levels of linguistic representation (such as semantics, syntax, and phonology). We show (a) how speakers and comprehenders use covert imitation and forward modeling to make predictions at these levels of representation, (b) how they interweave production and comprehension processes, and (c) how they use these predictions to monitor the upcoming utterances. We show how these accounts explain a range of behavioral and neuroscientific data on language processing and discuss some of the implications of our proposal.

Talking to each other and talking together: joint language tasks and degrees of interactivity.

A second-person perspective in neuroscience is particularly appropriate for the study of communication. We describe how the investigation of joint language tasks can contribute to our understanding of the mechanisms underlying interaction.

Response planning during question-answering: does deciding what to say involve deciding how to say it?

To answer a question, speakers must determine their response and formulate it in words. But do they decide on a response before formulation, or do they formulate different potential answers before selecting one? We addressed this issue in a verbal question-answering experiment. Participants answered questions more quickly when they had one potential answer (e.g., Which tourist attraction in Paris is very tall?) than when they had multiple potential answers (e.g., What is the name of a Shakespeare play?). Participants also answered more quickly when the set of potential answers were on average short rather than long, regardless of whether there was only one or multiple potential answers. Thus, participants were not affected by the linguistic complexity of unselected but plausible answers. These findings suggest that participants select a single answer before formulation.

Representation of others' synchronous and asynchronous sentences interferes with sentence production.

In dialogue, people represent each other's utterances to take turns and communicate successfully. In previous work, speakers who were naming single pictures or picture pairs represented whether another speaker was engaged in the same task (vs a different or no task) concurrently but did not represent in detail the content of the other speaker's utterance. Here, we investigate the co-representation of whole sentences. In three experiments, pairs of speakers imagined each other producing active or passive descriptions of transitive events. Speakers took longer to begin speaking when they believed their partner was also preparing to speak, compared to when they did not. Interference occurred when speakers believed their partners were preparing to speak at the same time as them (synchronous production and co-representation; Experiment 1), and also when speakers believed that their partner would speak only after them (asynchronous production and co-representation; Experiments 2a and 2b). However, interference was generally no greater when speakers believed their partner was preparing a different compared to a similar utterance, providing no consistent evidence that speakers represented what their partners were preparing to say. Taken together, these findings indicate that speakers can represent another's intention to speak even as they are themselves preparing to speak, but that such representation tends to lack detail.

Mechanisms of alignment: shared control, social cognition and metacognition.

In dialogue, speakers process a great deal of information, take and give the floor to each other, and plan and adjust their contributions on the fly. Despite the level of coordination and control that it requires, dialogue is the easiest way speakers possess to come to similar conceptualizations of the world. In this paper, we show how speakers align with each other by mutually controlling the flow of the dialogue and constantly monitoring their own and their interlocutors' way of representing information. Through examples of conversation, we introduce the notions of shared control, meta-representations of alignment and commentaries on alignment, and show how they support mutual understanding and the collaborative creation of abstract concepts. Indeed, whereas speakers can share similar representations of concrete concepts just by mutually attending to a tangible referent or by recalling it, they are likely to need more negotiation and mutual monitoring to build similar representations of abstract concepts. This article is part of the theme issue 'Concepts in interaction: social engagement and inner experiences'.

Evidence against egocentric prediction during language comprehension.

Although previous research has demonstrated that language comprehension can be egocentric, there is little evidence for egocentricity during prediction. In particular, comprehenders do not appear to predict egocentrically when the context makes it clear what the speaker is likely to refer to. But do comprehenders predict egocentrically when the context does not make it clear? We tested this hypothesis using a visual-world eye-tracking paradigm, in which participants heard sentences containing the gender-neutral pronoun They (e.g. They would like to wear…) while viewing four objects (e.g. tie, dress, drill, hairdryer). Two of these objects were plausible targets of the verb (tie and dress), and one was stereotypically compatible with the participant's gender (tie if the participant was male; dress if the participant was female). Participants rapidly fixated targets more than distractors, but there was no evidence that participants ever predicted egocentrically, fixating objects stereotypically compatible with their own gender. These findings suggest that participants do not fall back on their own egocentric perspective when predicting, even when they know that context does not make it clear what the speaker is likely to refer to.

Purposeful listening in challenging conditions: A study of prediction during consecutive interpreting in noise.

Prediction is often used during language comprehension. However, studies of prediction have tended to focus on L1 listeners in quiet conditions. Thus, it is unclear how listeners predict outside the laboratory and in specific communicative settings. Here, we report two eye-tracking studies which used a visual-world paradigm to investigate whether prediction during a consecutive interpreting task differs from prediction during a listening task in L2 listeners, and whether L2 listeners are able to predict in the noisy conditions that might be associated with this communicative setting. In a first study, thirty-six Dutch-English bilinguals either just listened to, or else listened to and then consecutively interpreted, predictable sentences presented on speech-shaped sound. In a second study, another thirty-six Dutch-English bilinguals carried out the same tasks in clear speech. Our results suggest that L2 listeners predict the meaning of upcoming words in noisy conditions. However, we did not find that predictive eye movements depended on task, nor that L2 listeners predicted upcoming word form. We also did not find a difference in predictive patterns when we compared our two studies. Thus, L2 listeners predict in noisy circumstances, supporting theories which posit that prediction regularly takes place in comprehension, but we did not find evidence that a subsequent production task or noise affects semantic prediction.

The effects of semantic similarity on Mandarin speakers' referential expressions.

Previous research has found apparently contradictory effects of a semantically similar competitor on how people refer to previously mentioned entities. To address this issue, we conducted two picture-description experiments in spoken Mandarin. In Experiment 1, participants saw pictures and heard sentences referring to both the target referent and a competitor, and then described actions involving only the target referent. They produced fewer omissions and more repeated noun phrases when the competitor was semantically similar to the target referent than otherwise. In Experiment 2, participants saw introductory pictures and heard sentences referring to only the target referent, and then described actions involving both the target referent and a competitor. They produced more omissions and fewer pronouns when the competitor was semantically similar to the target referent than otherwise. We interpret the results in terms of the representation of discourse entities and the stages of language production.