Speaker-listener neural coupling correlates with semantic and acoustic features of naturalistic speech.

Recent research has extensively reported the phenomenon of inter-brain neural coupling between speakers and listeners during speech communication. Yet, the specific speech processes underlying this neural coupling remain elusive. To bridge this gap, this study estimated the correlation between the temporal dynamics of speaker-listener neural coupling with speech features, utilizing two inter-brain datasets accounting for different noise levels and listener's language experiences (native vs. non-native). We first derived time-varying speaker-listener neural coupling, extracted acoustic feature (envelope) and semantic features (entropy and surprisal) from speech, and then explored their correlational relationship. Our findings reveal that in clear conditions, speaker-listener neural coupling correlates with semantic features. However, as noise increases, this correlation is only significant for native listeners. For non-native listeners, neural coupling correlates predominantly with acoustic feature rather than semantic features. These results revealed how speaker-listener neural coupling is associated with the acoustic and semantic features under various scenarios, enriching our understanding of the inter-brain neural mechanisms during natural speech communication. We therefore advocate for more attention on the dynamic nature of speaker-listener neural coupling and its modeling with multilevel speech features.

How does the human brain process noisy speech in real life? Insights from the second-person neuroscience perspective.

Comprehending speech with the existence of background noise is of great importance for human life. In the past decades, a large number of psychological, cognitive and neuroscientific research has explored the neurocognitive mechanisms of speech-in-noise comprehension. However, as limited by the low ecological validity of the speech stimuli and the experimental paradigm, as well as the inadequate attention on the high-order linguistic and extralinguistic processes, there remains much unknown about how the brain processes noisy speech in real-life scenarios. A recently emerging approach, i.e., the second-person neuroscience approach, provides a novel conceptual framework. It measures both of the speaker's and the listener's neural activities, and estimates the speaker-listener neural coupling with regarding of the speaker's production-related neural activity as a standardized reference. The second-person approach not only promotes the use of naturalistic speech but also allows for free communication between speaker and listener as in a close-to-life context. In this review, we first briefly review the previous discoveries about how the brain processes speech in noise; then, we introduce the principles and advantages of the second-person neuroscience approach and discuss its implications to unravel the linguistic and extralinguistic processes during speech-in-noise comprehension; finally, we conclude by proposing some critical issues and calls for more research interests in the second-person approach, which would further extend the present knowledge about how people comprehend speech in noise.

Exploring How the Arms Can Help the Legs in Facilitating Gait Rehabilitation.

Inspired by the ideas from the fields of gait rehabilitation, neuroscience, and locomotion biomechanics and energetics, a body of work is reviewed that has led to propose a conceptual framework for novel "self-assistive" walking devices that could further promote walking recovery from incomplete spinal cord injuries. The underlying rationale is based on a neural coupling mechanism that governs the coordinated movements of the arms and legs during walking, and that the excitability of these neural pathways can be exploited by actively engaging the arms during locomotor training. Self-assistive treadmill walking rehabilitation devices are envisioned as an approach that would allow an individual to actively use their arms to help the legs during walking. It is hoped that the conceptual framework inspires the design and use of self-assistive walking devices that are tailored to assist individuals with an incomplete spinal cord injury to regain their functional walking ability.

Neural coordination of bilateral hand movements: evidence for an involvement of brainstem motor centres.

Bilateral hand movements are assumed to be coordinated by a neural coupling mechanism. Neural coupling is experimentally reflected in complex electromyographic (EMG) responses in the forearm muscles of both sides to unilateral electrical arm nerve stimulation (ES). The aim of this study was to examine a potential involvement of the reticulospinal system in neural coupling by the application of loud acoustic stimuli (LAS) known to activate neurons of this system. LAS, ES and combined LAS/ES were applied to healthy subjects during visually guided bilateral hand flexion-extension movements. Muscle responses to the different stimuli were evaluated by electrophysiological recordings. Unilateral electrical ulnar nerve stimulation resulted in neural coupling responses in the forearm extensors (FE) of both sides. Interestingly, LAS evoked bilateral EMG responses that were similar in their configuration to those induced by ES. The presence of startles was associated with a shift of the onset and enhanced amplitude of LAS-induced coupling-like responses. Upon combined LAS/ES application, ES facilitated ipsilateral startles and coupling-like responses. Modulation of coupling-like responses by startles, the similarity of the responses to ES and LAS, and their interaction following combined stimulation suggests that both responses are mediated by the reticulospinal system. Our findings provide novel indirect evidence that the reticulospinal system is involved in the neural coupling of hand movements. This becomes clinically relevant in subjects with a damaged corticospinal system where a dominant reticulospinal system leads to involuntary limb coupling, referred to as associated movements. KEY POINTS: Automatic coordination of hand movements is assumed to be mediated by a neural coupling mechanism reflected by bilateral reflex responses in forearm muscles to unilateral electrical arm nerve stimulation (ES). Loud acoustic stimuli (LAS) were applied to assess a potential involvement of the reticulospinal system in the neural coupling mechanism. LAS evoked a bilateral reflex response in the forearm extensors that was similar to the neural coupling response to ES, and which could be separated from the acoustic startle response. Combined application of LAS and ES resulted in a facilitation of startle and coupling-like responses ipsilateral to ES, thus indicating an interaction of afferences from both stimuli. These novel findings provide indirect evidence that the reticulospinal system is a key motor structure for the coupling of bilateral hand movements.

Neural Coupling between Interhemispheric and Frontoparietal Functional Connectivity during Semantic Processing.

Interhemispheric and frontoparietal functional connectivity have been reported to increase during explicit information processing. However, it is unclear how and when interhemispheric and frontoparietal functional connectivity interact during explicit semantic processing. Here, we tested the neural coupling hypothesis that explicit semantic processing promotes neural activity in the nondominant right hemispheric areas, owing to synchronization with enhanced frontoparietal functional connectivity at later processing stages. We analyzed electroencephalogram data obtained using a semantic priming paradigm, which comprised visual priming and target words successively presented under direct or indirect attention to semantic association. Scalp potential analysis demonstrated that the explicit processing of congruent targets reduced negative event-related potentials, as previously reported. Current source density analysis showed that explicit semantic processing activated the right temporal area during later temporal intervals. Subsequent dynamic functional connectivity and neural coupling analyses revealed that explicit semantic processing increased the correlation between right temporal source activities and frontoparietal functional connectivity in later temporal intervals. These findings indicate that explicit semantic processing increases neural coupling between the interhemispheric and frontoparietal functional connectivity during later processing stages.

From correlation to communication: Disentangling hidden factors from functional connectivity changes.

While correlations in the BOLD fMRI signal are widely used to capture functional connectivity (FC) and its changes across contexts, its interpretation is often ambiguous. The entanglement of multiple factors including local coupling of two neighbors and nonlocal inputs from the rest of the network (affecting one or both regions) limits the scope of the conclusions that can be drawn from correlation measures alone. Here we present a method of estimating the contribution of nonlocal network input to FC changes across different contexts. To disentangle the effect of task-induced coupling change from the network input change, we propose a new metric, "communication change," utilizing BOLD signal correlation and variance. With a combination of simulation and empirical analysis, we demonstrate that (1) input from the rest of the network accounts for a moderate but significant amount of task-induced FC change and (2) the proposed "communication change" is a promising candidate for tracking the local coupling in task context-induced change. Additionally, when compared to FC change across three different tasks, communication change can better discriminate specific task types. Taken together, this novel index of local coupling may have many applications in improving our understanding of local and widespread interactions across large-scale functional networks.

The effect of speech-gesture asynchrony on the neural coupling of interlocutors in interpreter-mediated communication.

In everyday face-to-face communication, speakers use speech to transfer information and rely on co-occurring nonverbal cues, such as hand and facial gestures. The integration of speech and gestures facilitates both language comprehension and the skill of the theory of mind. Consecutive dialogue interpreting (DI) allows dyads of different linguistic backgrounds to communicate with each other. The interpreter interprets after the interlocutor has finished a turn, so the interlocutor watches the gesture first and hears the target language a few seconds later, resulting in speech-gesture asynchrony. In this study, we used the functional near-infrared spectroscopy hyperscanning technique to investigate the influence of speech-gesture asynchrony on different levels of communication. Twenty groups were recruited for the DI experiments. The results showed that when the interpreter performed consecutive interpreting, the time-lagged neural coupling at the temporoparietal junction decreased compared to simultaneous interpreting. It suggests that speech-gesture asynchrony significantly weakened the ability of interlocutors to understand each other's mental state, and the decreased neural coupling was significantly correlated with the interpreter's interpretation skill. In addition, the time-aligned neural coupling at the left inferior frontal gyrus increased, which suggests that, as compensation, the interlocutor's verbal working memory increases in line with the communication process.

The hyper-brain neural couplings distinguishing high-creative group dynamics: an fNIRS hyperscanning study.

This hyperscanning study aimed to identify a neural coupling profile that distinguishes high-creative group dynamics through functional near infrared spectroscopy. A total of 123 dyads completed one creativity task (alternative uses task, AUT) and contrast task (objective characteristics task). A K-means clustering analysis on AUT performance grouped 31/29 dyads into high/low-creative group, respectively. In comparison with the low-creative group, the high-creative group showed: (i) higher collective flexibility and delayed perspective-taking behaviors, but lower immediate perspective-taking behaviors; (ii) enhanced interpersonal brain synchronization (IBS) between the left inferior frontal gyrus (lIFG) and right motor cortex, and nodal Eloc at the right superior temporal gyrus (rSTG); (iii) declined intrapersonal functional connectivity between the right angular gyrus (rAG) and rSTG, and IBS between the lIFG and rAG. The enhanced neural couplings positively correlated with group creative performance, whereas a reverse correlation pattern existed in the declined ones. A leave-one-out cross-validation analysis showed these neural couplings reliably predicted group creative performance within the sample. These indicate that high-creative group dynamics are characterized by utilizing partners' shared information when necessary (e.g. encountering idea exhaustion). A neural coupling profile consisting of sophisticated interplays between regions within frontal, temporal, and parietal lobes may underlie high-creative creative dynamics.

An Approach to Neuroimaging Interpersonal Interactions in Mental Health Interventions.

BACKGROUND: Conventional paradigms in clinical neuroscience tend to be constrained in terms of ecological validity, raising several challenges to studying the mechanisms mediating treatments and outcomes in clinical settings. Addressing these issues requires real-world neuroimaging techniques that are capable of continuously collecting data during free-flowing interpersonal interactions and that allow for experimental designs that are representative of the clinical situations in which they occur. METHODS: In this work, we developed a paradigm that fractionates the major components of human-to-human verbal interactions occurring in clinical situations and used functional near-infrared spectroscopy to assess the brain systems underlying clinician-client discourse (N = 30). RESULTS: Cross-brain neural coupling between people was significantly greater during clinical interactions compared with everyday life verbal communication, particularly between the prefrontal cortex (e.g., inferior frontal gyrus) and inferior parietal lobule (e.g., supramarginal gyrus). The clinical tasks revealed extensive increases in activity across the prefrontal cortex, especially in the rostral prefrontal cortex (area 10), during periods in which participants were required to silently reason about the dysfunctional cognitions of the other person. CONCLUSIONS: This work demonstrates a novel experimental approach to investigating the neural underpinnings of interpersonal interactions that typically occur in clinical settings, and its findings support the idea that particular prefrontal systems might be critical to cultivating mental health.

The "two-brain" approach reveals the active role of task-deactivated default mode network in speech comprehension.

Exhibiting deactivation and anticorrelation with task-positive networks, the default mode network (DMN) has been traditionally thought to be suppressed to support externally oriented cognitive processes during spoken language comprehension. In contrast, recent studies examining listener-listener intersubject correlation (ISC) have proposed an active role of DMN in language comprehension. How can we reconcile those seemingly conflicting results? This study adopted a "two-brain" paradigm and combined "within-brain" and "across-brain" analyses to address this issue. We found, despite being deactivated and anticorrelated with the language network (LN) and executive control network (ECN), both the anterior and posterior DMN in the listeners' brains were temporally coupled with the homologous networks in the speaker's brain. Interestingly, the listener-speaker neural couplings persisted even after controlling for listener-listener ISC. Moreover, the coupling strength of posterior DMN positively correlated with the listeners' speech comprehension. Further dynamic causal modeling showed that the LN and ECN, the anterior DMN, and the posterior DMN occupied the bottom, intermediate, and top layers of a hierarchical system, respectively. We suggest the DMN may primarily serve as an internal module that cooperates with the externally oriented modules, potentially supporting the transformation of external acoustic signals into internal mental representations during successful language comprehension.

Speaker-Listener Neural Coupling Reveals an Adaptive Mechanism for Speech Comprehension in a Noisy Environment.

Comprehending speech in noise is an essential cognitive skill for verbal communication. However, it remains unclear how our brain adapts to the noisy environment to achieve comprehension. The present study investigated the neural mechanisms of speech comprehension in noise using an functional near-infrared spectroscopy-based inter-brain approach. A group of speakers was invited to tell real-life stories. The recorded speech audios were added with meaningless white noise at four signal-to-noise levels and then played to listeners. Results showed that speaker-listener neural couplings of listener's left inferior frontal gyri (IFG), that is, sensorimotor system, and right middle temporal gyri (MTG), angular gyri (AG), that is, auditory system, were significantly higher in listening conditions than in the baseline. More importantly, the correlation between neural coupling of listener's left IFG and the comprehension performance gradually became more positive with increasing noise level, indicating an adaptive role of sensorimotor system in noisy speech comprehension; however, the top behavioral correlations for the coupling of listener's right MTG and AG were only obtained in mild noise conditions, indicating a different and less robust mechanism. To sum up, speaker-listener coupling analysis provides added value and new sight to understand the neural mechanism of speech-in-noise comprehension.

Effect of Thoracic Connective Lesion on Inter-Leg Coordination in Freely Walking Stick Insects.

Multi-legged locomotion requires appropriate coordination of all legs with coincident ground contact. Whereas behaviourally derived coordination rules can adequately describe many aspects of inter-leg coordination, the neural mechanisms underlying these rules are still not entirely clear. The fact that inter-leg coordination is strongly affected by cut thoracic connectives in tethered walking insects, shows that neural information exchange among legs is important. As yet, recent studies have shown that load transfer among legs can contribute to inter-leg coordination through mechanical coupling alone, i.e., without neural information exchange among legs. Since naturalistic load transfer among legs works only in freely walking animals but not in tethered animals, we tested the hypothesis that connective lesions have less strong effects if mechanical coupling through load transfer among legs is possible. To do so, we recorded protraction/retraction angles of all legs in unrestrained walking stick insects that either had one thoracic connective cut or had undergone a corresponding sham operation. In lesioned animals, either a pro-to-mesothorax or a meso-to-metathorax connective was cut. Overall, our results on temporal coordination were similar to published reports on tethered walking animals, in that the phase relationship of the legs immediately adjacent to the lesion was much less precise, although the effect on mean phase was relatively weak or absent. Lesioned animals could walk at the same speed as the control group, though with a significant sideward bias toward the intact side. Detailed comparison of lesion effects in free-walking and supported animals reveal that the strongest differences concern the spatial coordination among legs. In free walking, lesioned animals, touch-down and lift-off positions shifted significantly in almost all legs, including legs of the intact body side. We conclude that insects with disrupted neural information transfer through one connective adjust to this disruption differently if they experience naturalistic load distribution. While mechanical load transfer cannot compensate for lesion-induced effects on temporal inter-leg coordination, several compensatory changes in spatial coordination occur only if animals carry their own weight.

Reduced listener-speaker neural coupling underlies speech understanding difficulty in older adults.

An increasing number of studies have highlighted the importance of listener-speaker neural coupling in successful verbal communication. Whether the brain-to-brain coupling changes with healthy aging and the possible role of this change in the speech comprehension of older adults remain unexplored. In this study, we scanned with fMRI a young and an older speaker telling real-life stories and then played the audio recordings to a group of young (N = 28, aged 19-27 year) and a group of older adults during scanning (N = 27, aged 53-75 year), respectively. The older listeners understood the speech less well than did the young listeners, and the age of the older listeners was negatively correlated with their level of speech understanding. Compared to the young listener-speaker dyads, the older dyads exhibited reduced neural couplings in both linguistic and extra-linguistic areas. Moreover, within the older group, the listener's age was negatively correlated with the overall strength of interbrain coupling, which in turn was associated with reduced speech understanding. These results reveal the deficits of older adults in achieving neural alignment with other brains, which may underlie the age-related decline in speech understanding.

Mechanical Design and Control System Development of a Rehabilitation Robotic System for Walking With Arm Swing.

Background: Interlimb neural coupling implies that arm swing should be included during gait training to improve rehabilitation outcomes. We previously developed several systems for production of walking with arm swing, but the reaction forces on the foot sole during usage of the systems were not satisfactory and there was potential to improve control system performance. This work aimed to design and technically evaluate a novel system for producing walking with synchronised arm and leg movement and with dynamic force loading on the foot soles. Methods: The robotic system included a passive curved treadmill and a trunk frame, upon which the rigs for the upper and lower limbs were mounted. Ten actuators and servocontrollers with EtherCAT communication protocol controlled the bilateral shoulder, elbow, hip, knee and ankle joints. Impedance control algorithms were developed and ran in an industrial PC. Flexible pressure sensors recorded the plantar forces on the foot soles. The criteria of implementation and responsiveness were used to formally evaluate the technical feasibility of the system. Results: Using impedance algorithms, the system produced synchronous walking with arm swing on the curved treadmill, with mean RMS angular tracking error <2° in the 10 joint profiles. The foot trajectories relative to the hip presented similar shapes to those during normal gait, with mean RMS displacement error <1.5 cm. A force pattern that started at the heel and finished at the forefoot was observed during walking using the system, which was similar to the pattern from overground walking. Conclusion: The robotic system produced walking-like kinematics in the 10 joints and in the foot trajectories. Integrated with the curved treadmill, the system also produced walking-like force patterns on the foot soles. The system is considered feasible as far as implementation and responsiveness are concerned. Future work will focus on improvement of the mechanical system for future clinical application.

Neural coordination of bilateral power and precision finger movements.

The dexterity of hands and fingers is related to the strength of control by cortico-motoneuronal connections which exclusively exist in primates. The cortical command is associated with a task-specific, rapid proprioceptive adaptation of forces applied by hands and fingers to an object. This neural control differs between "power grip" movements (e.g., reach and grasp of a cup) where hand and fingers act as a unity and "precision grip" movements (e.g., picking up a raspberry) where fingers move independently from the hand. In motor tasks requiring hands and fingers of both sides a "neural coupling" (reflected in bilateral reflex responses to unilateral stimulations) coordinates power grip movements (e.g., opening a bottle). In contrast, during bilateral precision movements, such as playing piano, the fingers of both hands move independently, due to a direct cortico-motoneuronal control, while the hands are coupled (e.g., to maintain the rhythm between the two sides). While most studies on prehension concern unilateral hand movements, many activities of daily life are tackled by bilateral power grips where a neural coupling serves for an automatic movement performance. In primates this mode of motor control is supplemented by a system that enables the uni- or bilateral performance of skilled individual finger movements.

Arm cycling increases the short-latency reflex from ankle dorsiflexor afferents to knee extensor muscles.

Low-intensity electrical stimulation of the common peroneal nerve (CPN) evokes a short latency reflex in the heteronymous knee extensor muscles (referred to as the CPN reflex). The CPN reflex is facilitated at a heel strike during walking, contributing to body weight support. However, the origin of the CPN reflex increase during walking remains unclear. We speculate that this increase originates from multiple sources due to a body of evidence suggesting the presence of neural coupling between the arms and legs. Therefore, we investigated the extent to which the CPN reflex is modulated during rhythmic arm cycling. Twenty-eight subjects sat in an armchair and were asked to perform arm cycling at a moderate cadence using a stationary ergometer while performing isometric contraction of the knee extensors, such that the CPN reflex was evoked. The CPN reflex was evoked by stimulating the CPN [0.9-2.0× the motor threshold (MT) in the tibialis anterior muscle] at the level of the neck of the fibula. The CPN-reflex amplitude was measured from the vastus lateralis (VL). The biphasic reflex response in the VL was evoked within 27-45 ms following CPN stimulation. The amplitude of the CPN reflex increased during arm cycling compared with that before cycling. The modulation of the CPN reflex during arm cycling was detected only for CPN stimulation intensity around 1.2× MT. Furthermore, CPN-reflex modulation was not observed during the isometric contraction of the arm or passive arm cycling. Our results suggest the presence of neural coupling between the CPN-reflex pathways and neural systems generating locomotive arm movement.NEW & NOTEWORTHY Whether locomotive arm movements contribute to the control of the reflex pathway from ankle dorsiflexor afferents to knee extensor muscles [common peroneal nerve (CPN)-reflex] is an unresolved issue. The CPN reflex in the stationary leg was facilitated only by arm cycling, and not by passive or isometric motor tasks. Our results suggest that the arm locomotor system modulates the reflex pathway from ankle dorsiflexor afferents to the knee extensor muscles.

Spontaneous Network Coupling Enables Efficient Task Performance without Local Task-Induced Activations.

Neurobehavioral studies in humans have long concentrated on changes in local activity levels during repetitive executions of a task. Spontaneous neural coupling within extended networks has latterly been found to also influence performance. Here, we intend to uncover the underlying mechanisms, the relative importance, and the interaction between spontaneous coupling and task-induced activations. To do so, we recorded two groups of healthy participants (male and female) during rest and while they performed either a visual perception or a motor sequence task. We demonstrate that, for both tasks, stronger activations during the task as well as greater network coupling through spontaneous α rhythms at rest predict performance. However, high performers present an absence of classical task-induced activations and, instead, stronger spontaneous network coupling. Activations were thus a compensation mechanism needed only in subjects with lower spontaneous network interactions. This challenges classical models of neural processing and calls for new strategies in attempts to train and enhance performance.SIGNIFICANCE STATEMENT Our findings challenge the widely accepted notion that task-induced activations are of paramount importance for behavior. This will have an important impact on interpretations of human neurobehavioral research. They further link the widely used techniques of quantifying network communication in the brain with classical neuroscience methods and demonstrate possible ways of how network communication influences human behavior. Traditional training methods attempt to enhance neural activations through task repetitions. Our findings suggest a more efficient neural target for learning: enhancing spontaneous neural interactions. This will be of major interest for a large variety of scientific fields with very broad applications in schools, work, and others.

Development of an Improved Rotational Orthosis for Walking With Arm Swing and Active Ankle Control.

Based on interlimb neural coupling, gait robotic systems should produce walking-like movement in both upper and lower limbs for effective walking restoration. Two orthoses were previously designed in our lab to provide passive walking with arm swing. However, an active system for walking with arm swing is desirable to serve as a testbed for investigation of interlimb neural coupling in response to voluntary input. Given the important function of the ankle joint during normal walking, this work aimed to develop an improved rotational orthosis for walking with arm swing, which is called ROWAS II, and especially to develop and evaluate the algorithms for active ankle control. After description of the mechanical structure and control schemes of the overall ROWAS II system, the closed-loop position control and adjustable admittance control algorithms were firstly deduced, then simulated in Matlab/Simulink and finally implemented in the ROWAS II system. Six able-bodied participants were recruited to use the ROWAS II system in passive mode, and then to estimate the active ankle mechanism. It was showed that the closed-loop position control algorithms enabled the ROWAS II system to track the target arm-leg walking movement patterns well in passive mode, with the tracking error of each joint <0.7°. The adjustable admittance control algorithms enabled the participants to voluntarily adjust the ankle movement by exerting various active force. Higher admittance gains enabled the participants to more easily adjust the movement trajectory of the ankle mechanism. The ROWAS II system is technically feasible to produce walking-like movement in the bilateral upper and lower limbs in passive mode, and the ankle mechanism has technical potential to provide various active ankle training during gait rehabilitation. This novel ROWAS II system can serve as a testbed for further investigation of interlimb neural coupling in response to voluntary ankle movement and is technically feasible to provide a new training paradigm of walking with arm swing and active ankle control.