Biohybrid Robotic Hand to Investigate Tactile Encoding and Sensorimotor Integration.

For people who have experienced a spinal cord injury or an amputation, the recovery of sensation and motor control could be incomplete despite noteworthy advances with invasive neural interfaces. Our objective is to explore the feasibility of a novel biohybrid robotic hand model to investigate aspects of tactile sensation and sensorimotor integration with a pre-clinical research platform. Our new biohybrid model couples an artificial hand with biological neural networks (BNN) cultured in a multichannel microelectrode array (MEA). We decoded neural activity to control a finger of the artificial hand that was outfitted with a tactile sensor. The fingertip sensations were encoded into rapidly adapting (RA) or slowly adapting (SA) mechanoreceptor firing patterns that were used to electrically stimulate the BNN. We classified the coherence between afferent and efferent electrodes in the MEA with a convolutional neural network (CNN) using a transfer learning approach. The BNN exhibited the capacity for functional specialization with the RA and SA patterns, represented by significantly different robotic behavior of the biohybrid hand with respect to the tactile encoding method. Furthermore, the CNN was able to distinguish between RA and SA encoding methods with 97.84% ± 0.65% accuracy when the BNN was provided tactile feedback, averaged across three days in vitro (DIV). This novel biohybrid research platform demonstrates that BNNs are sensitive to tactile encoding methods and can integrate robotic tactile sensations with the motor control of an artificial hand. This opens the possibility of using biohybrid research platforms in the future to study aspects of neural interfaces with minimal human risk.

Human neuromarkers of tactile perception: state of the art in methods and findings.

Tactile perception is a multifaceted sense with complicated convergent/divergent peripheral pathways. Its neuromarkers remain poorly understood, due to the sense's inherent complexity and the confounding factor of intricate motor, cognitive and affective correlates. This gap hinders research evaluating interventions to restore touch in artificial hands. We inventorize state-of-the-art and recent innovations in control systems with soft and hard robotics that are poised to unlock more targeted non-invasive stimulations. We review neuromarkers observed for pressure, vibration, brushing, texture discrimination, pain, heat and cold, complemented with the covariates from movement, attention, working memory, multisensory and sensorimotor integration or competition (audition, vision) and affect. We analyze neural oscillations during sensory and (peripheral and central) electro-magnetic stimulation. This review matures a framework of reverse prediction, in which non-invasive observation of neural activity robustly and unobtrusively quantifies tactile perception.

Multichannel haptic feedback unlocks prosthetic hand dexterity.

Loss of tactile sensations is a major roadblock preventing upper limb-absent people from multitasking or using the full dexterity of their prosthetic hands. With current myoelectric prosthetic hands, limb-absent people can only control one grasp function at a time even though modern artificial hands are mechanically capable of individual control of all five digits. In this paper, we investigated whether people could precisely control the grip forces applied to two different objects grasped simultaneously with a dexterous artificial hand. Toward that end, we developed a novel multichannel wearable soft robotic armband to convey artificial sensations of touch to the robotic hand users. Multiple channels of haptic feedback enabled subjects to successfully grasp and transport two objects simultaneously with the dexterous artificial hand without breaking or dropping them, even when their vision of both objects was obstructed. Simultaneous transport of the objects provided a significant time savings to perform the deliveries in comparison to a one-at-a-time approach. This paper demonstrated that subjects were able to integrate multiple channels of haptic feedback into their motor control strategies to perform a complex simultaneous object grasp control task with an artificial limb, which could serve as a paradigm shift in the way prosthetic hands are operated.

Robotically Embodied Biological Neural Networks to Investigate Haptic Restoration with Neuroprosthetic Hands.

Neuroprosthetic limbs reconnect severed neural pathways for control of (and increasingly sensation from) an artificial limb. However, the plastic interaction between robotic and biological components is poorly understood. To gain such insight, we developed a novel noninvasive neuroprosthetic research platform that enables bidirectional electrical communications (action, sensory perception) between a dexterous artificial hand and neuronal cultures living in a multichannel microelectrode array (MEA) chamber. Artificial tactile sensations from robotic fingertips were encoded to mimic slowly adapting (SA) or rapidly adapting (RA) mechanoreceptors. Afferent spike trains were used to stimulate neurons in a region of the neuronal culture. Electrical activity from neurons at another region in the MEA chamber was used as the motor control signal for the artificial hand. Results from artificial neural networks (ANNs) showed that the haptic model used to encode RA or SA fingertip sensations affected biological neural network (BNN) activity patterns, which in turn impacted the behavior of the artificial hand. That is, the exhibited finger tapping behavior of this closed-loop neurorobotic system showed statistical significance (p<0.01) between the haptic encoding methods across two different neuronal cultures and over multiple days. These findings suggest that our noninvasive neuroprosthetic research platform can be used to devise high-throughput experiments exploring how neural plasticity is affected by the mutual interactions between perception and action.

Third party stabilization of unstable coordination in systems of coupled oscillators.

The Haken-Kelso-Bunz (HKB) system of equations is a well-developed model for dyadic rhythmic coordination in biological systems. It captures ubiquitous empirical observations of bistability - the coexistence of in-phase and antiphase motion - in neural, behavioral, and social coordination. Recent work by Zhang and colleagues has generalized HKB to many oscillators to account for new empirical phenomena observed in multiagent interaction. Utilising this generalization, the present work examines how the coordination dynamics of a pair of oscillators can be augmented by virtue of their coupling to a third oscillator. We show that stable antiphase coordination emerges in pairs of oscillators even when their coupling parameters would have prohibited such coordination in their dyadic relation. We envision two lines of application for this theoretical work. In the social sciences, our model points toward the development of intervention strategies to support coordination behavior in heterogeneous groups (for instance in gerontology, when younger and older individuals interact). In neuroscience, our model will advance our understanding of how the direct functional connection of mesoscale or microscale neural ensembles might be switched by their changing coupling to other neural ensembles. Our findings illuminate a crucial property of complex systems: how the whole is different than the system's parts.

Degeneracy and Complexity in Neuro-Behavioral Correlates of Team Coordination.

Team coordination-members of a group acting together rather than performing specific actions individually-is essential for success in many real-world tasks such as military missions, sports, workplace, or school interactions. However, team coordination is highly variable, which is one reason why its underlying neural processes are largely unknown. Here we used dual electroencephalography (EEG) in dyads to study the neurobehavioral dynamics of team coordination in an ecologically valid task that places intensive demands on joint performance. We present a novel conceptual framework to interpret neurobehavioral variability in terms of degeneracy, a fundamental property of complex biological systems said to enhance flexibility and robustness. We characterize degeneracy conceptually in terms of a manifold representing the geometric locus of the dynamics in the high dimensional state-space of neurobehavioral signals. The geometry and dimensionality of the manifold are determined by task constraints and team coordination requirements which restrict the manifold to trajectories that are conducive to successful task performance. Our results indicate that team coordination is associated with dimensionality reduction of the manifold as evident in increased inter-brain phase coherence of beta and gamma rhythms during critical phases of task performance where subjects exchange information. Team coordination was also found to affect the shape of the manifold manifested as a symmetry breaking of centro-parietal wavelet power patterns across subjects in trials with high team coordination. These results open a conceptual and empirical path to identifying the mechanisms underlying team performance in complex tasks.

Coordination Dynamics: A Foundation for Understanding Social Behavior.

Humans' interactions with each other or with socially competent machines exhibit lawful coordination patterns at multiple levels of description. According to Coordination Dynamics, such laws specify the flow of coordination states produced by functional synergies of elements (e.g., cells, body parts, brain areas, people…) that are temporarily organized as single, coherent units. These coordinative structures or synergies may be mathematically characterized as informationally coupled self-organizing dynamical systems (Coordination Dynamics). In this paper, we start from a simple foundation, an elemental model system for social interactions, whose behavior has been captured in the Haken-Kelso-Bunz (HKB) model. We follow a tried and tested scientific method that tightly interweaves experimental neurobehavioral studies and mathematical models. We use this method to further develop a body of empirical research that advances the theory toward more generalized forms. In concordance with this interdisciplinary spirit, the present paper is written both as an overview of relevant advances and as an introduction to its mathematical underpinnings. We demonstrate HKB's evolution in the context of social coordination along several directions, with its applicability growing to increasingly complex scenarios. In particular, we show that accommodating for symmetry breaking in intrinsic dynamics and coupling, multiscale generalization and adaptation are principal evolutions. We conclude that a general framework for social coordination dynamics is on the horizon, in which models support experiments with hypothesis generation and mechanistic insights.

Topological portraits of multiscale coordination dynamics.

Living systems exhibit complex yet organized behavior on multiple spatiotemporal scales. To investigate the nature of multiscale coordination in living systems, one needs a meaningful and systematic way to quantify the complex dynamics, a challenge in both theoretical and empirical realms. The present work shows how integrating approaches from computational algebraic topology and dynamical systems may help us meet this challenge. In particular, we focus on the application of multiscale topological analysis to coordinated rhythmic processes. First, theoretical arguments are introduced as to why certain topological features and their scale-dependency are highly relevant to understanding complex collective dynamics. Second, we propose a method to capture such dynamically relevant topological information using persistent homology, which allows us to effectively construct a multiscale topological portrait of rhythmic coordination. Finally, the method is put to test in detecting transitions in real data from an experiment of rhythmic coordination in ensembles of interacting humans. The recurrence plots of topological portraits highlight collective transitions in coordination patterns that were elusive to more traditional methods. This sensitivity to collective transitions would be lost if the behavioral dynamics of individuals were treated as separate degrees of freedom instead of constituents of the topology that they collectively forge. Such multiscale topological portraits highlight collective aspects of coordination patterns that are irreducible to properties of individual parts. The present work demonstrates how the analysis of multiscale coordination dynamics can benefit from topological methods, thereby paving the way for further systematic quantification of complex, high-dimensional dynamics in living systems.

Spectral dissociation of lateralized brain rhythms.

Using high resolution spectral methods to uncover neuromarkers of social, cognitive and behavioral function, we have found that hemi-lateralized pairs of oscillations such as left and right occipital alpha or left and right rolandic mu dissociate spectrally. That is, they show a shifted frequency distribution, with one member of the pair peaking at a slightly lower frequency than the other. Resorting to the analysis of EEG spatio-spectral patterns, we provide examples of dissociations in the 10Hz frequency band. Occasionally, hemi-lateralized pairs blend into medial aggregates, probably when functional interactions lead to strongly coherent dynamics through frequency-locking or metastability. Our observations support the hypothesis that homologous pairs of neuromarkers have characteristically distinct intrinsic frequencies and coordinate their oscillations into synchronous ensembles only transiently. This property could play a role in the balance of integration and segregation in the brain: spectral separation of the oscillations from homologous cortical areas allows them to function independently under certain circumstances, all the while preserving a potential for stronger interactions supported by structural and functional symmetries. Spectral dissociation (and its methodological corollary: spectral analysis with high frequency resolution) may be harnessed to better track the individual power of each member of a hemi-lateralized pair and their respective time-course, leading to enhanced internal validity and reproducibility of research on neural oscillations. Resulting insights may shed light on the functional interaction between homologous cortices in studies of attention (alpha), e.g. during perceptual and social interaction tasks, and in studies of somatomotor processing (mu), e.g. in bimanual coordination and neuroprosthetics.

More than Meets the Mind's Eye? Preliminary Observations Hint at Heterogeneous Alpha Neuromarkers for Visual Attention.

With their salient power distribution and privileged timescale for cognition and behavior, brainwaves within the 10 Hz band are special in human waking electroencephalography (EEG). From the inception of electroencephalographic technology, the contribution of alpha rhythm to attention is well-known: Its amplitude increases when visual attention wanes or visual input is removed. However, alpha is not alone in the 10 Hz frequency band. A number of other 10 Hz neuromarkers have function and topography clearly distinct from alpha. In small pilot studies, an activity that we named xi was found over left centroparietal scalp regions when subjects held their attention to spatially peripheral locations while maintaining their gaze centrally ("looking from the corner of the eyes"). I outline several potential functions for xi as a putative neuromarker of covert attention distinct from alpha. I review methodological aids to test and validate their functional role. They emphasize high spectral resolution, sufficient spatial resolution to provide topographical separation, and an acute attention to dynamics that caters to neuromarkers' transiency.

Connecting empirical phenomena and theoretical models of biological coordination across scales.

Coordination in living systems-from cells to people-must be understood at multiple levels of description. Analyses and modelling of empirically observed patterns of biological coordination often focus either on ensemble-level statistics in large-scale systems with many components, or on detailed dynamics in small-scale systems with few components. The two approaches have proceeded largely independent of each other. To bridge this gap between levels and scales, we have recently conducted a human experiment of mid-scale social coordination specifically designed to reveal coordination at multiple levels (ensemble, subgroups and dyads) simultaneously. Based on this experiment, the present work shows that, surprisingly, a single system of equations captures key observations at all relevant levels. It also connects empirically validated models of large- and small-scale biological coordination-the Kuramoto and extended Haken-Kelso-Bunz (HKB) models-and the hallmark phenomena that each is known to capture. For example, it exhibits both multistability and metastability observed in small-scale empirical research (via the second-order coupling and symmetry breaking in extended HKB) and the growth of biological complexity as a function of scale (via the scalability of the Kuramoto model). Only by incorporating both of these features simultaneously can we reproduce the essential coordination behaviour observed in our experiment.

Critical diversity: Divided or united states of social coordination.

Much of our knowledge of coordination comes from studies of simple, dyadic systems or systems containing large numbers of components. The huge gap 'in between' is seldom addressed, empirically or theoretically. We introduce a new paradigm to study the coordination dynamics of such intermediate-sized ensembles with the goal of identifying key mechanisms of interaction. Rhythmic coordination was studied in ensembles of eight people, with differences in movement frequency ('diversity') manipulated within the ensemble. Quantitative change in diversity led to qualitative changes in coordination, a critical value separating régimes of integration and segregation between groups. Metastable and multifrequency coordination between participants enabled communication across segregated groups within the ensemble, without destroying overall order. These novel findings reveal key factors underlying coordination in ensemble sizes previously considered too complicated or 'messy' for systematic study and supply future theoretical/computational models with new empirical checkpoints.

A roadmap to computational social neuroscience.

To complement experimental efforts toward understanding human social interactions at both neural and behavioral levels, two computational approaches are presented: (1) a fully parameterizable mathematical model of a social partner, the Human Dynamic Clamp which, by virtue of experimentally controlled interactions between Virtual Partners and real people, allows for emergent behaviors to be studied; and (2) a multiscale neurocomputational model of social coordination that enables exploration of social self-organization at all levels-from neuronal patterns to people interacting with each other. These complementary frameworks and the cross product of their analysis aim at understanding the fundamental principles governing social behavior.

How Interpersonal Coordination Affects Individual Behavior (and Vice Versa): Experimental analysis and adaptive HKB model of social memory.

How one behaves after interacting with a friend may not be the same as before the interaction. The present study investigated which spontaneous coordination patterns formed between two persons and whether a remnant of the interaction remained ("social memory"). Pairs of people sat face-to-face and continuously flexed index fingers while vision between partners was manipulated to allow or prevent information exchange. Trials consisted of three successive twenty-second intervals: without vision, with vision, and again without vision. Steady, transient, or absent phase coupling was observed during vision. In support of social memory, participants tended to remain near each other's movement frequency after the interaction ended. Furthermore, the greater the stability of interpersonal coordination, the more similar partners' post-interactional frequencies became. Proposing that social memory resulted from prior frequency adaptation, a model based on Haken-Kelso-Bunz oscillators reproduced the experimental findings, even for patterns observed on individual trials. Parametric manipulations revealed multiple routes to social memory through the interplay of adaptation and other model parameters. The experimental results, model, and interpretation motivate potential future research and therapeutic applications.

Armband with Soft Robotic Actuators and Vibrotactile Stimulators for Bimodal Haptic Feedback from a Dexterous Artificial Hand.

The haptic sense relies upon a plurality of receptors and pathways to produce a complex perceptual experience of contact, pressure, taps, vibrations and flutters. This complexity is yet to be reproduced in haptic feedback interfaces that are used by people controlling a dexterous robotic hand, be it for limb-absence or teleoperation. The goal of the present bimodal haptic armband is to convey both low-frequency pressure changes and high-frequency vibrations from a dexterous robotic hand to a human's upper arm, so as to guide his/her control of the artificial limb. To that end, we design and manufacture four novel soft robotic armbands combining inflatable air chambers and vibrotactile stimulators. We develop control systems for both pathways. We conduct a series of benchtop tests to determine the pneumatic and vibrotactile performance and select from competing designs and materials. We test two of the resulting bimodal haptic armband on human subjects and confirm their ability to use both aspects of this haptic information. Arguing that dexterous artificial hands are presently not used to their fullest capability by the dearth of haptic information in users, this work aims to achieve a more realistic tactile experience for a fluent, more natural usage of robotic artificial hands.

Enhanced emotional responses during social coordination with a virtual partner.

Emotion and motion, though seldom studied in tandem, are complementary aspects of social experience. This study investigates variations in emotional responses during movement coordination between a human and a Virtual Partner (VP), an agent whose virtual finger movements are driven by the Haken-Kelso-Bunz (HKB) equations of Coordination Dynamics. Twenty-one subjects were instructed to coordinate finger movements with the VP in either inphase or antiphase patterns. By adjusting model parameters, we manipulated the 'intention' of VP as cooperative or competitive with the human's instructed goal. Skin potential responses (SPR) were recorded to quantify the intensity of emotional response. At the end of each trial, subjects rated the VP's intention and whether they thought their partner was another human being or a machine. We found greater emotional responses when subjects reported that their partner was human and when coordination was stable. That emotional responses are strongly influenced by dynamic features of the VP's behavior, has implications for mental health, brain disorders and the design of socially cooperative machines.

The coordination dynamics of social neuromarkers.

Social behavior is a complex integrative function that entails many aspects of the brain's sensory, cognitive, emotional and movement capacities. Its neural processes are seldom simultaneous but occur according to precise spatiotemporal choreographies, manifested by the coordination of their oscillations within and between brains. Methods with good temporal resolution can help to identify so-called "neuromarkers" of social function and aid in disentangling the dynamical architecture of social brains. In our ongoing research, we have used dual-electroencephalography (EEG) to study neuromarker dynamics during synchronic interactions in which pairs of subjects coordinate behavior spontaneously and intentionally (social coordination) and during diachronic transactions that require subjects to perceive or behave in turn (action observation, delayed imitation). In this paper, after outlining our dynamical approach to the neurophysiological basis of social behavior, we examine commonalities and differences in the neuromarkers that are recruited for both kinds of tasks. We find the neuromarker landscape to be task-specific: synchronic paradigms of social coordination reveal medial mu, alpha and the phi complex as contributing neuromarkers. Diachronic tasks recruit alpha as well, in addition to lateral mu rhythms and the newly discovered nu and kappa rhythms whose functional significance is still unclear. Social coordination, observation, and delayed imitation share commonality of context: in each of our experiments, subjects exchanged information through visual perception and moved in similar ways. Nonetheless, there was little overlap between their neuromarkers, a result that hints strongly of task-specific neural mechanisms for social behavior. The only neuromarker that transcended both synchronic and diachronic social behaviors was the ubiquitous alpha rhythm, which appears to be a key signature of visually-mediated social behaviors. The present paper is both an entry point and a challenge: much work remains to determine the nature and scope of recruitment of other neuromarkers, and to create theoretical models of their within- and between-brain dynamics during social interaction.

The human dynamic clamp as a paradigm for social interaction.

Social neuroscience has called for new experimental paradigms aimed toward real-time interactions. A distinctive feature of interactions is mutual information exchange: One member of a pair changes in response to the other while simultaneously producing actions that alter the other. Combining mathematical and neurophysiological methods, we introduce a paradigm called the human dynamic clamp (HDC), to directly manipulate the interaction or coupling between a human and a surrogate constructed to behave like a human. Inspired by the dynamic clamp used so productively in cellular neuroscience, the HDC allows a person to interact in real time with a virtual partner itself driven by well-established models of coordination dynamics. People coordinate hand movements with the visually observed movements of a virtual hand, the parameters of which depend on input from the subject's own movements. We demonstrate that HDC can be extended to cover a broad repertoire of human behavior, including rhythmic and discrete movements, adaptation to changes of pacing, and behavioral skill learning as specified by a virtual "teacher." We propose HDC as a general paradigm, best implemented when empirically verified theoretical or mathematical models have been developed in a particular scientific field. The HDC paradigm is powerful because it provides an opportunity to explore parameter ranges and perturbations that are not easily accessible in ordinary human interactions. The HDC not only enables to test the veracity of theoretical models, it also illuminates features that are not always apparent in real-time human social interactions and the brain correlates thereof.

Enlarging the scope: grasping brain complexity.

To further advance our understanding of the brain, new concepts and theories are needed. In particular, the ability of the brain to create information flows must be reconciled with its propensity for synchronization and mass action. The theoretical and empirical framework of Coordination Dynamics, a key aspect of which is metastability, are presented as a starting point to study the interplay of integrative and segregative tendencies that are expressed in space and time during the normal course of brain and behavioral function. Some recent shifts in perspective are emphasized, that may ultimately lead to a better understanding of brain complexity.

The metastable brain.

Neural ensembles oscillate across a broad range of frequencies and are transiently coupled or "bound" together when people attend to a stimulus, perceive, think, and act. This is a dynamic, self-assembling process, with parts of the brain engaging and disengaging in time. But how is it done? The theory of Coordination Dynamics proposes a mechanism called metastability, a subtle blend of integration and segregation. Tendencies for brain regions to express their individual autonomy and specialized functions (segregation, modularity) coexist with tendencies to couple and coordinate globally for multiple functions (integration). Although metastability has garnered increasing attention, it has yet to be demonstrated and treated within a fully spatiotemporal perspective. Here, we illustrate metastability in continuous neural and behavioral recordings, and we discuss theory and experiments at multiple scales, suggesting that metastable dynamics underlie the real-time coordination necessary for the brain's dynamic cognitive, behavioral, and social functions.