User Training With Error Augmentation for sEMG-Based Gesture Classification.

We designed and tested a system for real-time control of a user interface by extracting surface electromyographic (sEMG) activity from eight electrodes in a wristband configuration. sEMG data were streamed into a machine-learning algorithm that classified hand gestures in real-time. After an initial model calibration, participants were presented with one of three types of feedback during a human-learning stage: veridical feedback, in which predicted probabilities from the gesture classification algorithm were displayed without alteration; modified feedback, in which we applied a hidden augmentation of error to these probabilities; and no feedback. User performance was then evaluated in a series of minigames, in which subjects were required to use eight gestures to manipulate their game avatar to complete a task. Experimental results indicated that relative to the baseline, the modified feedback condition led to significantly improved accuracy. Class separation also improved, though this trend was not significant. These findings suggest that real-time feedback in a gamified user interface with manipulation of feedback may enable intuitive, rapid, and accurate task acquisition for sEMG-based gesture recognition applications.

Multimodal fusion of EMG and vision for human grasp intent inference in prosthetic hand control.

Objective: For transradial amputees, robotic prosthetic hands promise to regain the capability to perform daily living activities. Current control methods based on physiological signals such as electromyography (EMG) are prone to yielding poor inference outcomes due to motion artifacts, muscle fatigue, and many more. Vision sensors are a major source of information about the environment state and can play a vital role in inferring feasible and intended gestures. However, visual evidence is also susceptible to its own artifacts, most often due to object occlusion, lighting changes, etc. Multimodal evidence fusion using physiological and vision sensor measurements is a natural approach due to the complementary strengths of these modalities. Methods: In this paper, we present a Bayesian evidence fusion framework for grasp intent inference using eye-view video, eye-gaze, and EMG from the forearm processed by neural network models. We analyze individual and fused performance as a function of time as the hand approaches the object to grasp it. For this purpose, we have also developed novel data processing and augmentation techniques to train neural network components. Results: Our results indicate that, on average, fusion improves the instantaneous upcoming grasp type classification accuracy while in the reaching phase by 13.66% and 14.8%, relative to EMG (81.64% non-fused) and visual evidence (80.5% non-fused) individually, resulting in an overall fusion accuracy of 95.3%. Conclusion: Our experimental data analyses demonstrate that EMG and visual evidence show complementary strengths, and as a consequence, fusion of multimodal evidence can outperform each individual evidence modality at any given time.

M2M-InvNet: Human Motor Cortex Mapping From Multi-Muscle Response Using TMS and Generative 3D Convolutional Network.

Transcranial magnetic stimulation (TMS) is often applied to the motor cortex to stimulate a collection of motor evoked potentials (MEPs) in groups of peripheral muscles. The causal interface between TMS and MEP is the selective activation of neurons in the motor cortex; moving around the TMS 'spot' over the motor cortex causes different MEP responses. A question of interest is whether a collection of MEP responses can be used to identify the stimulated locations on the cortex, which could potentially be used to then place the TMS coil to produce chosen sets of MEPs. In this work we leverage our previous report on a 3D convolutional neural network (CNN) architecture that predicted MEPs from the induced electric field, to tackle an inverse imaging task in which we start with the MEPs and estimate the stimulated regions on the motor cortex. We present and evaluate five different inverse imaging CNN architectures, both conventional and generative, in terms of several measures of reconstruction accuracy. We found that one architecture, which we propose as M2M-InvNet, consistently achieved the best performance.

Neurorehabilitation robotics: how much control should therapists have?

Robotic technologies for rehabilitating motor impairments from neurological injuries have been the focus of intensive research and capital investment for more than 30 years. However, these devices have failed to convincingly demonstrate greater restoration of patient function compared to conventional therapy. Nevertheless, robots have value in reducing the manual effort required for physical therapists to provide high-intensity, high-dose interventions. In most robotic systems, therapists remain outside the control loop to act as high-level supervisors, selecting and initiating robot control algorithms to achieve a therapeutic goal. The low-level physical interactions between the robot and the patient are handled by adaptive algorithms that can provide progressive therapy. In this perspective, we examine the physical therapist's role in the control of rehabilitation robotics and whether embedding therapists in lower-level robot control loops could enhance rehabilitation outcomes. We discuss how the features of many automated robotic systems, which can provide repeatable patterns of physical interaction, may work against the goal of driving neuroplastic changes that promote retention and generalization of sensorimotor learning in patients. We highlight the benefits and limitations of letting therapists physically interact with patients through online control of robotic rehabilitation systems, and explore the concept of trust in human-robot interaction as it applies to patient-robot-therapist relationships. We conclude by highlighting several open questions to guide the future of therapist-in-the-loop rehabilitation robotics, including how much control to give therapists and possible approaches for having the robotic system learn from therapist-patient interactions.

Investigating and acquiring motor expertise using virtual reality.

After just months of simulated training, on January 19, 2019 a 23-year-old E-sports pro-gamer, Enzo Bonito, took to the racetrack and beat Lucas di Grassi, a Formula E and ex-Formula 1 driver with decades of real-world racing experience. This event raised the possibility that practicing in virtual reality can be surprisingly effective for acquiring motor expertise in real-world tasks. Here, we evaluate the potential of virtual reality to serve as a space for training to expert levels in highly complex real-world tasks in time windows much shorter than those required in the real world and at much lower financial cost without the hazards of the real world. We also discuss how VR can also serve as an experimental platform for exploring the science of expertise more generally.

Correspondence of functional connectivity gradients across human isocortex, cerebellum, and hippocampus.

Gradient mapping is an important technique to summarize high dimensional biological features as low dimensional manifold representations in exploring brain structure-function relationships at various levels of the cerebral cortex. While recent studies have characterized the major gradients of functional connectivity in several brain structures using this technique, very few have systematically examined the correspondence of such gradients across structures under a common systems-level framework. Using resting-state functional magnetic resonance imaging, here we show that the organizing principles of the isocortex, and those of the cerebellum and hippocampus in relation to the isocortex, can be described using two common functional gradients. We suggest that the similarity in functional connectivity gradients across these structures can be meaningfully interpreted within a common computational framework based on the principles of predictive processing. The present results, and the specific hypotheses that they suggest, represent an important step toward an integrative account of brain function.

Evaluation of Changes in Kinematic Measures of Three Dimensional Reach to Grasp Movements in the Early Subacute Period of Recovery from Stroke.

This study examines longitudinal data of subjects initially examined in the early subacute period of recovery following a stroke with a test of reach to grasp (RTG) kinematics in an attempt to identify changes in movement patterns during the period of heightened neural recovery following a stroke. Subjects (n=8) were a convenience sample of persons with stroke that participated in an intervention trial. Baseline Upper Extremity Fugl Meyer Assessment (UEFMA) scores ranged between 31 and 52 and ages were between 49 and 83. The UEFMA and RTG test were collected prior to intervention, immediately after the intervention (approximately 18 days later post baseline) and one month after the intervention. RTG data for the uninvolved UE was collected at the one-month session. Subjects reached for objects placed on a table 10 cm from their sternums, picking them up and placing them on a target 30 cm from their acromioclavicular joints. Data was collected using an optical motion capture system. Active makers were placed on each fingertip, metacarpophalangeal, and proximal interphalangeal joint. Four additional passive markers were placed on the dorsum of the hand, the elbow, the shoulder, and the sternum. Subjects demonstrated statistically significant improvements in reaching duration, reaching trajectory smoothness, time after peak velocity and peak grip aperture. All of these measures correlated significantly with improvements in UEFMA. Clinical Relevance- Kinematic measures of reaching and grasping collected early in the subacute period of recovery from stroke may offer insight into specific aspects of the recovery of upper extremity motor function that differ from the information gleaned from clinical scales.

Similarity of hand muscle synergies elicited by transcranial magnetic stimulation and those found during voluntary movement.

Converging evidence in human and animal models suggests that exogenous stimulation of the motor cortex (M1) elicits responses in the hand with similar modular structure to that found during voluntary grasping movements. The aim of this study was to establish the extent to which modularity in muscle responses to transcranial magnetic stimulation (TMS) to M1 resembles modularity in muscle activation during voluntary hand movements involving finger fractionation. Electromyography (EMG) was recorded from eight hand-forearm muscles in eight healthy individuals. Modularity was defined using non-negative matrix factorization to identify low-rank approximations (spatial muscle synergies) of the complex activation patterns of EMG data recorded during high-density TMS mapping of M1 and voluntary formation of gestures in the American Sign Language alphabet. Analysis of synergies revealed greater than chance similarity between those derived from TMS and those derived from voluntary movement. Both data sets included synergies dominated by single intrinsic hand muscles presumably to meet the demand for highly fractionated finger movement. These results suggest that corticospinal connectivity to individual intrinsic hand muscles may be combined with modular multimuscle activation via synergies in the formation of hand postures.NEW & NOTEWORTHY This is the first work to examine the similarity of modularity in hand muscle responses to transcranial magnetic stimulation (TMS) of the motor cortex and that derived from voluntary hand movement. We show that TMS-elicited muscle synergies of the hand, measured at rest, reflect those found in voluntary behavior involving finger fractionation. This work provides a basis for future work using TMS to investigate muscle activation modularity in the human motor system.

A kinematic and EMG dataset of online adjustment of reach-to-grasp movements to visual perturbations.

Control of reach-to-grasp movements for deft and robust interactions with objects requires rapid sensorimotor updating that enables online adjustments to changing external goals (e.g., perturbations or instability of objects we interact with). Rarely do we appreciate the remarkable coordination in reach-to-grasp, until control becomes impaired by neurological injuries such as stroke, neurodegenerative diseases, or even aging. Modeling online control of human reach-to-grasp movements is a challenging problem but fundamental to several domains, including behavioral and computational neuroscience, neurorehabilitation, neural prostheses, and robotics. Currently, there are no publicly available datasets that include online adjustment of reach-to-grasp movements to object perturbations. This work aims to advance modeling efforts of reach-to-grasp movements by making publicly available a large kinematic and EMG dataset of online adjustment of reach-to-grasp movements to instantaneous perturbations of object size and distance performed in immersive haptic-free virtual environment (hf-VE). The presented dataset is composed of a large number of perturbation types (10 for both object size and distance) applied at three different latencies after the start of the movement.

Classifications of Dynamic EMG in Hand Gesture and Unsupervised Grasp Motion Segmentation.

The electromyography (EMG) signals have been widely utilized in human-robot interaction for extracting user hand/arm motion instructions. A major challenge of the online interaction with robots is the reliable EMG recognition from real-time data. However, previous studies mainly focused on using steady-state EMG signals with a small number of grasp patterns to implement classification algorithms, which is insufficient to generate robust control regarding the dynamic muscular activity variation in practice. Introducing more EMG variability during training and validation could implement a better dynamic-motion detection, but only limited research focused on such grasp-movement identification, and all of those assessments on the non-static EMG classification require supervised ground-truth label of the movement status. In this study, we propose a framework for classifying EMG signals generated from continuous grasp movements with variations on dynamic arm/hand postures, using an unsupervised motion status segmentation method. We collected data from large gesture vocabularies with multiple dynamic motion phases to encode the transitions from one intent to another based on common sequences of the grasp movements. Two classifiers were constructed for identifying the motion-phase label and grasptype label, where the dynamic motion phases were segmented and labeled in an unsupervised manner. The proposed framework was evaluated in real-time with the accuracy variation over time presented, which was shown to be efficient due to the high degree of freedom of the EMG data.

Investigating a classical neuropsychological test in a real world context.

This study was performed to investigate the validity of a real world version of the Trail Making Test (TMT) across age strata, compared to the current standard TMT which is delivered using a pen-paper protocol. We developed a real world version of the TMT, the Can-TMT, that involves the retrieval of food cans, with numeric or alphanumerical labels, from a shelf in ascending order. Eye tracking data was acquired during the Can-TMT to calculate task completion time and compared to that of the Paper-TMT. Results indicated a strong significant correlation between the real world and paper tasks for both TMTA and TMTB versions of the tasks, indicative of the validity of the real world task. Moreover, the two age groups exhibited significant differences on the TMTA and TMTB versions of both task modalities (paper and can), further supporting the validity of the real world task. This work will have a significant impact on our ability to infer skill or impairment with visual search, spatial reasoning, working memory, and motor proficiency during complex real-world tasks. Thus, we hope to fill a critical need for an exam with the resolution capable of determining deficits which subjective or reductionist assessments may otherwise miss.

Efficient TMS-Based Motor Cortex Mapping Using Gaussian Process Active Learning.

Transcranial Magnetic Stimulation (TMS) can be used to map cortical motor topography by spatially sampling the sensorimotor cortex while recording Motor Evoked Potentials (MEP) with surface electromyography (EMG). Traditional sampling strategies are time-consuming and inefficient, as they ignore the fact that responsive sites are typically sparse and highly spatially correlated. An alternative approach, commonly employed when TMS mapping is used for presurgical planning, is to leverage the expertise of the coil operator to use MEPs elicited by previous stimuli as feedback to decide which loci to stimulate next. In this paper, we propose to automatically infer optimal future stimulus loci using active learning Gaussian Process-based sampling in place of user expertise. We first compare the user-guided (USRG) method to the traditional grid selection method and randomized sampling to verify that the USRG approach has superior performance. We then compare several novel active Gaussian Process (GP) strategies with the USRG approach. Experimental results using real data show that, as expected, the USRG method is superior to the grid and random approach in both time efficiency and MEP map accuracy. We also found that an active warped GP entropy and a GP random-based strategy performed equally as well as, or even better than, the USRG method. These methods were completely automatic, and succeeded in efficiently sampling the regions in which the MEP response variations are largely confined. This work provides the foundation for highly efficient, fully automatized TMS mapping, especially when considered in the context of advances in robotic coil operation.

Control of aperture closure during reach-to-grasp movements in immersive haptic-free virtual reality.

Virtual reality (VR) has garnered much interest as a training environment for motor skill acquisition, including for neurological rehabilitation of upper extremities. While the focus has been on gross upper limb motion, VR applications that involve reaching for, and interacting with, virtual objects are growing. The absence of true haptics in VR when it comes to hand-object interactions raises a fundamentally important question: can haptic-free immersive virtual environments (hf-VEs) support naturalistic coordination of reach-to-grasp movements? This issue has been grossly understudied, and yet is of significant importance in the development and application of VR across a number of sectors. In a previous study (Furmanek et al., J Neuroeng Rehabil 16:78, 2019), we reported that reach-to-grasp movements are similarly coordinated in both the physical environment (PE) and hf-VE. The most noteworthy difference was that the closure phase-which begins at maximum aperture and lasts through the end of the movement-was longer in hf-VE than in PE, suggesting that different control laws might govern the initiation of closure between the two environments. To do so, we reanalyzed data from Furmanek et al. (J Neuroeng Rehabil 16:78, 2019), in which the participants reached to grasp three differently sized physical objects, and matching 3D virtual object renderings, placed at three different locations. Our analysis revealed two key findings pertaining to the initiation of closure in PE and hf-VE. First, the respective control laws governing the initiation of aperture closure in PE and hf-VE both included state estimates of transport velocity and acceleration, supporting a general unified control policy for implementing reach-to-grasp across physical and virtual environments. Second, the aperture was less informative to the control law in hf-VE. We suggest that the latter was likely because transport velocity at closure onset and aperture at closure onset were less independent in hf-VE than in PE, ultimately resulting in an aperture at closure onset having a weaker influence on the initiation of closure. In this way, the excess time and muscular effort needed to actively bring the fingers to a stop at the interface of a virtual object was factored into the control law governing the initiation of closure in hf-VE. Critically, this control law remained applicable, albeit with different weights in hf-VE, despite the absence of terminal haptic feedback and potential perceptual differences.

Virtual Rehabilitation of the Paretic Hand and Arm in Persons With Stroke: Translation From Laboratory to Rehabilitation Centers and the Patient's Home.

The anatomical and physiological heterogeneity of strokes and persons with stroke, along with the complexity of normal upper extremity movement make the possibility that any single treatment approach will become the definitive solution for all persons with upper extremity hemiparesis due to stroke unlikely. This situation and the non-inferiority level outcomes identified by many studies of virtual rehabilitation are considered by some to indicate that it is time to consider other treatment modalities. Our group, among others, has endeavored to build on the initial positive outcomes in studies of virtual rehabilitation by identifying patient populations, treatment settings and training schedules that will best leverage virtual rehabilitation's strengths. We feel that data generated by our lab and others suggest that (1) persons with stroke may adapt to virtual rehabilitation of hand function differently based on their level of impairment and stage of recovery and (2) that less expensive, more accessible home based equipment seems to be an effective alternative to clinic based treatment that justifies continued optimism and study.

Synergistic Activation Patterns of Hand Muscles in Left-and Right-Hand Dominant Individuals.

Handedness has been associated with behavioral asymmetries between limbs that suggest specialized function of dominant and non-dominant hand. Whether patterns of muscle co-activation, representing muscle synergies, also differ between the limbs remains an open question. Previous investigations of proximal upper limb muscle synergies have reported little evidence of limb asymmetry; however, whether the same is true of the distal upper limb and hand remains unknown. This study compared forearm and hand muscle synergies between the dominant and non-dominant limb of left-handed and right-handed participants. Participants formed their hands into the postures of the American Sign Language (ASL) alphabet, while EMG was recorded from hand and forearm muscles. Muscle synergies were extracted for each limb individually by applying non-negative-matrix-factorization (NMF). Extracted synergies were compared between limbs for each individual, and between individuals to assess within and across participant differences. Results indicate no difference between the limbs for individuals, but differences in limb synergies at the population level. Left limb synergies were found to be more similar than right limb synergies across left- and right-handed individuals. Synergies of the left hand of left dominant individuals were found to have greater population level similarity than the other limbs tested. Results are interpreted with respect to known differences in the neuroanatomy and neurophysiology of proximal and distal upper limb motor control. Implications for skill training in sports requiring dexterous control of the hand are discussed.

Hand Focused Upper Extremity Rehabilitation in the Subacute Phase Post-stroke Using Interactive Virtual Environments.

Introduction: Innovative motor therapies have attempted to reduce upper extremity impairment after stroke but have not made substantial improvement as over 50% of people post-stroke continue to have sensorimotor deficits affecting their self-care and participation in daily activities. Intervention studies have focused on the role of increased dosing, however recent studies have indicated that timing of rehabilitation interventions may be as important as dosing and importantly, that dosing and timing interact in mediating effectiveness. This study is designed to empirically test dosing and timing. Methods and Analysis: In this single-blinded, interventional study, subjects will be stratified on two dimensions, impairment level (Fugl-Meyer Upper Extremity Assessment (FM) and presence or absence of Motor Evoked Potentials (MEPs) as follows; (1) Severe, FM score 10-19, MEP+, (2) Severe, FM score 10-19, MEP-, (3) Moderate, FM score 20-49, MEP+, (4) Moderate, FM score 20-49, MEP-. Subjects not eligible for TMS will be assigned to either group 2 (if severe) or group 3 (if moderate). Stratified block randomization will then be used to achieve a balanced assignment. Early Robotic/VR Therapy (EVR) experimental group will receive in-patient usual care therapy plus an extra 10 h of intensive upper extremity therapy focusing on the hand using robotically facilitated rehabilitation interventions presented in virtual environments and initiated 5-30 days post-stroke. Delayed Robotic/VR Therapy (DVR) experimental group will receive the same intervention but initiated 30-60 days post-stroke. Dose-matched usual care group (DMUC) will receive an extra 10 h of usual care initiated 5-30 days post-stroke. Usual Care Group (UC) will receive the usual amount of physical/occupational therapy. Outcomes: There are clinical, neurophysiological, and kinematic/kinetic measures, plus measures of daily arm use and quality of life. Primary outcome is the Action Research Arm Test (ARAT) measured at 4 months post-stroke. Discussion: Outcome measures will be assessed to determine whether there is an early time period in which rehabilitation will be most effective, and whether there is a difference in the recapture of premorbid patterns of movement vs. the development of an efficient, but compensatory movement strategy. Ethical Considerations: The IRBs of New Jersey Institute of Technology, Rutgers University, Northeastern University, and Kessler Foundation reviewed and approved all study protocols. Study was registered in https://ClinicalTrials.gov (NCT03569059) prior to recruitment. Dissemination will include submission to peer-reviewed journals and professional presentations.

A Muscle Synergy Framework for Cross-Limb Reconstruction of Hand Muscle Activity Distal to a Virtual Wrist-Level Disarticulation.

Currently, myoelectric prostheses lack dexterity and ease of control, in part because of inadequate schemes to extract relevant muscle features that can approximate muscle activation patterns that enable individuated dexterous finger motion. This project seeks to apply a novel algorithm pipeline that extracts muscle activation patterns from one limb, as well as from forearm muscles of the opposite limb, to predict muscle activation data of opposite limb intrinsic hand muscles, with the long-range goal of informing dexterous prosthetic control.

Mapping Motor Cortex Stimulation to Muscle Responses: A Deep Neural Network Modeling Approach.

A deep neural network (DNN) that can reliably model muscle responses from corresponding brain stimulation has the potential to increase knowledge of coordinated motor control for numerous basic science and applied use cases. Such cases include the understanding of abnormal movement patterns due to neurological injury from stroke, and stimulation based interventions for neurological recovery such as paired associative stimulation. In this work, potential DNN models are explored and the one with the minimum squared errors is recommended for the optimal performance of the M2M-Net, a network that maps transcranial magnetic stimulation of the motor cortex to corresponding muscle responses, using: a finite element simulation, an empirical neural response profile, a convolutional autoencoder, a separate deep network mapper, and recordings of multi-muscle activation. We discuss the rationale behind the different modeling approaches and architectures, and contrast their results. Additionally, to obtain a comparative insight of the trade-o between complexity and performance analysis, we explore different techniques, including the extension of two classical information criteria for M2M-Net. Finally, we find that the model analogous to mapping the motor cortex stimulation to a combination of direct and synergistic connection to the muscles performs the best, when the neural response profile is used at the input.

Motor Cortex Mapping using Active Gaussian Processes.

One important application of transcranial magnetic stimulation (TMS) is to map cortical motor topography by spatially sampling the motor cortex, and recording motor evoked potentials (MEP) with surface electromyography. Standard approaches to TMS mapping involve repetitive stimulations at different loci spaced on a (typically 1 cm) grid on the scalp. These mappings strategies are time consuming and responsive sites are typically sparse. Furthermore, the long time scale prevents measurement of transient cortical changes, and is poorly tolerated in clinical populations. An alternative approach involves using the TMS mapper expertise to exploit the map's sparsity through the use of feedback of MEPs to decide which loci to stimulate. In this investigation, we propose a novel active learning method to automatically infer optimal future stimulus loci in place of user expertise. Specifically, we propose an active Gaussian Process (GP) strategy with loci selection criteria such as entropy and mutual information (MI). The proposed method twists the usual entropy- and MI-based selection criteria by modeling the estimated MEP field, i.e., the GP mean, as a Gaussian random variable itself. By doing so, we include MEP amplitudes in the loci selection criteria which would be otherwise completely independent of the MEP values. Experimental results using real data shows that the proposed strategy can greatly outperform competing methods when the MEP variations are mostly conned in a sub-region of the space.