An Exploratory Multi-Session Study of Learning High-Dimensional Body-Machine Interfacing for Assistive Robot Control.

Individuals who suffer from severe paralysis often lose the capacity to perform fundamental body movements and everyday activities. Empowering these individuals with the ability to operate robotic arms, in high degrees-of-freedom (DoFs), can help to maximize both functional utility and independence. However, robot teleoperation in high DoFs currently lacks accessibility due to the challenge in capturing high-dimensional control signals from the human, especially in the face of motor impairments. Body-machine interfacing is a viable option that offers the necessary high-dimensional motion capture, and it moreover is noninvasive, affordable, and promotes movement and motor recovery. Nevertheless, to what extent body-machine interfacing is able to scale to high-DoF robot control, and whether it is feasible for humans to learn, remains an open question. In this exploratory multi-session study, we demonstrate the feasibility of human learning to operate a body-machine interface to control a complex, assistive robotic arm. We use a sensor net of four inertial measurement unit sensors, bilaterally placed on the scapulae and humeri. Ten uninjured participants are familiarized, trained, and evaluated in reaching and Activities of Daily Living tasks, using the body- machine interface. Our results suggest the manner of control space mapping (joint-space control versus task-space control), from interface to robot, plays a critical role in the evolution of human learning. Though joint-space control shows to be more intuitive initially, task-space control is found to have a greater capacity for longer-term improvement and learning.

Autoencoder-based myoelectric controller for prosthetic hands.

In the past, linear dimensionality-reduction techniques, such as Principal Component Analysis, have been used to simplify the myoelectric control of high-dimensional prosthetic hands. Nonetheless, their nonlinear counterparts, such as Autoencoders, have been shown to be more effective at compressing and reconstructing complex hand kinematics data. As a result, they have a potential of being a more accurate tool for prosthetic hand control. Here, we present a novel Autoencoder-based controller, in which the user is able to control a high-dimensional (17D) virtual hand via a low-dimensional (2D) space. We assess the efficacy of the controller via a validation experiment with four unimpaired participants. All the participants were able to significantly decrease the time it took for them to match a target gesture with a virtual hand to an average of 6.9s and three out of four participants significantly improved path efficiency. Our results suggest that the Autoencoder-based controller has the potential to be used to manipulate high-dimensional hand systems via a myoelectric interface with a higher accuracy than PCA; however, more exploration needs to be done on the most effective ways of learning such a controller.

Learning to operate a high-dimensional hand via a low-dimensional controller.

Dimensionality reduction techniques have proven useful in simplifying complex hand kinematics. They may allow for a low-dimensional kinematic or myoelectric interface to be used to control a high-dimensional hand. Controlling a high-dimensional hand, however, is difficult to learn since the relationship between the low-dimensional controls and the high-dimensional system can be hard to perceive. In this manuscript, we explore how training practices that make this relationship more explicit can aid learning. We outline three studies that explore different factors which affect learning of an autoencoder-based controller, in which a user is able to operate a high-dimensional virtual hand via a low-dimensional control space. We compare computer mouse and myoelectric control as one factor contributing to learning difficulty. We also compare training paradigms in which the dimensionality of the training task matched or did not match the true dimensionality of the low-dimensional controller (both 2D). The training paradigms were a) a full-dimensional task, in which the user was unaware of the underlying controller dimensionality, b) an implicit 2D training, which allowed the user to practice on a simple 2D reaching task before attempting the full-dimensional one, without establishing an explicit connection between the two, and c) an explicit 2D training, during which the user was able to observe the relationship between their 2D movements and the higher-dimensional hand. We found that operating a myoelectric interface did not pose a big challenge to learning the low-dimensional controller and was not the main reason for the poor performance. Implicit 2D training was found to be as good, but not better, as training directly on the high-dimensional hand. What truly aided the user's ability to learn the controller was the 2D training that established an explicit connection between the low-dimensional control space and the high-dimensional hand movements.

A Non-Linear Body Machine Interface for Controlling Assistive Robotic Arms.

OBJECTIVE: Body machine interfaces (BoMIs) enable individuals with paralysis to achieve a greater measure of independence in daily activities by assisting the control of devices such as robotic manipulators. The first BoMIs relied on Principal Component Analysis (PCA) to extract a lower dimensional control space from information in voluntary movement signals. Despite its widespread use, PCA might not be suited for controlling devices with a large number of degrees of freedom, as because of PCs' orthonormality the variance explained by successive components drops sharply after the first. METHODS: Here, we propose an alternative BoMI based on non-linear autoencoder (AE) networks that mapped arm kinematic signals into joint angles of a 4D virtual robotic manipulator. First, we performed a validation procedure that aimed at selecting an AE structure that would allow to distribute the input variance uniformly across the dimensions of the control space. Then, we assessed the users' proficiency practicing a 3D reaching task by operating the robot with the validated AE. RESULTS: All participants managed to acquire an adequate level of skill when operating the 4D robot. Moreover, they retained the performance across two non-consecutive days of training. CONCLUSION: While providing users with a fully continuous control of the robot, the entirely unsupervised nature of our approach makes it ideal for applications in a clinical context since it can be tailored to each user's residual movements. SIGNIFICANCE: We consider these findings as supporting a future implementation of our interface as an assistive tool for people with motor impairments.

Recovery of Distal Arm Movements in Spinal Cord Injured Patients with a Body-Machine Interface: A Proof-of-Concept Study.

BACKGROUND: The recovery of upper limb mobility and functions is essential for people with cervical spinal cord injuries (cSCI) to maximize independence in daily activities and ensure a successful return to normality. The rehabilitative path should include a thorough neuromotor evaluation and personalized treatments aimed at recovering motor functions. Body-machine interfaces (BoMI) have been proven to be capable of harnessing residual joint motions to control objects like computer cursors and virtual or physical wheelchairs and to promote motor recovery. However, their therapeutic application has still been limited to shoulder movements. Here, we expanded the use of BoMI to promote the whole arm's mobility, with a special focus on elbow movements. We also developed an instrumented evaluation test and a set of kinematic indicators for assessing residual abilities and recovery. METHODS: Five inpatient cSCI subjects (four acute, one chronic) participated in a BoMI treatment complementary to their standard rehabilitative routine. The subjects wore a BoMI with sensors placed on both proximal and distal arm districts and practiced for 5 weeks. The BoMI was programmed to promote symmetry between right and left arms use and the forearms' mobility while playing games. To evaluate the effectiveness of the treatment, the subjects' kinematics were recorded while performing an evaluation test that involved functional bilateral arms movements, before, at the end, and three months after training. RESULTS: At the end of the training, all subjects learned to efficiently use the interface despite being compelled by it to engage their most impaired movements. The subjects completed the training with bilateral symmetry in body recruitment, already present at the end of the familiarization, and they increased the forearm activity. The instrumental evaluation confirmed this. The elbow motion's angular amplitude improved for all subjects, and other kinematic parameters showed a trend towards the normality range. CONCLUSION: The outcomes are preliminary evidence supporting the efficacy of the proposed BoMI as a rehabilitation tool to be considered for clinical practice. It also suggests an instrumental evaluation protocol and a set of indicators to assess and evaluate motor impairment and recovery in cSCI.

Building an adaptive interface via unsupervised tracking of latent manifolds.

In human-machine interfaces, decoder calibration is critical to enable an effective and seamless interaction with the machine. However, recalibration is often necessary as the decoder off-line predictive power does not generally imply ease-of-use, due to closed loop dynamics and user adaptation that cannot be accounted for during the calibration procedure. Here, we propose an adaptive interface that makes use of a non-linear autoencoder trained iteratively to perform online manifold identification and tracking, with the dual goal of reducing the need for interface recalibration and enhancing human-machine joint performance. Importantly, the proposed approach avoids interrupting the operation of the device and it neither relies on information about the state of the task, nor on the existence of a stable neural or movement manifold, allowing it to be applied in the earliest stages of interface operation, when the formation of new neural strategies is still on-going. In order to more directly test the performance of our algorithm, we defined the autoencoder latent space as the control space of a body-machine interface. After an initial offline parameter tuning, we evaluated the performance of the adaptive interface versus that of a static decoder in approximating the evolving low-dimensional manifold of users simultaneously learning to perform reaching movements within the latent space. Results show that the adaptive approach increased the representational efficiency of the interface decoder. Concurrently, it significantly improved users' task-related performance, indicating that the development of a more accurate internal model is encouraged by the online co-adaptation process.

Development of a Planar Haptic Robot With Minimized Impedance.

Several studies have reported that stroke survivors displayed improved voluntary planar movements when forces supporting the upper limb increased, and when impeding forces decreased. Earlier haptic devices interacting with the human upper limb were potentially impacted by undesired residual friction force and device inertia. To explore natural, undisturbed voluntary motor control in stroke survivors, we describe the development of a Decoupled-Operational space Robot for wide Impedance Switching (DORIS) with minimized mechanical impedances. This design is based on a novel decoupling mechanism separating the end effector from a manipulator. While the user manipulates the end effector freely inside the workspace of the decoupling mechanism, to which a manipulator of the robot is attached, the robot detects such change in position using a lightweight linkage system. The manipulator of the robot then follows such movements of the end effector swiftly. Consequently, the user can explore the extended workspace, which can be as large as the manipulator's workspace. Since the end effector is mechanically decoupled from the manipulators and actuators, the user can remain unaffected by the mechanical impedances of the manipulator. Mechanical impedances perceived by the user and bandwidth of the control system were estimated. The developed robot was capable of detecting larger maximum acceleration and larger jerk of the reaching movement in chronic stroke survivors with hemiparesis. We propose that this device can be utilized for evaluating voluntary motor control of the upper limb while minimizing the impact of robot inertia and friction forces on limb behavior.

Linear and Non-linear Dimensionality-Reduction Techniques on Full Hand Kinematics.

The purpose of this study was to find a parsimonious representation of hand kinematics data that could facilitate prosthetic hand control. Principal Component Analysis (PCA) and a non-linear Autoencoder Network (nAEN) were compared in their effectiveness at capturing the essential characteristics of a wide spectrum of hand gestures and actions. Performance of the two methods was compared on (a) the ability to accurately reconstruct hand kinematic data from a latent manifold of reduced dimension, (b) variance distribution across latent dimensions, and (c) the separability of hand movements in compressed and reconstructed representations derived using a linear classifier. The nAEN exhibited higher performance than PCA in its ability to more accurately reconstruct hand kinematic data from a latent manifold of reduced dimension. Whereas, for two dimensions in the latent manifold, PCA was able to account for 78% of input data variance, nAEN accounted for 94%. In addition, the nAEN latent manifold was spanned by coordinates with more uniform share of signal variance compared to PCA. Lastly, the nAEN was able to produce a manifold of more separable movements than PCA, as different tasks, when reconstructed, were more distinguishable by a linear classifier, SoftMax regression. It is concluded that non-linear dimensionality reduction may offer a more effective platform than linear methods to control prosthetic hands.

Guiding functional reorganization of motor redundancy using a body-machine interface.

BACKGROUND: Body-machine interfaces map movements onto commands to external devices. Redundant motion signals derived from inertial sensors are mapped onto lower-dimensional device commands. Then, the device users face two problems, a) the structural problem of understanding the operation of the interface and b) the performance problem of controlling the external device with high efficiency. We hypothesize that these problems, while being distinct are connected in that aligning the space of body movements with the space encoded by the interface, i.e. solving the structural problem, facilitates redundancy resolution towards increasing efficiency, i.e. solving the performance problem. METHODS: Twenty unimpaired volunteers practiced controlling the movement of a computer cursor by moving their arms. Eight signals from four inertial sensors were mapped onto the two cursor's coordinates on a screen. The mapping matrix was initialized by asking each user to perform free-form spontaneous upper-limb motions and deriving the two main principal components of the motion signals. Participants engaged in a reaching task for 18 min, followed by a tracking task. One group of 10 participants practiced with the same mapping throughout the experiment, while the other 10 with an adaptive mapping that was iteratively updated by recalculating the principal components based on ongoing movements. RESULTS: Participants quickly reduced reaching time while also learning to distribute most movement variance over two dimensions. Participants with the fixed mapping distributed movement variance over a subspace that did not match the potent subspace defined by the interface map. In contrast, participant with the adaptive map reduced the difference between the two subspaces, resulting in a smaller amount of arm motions distributed over the null space of the interface map. This, in turn, enhanced movement efficiency without impairing generalization from reaching to tracking. CONCLUSIONS: Aligning the potent subspace encoded by the interface map to the user's movement subspace guides redundancy resolution towards increasing movement efficiency, with implications for controlling assistive devices. In contrast, in the pursuit of rehabilitative goals, results would suggest that the interface must change to drive the statistics of user's motions away from the established pattern and toward the engagement of movements to be recovered. TRIAL REGISTRATION: ClinicalTrials.gov, NCT01608438, Registered 16 April 2012.

Stretching the skin immediately enhances perceived stiffness and gradually enhances the predictive control of grip force.

When manipulating objects, we use kinesthetic and tactile information to form an internal representation of their mechanical properties for cognitive perception and for preventing their slippage using predictive control of grip force. A major challenge in understanding the dissociable contributions of tactile and kinesthetic information to perception and action is the natural coupling between them. Unlike previous studies that addressed this question either by focusing on impaired sensory processing in patients or using local anesthesia, we used a behavioral study with a programmable mechatronic device that stretches the skin of the fingertips to address this issue in the intact sensorimotor system. We found that artificial skin-stretch increases the predictive grip force modulation in anticipation of the load force. Moreover, the stretch causes an immediate illusion of touching a harder object that does not depend on the gradual development of the predictive modulation of grip force.

The dynamics of motor learning through the formation of internal models.

A medical student learning to perform a laparoscopic procedure or a recently paralyzed user of a powered wheelchair must learn to operate machinery via interfaces that translate their actions into commands for an external device. Since the user's actions are selected from a number of alternatives that would result in the same effect in the control space of the external device, learning to use such interfaces involves dealing with redundancy. Subjects need to learn an externally chosen many-to-one map that transforms their actions into device commands. Mathematically, we describe this type of learning as a deterministic dynamical process, whose state is the evolving forward and inverse internal models of the interface. The forward model predicts the outcomes of actions, while the inverse model generates actions designed to attain desired outcomes. Both the mathematical analysis of the proposed model of learning dynamics and the learning performance observed in a group of subjects demonstrate a first-order exponential convergence of the learning process toward a particular state that depends only on the initial state of the inverse and forward models and on the sequence of targets supplied to the users. Noise is not only present but necessary for the convergence of learning through the minimization of the difference between actual and predicted outcomes.

Effects of visuomotor delays on the control of movement and on perceptual localization in the presence and absence of visual targets.

The sensory system constantly deals with delayed feedback. Recent studies showed that playing a virtual game of pong with delayed feedback caused hypermetric reaching movements. We investigated whether this effect is associated with a perceptual bias. In addition, we examined the importance of the target in causing hypermetric movements. In a first experiment, participants played a delayed pong game and blindly reached to presented targets. Following each reaching movement, they assessed the position of the invisible cursor. We found that participants performed hypermetric movements but reported that the invisible cursor reached the target, suggesting that they were unaware of the hypermetria and that their perception was biased toward the target rather than toward their hand position. In a second experiment, we removed the visual target, and strikingly, the hypermetria vanished. Moreover, participants reported that the invisible cursor was located with their hand. Taking these results together, we conclude that the adaptation to the visuomotor delay during the pong game selectively affected the execution of goal directed movements, resulting in hypermetria and perceptual bias when movements are directed toward visual targets but not when such targets are absent.NEW & NOTEWORTHY Recent studies showed that adaptation to visuomotor delays causes hypermetric movements in the absence of visual feedback, suggesting that visuomotor delay is represented using current state information. We report that this adaptation also affects perception. Importantly, both the motor and perceptual effects are selective to the representations that are used in the execution of goal-directed movements toward visual targets.

Adaptation to Laterally Asymmetrical Visuomotor Delay Has an Effect on Action But Not on Perception.

When interacting with the environment, the sensorimotor system faces temporal and spatial discrepancies between sensory inputs, such as delay in sensory information transmission, and asymmetrical visual inputs across space. These discrepancies can affect motor control and the representation of space. We recently showed that adaptation to a laterally asymmetric delay in the visual feedback induces neglect-like effects in blind drawing movements, expressed by asymmetrical elongation of circles that are drawn in different workspaces and directions; this establishes a possible connection between delayed feedback and asymmetrical spatial processing in the control of action. In the current study, we investigate whether such adaptation also influences visual perception. In addition, we examined transfer to another motor task - a line bisection task that is commonly used to detect spatial disorders, and extend these results to examine the mapping of these neglect-like effects. We performed two sets of experiments in which participants executed lateral reaching movements, and were exposed to visual feedback delay only in the left workspace. We examined transfer of adaptation to a perceptual line bisection task - answers about the perceived midline of lines that were presented in different directions and workspaces, and to a blind motor line bisection task - reaching movements toward the centers of similar lines. We found that the adaptation to the asymmetrical delay transferred to the control of lateral movements, but did not affect the perceived location of the midlines. Our results clarify the effect of asymmetrical delayed visual feedback on perception and action, and provide potential insights on the link between visuomotor delay and neurological disorders such as the hemispatial neglect syndrome.

Development of an EMG-Controlled Serious Game for Rehabilitation.

A majority of the seven million stroke survivors in the U.S. have hand impairments, adversely affecting performance of a variety of activities of daily living, because of the fundamental role of the hand in performing functional tasks. Disability in stroke survivors is largely attributable to damaged neuronal pathways, which result in inappropriate activation of muscles, a condition prevalent in distal upper extremity muscles following stroke. While conventional rehabilitation methods focus on the amplification of existing muscle activation, the effectiveness of therapy targeting the reorganization of pathological activation patterns is often unexplored. To encourage modulation of activation level and exploration of the activation workspace, we developed a novel platform for playing a serious game through electromyographic control. This system was evaluated by a group of neurologically intact subjects over multiple sessions held on different days. Subjects were assigned to one of two groups, training either with their non-dominant hand only (unilateral) or with both hands (bilateral). Both groups of subjects displayed improved performance in controlling the cursor with their non-dominant hand, with retention from one session to the next. The system holds promise for rehabilitation of control of muscle activation patterns.

Combining Metrics From Clinical Simulators and Sensorimotor Tasks Can Reveal the Training Background of Surgeons.

BACKGROUND: Skill assessment in surgery traditionally has relied on the expert observation and qualitative scoring. Our novel study design demonstrates how analysis of performance in sensorimotor tasks and bench-top surgical simulators can provide inferences about the technical proficiency as well as the training history of surgeons. METHODS: We examined metrics for basic sensorimotor tasks in a virtual reality interface as well as motion metrics in clinical scenario simulations. As indicators of the training level, we considered survey responses from surgery residents, including the number of postgraduation years (PGY, four levels), research years (RY, three levels), and clinical years (CY, three levels). Next, we performed a linear discriminant analysis with cross-validation (90% training, 10% testing) to relate the training levels to the selected metrics. RESULTS: Using combined metrics from all stations, we found greater than chance predictions for each survey category, with an overall accuracy of 43.4 ± 2.9% for identifying the level for post-graduate years, 79.1 ± 1.0% accuracy for research training years, and 64.2 ± 1.0% for clinical training years. Our main finding was that combining metrics from all stations resulted in more accurate predictions than using only sensorimotor or clinical scenario tasks. In addition, we found that metrics related to the ability to cope with changes in the task environment were the most important predictors of training level. CONCLUSIONS: These results suggest that each simulator-type provided crucial information for evaluating surgical proficiency. The methods developed in this paper could improve evaluations of a surgeon's clinical proficiency as well as training potential in terms of basic sensorimotor ability.

The Concurrent Control of Motion and Contact Force in the Presence of Predictable Disturbances.

The simultaneous control of force and motion is important in everyday activities when humans interact with objects. While many studies have analyzed the control of movement within a perturbing force field, few have investigated its dual aspects of controlling a contact force in nonisometric conditions. The mechanism by which the central nervous system controls forces during movements is still unclear, and it can be elucidated by estimating the mechanical properties of the arm during tasks with concurrent motion and contact force goals. We investigate how arm mechanics change when a force control task is accomplished during low-frequency positional perturbations of the arm. Contrary to many force regulation algorithms implemented in robotics, where contact impedance is decreased to reduce force fluctuations in response to position disturbances, we observed a steady increase of arm endpoint stiffness as the task progressed. Based on this evidence, we propose a theoretical framework suggesting that an internal model of the perturbing trajectory is formed. We observed that force regulation in the presence of predictable positional disturbances is implemented using a position control strategy together with the modulation of the endpoint stiffness magnitude, where the direction of the endpoint stiffness ellipse's major axis is oriented toward the desired force.

Neglect-Like Effects on Drawing Symmetry Induced by Adaptation to a Laterally Asymmetric Visuomotor Delay.

In daily interactions, our sensorimotor system accounts for spatial and temporal discrepancies between the senses. Functional lateralization between hemispheres causes differences in attention and in the control of action across the left and right workspaces. In addition, differences in transmission delays between modalities affect movement control and internal representations. Studies on motor impairments such as hemispatial neglect syndrome suggested a link between lateral spatial biases and temporal processing. To understand this link, we computationally modeled and experimentally validated the effect of laterally asymmetric delay in visual feedback on motor learning and its transfer to the control of drawing movements without visual feedback. In the behavioral experiments, we asked healthy participants to perform lateral reaching movements while adapting to delayed visual feedback in either left, right, or both workspaces. We found that the adaptation transferred to blind drawing and caused movement elongation, which is consistent with a state representation of the delay. However, the pattern of the spatial effect varied between conditions: whereas adaptation to delay in only the left workspace or in the whole workspace caused selective leftward elongation, adaptation to delay in only the right workspace caused drawing elongation in both directions. We simulated arm movements according to different models of perceptual and motor spatial asymmetry in the representation of delay and found that the best model that accounts for our results combines both perceptual and motor asymmetry between the hemispheres. These results provide direct evidence for an asymmetrical processing of delayed visual feedback that is associated with both perceptual and motor biases that are similar to those observed in hemispatial neglect syndrome.

Energy exchanges at contact events guide sensorimotor integration.

The brain must consider the arm's inertia to predict the arm's movements elicited by commands impressed upon the muscles. Here, we present evidence suggesting that the integration of sensory information leading to the representation of the arm's inertia does not take place continuously in time but only at discrete transient events, in which kinetic energy is exchanged between the arm and the environment. We used a visuomotor delay to induce cross-modal variations in state feedback and uncovered that the difference between visual and proprioceptive velocity estimations at isolated collision events was compensated by a change in the representation of arm inertia. The compensation maintained an invariant estimate across modalities of the expected energy exchange with the environment. This invariance captures different types of dysmetria observed across individuals following prolonged exposure to a fixed intermodal temporal perturbation and provides a new interpretation for cerebellar ataxia.

The Mechanical Representation of Temporal Delays.

When we knock on a door, we perceive the impact as a collection of simultaneous events, combining sound, sight, and tactile sensation. In reality, information from different modalities but from a single source is flowing inside the brain along different pathways, reaching processing centers at different times. Therefore, interpreting different sensory modalities which seem to occur simultaneously requires information processing that accounts for these different delays. As in a computer-based robotic system, does the brain use some explicit estimation of the time delay, to realign the sensory flows? Or does it compensate for temporal delays by representing them as changes in the body/environment mechanics? Using delayed-state or an approximation for delayed-state manipulations between visual and proprioceptive feedback during a tracking task, we show that tracking errors, grip forces, and learning curves are consistent with predictions of a representation that is based on approximation for delay, refuting an explicit delayed-state representation. Delayed-state representations are based on estimating the time elapsed between the movement commands and their observed consequences. In contrast, an approximation for delay representations result from estimating the instantaneous relation between the expected and observed motion variables, without explicit reference to time.

Learning new movements after paralysis: Results from a home-based study.

Body-machine interfaces (BMIs) decode upper-body motion for operating devices, such as computers and wheelchairs. We developed a low-cost portable BMI for survivors of cervical spinal cord injury and investigated it as a means to support personalized assistance and therapy within the home environment. Depending on the specific impairment of each participant, we modified the interface gains to restore a higher level of upper body mobility. The use of the BMI over one month led to increased range of motion and force at the shoulders in chronic survivors. Concurrently, subjects learned to reorganize their body motions as they practiced the control of a computer cursor to perform different tasks and games. The BMI allowed subjects to generate any movement of the cursor with different motions of their body. Through practice subjects demonstrated a tendency to increase the similarity between the body motions used to control the cursor in distinct tasks. Nevertheless, by the end of learning, some significant and persistent differences appeared to persist. This suggests the ability of the central nervous system to concurrently learn operating the BMI while exploiting the possibility to adapt the available mobility to the specific spatio-temporal requirements of each task.