Toward higher-performance bionic limbs for wider clinical use.

Most prosthetic limbs can autonomously move with dexterity, yet they are not perceived by the user as belonging to their own body. Robotic limbs can convey information about the environment with higher precision than biological limbs, but their actual performance is substantially limited by current technologies for the interfacing of the robotic devices with the body and for transferring motor and sensory information bidirectionally between the prosthesis and the user. In this Perspective, we argue that direct skeletal attachment of bionic devices via osseointegration, the amplification of neural signals by targeted muscle innervation, improved prosthesis control via implanted muscle sensors and advanced algorithms, and the provision of sensory feedback by means of electrodes implanted in peripheral nerves, should all be leveraged towards the creation of a new generation of high-performance bionic limbs. These technologies have been clinically tested in humans, and alongside mechanical redesigns and adequate rehabilitation training should facilitate the wider clinical use of bionic limbs.

Tactile feedback is an effective instrument for the training of grasping with a prosthesis at low- and medium-force levels.

Grasping is a complex task routinely performed in an anticipatory (feedforward) manner, where sensory feedback is responsible for learning and updating the internal model of grasp dynamics. This study aims at evaluating whether providing a proportional tactile force feedback during the myoelectric control of a prosthesis facilitates learning a stable internal model of the prosthesis force control. Ten able-bodied subjects controlled a sensorized myoelectric prosthesis performing four blocks of consecutive grasps at three levels of target force (30, 50, and 70%), repeatedly closing the fully opened hand. In the first and third block, the subjects received tactile and visual feedback, respectively, while during the second and fourth block, the feedback was removed. The subjects also performed an additional block with no feedback 1 day after the training (Retest). The median and interquartile range of the generated forces was computed to assess the accuracy and precision of force control. The results demonstrated that the feedback was indeed an effective instrument for the training of prosthesis control. After the training, the subjects were still able to accurately generate the desired force for the low and medium target (30 and 50% of maximum force available in a prosthesis), despite the feedback being removed within the session and during the retest (low target force). However, the training was substantially less successful for high forces (70% of prosthesis maximum force), where subjects exhibited a substantial loss of accuracy as soon as the feedback was removed. The precision of control decreased with higher forces and it was consistent across conditions, determined by an intrinsic variability of repeated myoelectric grasping. This study demonstrated that the subject could rely on the tactile feedback to adjust the motor command to the prosthesis across trials. The subjects adjusted the mean level of muscle activation (accuracy), whereas the precision could not be modulated as it depends on the intrinsic myoelectric variability. They were also able to maintain the feedforward command even after the feedback was removed, demonstrating thereby a stable learning, but the retention depended on the level of the target force. This is an important insight into the role of feedback as an instrument for learning of anticipatory prosthesis force control.

GLIMPSE: Google Glass interface for sensory feedback in myoelectric hand prostheses.

OBJECTIVE: Providing sensory feedback to the user of the prosthesis is an important challenge. The common approach is to use tactile stimulation, which is easy to implement but requires training and has limited information bandwidth. In this study, we propose an alternative approach based on augmented reality. APPROACH: We have developed the GLIMPSE, a Google Glass application which connects to the prosthesis via a Bluetooth interface and renders the prosthesis states (EMG signals, aperture, force and contact) using augmented reality (see-through display) and sound (bone conduction transducer). The interface was tested in healthy subjects that used the prosthesis with (FB group) and without (NFB group) feedback during a modified clothespins test that allowed us to vary the difficulty of the task. The outcome measures were the number of unsuccessful trials, the time to accomplish the task, and the subjective ratings of the relevance of the feedback. MAIN RESULTS: There was no difference in performance between FB and NFB groups in the case of a simple task (basic, same-color clothespins test), but the feedback significantly improved the performance in a more complex task (pins of different resistances). Importantly, the GLIMPSE feedback did not increase the time to accomplish the task. Therefore, the supplemental feedback might be useful in the tasks which are more demanding, and thereby less likely to benefit from learning and feedforward control. The subjects integrated the supplemental feedback with the intrinsic sources (vision and muscle proprioception), developing their own idiosyncratic strategies to accomplish the task. SIGNIFICANCE: The present study demonstrates a novel self-contained, ready-to-deploy, wearable feedback interface. The interface was successfully tested and was proven to be feasible and functionally beneficial. The GLIMPSE can be used as a practical solution but also as a general and flexible instrument to investigate closed-loop prosthesis control.

Electrotactile EMG feedback improves the control of prosthesis grasping force.

OBJECTIVE: A drawback of active prostheses is that they detach the subject from the produced forces, thereby preventing direct mechanical feedback. This can be compensated by providing somatosensory feedback to the user through mechanical or electrical stimulation, which in turn may improve the utility, sense of embodiment, and thereby increase the acceptance rate. APPROACH: In this study, we compared a novel approach to closing the loop, namely EMG feedback (emgFB), to classic force feedback (forceFB), using electrotactile interface in a realistic task setup. Eleven intact-bodied subjects and one transradial amputee performed a routine grasping task while receiving emgFB or forceFB. The two feedback types were delivered through the same electrotactile interface, using a mixed spatial/frequency coding to transmit 8 discrete levels of the feedback variable. In emgFB, the stimulation transmitted the amplitude of the processed myoelectric signal generated by the subject (prosthesis input), and in forceFB the generated grasping force (prosthesis output). The task comprised 150 trials of routine grasping at six forces, randomly presented in blocks of five trials (same force). Interquartile range and changes in the absolute error (AE) distribution (magnitude and dispersion) with respect to the target level were used to assess precision and overall performance, respectively. MAIN RESULTS: Relative to forceFB, emgFB significantly improved the precision of myoelectric commands (min/max of the significant levels) for 23%/36% as well as the precision of force control for 12%/32%, in intact-bodied subjects. Also, the magnitude and dispersion of the AE distribution were reduced. The results were similar in the amputee, showing considerable improvements. SIGNIFICANCE: Using emgFB, the subjects therefore decreased the uncertainty of the forward pathway. Since there is a correspondence between the EMG and force, where the former anticipates the latter, the emgFB allowed for predictive control, as the subjects used the feedback to adjust the desired force even before the prosthesis contacted the object. In conclusion, the online emgFB was superior to the classic forceFB in realistic conditions that included electrotactile stimulation, limited feedback resolution (8 levels), cognitive processing delay, and time constraints (fast grasping).

Context-Dependent Upper Limb Prosthesis Control for Natural and Robust Use.

Pattern recognition and regression methods applied to the surface EMG have been used for estimating the user intended motor tasks across multiple degrees of freedom (DOF), for prosthetic control. While these methods are effective in several conditions, they are still characterized by some shortcomings. In this study we propose a methodology that combines these two approaches for mutually alleviating their limitations. This resulted in a control method capable of context-dependent movement estimation that switched automatically between sequential (one DOF at a time) or simultaneous (multiple DOF) prosthesis control, based on an online estimation of signal dimensionality. The proposed method was evaluated in scenarios close to real-life situations, with the control of a physical prosthesis in applied tasks of varying difficulties. Test prostheses were individually manufactured for both able-bodied and transradial amputee subjects. With these prostheses, two amputees performed the Southampton Hand Assessment Procedure test with scores of 58 and 71 points. The five able-bodied individuals performed standardized tests, such as the box&block and clothes pin test, reducing the completion times by up to 30%, with respect to using a state-of-the-art pure sequential control algorithm. Apart from facilitating fast simultaneous movements, the proposed control scheme was also more intuitive to use, since human movements are predominated by simultaneous activations across joints. The proposed method thus represents a significant step towards intelligent, intuitive and natural control of upper limb prostheses.

Sensor fusion and computer vision for context-aware control of a multi degree-of-freedom prosthesis.

OBJECTIVE: Myoelectric activity volitionally generated by the user is often used for controlling hand prostheses in order to replicate the synergistic actions of muscles in healthy humans during grasping. Muscle synergies in healthy humans are based on the integration of visual perception, heuristics and proprioception. Here, we demonstrate how sensor fusion that combines artificial vision and proprioceptive information with the high-level processing characteristics of biological systems can be effectively used in transradial prosthesis control. APPROACH: We developed a novel context- and user-aware prosthesis (CASP) controller integrating computer vision and inertial sensing with myoelectric activity in order to achieve semi-autonomous and reactive control of a prosthetic hand. The presented method semi-automatically provides simultaneous and proportional control of multiple degrees-of-freedom (DOFs), thus decreasing overall physical effort while retaining full user control. The system was compared against the major commercial state-of-the art myoelectric control system in ten able-bodied and one amputee subject. All subjects used transradial prosthesis with an active wrist to grasp objects typically associated with activities of daily living. MAIN RESULTS: The CASP significantly outperformed the myoelectric interface when controlling all of the prosthesis DOF. However, when tested with less complex prosthetic system (smaller number of DOF), the CASP was slower but resulted with reaching motions that contained less compensatory movements. Another important finding is that the CASP system required minimal user adaptation and training. SIGNIFICANCE: The CASP constitutes a substantial improvement for the control of multi-DOF prostheses. The application of the CASP will have a significant impact when translated to real-life scenarious, particularly with respect to improving the usability and acceptance of highly complex systems (e.g., full prosthetic arms) by amputees.

EMG Biofeedback for online predictive control of grasping force in a myoelectric prosthesis.

BACKGROUND: Active hand prostheses controlled using electromyography (EMG) signals have been used for decades to restore the grasping function, lost after an amputation. Although myocontrol is a simple and intuitive interface, it is also imprecise due to the stochastic nature of the EMG recorded using surface electrodes. Furthermore, the sensory feedback from the prosthesis to the user is still missing. In this study, we present a novel concept to close the loop in myoelectric prostheses. In addition to conveying the grasping force (system output), we provided to the user the online information about the system input (EMG biofeedback). METHODS: As a proof-of-concept, the EMG biofeedback was transmitted in the current study using a visual interface (ideal condition). Ten able-bodied subjects and two amputees controlled a state-of-the-art myoelectric prosthesis in routine grasping and force steering tasks using EMG and force feedback (novel approach) and force feedback only (classic approach). The outcome measures were the variability of the generated forces and absolute deviation from the target levels in the routine grasping task, and the root mean square tracking error and the number of sudden drops in the force steering task. RESULTS: During the routine grasping, the novel method when used by able-bodied subjects decreased twofold the force dispersion as well as absolute deviations from the target force levels, and also resulted in a more accurate and stable tracking of the reference force profiles during the force steering. Furthermore, the force variability during routine grasping did not increase for the higher target forces with EMG biofeedback. The trend was similar in the two amputees. CONCLUSIONS: The study demonstrated that the subjects, including the two experienced users of a myoelectric prosthesis, were able to exploit the online EMG biofeedback to observe and modulate the myoelectric signals, generating thereby more consistent commands. This allowed them to control the force predictively (routine grasping) and with a finer resolution (force steering). The future step will be to implement this promising and simple approach using an electrotactile interface. A prosthesis with a reliable response, following faithfully user intentions, would improve the utility during daily-life use and also facilitate the embodiment of the assistive system.

The impact of the stimulation frequency on closed-loop control with electrotactile feedback.

BACKGROUND: Electrocutaneous stimulation can restore the missing sensory information to prosthetic users. In electrotactile feedback, the information about the prosthesis state is transmitted in the form of pulse trains. The stimulation frequency is an important parameter since it influences the data transmission rate over the feedback channel as well as the form of the elicited tactile sensations. METHODS: We evaluated the influence of the stimulation frequency on the subject's ability to utilize the feedback information during electrotactile closed-loop control. Ten healthy subjects performed a real-time compensatory tracking (standard test bench) of sinusoids and pseudorandom signals using either visual feedback (benchmark) or electrocutaneous feedback in seven conditions characterized by different combinations of the stimulation frequency (FSTIM) and tracking error sampling rate (FTE). The tracking error was transmitted using two concentric electrodes placed on the forearm. The quality of tracking was assessed using the Squared Pearson Correlation Coefficient (SPCC), the Normalized Root Mean Square Tracking Error (NRMSTE) and the time delay between the reference and generated trajectories (TDIO). RESULTS: The results demonstrated that FSTIM was more important for the control performance than FTE. The quality of tracking deteriorated with a decrease in the stimulation frequency, SPCC and NRMSTE (mean) were 87.5% and 9.4% in the condition 100/100 (FTE/FSTIM), respectively, and deteriorated to 61.1% and 15.3% in 5/5, respectively, while the TDIO increased from 359.8 ms in 100/100 to 1009 ms in 5/5. However, the performance recovered when the tracking error sampled at a low rate was delivered using a high stimulation frequency (SPCC = 83.6%, NRMSTE = 10.3%, TDIO = 415.6 ms, in 5/100). CONCLUSIONS: The likely reason for the performance decrease and recovery was that the stimulation frequency critically influenced the tactile perception quality and thereby the effective rate of information transfer through the feedback channel. The outcome of this study can facilitate the selection of optimal system parameters for somatosensory feedback in upper limb prostheses. The results imply that the feedback variables (e.g., grasping force) should be transmitted at relatively high frequencies of stimulation (>25 Hz), but that they can be sampled at much lower rates (e.g., 5 Hz).

A Multi-Class Proportional Myocontrol Algorithm for Upper Limb Prosthesis Control: Validation in Real-Life Scenarios on Amputees.

Functional replacement of upper limbs by means of dexterous prosthetic devices remains a technological challenge. While the mechanical design of prosthetic hands has advanced rapidly, the human-machine interfacing and the control strategies needed for the activation of multiple degrees of freedom are not reliable enough for restoring hand function successfully. Machine learning methods capable of inferring the user intent from EMG signals generated by the activation of the remnant muscles are regarded as a promising solution to this problem. However, the lack of robustness of the current methods impedes their routine clinical application. In this study, we propose a novel algorithm for controlling multiple degrees of freedom sequentially, inherently proportionally and with high robustness, allowing a good level of prosthetic hand function. The control algorithm is based on the spatial linear combinations of amplitude-related EMG signal features. The weighting coefficients in this combination are derived from the optimization criterion of the common spatial patterns filters which allow for maximal discriminability between movements. An important component of the study is the validation of the method which was performed on both able-bodied and amputee subjects who used physical prostheses with customized sockets and performed three standardized functional tests mimicking daily-life activities of varying difficulty. Moreover, the new method was compared in the same conditions with one clinical/industrial and one academic state-of-the-art method. The novel algorithm outperformed significantly the state-of-the-art techniques in both subject groups for tests that required the activation of more than one degree of freedom. Because of the evaluation in real time control on both able-bodied subjects and final users (amputees) wearing physical prostheses, the results obtained allow for the direct extrapolation of the benefits of the proposed method for the end users. In conclusion, the method proposed and validated in real-life use scenarios, allows the practical usability of multifunctional hand prostheses in an intuitive way, with significant advantages with respect to previous systems.

The extraction of neural information from the surface EMG for the control of upper-limb prostheses: emerging avenues and challenges.

Despite not recording directly from neural cells, the surface electromyogram (EMG) signal contains information on the neural drive to muscles, i.e., the spike trains of motor neurons. Using this property, myoelectric control consists of the recording of EMG signals for extracting control signals to command external devices, such as hand prostheses. In commercial control systems, the intensity of muscle activity is extracted from the EMG and used for single degrees of freedom activation (direct control). Over the past 60 years, academic research has progressed to more sophisticated approaches but, surprisingly, none of these academic achievements has been implemented in commercial systems so far. We provide an overview of both commercial and academic myoelectric control systems and we analyze their performance with respect to the characteristics of the ideal myocontroller. Classic and relatively novel academic methods are described, including techniques for simultaneous and proportional control of multiple degrees of freedom and the use of individual motor neuron spike trains for direct control. The conclusion is that the gap between industry and academia is due to the relatively small functional improvement in daily situations that academic systems offer, despite the promising laboratory results, at the expense of a substantial reduction in robustness. None of the systems so far proposed in the literature fulfills all the important criteria needed for widespread acceptance by the patients, i.e. intuitive, closed-loop, adaptive, and robust real-time ( 200 ms delay) control, minimal number of recording electrodes with low sensitivity to repositioning, minimal training, limited complexity and low consumption. Nonetheless, in recent years, important efforts have been invested in matching these criteria, with relevant steps forwards.

Self-correcting pattern recognition system of surface EMG signals for upper limb prosthesis control.

Pattern recognition methods for classifying user motion intent based on surface electromyography developed by research groups in well-controlled laboratory conditions are not yet clinically viable for upper limb prosthesis control, due to their limited robustness in users' real-life situations. To address this problem, a novel postprocessing algorithm, aiming to detect and remove misclassifications of a pattern recognition system of forearm and hand motions, is proposed. Using the maximum likelihood calculated by a classifier and the mean global muscle activity of the forearm, an artificial neural network was trained to detect potentially erroneous classification decisions. This system was compared to four previously proposed classification postprocessing methods, in both able-bodied and amputee subjects. Various nonstationarities were included in the experimental protocol to account for challenges posed in real-life settings, such as different contraction levels, static and dynamic motion phases, and effects induced by day-to-day transfers, such as electrode shifts, impedance changes, and psychometric user variability. The improvement in classification accuracy with respect to the unprocessed classifier ranged from 4.8% to 31.6%, depending on the scenarios investigated. The system significantly reduced misclassifications to wrong active classes and is thus a promising approach for improving the robustness of hand prosthesis controllability.

Covariate shift adaptation in EMG pattern recognition for prosthetic device control.

Ensuring robustness of myocontrol algorithms for prosthetic devices is an important challenge. Robustness needs to be maintained under nonstationarities, e.g. due to electrode shifts after donning and doffing, sweating, additional weight or varying arm positions. Such nonstationary behavior changes the signal distributions - a scenario often referred to as covariate shift. This circumstance causes a significant decrease in classification accuracy in daily life applications. Re-training is possible but it is time consuming since it requires a large number of trials. In this paper, we propose to adapt the EMG classifier by a small calibration set only, which is able to capture the relevant aspects of the nonstationarities, but requires re-training data of only very short duration. We tested this strategy on signals acquired across 5 days in able-bodied individuals. The results showed that an estimator that shrinks the training model parameters towards the calibration set parameters significantly increased the classifier performance across different testing days. Even when using only one trial per class as re-training data for each day, the classification accuracy remained > 92% over five days. These results indicate that the proposed methodology can be a practical means for improving robustness in pattern recognition methods for myocontrol.

Intuitive, online, simultaneous, and proportional myoelectric control over two degrees-of-freedom in upper limb amputees.

We propose an approach for online simultaneous and proportional myoelectric control of two degrees-of-freedom (DoF) of the wrist, using surface electromyographic signals. The method is based on the nonnegative matrix factorization (NMF) of the wrist muscle activation to extract low-dimensional control signals translated by the user into kinematic variables. This procedure does not need a training set of signals for which the kinematics is known (labeled dataset) and is thus unsupervised (although it requires an initial calibration without labeled signals). The estimated control signals using NMF are used to directly control two DoFs of wrist. The method was tested on seven subjects with upper limb deficiency and on seven able-bodied subjects. The subjects performed online control of a virtual object with two DoFs to achieve goal-oriented tasks. The performance of the two subject groups, measured as the task completion rate, task completion time, and execution efficiency, was not statistically different. The approach was compared, and demonstrated to be superior to the online control by the industrial state-of-the-art approach. These results show that this new approach, which has several advantages over the previous myoelectric prosthetic control systems, has the potential of providing intuitive and dexterous control of artificial limbs for amputees.

Is accurate mapping of EMG signals on kinematics needed for precise online myoelectric control?

In this paper, we present a systematic analysis of the relationship between the accuracy of the mapping between EMG and hand kinematics and the control performance in goal-oriented tasks of three simultaneous and proportional myoelectric control algorithms: nonnegative matrix factorization (NMF), linear regression (LR), and artificial neural networks (ANN). The purpose was to investigate the impact of the precision of the kinematics estimation by a myoelectric controller for accurately complete goal-directed tasks. Nine naïve subjects performed a series of goal-directed myoelectric control tasks using the three algorithms, and their online performance was characterized by 6 indexes. The results showed that, although the three algorithms' mapping accuracies were significantly different, their online performance was similar. Moreover, for LR and ANN, the offline performance was not correlated to any of the online performance indexes, and only a weak correlation was found with three of them for NMF . We conclude that for reliable simultaneous and proportional myoelectric control, it is not necessary to achieve high accuracy in the mapping between EMG and kinematics. Rather, good online myoelectric control is achieved by the continuous interaction and adaptation of the user with the myoelectric controller through feedback (visual in the current study). Control signals generated by EMG with rather poor association with kinematic variables can still be fully exploited by the user for precise control. This conclusion explains the possibility of accurate simultaneous and proportional control over multiple degrees of freedom when using unsupervised algorithms, such as NMF.

Extending mode switching to multiple degrees of freedom in hand prosthesis control is not efficient.

In recent years, many sophisticated control strategies for multifunctional dexterous hand prostheses have been developed. It was indeed assumed that control mechanisms based on switching between degrees of freedom, which are in use since the 1960's, could not be extended to efficient control of more than two degrees of freedom. However, quantitative proof for this assumption has not been shown. In this study, we adopted the mode switching paradigm available in commercial prostheses for two degree of freedom control and we extended it for the control of seven functions (3.5 degrees of freedom) in a modern robotic hand. We compared the controllability of this scaled version of the standard method to a state of the art pattern recognition based control in an applied online study. The aim was to quantify whether multi-functional prosthetic control with mode switching outperformed pattern recognition in the control of a real prosthetic hand for daily life activities online. Although in simple grasp-release tasks the conventional method performed best, tasks requiring more complex control of multiple degrees of freedom required a more intuitive control method, such as pattern recognition, for achieving high performance.

Long term stability of surface EMG pattern classification for prosthetic control.

Long-term functioning of a hand prosthesis is crucial for its acceptance by patients with upper limb deficit. In this study the reliability over days of the performance of pattern classification approaches based on surface electromyography (sEMG) signal for the control of upper limb prostheses was investigated. Recordings of sEMG from the forearm muscles were obtained across five consecutive days from five healthy subjects. It was demonstrated that the classification performance decreased monotonically on average by 4.1% per day. It was also found that the accumulated error was confined to three of the eight movement classes investigated. This contribution gives insight on the long term behavior of pattern classification, which is crucial for commercial viability.

Effect of arm position on the prediction of kinematics from EMG in amputees.

Myoelectric control has been extensively applied in multi-function hand/wrist prostheses. The performance of this type of control is however, influenced by several practical factors that still limit its clinical applicability. One of these factors is the change in arm posture during the daily use of prostheses. In this study, we investigate the effect of arm position on the performance of a simultaneous and proportional myoelectric control algorithm, both on trans-radial amputees and able-bodied subjects. The results showed that changing arm position adversely influences the performance of the algorithm for both subject groups, but that this influence is less pronounced in amputee subjects with respect to able-bodied subjects. Thus, the impact of arm posture on myoelectric control cannot be inferred from results on able-bodied subjects and should be directly investigated in amputee subjects.

Spatial filtering for robust myoelectric control.

Pattern recognition techniques have been applied to extract information from electromyographic (EMG) signals that can be used to control electrical powered hand prostheses. In this paper, optimized spatial filters that enhance separation properties of EMG signals are investigated. In particular, different multiclass extensions of the common spatial patterns algorithm are applied to high-density surface EMG signals acquired from the forearms of ten healthy subjects. Visualization of the obtained filter coefficients provides insight into the physiology of the muscles related to the performed contractions. The CSP methods are compared with a commonly used pattern recognition approach in a six-class classification task. Cross-validation results show a significant improvement in performance and a higher robustness against noise than commonly used pattern recognition methods.

Real time simultaneous and proportional control of multiple degrees of freedom from surface EMG: Preliminary results on subjects with limb deficiency.

We present the real time simultaneous and proportional control of two degrees of freedom (DoF), using surface electromyographic signals from the residual limbs of three subject with limb deficiency. Three subjects could control a virtual object in two dimensions using their residual muscle activities to achieve goal-oriented tasks. The subjects indicated that they found the control intuitive and useful. These results show that such a simultaneous and proportional control paradigm is a promising direction for multi-functional prosthetic control.

Myoelectric control in neurorehabilitation.

A myoelectric signal, or electromyogram (EMG), is the electrical manifestation of a muscle contraction. Through advanced signal processing techniques, information on the neural control of muscles can be extracted from the EMG, and the state of the neuromuscular system can be inferred. Because of its easy accessibility and relatively high signal-to-noise ratio, EMG has been applied as a control signal in several neurorehabilitation devices and applications, such as multi-function prostheses and orthoses, rehabilitation robots, and functional electrical stimulation/therapy. These EMG-based neurorehabilitation modules, which constitute muscle-machine interfaces, are applied for replacement, restoration, or modulation of lost or impaired function in research and clinical settings. The purpose of this review is to discuss the assumptions of EMG-based control and its applications in neurorehabilitation.