A Systematic Review of Sensor Fusion Methods Using Peripheral Bio-Signals for Human Intention Decoding.

Humans learn about the environment by interacting with it. With an increasing use of computer and virtual applications as well as robotic and prosthetic devices, there is a need for intuitive interfaces that allow the user to have an embodied interaction with the devices they are controlling. Muscle-machine interfaces can provide an intuitive solution by decoding human intentions utilizing myoelectric activations. There are several different methods that can be utilized to develop MuMIs, such as electromyography, ultrasonography, mechanomyography, and near-infrared spectroscopy. In this paper, we analyze the advantages and disadvantages of different myography methods by reviewing myography fusion methods. In a systematic review following the PRISMA guidelines, we identify and analyze studies that employ the fusion of different sensors and myography techniques, while also considering interface wearability. We also explore the properties of different fusion techniques in decoding user intentions. The fusion of electromyography, ultrasonography, mechanomyography, and near-infrared spectroscopy as well as other sensing such as inertial measurement units and optical sensing methods has been of continuous interest over the last decade with the main focus decoding the user intention for the upper limb. From the systematic review, it can be concluded that the fusion of two or more myography methods leads to a better performance for the decoding of a user's intention. Furthermore, promising sensor fusion techniques for different applications were also identified based on the existing literature.

How Positioning Wearable Haptic Interfaces on Limbs Influences Virtual Embodiment.

With increasing use of computer applications and robotic devices in our everyday life, and with the advent of metaverse, there is an urgent need of developing new types of interfaces that facilitate a more intuitive interaction in physical and virtual space. In this work, we investigate the influence of the location of haptic feedback devices on embodiment of virtual hands and user load during an interactive pick-and-place task. To do this, we conducted a user study with a 3x2 repeated measure experiment design: feedback position is varied between the distal phalanx of the index finger and the thumb, the proximal phalanx of the index finger and the thumb, and the wrist. These conditions of feedback are tested with the stimuli applied synchronously to the participant in one case, and with an additional delay of 350 ms in the second case. The results show that the location of the haptic feedback device does not affect embodiment, whereas the delay, i.e., whether the feedback is applied synchronously or asynchronously, affects embodiment. This suggests that for pick-and-place tasks, haptic feedback devices can be placed on the user's wrist without compromising performance making the hands to remain free, allowing unobstructed hand visibility for precise motion tracking, thereby improving accuracy.

On lightmyography based muscle-machine interfaces for the efficient decoding of human gestures and forces.

Conventional muscle-machine interfaces like Electromyography (EMG), have significant drawbacks, such as crosstalk, a non-linear relationship between the signal and the corresponding motion, and increased signal processing requirements. In this work, we introduce a new muscle-machine interfacing technique called lightmyography (LMG), that can be used to efficiently decode human hand gestures, motion, and forces from the detected contractions of the human muscles. LMG utilizes light propagation through elastic media and human tissue, measuring changes in light luminosity to detect muscle movement. Similar to forcemyography, LMG infers muscular contractions through tissue deformation and skin displacements. In this study, we look at how different characteristics of the light source and silicone medium affect the performance of LMG and we compare LMG and EMG based gesture decoding using various machine learning techniques. To do that, we design an armband equipped with five LMG modules, and we use it to collect the required LMG data. Three different machine learning methods are employed: Random Forests, Convolutional Neural Networks, and Temporal Multi-Channel Vision Transformers. The system has also been efficiently used in decoding the forces exerted during power grasping. The results demonstrate that LMG outperforms EMG for most methods and subjects.

Multi-Grasp Classification for the Control of Robot Hands Employing Transformers and Lightmyography Signals.

The increasing use of smart technical devices in our everyday lives has necessitated the use of muscle-machine interfaces (MuMI) that are intuitive and that can facilitate immersive interactions with these devices. The most common method to develop MuMIs is using Electromyography (EMG) based signals. However, due to several drawbacks of EMG-based interfaces, alternative methods to develop MuMI are being explored. In our previous work, we presented a new MuMI called Lightmyography (LMG), which achieved outstanding results compared to a classic EMG-based interface in a five-gesture classification task. In this study, we extend our previous work experimentally validating the efficiency of the LMG armband in classifying thirty-two different gestures from six participants using a deep learning technique called Temporal Multi-Channel Vision Transformers (TMC-ViT). The efficiency of the proposed model was assessed using accuracy. Moreover, two different undersampling techniques are compared. The proposed thirty-two-gesture classifiers achieve accuracies as high as 92%. Finally, we employ the LMG interface in the real-time control of a robotic hand using ten different gestures, successfully reproducing several grasp types from taxonomy grasps presented in the literature.

Electromyography Based Decoding of Dexterous, In-Hand Manipulation Motions With Temporal Multichannel Vision Transformers.

Electromyography (EMG) signals have been used in designing muscle-machine interfaces (MuMIs) for various applications, ranging from entertainment (EMG controlled games) to human assistance and human augmentation (EMG controlled prostheses and exoskeletons). For this, classical machine learning methods such as Random Forest (RF) models have been used to decode EMG signals. However, these methods depend on several stages of signal pre-processing and extraction of hand-crafted features so as to obtain the desired output. In this work, we propose EMG based frameworks for the decoding of object motions in the execution of dexterous, in-hand manipulation tasks using raw EMG signals input and two novel deep learning (DL) techniques called Temporal Multi-Channel Transformers and Vision Transformers. The results obtained are compared, in terms of accuracy and speed of decoding the motion, with RF-based models and Convolutional Neural Networks as a benchmark. The models are trained for 11 subjects in a motion-object specific and motion-object generic way, using the 10-fold cross-validation procedure. This study shows that the performance of MuMIs can be improved by employing DL-based models with raw myoelectric activations instead of developing DL or classic machine learning models with hand-crafted features.

Comparing Machine Learning Methods and Feature Extraction Techniques for the EMG Based Decoding of Human Intention.

With an increasing number of robotic and prosthetic devices, there is a need for intuitive Muscle-Machine Interfaces (MuMIs) that allow the user to have an embodied interaction with the devices they are controlling. Such MuMIs can be developed using machine learning based methods that utilize myoelectric activations from the muscles of the user to decode their intention. However, the choice of the learning method is subjective and depends on the features extracted from the raw Electromyography signals as well as on the intended application. In this work, we compare the performance of five machine learning methods and eight time-domain feature extraction techniques in discriminating between different gestures executed by the user of an EMG based MuMI. From the results, it can be seen that the Willison Amplitude performs consistently better for all the machine learning methods compared in this study, while the Zero Crossings achieves the worst results for the Decision Trees and the Random Forests and the Variance offers the worst performance for all the other learning methods. The Random Forests method is shown to achieve the best results in terms of achieved accuracies (has the lowest variance between subjects). In order to experimentally validate the efficiency of the Random Forest classifier and the Willison Amplitude technique, a series of gestures were decoded in a real-time manner from the myoelectric activations of the operator and they were used to control a robot hand.

On Lightmyography: A New Muscle Machine Interfacing Method for Decoding Human Intention and Motion.

Recognising and classifying human hand gestures is important for effective communication between humans and machines in applications such as human-robot interaction, human to robot skill transfer, and control of prosthetic devices. Although there are already many interfaces that enable decoding of the intention and action of humans, they are either bulky or they rely on techniques that need careful positioning of the sensors, causing inconvenience when the system needs to be used in real-life scenarios and environments. Moreover, electromyography (EMG), which is the most commonly used technique, captures EMG signals that have a nonlinear relationship with the human intention and motion. In this work, we present lightmyography (LMG) a new muscle machine interfacing method for decoding human intention and motion. Lightmyography utilizes light propagation through elastic media and the change of light luminosity to detect silicone deformation. Lightmyography is similar to forcemyography in the sense that they both record muscular contractions through skin displacements. In order to experimentally validate the efficiency of the proposed method, we designed an interface consisting of five LMG sensors to perform gesture classification experiments. Using this device, we were able to accurately detect a series of different hand postures and gestures. We also compared LMG data with processed EMG data.

A Learning Scheme for EMG Based Decoding of Dexterous, In-Hand Manipulation Motions.

Electromyography (EMG) based interfaces are the most common solutions for the control of robotic, orthotic, prosthetic, assistive, and rehabilitation devices, translating myoelectric activations into meaningful actions. Over the last years, a lot of emphasis has been put into the EMG based decoding of human intention, but very few studies have been carried out focusing on the continuous decoding of human motion. In this work, we present a learning scheme for the EMG based decoding of object motions in dexterous, in-hand manipulation tasks. We also study the contribution of different muscles while performing these tasks and the effect of the gender and hand size in the overall decoding accuracy. To do that, we use EMG signals derived from 16 muscle sites (8 on the hand and 8 on the forearm) from 11 different subjects and an optical motion capture system that records the object motion. The object motion decoding is formulated as a regression problem using the Random Forests methodology. Regarding feature selection, we use the following time-domain features: root mean square, waveform length and zero crossings. A 10-fold cross validation procedure is used for model assessment purposes and the feature variable importance values are calculated for each feature. This study shows that subject specific, hand specific, and object specific decoding models offer better decoding accuracy that the generic models.

On Muscle Selection for EMG Based Decoding of Dexterous, In-Hand Manipulation Motions.

The field of Brain Machine Interfaces (BMI) has attracted an increased interest due to its multiple applications in the health and entertainment domains. A BMI enables a direct interface between the brain and machines and is capable of translating neuronal information into meaningful actions (e.g., Electromyography based control of a prosthetic hand). One of the biggest challenges in developing a surface Electromyography (sEMG) based interface is the selection of the right muscles for the execution of a desired task. In this work, we investigate optimal muscle selections for sEMG based decoding of dexterous in-hand manipulation motions. To do that, we use EMG signals derived from 14 muscle sites of interest (7 on the hand and 7 on the forearm) and an optical motion capture system that records the object motion. The regression problem is formulated using the Random Forests methodology that is based on decision trees. Regarding features selection, we use the following time-domain features: root mean square, waveform length and zero crossings. A 5-fold cross validation procedure is used for model assessment purposes and the importance values are calculated for each feature. This pilot study shows that the muscles of the hand contribute more than the muscles of the forearm to the execution of inhand manipulation tasks and that the myoelectric activations of the hand muscles provide better estimation accuracies for the decoding of manipulation motions. These outcomes suggest that the loss of the hand muscles in certain amputations limits the amputees' ability to perform a dexterous, EMG based control of a prosthesis in manipulation tasks. The results discussed can also be used for improving the efficiency and intuitiveness of EMG based interfaces for healthy subjects.