[Enhancement algorithm for surface electromyographic-based gesture recognition based on real-time fusion of muscle fatigue features].

This study aims to optimize surface electromyography-based gesture recognition technique, focusing on the impact of muscle fatigue on the recognition performance. An innovative real-time analysis algorithm is proposed in the paper, which can extract muscle fatigue features in real time and fuse them into the hand gesture recognition process. Based on self-collected data, this paper applies algorithms such as convolutional neural networks and long short-term memory networks to provide an in-depth analysis of the feature extraction method of muscle fatigue, and compares the impact of muscle fatigue features on the performance of surface electromyography-based gesture recognition tasks. The results show that by fusing the muscle fatigue features in real time, the algorithm proposed in this paper improves the accuracy of hand gesture recognition at different fatigue levels, and the average recognition accuracy for different subjects is also improved. In summary, the algorithm in this paper not only improves the adaptability and robustness of the hand gesture recognition system, but its research process can also provide new insights into the development of gesture recognition technology in the field of biomedical engineering.

Idiosyncratic gesture use in a mother-infant dyad in chimpanzees (Pan troglodytes schweinfurthii) in the wild.

One promising method to tackle the question, "In which modality did language evolve?" is by studying the ontogenetic trajectory of signals in human's closest living relatives, including chimpanzees (Pan troglodytes). Concerning gestures, current debates centre on four different hypotheses: "phylogenetic ritualization", "social transmission through imitation", "ontogenetic ritualization", and "social negotiation". These differ in their predictions regarding idiosyncratic gestures, making such occurrences a crucial area of investigation. Here, we describe a novel and potential idiosyncratic behaviour - 'hand-on-eye' - which was initially observed in one mother-infant dyad in a community of chimpanzees living in the wild. We systematically investigated the form, sequential organisation, intentionality, usage, function, and distribution of the behaviour over a five-year period. The results showed that 'hand-on-eye' was nearly exclusively deployed in a single mother-infant dyad, was accompanied by hallmarks of intentionality, and served to initiate or resume joint dorsal travel. Although the behaviour was observed once in each of three other mother-infant dyads, these lacked the same frequency and hallmarks of intentionality. 'Hand-on-eye' thus qualifies as an idiosyncratic gesture. The proposed developmental pathway gives support to both the "ontogenetic ritualization" and "social negotiation" hypotheses. It also stresses the crucial need for longitudinal approaches to tackle developmental processes that are triggered by unique circumstances and unfold over relatively long time windows.

Analysis of electrode locations on limb condition effect for myoelectric pattern recognition.

BACKGROUND: Gesture recognition using surface electromyography (sEMG) has garnered significant attention due to its potential for intuitive and natural control in wearable human-machine interfaces. However, ensuring robustness remains essential and is currently the primary challenge for practical applications. METHODS: This study investigates the impact of limb conditions and analyzes the influence of electrode placement. Both static and dynamic limb conditions were examined using electrodes positioned on the wrist, elbow, and the midpoint between them. Initially, we compared classification performance across various training conditions at these three electrode locations. Subsequently, a feature space analysis was conducted to quantify the effects of limb conditions. Finally, strategies for group training and feature selection were explored to mitigate these effects. RESULTS: The results indicate that with the state-of-the-art method, classification performance at the wrist was comparable to that at the middle position, both of which outperformed the elbow, consistent with the findings from the feature space analysis. In inter-condition classification, training under dynamic limb conditions yielded better results than training under static conditions, especially at the positions covered by dynamic training. Additionally, fast and slow movement speeds produced similar performance outcomes. To mitigate the effects of limb conditions, adding more training conditions reduced classification errors; however, this reduction plateaued after four conditions, resulting in classification errors of 22.72%, 22.65%, and 26.58% for the wrist, middle, and elbow, respectively. Feature selection further improved classification performance, reducing errors to 19.98%, 19.75%, and 27.14% at the respective electrode locations, using three optimal features derived from single-condition training. CONCLUSIONS: The study demonstrated that the impact of limb conditions was mitigated when electrodes were placed near the wrist. Dynamic limb condition training, combined with feature optimization, proved to be an effective strategy for reducing this effect. This work contributes to enhancing the robustness of myoelectric-controlled interfaces, thereby advancing the development of wearable intelligent devices.

Gender differences in the observation of gesture direction: a physiological study.

Intransitive gestures are expressive and symbolic, whereas pantomimes are object-related actions. These gestures convey different meanings depending on whether they are directed toward (TB) or away from the body (AB). TB gestures express mental states (intransitive) or hygiene/nutritional activities (pantomime), while AB gestures modify the behaviour of the observer (intransitive) or demonstrate tool use with an object (pantomime). A substantial body of literature suggests that females exhibit stronger social cue processing compared to males. Considering the social significance of gestures, this study aimed to explore the physiological gender differences in the observation of AB and TB gestures. Pupil dilation and High Frequency Heart Rate Variability (HF-HRV) were measured in 54 participants (27 female) while observing TB and AB gestures. The Interpersonal Reactivity Index (IRI) and the Vicarious Distress Questionnaire (VDQ) were used to assess social-emotional processes. Results showed greater pupil dilation in females for TB gestures, but no significant gender differences for HF-HRV. Males showed a significant correlation between increased pupil dilation to both TB and AB gestures and empathy levels (IRI). The support scale of the VDQ correlated significantly with TB gestures in males. These findings provide insight into the neurobiological basis of gender differences in perceiving social gestures.

Early gesture development as a predictor of autism spectrum disorder in elevated-likelihood infants of ASD.

BACKGROUND: Gesture difficulties have been reported in later-born siblings of children with autism spectrum disorder (ASD). Careful observation of gesture development during the first two years of children at elevated likelihood (EL) of developing ASD may identify behavioral indicators that facilitate early diagnosis. METHODS: This study enrolled 47 EL infants and 27 low-likelihood (LL) infants to explore gesture developmental trajectories and the predictive value of gesture to expedite the early detection of core characteristics of ASD. Gesture frequency, communication function, and integration ability were observed and coded from a semi-structured assessment administered longitudinally across 9-19 months of age. We conducted the Autism Diagnostic Observation Schedule assessment at 18-19 months for ASD's core characteristics. RESULTS: The development of joint attention (JA) gestures was slower in the EL than in the LL group. The trajectories of the two groups began to diverge at 14-18 months. Children who reached the diagnostic cutoff point for ASD showed reductions in social interaction gestures at 12-13 months, in gestures integrated with any two communication skills (G-M) at 15-16 months; and in gestures integrated with eye contact (G-E) at 18-19 months. Overall gesture and G-M integration were associated with an overall ADOS communication and social interaction score. CONCLUSIONS: The developmental trajectories of JA gestures of EL and LL children differed. G-M gestures represent early indicators that may be a predictor of ASD.

Design of a Capacitive Tactile Sensor Array System for Human-Computer Interaction.

This paper introduces a novel capacitive sensor array designed for tactile perception applications. Utilizing an all-in-one inkjet deposition printing process, the sensor array exhibited exceptional flexibility and accuracy. With a resolution of up to 32.7 dpi, the sensor array was capable of capturing the fine details of touch inputs, making it suitable for applications requiring high spatial resolution. The design incorporates two multiplexers to achieve a scanning rate of 100 Hz, ensuring the rapid and responsive data acquisition that is essential for real-time feedback in interactive applications, such as gesture recognition and haptic interfaces. To evaluate the performance of the capacitive sensor array, an experiment that involved handwritten number recognition was conducted. The results demonstrated that the sensor accurately captured fingertip inputs with a high precision. When combined with an Auxiliary Classifier Generative Adversarial Network (ACGAN) algorithm, the sensor system achieved a recognition accuracy of 98% for various handwritten numbers from "0" to "9". These results show the potential of the capacitive sensor array for advanced human-computer interaction applications.

Development of Dual-Arm Human Companion Robots That Can Dance.

As gestures play an important role in human communication, there have been a number of service robots equipped with a pair of human-like arms for gesture-based human-robot interactions. However, the arms of most human companion robots are limited to slow and simple gestures due to the low maximum velocity of the arm actuators. In this work, we present the JF-2 robot, a mobile home service robot equipped with a pair of torque-controlled anthropomorphic arms. Thanks to the low inertia design of the arm, responsive Quasi-Direct Drive (QDD) actuators, and active compliant control of the joints, the robot can replicate fast human dance motions while being safe in the environment. In addition to the JF-2 robot, we also present the JF-mini robot, a scaled-down, low-cost version of the JF-2 robot mainly targeted for commercial use at kindergarten and childcare facilities. The suggested system is validated by performing three experiments, a safety test, teaching children how to dance along to the music, and bringing a requested item to a human subject.

Pattern Recognition in the Processing of Electromyographic Signals for Selected Expressions of Polish Sign Language.

Gesture recognition has become a significant part of human-machine interaction, particularly when verbal interaction is not feasible. The rapid development of biomedical sensing and machine learning algorithms, including electromyography (EMG) and convolutional neural networks (CNNs), has enabled the interpretation of sign languages, including the Polish Sign Language, based on EMG signals. The objective was to classify the game control gestures and Polish Sign Language gestures recorded specifically for this study using two different data acquisition systems: BIOPAC MP36 and MyoWare 2.0. We compared the classification performance of various machine learning algorithms, with a particular emphasis on CNNs on the dataset of EMG signals representing 24 gestures, recorded using both types of EMG sensors. The results (98.324% versus ≤7.8571% and 95.5307% versus ≤10.2697% of accuracy for CNNs and other classifiers in data recorded with BIOPAC MP36 and MyoWare, respectively) indicate that CNNs demonstrate superior accuracy. These results suggest the feasibility of using lower-cost sensors for effective gesture classification and the viability of integrating affordable EMG-based technologies into broader gesture recognition frameworks, providing a cost-effective solution for real-world applications. The dataset created during the study offers a basis for future studies on EMG-based recognition of Polish Sign Language.

Hand Gesture Recognition Using Ultrasonic Array with Machine Learning.

In the field of gesture recognition technology, accurately detecting human gestures is crucial. In this research, ultrasonic transducers were utilized for gesture recognition. Due to the wide beamwidth of ultrasonic transducers, it is difficult to effectively distinguish between multiple objects within a single beam. However, they are effective at accurately identifying individual objects. To leverage this characteristic of the ultrasonic transducer as an advantage, this research involved constructing an ultrasonic array. This array was created by arranging eight transmitting transducers in a circular formation and placing a single receiving transducer at the center. Through this, a wide beam area was formed extensively, enabling the measurement of unrestricted movement of a single hand in the X, Y, and Z axes. Hand gesture data were collected at distances of 10 cm, 30 cm, 50 cm, 70 cm, and 90 cm from the array. The collected data were trained and tested using a customized Convolutional Neural Network (CNN) model, demonstrating high accuracy on raw data, which is most suitable for immediate interaction with computers. The proposed system achieved over 98% accuracy.

The Key Role of Gestures in Spatial Tasks for Students With Intellectual Disability.

The gestures produced by children with intellectual disability (ID) in spatial tasks are rarely considered, although they have a supporting role in the formation of thought. In this research study, we analyzed the number of gestures, the type of gestures, and their role in the expression of knowledge of students with ID. Twenty students (12-17 years old) with ID and 40 students with typical development (TD) matched on visual-spatial level (n = 20) and on language level (n = 20) participated in the research. Students with ID made significantly more gestures in relation to the number of words spoken compared to their peers with TD. Thirty percent of the expressive communication of students with ID came from gestures alone, and 60% of the responses contained at least one gesture. Finally, the higher the level of task difficulty, the more gestures the students made.

Social negotiation and "accents" in Western lowland gorillas' gestural communication.

Recent findings on chimpanzee infants' gestural development show that they use some gesture types flexibly and adjust them depending on their interaction partner and social context, suggesting that gestural communication is partly learnt and partly genetically determined. However, how gesture types are shaped by social and demographic factors remains unclear. We addressed this question by focusing on gesture type morphology and conducted a fined-grained analysis of gestural form during intraspecific social-play interactions in two captive groups of Western lowland gorillas (Gorilla gorilla gorilla). We focused on the most frequent gesture types (BEAT CHEST, SLAP BODY, SLAP GROUND and TOUCH BODY) produced by subadults (infants, juveniles and adolescents). We considered twelve morphological gesture characteristics (e.g., horizontal and vertical hand trajectories, fingers flexion and spread). Our multifactorial investigation shows that morphological characteristics of distinct gesture types can be shaped by social factors, namely signaller's sociodemographic characteristics (group and kinship), signaller's behavioural characteristics (body posture) and context-related characteristics (recipient's sex, attentional state and position in the signaller's visual field). We nurtured the lively debate concerning gesture origins by revealing the existence of "accents" in non-verbal communication and the highly variable adjustment of gestural form to different conspecifics and interactional characteristics, which supports the revised social negotiation hypothesis.

A Novel Multi-Feature Fusion Network With Spatial Partitioning Strategy and Cross-Attention for Armband-Based Gesture Recognition.

Effectively integrating the time-space-frequency information of multi-modal signals from armband sensor, including surface electromyogram (sEMG) and accelerometer data, is critical for accurate gesture recognition. Existing approaches often neglect the abundant spatial relationships inherent in multi-channel sEMG signals obtained via armband sensors and face challenges in harnessing the correlations across multiple feature domains. To address this issue, we propose a novel multi-feature fusion network with spatial partitioning strategy and cross-attention (MFN-SPSCA) to improve the accuracy and robustness of gesture recognition. Specifically, a spatiotemporal graph convolution module with a spatial partitioning strategy is designed to capture potential spatial feature of multi-channel sEMG signals. Additionally, we design a cross-attention fusion module to learn and prioritize the importance and correlation of multi-feature domain. Extensive experiment demonstrate that the MFN-SPSCA method outperforms other state-of-the-art methods on self-collected dataset and the Ninapro DB5 dataset. Our work addresses the challenge of recognizing gestures from the multi-modal data collected by armband sensor, emphasizing the importance of integrating time-space-frequency information. Codes are available at https://github.com/ZJUTofBrainIntelligence/MFN-SPSCA.

A Multimodal Bracelet to Acquire Muscular Activity and Gyroscopic Data to Study Sensor Fusion for Intent Detection.

Researchers have attempted to control robotic hands and prostheses through biosignals but could not match the human hand. Surface electromyography records electrical muscle activity using non-invasive electrodes and has been the primary method in most studies. While surface electromyography-based hand motion decoding shows promise, it has not yet met the requirements for reliable use. Combining different sensing modalities has been shown to improve hand gesture classification accuracy. This work introduces a multimodal bracelet that integrates a 24-channel force myography system with six commercial surface electromyography sensors, each containing a six-axis inertial measurement unit. The device's functionality was tested by acquiring muscular activity with the proposed device from five participants performing five different gestures in a random order. A random forest model was then used to classify the performed gestures from the acquired signal. The results confirmed the device's functionality, making it suitable to study sensor fusion for intent detection in future studies. The results showed that combining all modalities yielded the highest classification accuracies across all participants, reaching 92.3±2.6% on average, effectively reducing misclassifications by 37% and 22% compared to using surface electromyography and force myography individually as input signals, respectively. This demonstrates the potential benefits of sensor fusion for more robust and accurate hand gesture classification and paves the way for advanced control of robotic and prosthetic hands.

Real-Time Hand Gesture Monitoring Model Based on MediaPipe's Registerable System.

Hand gesture recognition plays a significant role in human-to-human and human-to-machine interactions. Currently, most hand gesture detection methods rely on fixed hand gesture recognition. However, with the diversity and variability of hand gestures in daily life, this paper proposes a registerable hand gesture recognition approach based on Triple Loss. By learning the differences between different hand gestures, it can cluster them and identify newly added gestures. This paper constructs a registerable gesture dataset (RGDS) for training registerable hand gesture recognition models. Additionally, it proposes a normalization method for transforming hand gesture data and a FingerComb block for combining and extracting hand gesture data to enhance features and accelerate model convergence. It also improves ResNet and introduces FingerNet for registerable single-hand gesture recognition. The proposed model performs well on the RGDS dataset. The system is registerable, allowing users to flexibly register their own hand gestures for personalized gesture recognition.

Machine Learning-Assisted Gesture Sensor Made with Graphene/Carbon Nanotubes for Sign Language Recognition.

Gesture sensors are essential to collect human movements for human-computer interfaces, but their application is normally hampered by the difficulties in achieving high sensitivity and an ultrawide response range simultaneously. In this article, inspired by the spider silk structure in nature, a novel gesture sensor with a core-shell structure is proposed. The sensor offers a high gauge factor of up to 340 and a wide response range of 60%. Moreover, the sensor combining with a deep learning technique creates a system for precise gesture recognition. The system demonstrated an impressive 99% accuracy in single gesture recognition tests. Meanwhile, by using the sliding window technology and large language model, a high performance of 97% accuracy is achieved in continuous sentence recognition. In summary, the proposed high-performance sensor significantly improves the sensitivity and response range of the gesture recognition sensor. Meanwhile, the neural network technology is combined to further improve the way of daily communication by sign language users.

RingGesture: A Ring-Based Mid-Air Gesture Typing System Powered by a Deep-Learning Word Prediction Framework.

Text entry is a critical capability for any modern computing experience, with lightweight augmented reality (AR) glasses being no exception. Designed for all-day wearability, a limitation of lightweight AR glass is the restriction to the inclusion of multiple cameras for extensive field of view in hand tracking. This constraint underscores the need for an additional input device. We propose a system to address this gap: a ring-based mid-air gesture typing technique, RingGesture, utilizing electrodes to mark the start and end of gesture trajectories and inertial measurement units (IMU) sensors for hand tracking. This method offers an intuitive experience similar to raycast-based mid-air gesture typing found in VR headsets, allowing for a seamless translation of hand movements into cursor navigation. To enhance both accuracy and input speed, we propose a novel deep-learning word prediction framework, Score Fusion, comprised of three key components: a) a word-gesture decoding model, b) a spatial spelling correction model, and c) a lightweight contextual language model. In contrast, this framework fuses the scores from the three models to predict the most likely words with higher precision. We conduct comparative and longitudinal studies to demonstrate two key findings: firstly, the overall effectiveness of RingGesture, which achieves an average text entry speed of 27.3 words per minute (WPM) and a peak performance of 47.9 WPM. Secondly, we highlight the superior performance of the Score Fusion framework, which offers a 28.2% improvement in uncorrected Character Error Rate over a conventional word prediction framework, Naive Correction, leading to a 55.2% improvement in text entry speed for RingGesture. Additionally, RingGesture received a System Usability Score of 83 signifying its excellent usability.

Self-Healing and Shear-Stiffening Electrodes for Wearable Biopotential Sensing and Gesture Recognition.

The achievement of flexible skin electrodes for dynamic monitoring of biopotential is one of the challenging issues in flexible electronics due to the interference of large acceleration and heavy sweat that influence the stability of skin-electrode interfaces. This work presents materials and techniques to achieve self-healing and shear-stiffening electrodes and an associated flexible system that can be used for multichannel biopotential measurement on the skin. The electrode that is based on a composite of silver (Ag) flakes, Ag nanowires, and polyborosiloxane offers an electrical conductivity of 9.71 × 104 S/m and a rheological characteristic that ensures stable and fully conformal contact with skin and easy removal under different shear rates. The electrode can maintain its conductivity even after being stretched by more than 60% and becomes self-healed after mechanical damage. The combination of the electrodes with a screen-printed multichannel flexible sensor allows stable monitoring of both static and dynamic electromyography signals, leading to the acquisition of high-quality multilead biopotential signals that can be readily extracted to yield gesture recognition results with over 97.42% accuracy. The conductive self-healing materials and flexible sensors may be utilized in various daily biopotential sensing applications, allowing highly stable dynamic measurement to facilitate artificial intelligence-enabled health condition diagnosis and human-computer interface.

Temporal Coordination of Articulatory Gestures in Nonnative Onset Clusters: Evidence From American English Speakers Using Electromagnetic Articulography.

PURPOSE: Research in cross-language speech production indicates that, although the production of nonnative consonant clusters is often difficult, speakers of American English can produce some nonnative clusters (e.g., /fn/) with high accuracy. This ease of production for select nonnative clusters may occur due to similarity of phonetic structure with native clusters (e.g., nonnative /fn/ and native /sm/ are both fricative-nasal sequences). The current study tested this hypothesis by examining the extent of transfer of articulatory coordination from phonetically similar native onset clusters (i.e., /fl/, /sm/) to nonnative /fn/ clusters. METHOD: Using electromagnetic articulography, lip, tongue, and jaw movements were recorded in nine participants during the production of 22 nonwords (eight tokens per nonword) containing the native and nonnative clusters in different carrier phrases. We examined the temporal lags between each consonantal gesture in a cluster and the flanking vowel gesture, which were compared to the matched singleton conditions. RESULTS: Analyses revealed that, as in native speech, when the syllable onset became more complex (i.e., CV ➔ CCV [C as consonant, V as vowel]), there was an increase in lag (less temporal overlap) between the leftmost consonantal gesture and the vocalic gesture, whereas there was a decrease in lag (more temporal overlap) between the rightmost consonant and the vocalic gesture (i.e., C-center timing). However, the segmental makeup of the cluster and type of carrier phrase used were also found to influence this change in temporal organization, raising new questions for future research. CONCLUSIONS: By and large, the findings are in agreement with the idea that the temporal coordination of articulator movements may be transferred from native clusters to phonetically similar nonnative clusters. However, kinematic measures of a broader range of nonnative clusters in different contexts are needed to fully explore this position.

Kin Cognition and Communication: What Talking, Gesturing, and Drawing About Family Can Tell us About the Way We Think About This Core Social Structure.

When people talk about kinship systems, they often use co-speech gestures and other representations to elaborate. This paper investigates such polysemiotic (spoken, gestured, and drawn) descriptions of kinship relations, to see if they display recurring patterns of conventionalization that capture specific social structures. We present an exploratory hypothesis-generating study of descriptions produced by a lesser-known ethnolinguistic community to the cognitive sciences: the Paamese people of Vanuatu. Forty Paamese speakers were asked to talk about their family in semi-guided kinship interviews. Analyses of the speech, gesture, and drawings produced during these interviews revealed that lineality (i.e., mother's side vs. father's side) is lateralized in the speaker's gesture space. In other words, kinship members of the speaker's matriline are placed on the left side of the speaker's body and those of the patriline are placed on their right side, when they are mentioned in speech. Moreover, we find that the gesture produced by Paamese participants during verbal descriptions of marital relations are performed significantly more often on two diagonal directions of the sagittal axis. We show that these diagonals are also found in the few diagrams that participants drew on the ground to augment their verbo-gestural descriptions of marriage practices with drawing. We interpret this behavior as evidence of a spatial template, which Paamese speakers activate to think and communicate about family relations. We therefore argue that extending investigations of kinship structures beyond kinship terminologies alone can unveil additional key factors that shape kinship cognition and communication and hereby provide further insights into the diversity of social structures.

Gesture Recognition Framework for Teleoperation of Infrared (IR) Consumer Devices Using a Novel pFMG Soft Armband.

Wearable technologies represent a significant advancement in facilitating communication between humans and machines. Powered by artificial intelligence (AI), human gestures detected by wearable sensors can provide people with seamless interaction with physical, digital, and mixed environments. In this paper, the foundations of a gesture-recognition framework for the teleoperation of infrared consumer electronics are established. This framework is based on force myography data of the upper forearm, acquired from a prototype novel soft pressure-based force myography (pFMG) armband. Here, the sub-processes of the framework are detailed, including the acquisition of infrared and force myography data; pre-processing; feature construction/selection; classifier selection; post-processing; and interfacing/actuation. The gesture recognition system is evaluated using 12 subjects' force myography data obtained whilst performing five classes of gestures. Our results demonstrate an inter-session and inter-trial gesture average recognition accuracy of approximately 92.2% and 88.9%, respectively. The gesture recognition framework was successfully able to teleoperate several infrared consumer electronics as a wearable, safe and affordable human-machine interface system. The contribution of this study centres around proposing and demonstrating a user-centred design methodology to allow direct human-machine interaction and interface for applications where humans and devices are in the same loop or coexist, as typified between users and infrared-communicating devices in this study.