Motion capture device reveals a quick learning curve in vascular anastomosis training.

PURPOSE: Surgical procedures are often evaluated subjectively, and an objective evaluation has been considered difficult to make and rarely reported, especially in open surgery, where the range of motion is wide. This study evaluated the effectiveness of surgical suturing training as an educational tool using the Leap Motion Controller (LMC), which can capture hand movements and reproduce them as data comprising parametric elements. METHODS: We developed an off-the-job training system (Off-JT) in our department, mainly using prosthetic grafts and various anastomotic methodologies with graded difficulty levels. We recruited 50 medical students (novice group) and 6 vascular surgeons (expert group) for the study. We evaluated four parameters for intraoperative skills: suturing time, slope of the roll, smoothness, and rate of excess motion. RESULTS: All 4 parameters distinguished the skill of the novice group at 1 and 10 h off-JT. After 10 h of off-JT, all 4 parameters of the novices were comparable to those of the expert group. CONCLUSION: Our education system using the LMC is relatively inexpensive and easy to set up, with a free application for analyses, serving as an effective and ubiquitous educational tool for young surgeons.

Use of low-cost virtual reality in the treatment of the upper extremity in chronic stroke: a randomized clinical trial.

BACKGROUND: Chronicity and lack of motivation often go together during the upper limb rehabilitation process in stroke. Virtual reality is a useful tool in this context, providing safe, intensive, individualised treatments in a playful environment. B-cost, easy-to-use devices with personalised and motivating games for a specific population seem to be the most effective option in the treatment of the upper limbs. METHODS: A randomised clinical study with follow-up was carried out to assess the effectiveness of the Leap Motion Controller® device in improving the functionality of the upper limb in patients with chronic stroke. Patients (n = 36) were randomised into a control group that performed conventional therapy and an experimental group that combined the virtual reality protocol with conventional therapy. The outcome measures used were grip strength; the Block and Box Test; the Action Research Arm Test; the Disabilities of the Arm, Shoulder and Hand; as well as a Technology Satisfaction Questionnaire and adherence to treatment. RESULTS: Inter-group statistical analysis showed no significant differences except in subsection D of the Action Research Arm Test. Intra-group analysis showed significant differences in both groups, but the experimental group reached significance in all long-term variables. Satisfaction and adherence levels were very high. CONCLUSIONS: The Leap Motion Controller® system, as a complementary tool, produces improvements in grip strength, dexterity and motor function in patients with chronic stroke. It is perceived as a safe, motivating, and easy-to-use device. CLINICAL REGISTRATION: NCT04166617 Clinical Trials.

Portable Head-Mounted System for Mobile Forearm Tracking.

Computer vision (CV)-based systems using cameras and recognition algorithms offer touchless, cost-effective, precise, and versatile hand tracking. These systems allow unrestricted, fluid, and natural movements without the constraints of wearable devices, gaining popularity in human-system interaction, virtual reality, and medical procedures. However, traditional CV-based systems, relying on stationary cameras, are not compatible with mobile applications and demand substantial computing power. To address these limitations, we propose a portable hand-tracking system utilizing the Leap Motion Controller 2 (LMC) mounted on the head and controlled by a single-board computer (SBC) powered by a compact power bank. The proposed system enhances portability, enabling users to interact freely with their surroundings. We present the system's design and conduct experimental tests to evaluate its robustness under variable lighting conditions, power consumption, CPU usage, temperature, and frame rate. This portable hand-tracking solution, which has minimal weight and runs independently of external power, proves suitable for mobile applications in daily life.

A Model of Multi-Finger Coordination in Keystroke Movement.

In multi-finger coordinated keystroke actions by professional pianists, movements are precisely regulated by multiple motor neural centers, exhibiting a certain degree of coordination in finger motions. This coordination enhances the flexibility and efficiency of professional pianists' keystrokes. Research on the coordination of keystrokes in professional pianists is of great significance for guiding the movements of piano beginners and the motion planning of exoskeleton robots, among other fields. Currently, research on the coordination of multi-finger piano keystroke actions is still in its infancy. Scholars primarily focus on phenomenological analysis and theoretical description, which lack accurate and practical modeling methods. Considering that the tendon of the ring finger is closely connected to adjacent fingers, resulting in limited flexibility in its movement, this study concentrates on coordinated keystrokes involving the middle and ring fingers. A motion measurement platform is constructed, and Leap Motion is used to collect data from 12 professional pianists. A universal model applicable to multiple individuals for multi-finger coordination in keystroke actions based on the backpropagation (BP) neural network is proposed, which is optimized using a genetic algorithm (GA) and a sparrow search algorithm (SSA). The angular rotation of the ring finger's MCP joint is selected as the model output, while the individual difference information and the angular data of the middle finger's MCP joint serve as inputs. The individual difference information used in this study includes ring finger length, middle finger length, and years of piano training. The results indicate that the proposed SSA-BP neural network-based model demonstrates superior predictive accuracy, with a root mean square error of 4.8328°. Based on this model, the keystroke motion of the ring finger's MCP joint can be accurately predicted from the middle finger's keystroke motion information, offering an evaluative method and scientific guidance for the training of multi-finger coordinated keystrokes in piano learners.

Analysis of the Leap Motion Controller's Performance in Measuring Wrist Rehabilitation Tasks Using an Industrial Robot Arm Reference.

The Leap Motion Controller (LMC) is a low-cost markerless optical sensor that performs measurements of various parameters of the hands that has been investigated for a wide range of different applications. Research attention still needs to focus on the evaluation of its precision and accuracy to fully understand its limitations and widen its range of applications. This paper presents the experimental validation of the LMC device to verify the feasibility of its use in assessing and tailoring wrist rehabilitation therapy for the treatment of physical disabilities through continuous exercises and integration with serious gaming environments. An experimental set up and analysis is proposed using an industrial robot as motion reference. The high repeatability of the selected robot is used for comparisons with the measurements obtained via a leap motion controller while performing the basic movements needed for rehabilitation exercises of the human wrist. Experimental tests are analyzed and discussed to demonstrate the feasibility of using the leap motion controller for wrist rehabilitation.

Hand Measurement System Based on Haptic and Vision Devices towards Post-Stroke Patients.

Diagnostics of a hand requires measurements of kinematics and joint limits. The standard tools for this purpose are manual devices such as goniometers which allow measuring only one joint simultaneously, making the diagnostics time-consuming. The paper presents a system for automatic measurement and computer presentation of essential parameters of a hand. Constructed software uses an integrated vision system, a haptic device for measurement, and has a web-based user interface. The system provides a simplified way to obtain hand parameters, such as hand size, wrist, and finger range of motions, using the homogeneous-matrix-based notation. The haptic device allows for active measurement of the wrist's range of motion and additional force measurement. A study was conducted to determine the accuracy and repeatability of measurements compared to the gold standard. The system functionality was confirmed on five healthy participants, with results showing comparable results to manual measurements regarding fingers' lengths. The study showed that the finger's basic kinematic structure could be measured by a vision system with a mean difference to caliper measurement of 4.5 mm and repeatability with the Standard Deviations up to 0.7 mm. Joint angle limits measurement achieved poorer results with a mean difference to goniometer of 23.6º. Force measurements taken by the haptic device showed the repeatability with a Standard Deviation of 0.7 N. The presented system allows for a unified measurement and a collection of important parameters of a human hand with therapist interface visualization and control with potential use for post-stroke patients' precise rehabilitation.

Classification of Activities of Daily Living Based on Grasp Dynamics Obtained from a Leap Motion Controller.

Stroke is one of the leading causes of mortality and disability worldwide. Several evaluation methods have been used to assess the effects of stroke on the performance of activities of daily living (ADL). However, these methods are qualitative. A first step toward developing a quantitative evaluation method is to classify different ADL tasks based on the hand grasp. In this paper, a dataset is presented that includes data collected by a leap motion controller on the hand grasps of healthy adults performing eight common ADL tasks. Then, a set of features with time and frequency domains is combined with two well-known classifiers, i.e., the support vector machine and convolutional neural network, to classify the tasks, and a classification accuracy of over 99% is achieved.

American Sign Language Words Recognition of Skeletal Videos Using Processed Video Driven Multi-Stacked Deep LSTM.

Complex hand gesture interactions among dynamic sign words may lead to misclassification, which affects the recognition accuracy of the ubiquitous sign language recognition system. This paper proposes to augment the feature vector of dynamic sign words with knowledge of hand dynamics as a proxy and classify dynamic sign words using motion patterns based on the extracted feature vector. In this method, some double-hand dynamic sign words have ambiguous or similar features across a hand motion trajectory, which leads to classification errors. Thus, the similar/ambiguous hand motion trajectory is determined based on the approximation of a probability density function over a time frame. Then, the extracted features are enhanced by transformation using maximal information correlation. These enhanced features of 3D skeletal videos captured by a leap motion controller are fed as a state transition pattern to a classifier for sign word classification. To evaluate the performance of the proposed method, an experiment is performed with 10 participants on 40 double hands dynamic ASL words, which reveals 97.98% accuracy. The method is further developed on challenging ASL, SHREC, and LMDHG data sets and outperforms conventional methods by 1.47%, 1.56%, and 0.37%, respectively.

Backhand-Approach-Based American Sign Language Words Recognition Using Spatial-Temporal Body Parts and Hand Relationship Patterns.

Most of the existing methods focus mainly on the extraction of shape-based, rotation-based, and motion-based features, usually neglecting the relationship between hands and body parts, which can provide significant information to address the problem of similar sign words based on the backhand approach. Therefore, this paper proposes four feature-based models. The spatial-temporal body parts and hand relationship patterns are the main feature. The second model consists of the spatial-temporal finger joint angle patterns. The third model consists of the spatial-temporal 3D hand motion trajectory patterns. The fourth model consists of the spatial-temporal double-hand relationship patterns. Then, a two-layer bidirectional long short-term memory method is used to deal with time-independent data as a classifier. The performance of the method was evaluated and compared with the existing works using 26 ASL letters, with an accuracy and F1-score of 97.34% and 97.36%, respectively. The method was further evaluated using 40 double-hand ASL words and achieved an accuracy and F1-score of 98.52% and 98.54%, respectively. The results demonstrated that the proposed method outperformed the existing works under consideration. However, in the analysis of 72 new ASL words, including single- and double-hand words from 10 participants, the accuracy and F1-score were approximately 96.99% and 97.00%, respectively.

Use of the Leap Motion Controller® System in the Rehabilitation of the Upper Limb in Stroke. A Systematic Review.

OBJECTIVES: Upper limb impairment is the most common motor impairment in stroke survivors. The use of new technologies in the field of rehabilitation aims to reduce the impact of functional problems. Our objective is to evaluate the effectiveness of using the Leap Motion Controller® virtual reality system in the treatment of upper limb functionality in people with stroke. MATERIALS AND METHODS: PRISMA guidelines were used to carry out the systematic review. The literature search was restricted to articles written in English or Spanish published from 2012 to December 2020 in Pubmed, Web of Science, Scopus, PEDro and Science Direct. Of the 309 search results, 230 unique references were reviewed after duplicates were removed. The Downs and Black and CONSORT scales were applied to evaluate the methodological quality of the included papers and the degree of evidence and level of recommendation were determined through the Oxford Centre for Evidence-Based Medicine. RESULTS: Six papers with a total of 144 participants were included in this review, with heterogeneity of the sample, assessment measures, protocols, number of sessions and diversity of games applied. The main results of the studies show favourable data after using the Leap Motion Controller® system in the improvement of upper limb functionality in people with stroke. CONCLUSIONS: There is a growing trend in the use of the Leap Motion Controller® device as a tool in the treatment of the upper limb in people with stroke. Nevertheless, the limitations encountered suggest the need for future research protocols with greater scientific rigor.

Comparison of the Performance of the Leap Motion ControllerTM with a Standard Marker-Based Motion Capture System.

Over the last few years, the Leap Motion Controller™ (LMC) has been increasingly used in clinical environments to track hand, wrist and forearm positions as an alternative to the gold-standard motion capture systems. Since the LMC is marker-less, portable, easy-to-use and low-cost, it is rapidly being adopted in healthcare services. This paper demonstrates the comparison of finger kinematic data between the LMC and a gold-standard marker-based motion capture system, Qualisys Track Manager (QTM). Both systems were time synchronised, and the participants performed abduction/adduction of the thumb and flexion/extension movements of all fingers. The LMC and QTM were compared in both static measuring finger segment lengths and dynamic flexion movements of all fingers. A Bland-Altman plot was used to demonstrate the performance of the LMC versus QTM with Pearson's correlation (r) to demonstrate trends in the data. Only the proximal interphalangeal joint (PIP) joint of the middle and ring finger during flexion/extension demonstrated acceptable agreement (r = 0.9062; r = 0.8978), but with a high mean bias. In conclusion, the study shows that currently, the LMC is not suitable to replace gold-standard motion capture systems in clinical settings. Further studies should be conducted to validate the performance of the LMC as it is updated and upgraded.

Research on Discrete Semantics in Continuous Hand Joint Movement Based on Perception and Expression.

Continuous movements of the hand contain discrete expressions of meaning, forming a variety of semantic gestures. For example, it is generally considered that the bending of the finger includes three semantic states of bending, half bending, and straightening. However, there is still no research on the number of semantic states that can be conveyed by each movement primitive of the hand, especially the interval of each semantic state and the representative movement angle. To clarify these issues, we conducted experiments of perception and expression. Experiments 1 and 2 focused on perceivable semantic levels and boundaries of different motion primitive units from the perspective of visual semantic perception. Experiment 3 verified and optimized the segmentation results obtained above and further determined the typical motion values of each semantic state. Furthermore, in Experiment 4, the empirical application of the above semantic state segmentation was illustrated by using Leap Motion as an example. We ended up with the discrete gesture semantic expression space both in the real world and Leap Motion Digital World, containing the clearly defined number of semantic states of each hand motion primitive unit and boundaries and typical motion angle values of each state. Construction of this quantitative semantic expression will play a role in guiding and advancing research in the fields of gesture coding, gesture recognition, and gesture design.

Hand Motion Capture from a 3D Leap Motion Controller for a Musculoskeletal Dynamic Simulation.

The AnyBody Modeling System™ (AMS) is a musculoskeletal software simulation solution using inverse dynamics analysis. It enables the determination of muscle and joint forces for a given bodily motion. The recording of the individual movement and the transfer into the AMS is a complex and protracted process. Researches indicated that the contactless, visual Leap Motion Controller (LMC) provides clinically meaningful motion data for hand tracking. Therefore, the aim of this study was to integrate the LMC hand motion data into the AMS in order to improve the process of recording a hand movement. A Python-based interface between the LMC and the AMS, termed ROSE Motion, was developed. This solution records and saves the data of the movement as Biovision Hierarchy (BVH) data and AnyScript vector files that are imported into the AMS simulation. Setting simulation parameters, initiating the calculation automatically, and fetching results is implemented by using the AnyPyTools library from AnyBody. The proposed tool offers a rapid and easy-to-use recording solution for elbow, hand, and finger movements. Features include animation, cutting/editing, exporting the motion, and remote controlling the AMS for the analysis and presentation of musculoskeletal simulation results. Comparing the motion tracking results with previous studies, covering problems when using the LMC limit the correctness of the motion data. However, fast experimental setup and intuitive and rapid motion data editing strengthen the use of marker less systems as the herein presented compared to marker based motion capturing.

Intuitive Cognition-Based Method for Generating Speech Using Hand Gestures.

Muteness at its various levels is a common disability. Most of the technological solutions to the problem creates vocal speech through the transition from mute languages to vocal acoustic sounds. We present a new approach for creating speech: a technology that does not require prior knowledge of sign language. This technology is based on the most basic level of speech according to the phonetic division into vowels and consonants. The speech itself is expected to be expressed through sensing of the hand movements, as the movements are divided into three rotations: yaw, pitch, and roll. The proposed algorithm converts these rotations through programming to vowels and consonants. For the hand movement sensing, we used a depth camera and standard speakers in order to produce the sounds. The combination of the programmed depth camera and the speakers, together with the cognitive activity of the brain, is integrated into a unique speech interface. Using this interface, the user can develop speech through an intuitive cognitive process in accordance with the ongoing brain activity, similar to the natural use of the vocal cords. Based on the performance of the presented speech interface prototype, it is substantiated that the proposed device could be a solution for those suffering from speech disabilities.

Leap Motion Controller Video Game-Based Therapy for Upper Extremity Motor Recovery in Patients with Central Nervous System Diseases. A Systematic Review with Meta-Analysis.

Leap Motion Controller (LMC) is a virtual reality device that can be used in the rehabilitation of central nervous system disease (CNSD) motor impairments. This review aimed to evaluate the effect of video game-based therapy with LMC on the recovery of upper extremity (UE) motor function in patients with CNSD. A systematic review with meta-analysis was performed in PubMed Medline, Web of Science, Scopus, CINAHL, and PEDro. We included five randomized controlled trials (RCTs) of patients with CNSD in which LMC was used as experimental therapy compared to conventional therapy (CT) to restore UE motor function. Pooled effects were estimated with Cohen's standardized mean difference (SMD) and its 95% confidence interval (95% CI). At first, in patients with stroke, LMC showed low-quality evidence of a large effect on UE mobility (SMD = 0.96; 95% CI = 0.47, 1.45). In combination with CT, LMC showed very low-quality evidence of a large effect on UE mobility (SMD = 1.34; 95% CI = 0.49, 2.19) and the UE mobility-oriented task (SMD = 1.26; 95% CI = 0.42, 2.10). Second, in patients with non-acute CNSD (cerebral palsy, multiple sclerosis, and Parkinson's disease), LMC showed low-quality evidence of a medium effect on grip strength (GS) (SMD = 0.47; 95% CI = 0.03, 0.90) and on gross motor dexterity (GMD) (SMD = 0.73; 95% CI = 0.28, 1.17) in the most affected UE. In combination with CT, LMC showed very low-quality evidence of a high effect in the most affected UE on GMD (SMD = 0.80; 95% CI = 0.06, 1.15) and fine motor dexterity (FMD) (SMD = 0.82; 95% CI = 0.07, 1.57). In stroke, LMC improved UE mobility and UE mobility-oriented tasks, and in non-acute CNSD, LMC improved the GS and GMD of the most affected UE and FMD when it was used with CT.

Effects of virtual reality associated with serious games for upper limb rehabilitation inpatients with multiple sclerosis: randomized controlled trial.

BACKGROUND: Dexterity and activities of daily living limitations on the upper limb (UL) represent one of the most common problems in patients with multiple sclerosis (MS). The aim of this study was to evaluate the effectiveness of the specially developed Serious Games that make use of the Leap Motion Controller (LMC) as main user interface for improving UL grip muscle strength, dexterity, fatigue, quality of life, satisfaction and compliance. METHODS: A single-blinded randomized controlled trial was conducted. The sample was randomized into two groups: an experimental group who received treatment based on serious games designed by the research team using the developed LMC based Serious Games for the UL plus conventional rehabilitation, and a control group who received the same conventional rehabilitation for the UL. Both groups received two 60 min sessions per week over a ten-week period. Grip muscle strength, coordination, speed of movements, fine and gross UL dexterity, fatigue, quality of life, satisfaction and compliance were assessed in both groups pre-treatment, post-treatment and in a follow-up period of 1 month without receiving any treatment. RESULTS: In the experimental group compared to the control group, significant improvements were observed in the post-treatment assessment for coordination, speed of movements, fine and gross UL dexterity. Also, significant results were found in the follow-up in coordination, speed of movements, fine and gross for the more affected side. CONCLUSIONS: An experimental protocol using an LMC based Serious Games designed for UL rehabilitation showed improvements for unilateral gross manual dexterity, fine manual dexterity, and coordination in MS patients with high satisfaction and excellent compliance. TRIAL REGISTRATION: This randomized controlled trial has been registered at ClinicalTrials.gov Identifier: NCT04171908 , Nov 2019.

Development of a Diagnosis and Evaluation System for Hemiplegic Patients Post-Stroke Based on Motion Recognition Tracking and Analysis of Wrist Joint Kinematics.

An inexperienced therapist lacks the analysis of a patient's movement. In addition, the patient does not receive objective feedback from the therapist due to the visual subjective judgment. The aim is to provide a guide for in-depth rehabilitation therapy in virtual space by continuously tracking the user's wrist joint during Leap Motion Controller (LMC) activities and present the basic data to confirm steady therapy results in real-time. The conventional Box and Block Test (BBT) is commonly used in upper extremity rehabilitation therapy. It was modeled in proportion to the actual size and Auto Desk Inventor was used to perform the 3D modeling work. The created 3D object was then implemented in C # through Unity5.6.2p4 based on LMC. After obtaining a wrist joint motion value, the motion was analyzed by 3D graph. Healthy subjects (23 males and 25 females, n = 48) were enrolled in this study. There was no statistically significant counting difference between conventional BBT and system BBT. This indicates the possibility of effective diagnosis and evaluation of hemiplegic patients post-stroke. We can keep track of wrist joints, check real-time continuous feedback in the implemented virtual space, and provide the basic data for an LMC-based quantitative rehabilitation therapy guide.

Analysis of Precision and Stability of Hand Tracking with Leap Motion Sensor.

In this analysis, we present results from measurements performed to determine the stability of a hand tracking system and the accuracy of the detected palm and finger's position. Measurements were performed for the evaluation of the sensor for an application in an industrial robot-assisted assembly scenario. Human-robot interaction is a relevant topic in collaborative robotics. Intuitive and straightforward control tools for robot navigation and program flow control are essential for effective utilisation in production scenarios without unnecessary slowdowns caused by the operator. For the hand tracking and gesture-based control, it is necessary to know the sensor's accuracy. For gesture recognition with a moving target, the sensor must provide stable tracking results. This paper evaluates the sensor's real-world performance by measuring the localisation deviations of the hand being tracked as it moves in the workspace.

Dynamic Hand Gesture Recognition Based on a Leap Motion Controller and Two-Layer Bidirectional Recurrent Neural Network.

Dynamic hand gesture recognition is one of the most significant tools for human-computer interaction. In order to improve the accuracy of the dynamic hand gesture recognition, in this paper, a two-layer Bidirectional Recurrent Neural Network for the recognition of dynamic hand gestures from a Leap Motion Controller (LMC) is proposed. In addition, based on LMC, an efficient way to capture the dynamic hand gestures is identified. Dynamic hand gestures are represented by sets of feature vectors from the LMC. The proposed system has been tested on the American Sign Language (ASL) datasets with 360 samples and 480 samples, and the Handicraft-Gesture dataset, respectively. On the ASL dataset with 360 samples, the system achieves accuracies of 100% and 96.3% on the training and testing sets. On the ASL dataset with 480 samples, the system achieves accuracies of 100% and 95.2%. On the Handicraft-Gesture dataset, the system achieves accuracies of 100% and 96.7%. In addition, 5-fold, 10-fold, and Leave-One-Out cross-validation are performed on these datasets. The accuracies are 93.33%, 94.1%, and 98.33% (360 samples), 93.75%, 93.5%, and 98.13% (480 samples), and 88.66%, 90%, and 92% on ASL and Handicraft-Gesture datasets, respectively. The developed system demonstrates similar or better performance compared to other approaches in the literature.

Perspective and Evolution of Gesture Recognition for Sign Language: A Review.

This review analyses the different gesture recognition systems through a timeline, showing the different types of technology, and specifying which are the most important features and their achieved recognition rates. At the end of the review, Leap Motion sensor possibilities are described in detail, in order to consider its application on the field of sign language. This device has many positive characteristics that make it a good option for sign language. One of the most important conclusions is the ability of the Leap Motion sensor to provide 3D information from the hands for due identification.