Portable Head-Mounted System for Mobile Forearm Tracking.

Computer vision (CV)-based systems using cameras and recognition algorithms offer touchless, cost-effective, precise, and versatile hand tracking. These systems allow unrestricted, fluid, and natural movements without the constraints of wearable devices, gaining popularity in human-system interaction, virtual reality, and medical procedures. However, traditional CV-based systems, relying on stationary cameras, are not compatible with mobile applications and demand substantial computing power. To address these limitations, we propose a portable hand-tracking system utilizing the Leap Motion Controller 2 (LMC) mounted on the head and controlled by a single-board computer (SBC) powered by a compact power bank. The proposed system enhances portability, enabling users to interact freely with their surroundings. We present the system's design and conduct experimental tests to evaluate its robustness under variable lighting conditions, power consumption, CPU usage, temperature, and frame rate. This portable hand-tracking solution, which has minimal weight and runs independently of external power, proves suitable for mobile applications in daily life.

Design of Virtual Hands for Natural Interaction in the Metaverse.

The emergence of the Metaverse is raising important questions in the field of human-machine interaction that must be addressed for a successful implementation of the new paradigm. Therefore, the exploration and integration of both technology and human interaction within this new framework are needed. This paper describes an innovative and technically viable proposal for virtual shopping in the fashion field. Virtual hands directly scanned from the real world have been integrated, after a retopology process, in a virtual environment created for the Metaverse, and have been integrated with digital nails. Human interaction with the Metaverse has been carried out through the acquisition of the real posture of the user's hands using an infrared-based sensor and mapping it in its virtualized version, achieving natural identification. The technique has been successfully tested in an immersive shopping experience with the Meta Quest 2 headset as a pilot project, where a transactions mechanism based on the blockchain technology (non-fungible tokens, NFTs) has allowed for the development of a feasible solution for massive audiences. The consumers' reactions were extremely positive, with a total of 250 in-person participants and 120 remote accesses to the Metaverse. Very interesting technical guidelines are raised in this project, the resolution of which may be useful for future implementations.

A methodological framework to assess the accuracy of virtual reality hand-tracking systems: A case study with the Meta Quest 2.

Optical markerless hand-tracking systems incorporated into virtual reality (VR) headsets are transforming the ability to assess fine motor skills in VR. This promises to have far-reaching implications for the increased applicability of VR across scientific, industrial, and clinical settings. However, so far, there are little data regarding the accuracy, delay, and overall performance of these types of hand-tracking systems. Here we present a novel methodological framework based on a fixed grid of targets, which can be easily applied to measure these systems' absolute positional error and delay. We also demonstrate a method to assess finger joint-angle accuracy. We used this framework to evaluate the Meta Quest 2 hand-tracking system. Our results showed an average fingertip positional error of 1.1cm, an average finger joint angle error of 9.6∘ and an average temporal delay of 45.0 ms. This methodological framework provides a powerful tool to ensure the reliability and validity of data originating from VR-based, markerless hand-tracking systems.

Hand Gesture Interface for Robot Path Definition in Collaborative Applications: Implementation and Comparative Study.

The article explores the possibilities of using hand gestures as a control interface for robotic systems in a collaborative workspace. The development of hand gesture control interfaces has become increasingly important in everyday life as well as professional contexts such as manufacturing processes. We present a system designed to facilitate collaboration between humans and robots in manufacturing processes that require frequent revisions of the robot path and that allows direct definition of the waypoints, which differentiates our system from the existing ones. We introduce a novel and intuitive approach to human-robot cooperation through the use of simple gestures. As part of a robotic workspace, a proposed interface was developed and implemented utilising three RGB-D sensors for monitoring the operator's hand movements within the workspace. The system employs distributed data processing through multiple Jetson Nano units, with each unit processing data from a single camera. MediaPipe solution is utilised to localise the hand landmarks in the RGB image, enabling gesture recognition. We compare the conventional methods of defining robot trajectories with their developed gesture-based system through an experiment with 20 volunteers. The experiment involved verification of the system under realistic conditions in a real workspace closely resembling the intended industrial application. Data collected during the experiment included both objective and subjective parameters. The results indicate that the gesture-based interface enables users to define a given path objectively faster than conventional methods. We critically analyse the features and limitations of the developed system and suggest directions for future research. Overall, the experimental results indicate the usefulness of the developed system as it can speed up the definition of the robot's path.

Patient-Therapist Cooperative Hand Telerehabilitation through a Novel Framework Involving the Virtual Glove System.

Telerehabilitation is important for post-stroke or post-surgery rehabilitation because the tasks it uses are reproducible. When combined with assistive technologies, such as robots, virtual reality, tracking systems, or a combination of them, it can also allow the recording of a patient's progression and rehabilitation monitoring, along with an objective evaluation. In this paper, we present the structure, from actors and functionalities to software and hardware views, of a novel framework that allows cooperation between patients and therapists. The system uses a computer-vision-based system named virtual glove for real-time hand tracking (40 fps), which is translated into a light and precise system. The novelty of this work lies in the fact that it gives the therapist quantitative, not only qualitative, information about the hand's mobility, for every hand joint separately, while at the same time providing control of the result of the rehabilitation by also quantitatively monitoring the progress of the hand mobility. Finally, it also offers a strategy for patient-therapist interaction and therapist-therapist data sharing.

Immersive virtual reality for upper limb rehabilitation: comparing hand and controller interaction.

Virtual reality shows great potential as an alternative to traditional therapies for motor rehabilitation given its ability to immerse the user in engaging scenarios that abstract them from medical facilities and tedious rehabilitation exercises. This paper presents a virtual reality application that includes three serious games and that was developed for motor rehabilitation. It uses a standalone headset and the user's hands without the need for any controller for interaction. Interacting with an immersive virtual reality environment using only natural hand gestures involves an interaction that is similar to that of real life, which would be especially desirable for patients with motor problems. A study involving 28 participants (4 with motor problems) was carried out to compare two types of interaction (hands vs. controllers). All of the participants completed the exercises. No significant differences were found in the number of attempts necessary to complete the games using the two types of interaction. The group that used controllers required less time to complete the exercise. The performance outcomes were independent of the gender and age of the participants. The subjective assessment of the participants with motor problems was not significantly different from the rest of the participants. With regard to the interaction type, the participants mostly preferred the interaction using their hands (78.5%). All four participants with motor problems preferred the hand interaction. These results suggest that the interaction with the user's hands together with standalone headsets could improve motivation, be well accepted by motor rehabilitation patients, and help to complete exercise therapy at home.

Attentional capture in goal-directed action during childhood, adolescence, and early adulthood.

Attentional capture occurs when salient but task-irrelevant information disrupts our ability to respond to task-relevant information. Although attentional capture costs have been found to decrease between childhood and adulthood, it is currently unclear the extent to which such age-related changes reflect an improved ability to recover from attentional capture or to avoid attentional capture. In addition, recent research using hand-tracking techniques with adults indicates that attentional capture by a distractor can generate response activations corresponding to the distractor's location, consistent with action-centered models of attention. However, it is unknown whether attentional capture can also result in the capture of action in children and adolescents. Therefore, we presented 5-year-olds, 9-year-olds, 13- and 14-year-olds, and adults (N = 96) with a singleton search task in which participants responded by reaching to touch targets on a digital display. Consistent with action-centered models of attention, distractor effects were evident in each age group's movement trajectories. In contrast to movement trajectories, movement times revealed significant age-related reductions in the costs of attentional capture, suggesting that age-related improvements in attentional control may be driven in part by an enhanced ability to recover from-as opposed to avoid-attentional capture. Children's performance was also significantly affected by response repetition effects, indicating that children may be more susceptible to interference from a wider range of task-irrelevant factors than adults. In addition to presenting novel insights into the development of attention and action, these results highlight the benefits of incorporating hand-tracking techniques into developmental research.

Proof of Concept for the Use of Immersive Virtual Reality in Upper Limb Rehabilitation of Multiple Sclerosis Patients.

Multiple sclerosis (MS) is a debilitating disease which gradually reduces motor function and mobility. Virtual reality (VR) has been successfully utilised in support of existing therapeutic approaches for many different conditions, and new innovative and experimental features could be the future of VR rehabilitation. The Quest is a new headset by Oculus, with its built-in tracking, relatively low cost, portability and lack of reliance on expensive processing heavy PCs to power it, and could be an ideal system to facilitate at-home or clinic-based upper limb rehabilitation. A hand-tracking-based rehabilitation game aimed at people with MS was developed for Oculus Quest using Unity. Two distinct games were made to replicate different types of hand exercises, piano playing for isolated finger flexion and maze tracking for coordination and arm flexion. This pilot study assesses the value of such approach along with evaluating intrinsic and extrinsic methods of providing feedback, namely, positive scoring, negative scoring and audio response. One physiotherapist and two individuals with MS were surveyed. Participant response was positive although small sample size impacts the user testing validity of the results. Future research is recommended to build off the data gathered as a pilot study and increase sample size to collect richer feedback.

Composition and decomposition of visuomotor maps during manual tracking.

Adapting hand movements to changes in our body or the environment is essential for skilled motor behavior, as is the ability to flexibly combine experience gathered in separate contexts. However, it has been shown that when adapting hand movements to two different visuomotor perturbations in succession, interference effects can occur. Here, we investigate whether these interference effects compromise our ability to adapt to the superposition of the two perturbations. Participants tracked with a joystick, a visual target that followed a smooth but an unpredictable trajectory. Four separate groups of participants (total n = 83) completed one block of 50 trials under each of three mappings: one in which the cursor was rotated by 90° (ROTATION), one in which the cursor mimicked the behavior of a mass-spring system (SPRING), and one in which the SPRING and ROTATION mappings were superimposed (SPROT). The order of the blocks differed across groups. Although interference effects were found when switching between SPRING and ROTATION, participants who performed these blocks first performed better in SPROT than participants who had no prior experience with SPRING and ROTATION (i.e., composition). Moreover, participants who started with SPROT exhibited better performance under SPRING and ROTATION than participants who had no prior experience with each of these mappings (i.e., decomposition). Additional analyses confirmed that these effects resulted from components of learning that were specific to the rotational and spring perturbations. These results show that interference effects do not preclude the ability to compose/decompose various forms of visuomotor adaptation.NEW & NOTEWORTHY The ability to compose/decompose task representations is critical for both cognitive and behavioral flexibility. Here, we show that this ability extends to two forms of visuomotor adaptation in which humans have to perform visually guided hand movements. Despite the presence of interference effects when switching between visuomotor maps, we show that participants are able to flexibly compose or decompose knowledge acquired in previous sessions. These results further demonstrate the flexibility of sensorimotor adaptation in humans.

3D Point-of-Intention Determination Using a Multimodal Fusion of Hand Pointing and Eye Gaze for a 3D Display.

This paper proposes a three-dimensional (3D) point-of-intention (POI) determination method using multimodal fusion between hand pointing and eye gaze for a 3D virtual display. In the method, the finger joint forms of the pointing hand sensed by a Leap Motion sensor are first detected as pointing intention candidates. Subsequently, differences with neighboring frames, which should be during hand pointing period, are checked by AND logic with the hand-pointing intention candidates. A crossing point between the eye gaze and hand pointing lines is finally decided by the closest distance concept. In order to evaluate the performance of the proposed method, experiments with ten participants, in which they looked at and pointed at nine test points for approximately five second each, were performed. The experimental results show the proposed method measures 3D POIs at 75 cm, 85 cm, and 95 cm with average distance errors of 4.67%, 5.38%, and 5.71%, respectively.

Analysis of Precision and Stability of Hand Tracking with Leap Motion Sensor.

In this analysis, we present results from measurements performed to determine the stability of a hand tracking system and the accuracy of the detected palm and finger's position. Measurements were performed for the evaluation of the sensor for an application in an industrial robot-assisted assembly scenario. Human-robot interaction is a relevant topic in collaborative robotics. Intuitive and straightforward control tools for robot navigation and program flow control are essential for effective utilisation in production scenarios without unnecessary slowdowns caused by the operator. For the hand tracking and gesture-based control, it is necessary to know the sensor's accuracy. For gesture recognition with a moving target, the sensor must provide stable tracking results. This paper evaluates the sensor's real-world performance by measuring the localisation deviations of the hand being tracked as it moves in the workspace.

Development of a Diagnosis and Evaluation System for Hemiplegic Patients Post-Stroke Based on Motion Recognition Tracking and Analysis of Wrist Joint Kinematics.

An inexperienced therapist lacks the analysis of a patient's movement. In addition, the patient does not receive objective feedback from the therapist due to the visual subjective judgment. The aim is to provide a guide for in-depth rehabilitation therapy in virtual space by continuously tracking the user's wrist joint during Leap Motion Controller (LMC) activities and present the basic data to confirm steady therapy results in real-time. The conventional Box and Block Test (BBT) is commonly used in upper extremity rehabilitation therapy. It was modeled in proportion to the actual size and Auto Desk Inventor was used to perform the 3D modeling work. The created 3D object was then implemented in C # through Unity5.6.2p4 based on LMC. After obtaining a wrist joint motion value, the motion was analyzed by 3D graph. Healthy subjects (23 males and 25 females, n = 48) were enrolled in this study. There was no statistically significant counting difference between conventional BBT and system BBT. This indicates the possibility of effective diagnosis and evaluation of hemiplegic patients post-stroke. We can keep track of wrist joints, check real-time continuous feedback in the implemented virtual space, and provide the basic data for an LMC-based quantitative rehabilitation therapy guide.

A Model-Based System for Real-Time Articulated Hand Tracking Using a Simple Data Glove and a Depth Camera.

Tracking detailed hand motion is a fundamental research topic in the area of human-computer interaction (HCI) and has been widely studied for decades. Existing solutions with single-model inputs either require tedious calibration, are expensive or lack sufficient robustness and accuracy due to occlusions. In this study, we present a real-time system to reconstruct the exact hand motion by iteratively fitting a triangular mesh model to the absolute measurement of hand from a depth camera under the robust restriction of a simple data glove. We redefine and simplify the function of the data glove to lighten its limitations, i.e., tedious calibration, cumbersome equipment, and hampering movement and keep our system lightweight. For accurate hand tracking, we introduce a new set of degrees of freedom (DoFs), a shape adjustment term for personalizing the triangular mesh model, and an adaptive collision term to prevent self-intersection. For efficiency, we extract a strong pose-space prior to the data glove to narrow the pose searching space. We also present a simplified approach for computing tracking correspondences without the loss of accuracy to reduce computation cost. Quantitative experiments show the comparable or increased accuracy of our system over the state-of-the-art with about 40% improvement in robustness. Besides, our system runs independent of Graphic Processing Unit (GPU) and reaches 40 frames per second (FPS) at about 25% Central Processing Unit (CPU) usage.

An Integrated Multi-Sensor Approach for the Remote Monitoring of Parkinson's Disease.

The increment of the prevalence of neurological diseases due to the trend in population aging demands for new strategies in disease management. In Parkinson's disease (PD), these strategies should aim at improving diagnosis accuracy and frequency of the clinical follow-up by means of decentralized cost-effective solutions. In this context, a system suitable for the remote monitoring of PD subjects is presented. It consists of the integration of two approaches investigated in our previous works, each one appropriate for the movement analysis of specific parts of the body: low-cost optical devices for the upper limbs and wearable sensors for the lower ones. The system performs the automated assessments of six motor tasks of the unified Parkinson's disease rating scale, and it is equipped with a gesture-based human machine interface designed to facilitate the user interaction and the system management. The usability of the system has been evaluated by means of standard questionnaires, and the accuracy of the automated assessment has been verified experimentally. The results demonstrate that the proposed solution represents a substantial improvement in PD assessment respect to the former two approaches treated separately, and a new example of an accurate, feasible and cost-effective mean for the decentralized management of PD.

A Tangible Solution for Hand Motion Tracking in Clinical Applications.

Objective real-time assessment of hand motion is crucial in many clinical applications including technically-assisted physical rehabilitation of the upper extremity. We propose an inertial-sensor-based hand motion tracking system and a set of dual-quaternion-based methods for estimation of finger segment orientations and fingertip positions. The proposed system addresses the specific requirements of clinical applications in two ways: (1) In contrast to glove-based approaches, the proposed solution maintains the sense of touch. (2) In contrast to previous work, the proposed methods avoid the use of complex calibration procedures, which means that they are suitable for patients with severe motor impairment of the hand. To overcome the limited significance of validation in lab environments with homogeneous magnetic fields, we validate the proposed system using functional hand motions in the presence of severe magnetic disturbances as they appear in realistic clinical settings. We show that standard sensor fusion methods that rely on magnetometer readings may perform well in perfect laboratory environments but can lead to more than 15 cm root-mean-square error for the fingertip distances in realistic environments, while our advanced method yields root-mean-square errors below 2 cm for all performed motions.

Using Surgeon Hand Motions to Predict Surgical Maneuvers.

OBJECTIVE: This study explores how common machine learning techniques can predict surgical maneuvers from a continuous video record of surgical benchtop simulations. BACKGROUND: Automatic computer vision recognition of surgical maneuvers (suturing, tying, and transition) could expedite video review and objective assessment of surgeries. METHOD: We recorded hand movements of 37 clinicians performing simple and running subcuticular suturing benchtop simulations, and applied three machine learning techniques (decision trees, random forests, and hidden Markov models) to classify surgical maneuvers every 2 s (60 frames) of video. RESULTS: Random forest predictions of surgical video correctly classified 74% of all video segments into suturing, tying, and transition states for a randomly selected test set. Hidden Markov model adjustments improved the random forest predictions to 79% for simple interrupted suturing on a subset of randomly selected participants. CONCLUSION: Random forest predictions aided by hidden Markov modeling provided the best prediction of surgical maneuvers. Training of models across all users improved prediction accuracy by 10% compared with a random selection of participants. APPLICATION: Marker-less video hand tracking can predict surgical maneuvers from a continuous video record with similar accuracy as robot-assisted surgical platforms, and may enable more efficient video review of surgical procedures for training and coaching.

Measurements by A LEAP-Based Virtual Glove for the Hand Rehabilitation.

Hand rehabilitation is fundamental after stroke or surgery. Traditional rehabilitation requires a therapist and implies high costs, stress for the patient, and subjective evaluation of the therapy effectiveness. Alternative approaches, based on mechanical and tracking-based gloves, can be really effective when used in virtual reality (VR) environments. Mechanical devices are often expensive, cumbersome, patient specific and hand specific, while tracking-based devices are not affected by these limitations but, especially if based on a single tracking sensor, could suffer from occlusions. In this paper, the implementation of a multi-sensors approach, the Virtual Glove (VG), based on the simultaneous use of two orthogonal LEAP motion controllers, is described. The VG is calibrated and static positioning measurements are compared with those collected with an accurate spatial positioning system. The positioning error is lower than 6 mm in a cylindrical region of interest of radius 10 cm and height 21 cm. Real-time hand tracking measurements are also performed, analysed and reported. Hand tracking measurements show that VG operated in real-time (60 fps), reduced occlusions, and managed two LEAP sensors correctly, without any temporal and spatial discontinuity when skipping from one sensor to the other. A video demonstrating the good performance of VG is also collected and presented in the Supplementary Materials. Results are promising but further work must be done to allow the calculation of the forces exerted by each finger when constrained by mechanical tools (e.g., peg-boards) and for reducing occlusions when grasping these tools. Although the VG is proposed for rehabilitation purposes, it could also be used for tele-operation of tools and robots, and for other VR applications.

A Self-Managed System for Automated Assessment of UPDRS Upper Limb Tasks in Parkinson's Disease.

A home-based, reliable, objective and automated assessment of motor performance of patients affected by Parkinson's Disease (PD) is important in disease management, both to monitor therapy efficacy and to reduce costs and discomforts. In this context, we have developed a self-managed system for the automated assessment of the PD upper limb motor tasks as specified by the Unified Parkinson's Disease Rating Scale (UPDRS). The system is built around a Human Computer Interface (HCI) based on an optical RGB-Depth device and a replicable software. The HCI accuracy and reliability of the hand tracking compares favorably against consumer hand tracking devices as verified by an optoelectronic system as reference. The interface allows gestural interactions with visual feedback, providing a system management suitable for motor impaired users. The system software characterizes hand movements by kinematic parameters of their trajectories. The correlation between selected parameters and clinical UPDRS scores of patient performance is used to assess new task instances by a machine learning approach based on supervised classifiers. The classifiers have been trained by an experimental campaign on cohorts of PD patients. Experimental results show that automated assessments of the system replicate clinical ones, demonstrating its effectiveness in home monitoring of PD.

Virtual reality assisted microscopy data visualization and colocalization analysis.

BACKGROUND: Confocal microscopes deliver detailed three-dimensional data and are instrumental in biological analysis and research. Usually, this three-dimensional data is rendered as a projection onto a two-dimensional display. We describe a system for rendering such data using a modern virtual reality (VR) headset. Sample manipulation is possible by fully-immersive hand-tracking and also by means of a conventional gamepad. We apply this system to the specific task of colocalization analysis, an important analysis tool in biological microscopy. We evaluate our system by means of a set of user trials. RESULTS: The user trials show that, despite inaccuracies which still plague the hand tracking, this is the most productive and intuitive interface. The inaccuracies nevertheless lead to a perception among users that productivity is low, resulting in a subjective preference for the gamepad. Fully-immersive manipulation was shown to be particularly effective when defining a region of interest (ROI) for colocalization analysis. CONCLUSIONS: Virtual reality offers an attractive and powerful means of visualization for microscopy data. Fully immersive interfaces using hand tracking show the highest levels of intuitiveness and consequent productivity. However, current inaccuracies in hand tracking performance still lead to a disproportionately critical user perception.

Decoding a wide range of hand configurations from macaque motor, premotor, and parietal cortices.

Despite recent advances in decoding cortical activity for motor control, the development of hand prosthetics remains a major challenge. To reduce the complexity of such applications, higher cortical areas that also represent motor plans rather than just the individual movements might be advantageous. We investigated the decoding of many grip types using spiking activity from the anterior intraparietal (AIP), ventral premotor (F5), and primary motor (M1) cortices. Two rhesus monkeys were trained to grasp 50 objects in a delayed task while hand kinematics and spiking activity from six implanted electrode arrays (total of 192 electrodes) were recorded. Offline, we determined 20 grip types from the kinematic data and decoded these hand configurations and the grasped objects with a simple Bayesian classifier. When decoding from AIP, F5, and M1 combined, the mean accuracy was 50% (using planning activity) and 62% (during motor execution) for predicting the 50 objects (chance level, 2%) and substantially larger when predicting the 20 grip types (planning, 74%; execution, 86%; chance level, 5%). When decoding from individual arrays, objects and grip types could be predicted well during movement planning from AIP (medial array) and F5 (lateral array), whereas M1 predictions were poor. In contrast, predictions during movement execution were best from M1, whereas F5 performed only slightly worse. These results demonstrate for the first time that a large number of grip types can be decoded from higher cortical areas during movement preparation and execution, which could be relevant for future neuroprosthetic devices that decode motor plans.