Tool-sensed object information effectively supports vision for multisensory grasping.

Tools enable humans to extend their sensing abilities beyond the natural limits of their hands, allowing them to sense objects as if they were using their hands directly. The similarities between direct hand interactions with objects (hand-based sensing) and the ability to extend sensory information processing beyond the hand (tool-mediated sensing) entail the existence of comparable processes for integrating tool- and hand-sensed information with vision, raising the question of whether tools support vision in bimanual object manipulations. Here, we investigated participants' performance while grasping objects either held with a tool or with their hand and compared these conditions with visually guided grasping (Experiment 1). By measuring reaction time, peak velocity, and peak of grip aperture, we found that actions were initiated earlier and performed with a smaller peak grip aperture when the object was seen and held with the tool or the contralateral hand compared to when it was only seen. Thus, tool-mediated sensing effectively supports vision in multisensory grasping and, even more intriguingly, resembles hand-based sensing. We excluded that results were due to the force exerted on the tool's handle (Experiment 2). Additionally, as for hand-based sensing, we found evidence that the tool supports vision by mainly providing object positional information (Experiment 3). Thus, integrating the tool-sensed position of the object with vision is sufficient to promote a multisensory advantage in grasping. Our findings indicate that multisensory integration mechanisms significantly improve grasping actions and fine-tune contralateral hand movements even when object information is only indirectly sensed through a tool. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Improving Hand Gesture Recognition Robustness to Dynamic Posture Variations by Multimodal Deep Feature Fusion.

Surface electromyography (sEMG), a human-machine interface for gesture recognition, has shown promising potential for decoding motor intentions, but a variety of nonideal factors restrict its practical application in assistive robots. In this paper, we summarized the current mainstream gesture recognition strategies and proposed a gesture recognition method based on multimodal canonical correlation analysis feature fusion classification (MCAFC) for a nonideal condition that occurs in daily life, i.e., posture variations. The deep features of the sEMG and acceleration signals were first extracted via convolutional neural networks. A canonical correlation analysis was subsequently performed to associate the deep features of the two modalities. The transformed features were utilized as inputs to a linear discriminant analysis classifier to recognize the corresponding gestures. Both offline and real-time experiments were conducted on eight non-disabled subjects. The experimental results indicated that MCAFC achieved an average classification accuracy, average motion completion rate, and average motion completion time of 93.44%, 94.05%, and 1.38 s, respectively, with multiple dynamic postures, indicating significantly better performance than that of comparable methods. The results demonstrate the feasibility and superiority of the proposed multimodal signal feature fusion method for gesture recognition with posture variations, providing a new scheme for myoelectric control.

A visual representation of the hand in the resting somatomotor regions of the human brain.

Hand visibility affects motor control, perception, and attention, as visual information is integrated into an internal model of somatomotor control. Spontaneous brain activity, i.e., at rest, in the absence of an active task, is correlated among somatomotor regions that are jointly activated during motor tasks. Recent studies suggest that spontaneous activity patterns not only replay task activation patterns but also maintain a model of the body's and environment's statistical regularities (priors), which may be used to predict upcoming behavior. Here, we test whether spontaneous activity in the human somatomotor cortex as measured using fMRI is modulated by visual stimuli that display hands vs. non-hand stimuli and by the use/action they represent. A multivariate pattern analysis was performed to examine the similarity between spontaneous activity patterns and task-evoked patterns to the presentation of natural hands, robot hands, gloves, or control stimuli (food). In the left somatomotor cortex, we observed a stronger (multivoxel) spatial correlation between resting state activity and natural hand picture patterns compared to other stimuli. No task-rest similarity was found in the visual cortex. Spontaneous activity patterns in somatomotor brain regions code for the visual representation of human hands and their use.

Wearable high-density EMG sleeve for complex hand gesture classification and continuous joint angle estimation.

High-density electromyography (HD-EMG) can provide a natural interface to enhance human-computer interaction (HCI). This study aims to demonstrate the capability of a novel HD-EMG forearm sleeve equipped with up to 150 electrodes to capture high-resolution muscle activity, decode complex hand gestures, and estimate continuous hand position via joint angle predictions. Ten able-bodied participants performed 37 hand movements and grasps while EMG was recorded using the HD-EMG sleeve. Simultaneously, an 18-sensor motion capture glove calculated 23 joint angles from the hand and fingers across all movements for training regression models. For classifying across the 37 gestures, our decoding algorithm was able to differentiate between sequential movements with 97.3 ± 0.3 % accuracy calculated on a 100 ms bin-by-bin basis. In a separate mixed dataset consisting of 19 movements randomly interspersed, decoding performance achieved an average bin-wise accuracy of 92.8 ± 0.8 % . When evaluating decoders for use in real-time scenarios, we found that decoders can reliably decode both movements and movement transitions, achieving an average accuracy of 93.3 ± 0.9 % on the sequential set and 88.5 ± 0.9 % on the mixed set. Furthermore, we estimated continuous joint angles from the EMG sleeve data, achieving a R 2 of 0.884 ± 0.003 in the sequential set and 0.750 ± 0.008 in the mixed set. Median absolute error (MAE) was kept below 10° across all joints, with a grand average MAE of 1.8 ± 0 . 04 ∘ and 3.4 ± 0 . 07 ∘ for the sequential and mixed datasets, respectively. We also assessed two algorithm modifications to address specific challenges for EMG-driven HCI applications. To minimize decoder latency, we used a method that accounts for reaction time by dynamically shifting cue labels in time. To reduce training requirements, we show that pretraining models with historical data provided an increase in decoding performance compared with models that were not pretrained when reducing the in-session training data to only one attempt of each movement. The HD-EMG sleeve, combined with sophisticated machine learning algorithms, can be a powerful tool for hand gesture recognition and joint angle estimation. This technology holds significant promise for applications in HCI, such as prosthetics, assistive technology, rehabilitation, and human-robot collaboration.

Stage-dependent role of interhemispheric pathway for motor recovery in primates.

Whether and how the non-lesional sensorimotor cortex is activated and contributes to post-injury motor recovery is controversial. Here, we investigated the role of interhemispheric pathway from the contralesional to ipsilesional premotor cortex in activating the ipsilesional sensorimotor cortex and promoting recovery after lesioning the lateral corticospinal tract at the cervical cord, by unidirectional chemogenetic blockade in macaques. The blockade impaired dexterous hand movements during the early recovery stage. Electrocorticographical recording showed that the low frequency band activity of the ipsilesional premotor cortex around movement onset was decreased by the blockade during the early recovery stage, while it was increased by blockade during the intact state and late recovery stage. These results demonstrate that action of the interhemispheric pathway changed from inhibition to facilitation, to involve the ipsilesional sensorimotor cortex in hand movements during the early recovery stage. The present study offers insights into the stage-dependent role of the interhemispheric pathway and a therapeutic target in the early recovery stage after lesioning of the corticospinal tract.

Switching between task control modes incurs greater reprogramming costs than programming the response de novo.

When performing synchronous hand and foot movements, the way the limbs are synchronized differs depending on the mode of control. When performed in a reaction time (RT) paradigm (reactive control), EMG onsets become synchronized resulting in asynchronous displacement onset. However, when the same movement is performed as an anticipation-timing task (predictive control), displacement onset is synchronized by unconsciously introducing a small delay between EMG onsets. The present experiment investigated the reprogramming costs associated with switching between predictive and reaction control modes. Participants (N = 12, 6F) were asked to simultaneously lift their right heel and right hand in an anticipation-timing task when a rotating clock hand reached a specified target. On some trials, an auditory stimulus was presented either 250 ms or 500 ms before the target and participants were instructed to execute the synchronous movement as quickly as possible after the signal (i.e., switch to reactive mode). Results showed that when the auditory stimulus was delivered 250 ms before the target, participants were unable to switch to a reactive control mode but did switch when the auditory stimulus was presented 500 ms before the target. As expected, the RT on switch trials was substantially longer (∼230 ms) than a simple RT control condition but was also significantly longer (∼130 ms) than a choice RT control condition. These results indicate that switching between control modes for a task involving the same musculature incurs reprogramming costs that are even greater than the time required to program the response de novo.

Visual background information modulates motor contagions in humans.

Motor contagions refer to implicit effects induced by the observation of actions made by others on one's own actions. A plethora of studies conducted over the last two decades have demonstrated that both observed and predicted actions can induce various kinds of motor contagions in a human observer. However, motor contagions have always been investigated with regard to different features of an observed action, and it remains unclear whether the background environment in which an observed action takes place modulates motor contagions as well. Here, we investigated participant movements in an empirical hand steering task during which the participants were required to move a cursor through a visual channel after being presented with videos of an actor performing the same task. We manipulated the congruency between the actions shown in the video and the background channels and examined whether and how they affected the participants' own movements. We observed a clear interaction between the observed action and its background. The movement time of the participants' actions tended to increase or decrease depending on whether they observed a faster or slower movement, respectively, and these changes were amplified if the background was not congruent with the action contained within it. These results suggest that background information can modulate motor contagions in humans.

Human tactile sensing and sensorimotor mechanism: from afferent tactile signals to efferent motor control.

In tactile sensing, decoding the journey from afferent tactile signals to efferent motor commands is a significant challenge primarily due to the difficulty in capturing population-level afferent nerve signals during active touch. This study integrates a finite element hand model with a neural dynamic model by using microneurography data to predict neural responses based on contact biomechanics and membrane transduction dynamics. This research focuses specifically on tactile sensation and its direct translation into motor actions. Evaluations of muscle synergy during in -vivo experiments revealed transduction functions linking tactile signals and muscle activation. These functions suggest similar sensorimotor strategies for grasping influenced by object size and weight. The decoded transduction mechanism was validated by restoring human-like sensorimotor performance on a tendon-driven biomimetic hand. This research advances our understanding of translating tactile sensation into motor actions, offering valuable insights into prosthetic design, robotics, and the development of next-generation prosthetics with neuromorphic tactile feedback.

A compact solution for vibrotactile proprioceptive feedback of wrist rotation and hand aperture.

BACKGROUND: Closing the control loop between users and their prostheses by providing artificial sensory feedback is a fundamental step toward the full restoration of lost sensory-motor functions. METHODS: We propose a novel approach to provide artificial proprioceptive feedback about two degrees of freedom using a single array of 8 vibration motors (compact solution). The performance afforded by the novel method during an online closed-loop control task was compared to that achieved using the conventional approach, in which the same information was conveyed using two arrays of 8 and 4 vibromotors (one array per degree of freedom), respectively. The new method employed Gaussian interpolation to modulate the intensity profile across a single array of vibration motors (compact feedback) to convey wrist rotation and hand aperture by adjusting the mean and standard deviation of the Gaussian, respectively. Ten able-bodied participants and four transradial amputees performed a target achievement control test by utilizing pattern recognition with compact and conventional vibrotactile feedback to control the Hannes prosthetic hand (test conditions). A second group of ten able-bodied participants performed the same experiment in control conditions with visual and auditory feedback as well as no-feedback. RESULTS: Conventional and compact approaches resulted in similar positioning accuracy, time and path efficiency, and total trial time. The comparison with control condition revealed that vibrational feedback was intuitive and useful, but also underlined the power of incidental feedback sources. Notably, amputee participants achieved similar performance to that of able-bodied participants. CONCLUSIONS: The study therefore shows that the novel feedback strategy conveys useful information about prosthesis movements while reducing the number of motors without compromising performance. This is an important step toward the full integration of such an interface into a prosthesis socket for clinical use.

Closed-Loop Force Control by Biorealistic Hand Prosthesis With Visual and Tactile Sensory Feedback.

The ability of a novel biorealistic hand prosthesis for grasp force control reveals improved neural compatibility between the human-prosthetic interaction. The primary purpose here was to validate a virtual training platform for amputee subjects and evaluate the respective roles of visual and tactile information in fundamental force control tasks. We developed a digital twin of tendon-driven prosthetic hand in the MuJoCo environment. Biorealistic controllers emulated a pair of antagonistic muscles controlling the index finger of the virtual hand by surface electromyographic (sEMG) signals from amputees' residual forearm muscles. Grasp force information was transmitted to amputees through evoked tactile sensation (ETS) feedback. Six forearm amputees participated in force tracking and holding tasks under different feedback conditions or using their intact hands. Test results showed that visual feedback played a predominant role than ETS feedback in force tracking and holding tasks. However, in the absence of visual feedback during the force holding task, ETS feedback significantly enhanced motor performance compared to feedforward control alone. Thus, ETS feedback still supplied reliable sensory information to facilitate amputee's ability of stable grasp force control. The effects of tactile and visual feedback on force control were subject-specific when both types of feedback were provided simultaneously. Amputees were able to integrate visual and tactile information to the biorealistic controllers and achieve a good sensorimotor performance in grasp force regulation. The virtual platform may provide a training paradigm for amputees to adapt the biorealistic hand controller and ETS feedback optimally.

EEG Analyses of visual cue effects on executed movements.

BACKGROUND: In electroencephalographic (EEG) or electrocorticographic (ECoG) experiments, visual cues are commonly used for timing synchronization but may inadvertently induce neural activity and cognitive processing, posing challenges when decoding self-initiated tasks. NEW METHOD: To address this concern, we introduced four new visual cues (Fade, Rotation, Reference, and Star) and investigated their impact on brain signals. Our objective was to identify a cue that minimizes its influence on brain activity, facilitating cue-effect free classifier training for asynchronous applications, particularly aiding individuals with severe paralysis. RESULTS: 22 able-bodied, right-handed participants aged 18-30 performed hand movements upon presentation of the visual cues. Analysis of time-variability between movement onset and cue-aligned data, grand average MRCP, and classification outcomes revealed significant differences among cues. Rotation and Reference cue exhibited favorable results in minimizing temporal variability, maintaining MRCP patterns, and achieving comparable accuracy to self-paced signals in classification. COMPARISON WITH EXISTING METHODS: Our study contrasts with traditional cue-based paradigms by introducing novel visual cues designed to mitigate unintended neural activity. We demonstrate the effectiveness of Rotation and Reference cue in eliciting consistent and accurate MRCPs during motor tasks, surpassing previous methods in achieving precise timing and high discriminability for classifier training. CONCLUSIONS: Precision in cue timing is crucial for training classifiers, where both Rotation and Reference cue demonstrate minimal variability and high discriminability, highlighting their potential for accurate classifications in online scenarios. These findings offer promising avenues for refining brain-computer interface systems, particularly for individuals with motor impairments, by enabling more reliable and intuitive control mechanisms.

Towards higher load capacity: innovative design of a robotic hand with soft jointed structure.

In this paper, the innovative design of a robotic hand with soft jointed structure is carried out and a tendon-driven mechanism, a master-slave motor coordinated drive mechanism, a thumb coupling transmission mechanism and a thumb steering mechanism are proposed. These innovative designs allow for more effective actuation in each finger, enhancing the load capacity of the robotic hand while maintaining key performance indicators such as dexterity and adaptability. A mechanical model of the robotic finger was made to determine the application limitations and load capacity. The robotic hand was then prototyped for a set of experiments. The experimental results showed that the proposed theoretical model were reliable. Also, the fingertip force of the robotic finger could reach up to 10.3 N, and the load force could reach up to 72.8 N. When grasping target objects of different sizes and shapes, the robotic hand was able to perform the various power grasping and precision grasping in the Cutkosky taxonomy. Moreover, the robotic hand had good flexibility and adaptability by means of adjusting the envelope state autonomously.

How bipedalism shapes humans' actions with hand tools.

The task for an embodied cognitive understanding of humans' actions with tools is to elucidate how the human body, as a whole, supports the perception of affordances and dexterous action with objects in relation to other objects. Here, we focus on the relationship between humans' actions with handheld tools and bipedal posture. Posture plays a pivotal role in shaping animals' perception and action dynamics. While humans stand and locomote bipedally, other primates predominantly employ quadrupedal postures and locomotion, relying on both hands and feet to support the body. Drawing upon evidence from evolutionary biology, developmental psychology and performance studies, we elucidate the influence of bipedalism on our actions with objects and on our proficiency in using tools. We use the metaphor of cascades to capture the dynamic, nonlinear transformations in morphology and behaviour associated with posture and the use of tools across evolutionary and developmental timescales. Recent work illustrates the promise of multifractal cascade analysis to reveal nonlinear, cross-scale interactions across the entire body in real-time, supporting the perception of affordances for actions with tools. Cascade analysis enriches our comprehension of real-time performance and facilitates exploration of the relationships among whole-body coordination, individual development, and evolutionary processes.This article is part of the theme issue 'Minds in movement: embodied cognition in the age of artificial intelligence'.

Time-independent relationship between digits closure velocity and hand transport acceleration during reach-to-grasp movements.

Prehension movements in primates have been extensively studied for decades, and hand transport and hand grip adjustment are usually considered as the main components of any object reach-to-grasp action. Evident temporal patterns were found for the velocity of the hand during the transport phase and for the digits kinematics during pre-shaping and enclosing phases. However, such kinematics were always analysed separately in regard to time, and never studied in terms of dependence one from another. Nevertheless, if a reliable one-to-one relationship is proven, it would allow reconstructing the digit velocity (and position) simply by knowing the hand acceleration during reaching motions towards the target object, ceasing the usual dependence seen in literature from time of movement and distance from the target. In this study, the aim was precisely to analyse reach-to-grasp motions to explore if such relationship exists and how it can be formulated. Offline and real-time results not only seem to suggest the existence of a time-independent, one-to-one relationship between hand transport and hand grip adjustment, but also that such relationship is quite resilient to the different intrinsic and extrinsic properties of the target objects such as size, shape and position.

A 3-D whole-body human thermoregulatory model to simulate cold-induced vasodilation in the hands and feet.

The cold-induced vasodilation (CIVD) response of the human body to Arctic-like environments helps delay or prevent cold injuries to peripheral regions, such as the hands and feet. To more comprehensively predict the thermal responses of these body regions to cold stress, here we extended our previously developed and validated anatomically accurate three-dimensional whole-body thermoregulatory human model by incorporating a new phenomenological formulation of the CIVD mechanism. In this formulation, we modulated the cyclic vasodilation and vasoconstriction flow of warm blood from the body core to the peripheral regions solely by determining the heat-transfer exchange between the skin and the surrounding environment, and deactivated it when the core body temperature decreased to 36.5 °C. In total, we calibrated and validated the model using eight distinct studies involving 153 unique male subjects exposed to 10 diverse experimental conditions, including cold-air exposure of the whole body as well as air exposure and cold-water immersion of the hand or the foot. With CIVD incorporated, the model predictions generally yielded root mean square errors (RMSEs) of <3.0 °C for skin temperature, which represented a reduction of up to 3.6 °C compared to when we did not consider CIVD. Similarly, the incorporation of CIVD increased the fraction of predictions within two standard errors of the measured data by up to 63 %. The model predictions yielded RMSEs for core body temperature of <0.2 °C. The model can be used to provide guidelines to reduce the risk of cold-related injuries during prolonged exposures to very-cold environments.

Validation of the Kinematic Assessment Protocol Used in the Technology-Supported Neurorehabilitation System, Rehabilitation Technologies for Hand and Arm (R3THA™), in Children and Teenagers with Cerebral Palsy.

This study evaluates the R3THA™ assessment protocol (R3THA-AP™), a technology-supported testing module for personalized rehabilitation in children with cerebral palsy (CP). It focuses on the reliability and validity of the R3THA-AP in assessing hand and arm function, by comparing kinematic assessments with standard clinical assessments. Conducted during a 4-week summer camp, the study assessed the functional and impairment levels of children with CP aged 3-18. The findings suggest that R3THA is more reliable for children aged 8 and older, indicating that age significantly influences the protocol's effectiveness. The results also showed that the R3THA-AP's kinematic measurements of hand and wrist movements are positively correlated with the Box and Blocks Test Index (BBTI), reflecting hand function and dexterity. Additionally, the R3THA-AP's accuracy metrics for hand and wrist activities align with the Melbourne Assessment 2's Range of Motion (MA2-ROM) scores, suggesting a meaningful relationship between R3THA-AP data and clinical assessments of motor skills. However, no significant correlations were observed between the R3THA-AP and MA2's accuracy and dexterity measurements, indicating areas for further research. These findings validate the R3THA-AP's utility in assessing motor abilities in CP patients, supporting its integration into clinical practice.

Pink noise promotes sooner state transitions during bimanual coordination.

The seemingly straightforward task of tying one's shoes requires a sophisticated interplay of joints, muscles, and neural pathways, posing a formidable challenge for researchers studying the intricacies of coordination. A widely accepted framework for measuring coordinated behavior is the Haken-Kelso-Bunz (HKB) model. However, a significant limitation of this model is its lack of accounting for the diverse variability structures inherent in the coordinated systems it frequently models. Variability is a pervasive phenomenon across various biological and physical systems, and it changes in healthy adults, older adults, and pathological populations. Here, we show, both empirically and with simulations, that manipulating the variability in coordinated movements significantly impacts the ability to change coordination patterns-a fundamental feature of the HKB model. Our results demonstrate that synchronized bimanual coordination, mirroring a state of healthy variability, instigates earlier transitions of coordinated movements compared to other variability conditions. This suggests a heightened adaptability when movements possess a healthy variability. We anticipate our study to show the necessity of adapting the HKB model to encompass variability, particularly in predictive applications such as neuroimaging, cognition, skill development, biomechanics, and beyond.

Distribution of control during bimanual movement and stabilization.

In two-handed actions like baseball batting, the brain can allocate the control to each arm in an infinite number of ways. According to hemispheric specialization theory, the dominant hemisphere is adept at ballistic control, while the non-dominant hemisphere is specialized at postural stabilization, so the brain should divide the control between the arms according to their respective specialization. Here, we tested this prediction by examining how the brain shares the control between the dominant and non-dominant arms during bimanual reaching and postural stabilization. Participants reached with both hands, which were tied together by a stiff virtual spring, to a target surrounded by an unstable repulsive force field. If the brain exploits each hemisphere's specialization, then the dominant arm should be responsible for acceleration early in the movement, and the non-dominant arm will be the prime actor at the end when holding steady against the force field. The power grasp force, which signifies the postural stability of each arm, peaked at movement termination but was equally large in both arms. Furthermore, the brain predominantly used the arm that could use the stronger flexor muscles to mainly accelerate the movement. These results point to the brain flexibly allocating the control to each arm according to the task goal without adhering to a strict specialization scheme.

Adaptation of the gain of the corrective lifting response in object manipulation transfers across the hand.

Our ability to skillfully manipulate objects is supported by rapid corrective responses that are initiated when we experience perturbations that interfere with movement goals. For example, the corrective lifting response is triggered when an object is heavier than expected and fails to lift off the surface. In this situation, the absence of expected sensory feedback signalling lift off initiates, within ~ 90 ms, an increase in lifting force. Importantly, when people repeatedly lift an object that, on occasional catch trials, is heavier than expected, the gain of the corrective response, defined as the rate of force increase, adapts to the 'catch' weight. In the present study, we investigated whether this response adaption transfers intermanually. In the training phase, participants used either their left or right hand (counterbalanced) to repeatedly lift a 3 N object that unexpectedly increased to 9 N on catch trials, leading to an increase in the gain of the lifting response across catch trials. Participants then lifted the object with their other hand. On the first catch trial, the gain remained elevated and thus transferred across the hands. This finding suggests that the history of lifts performed by one hand updates the corrective responses for both hands.

Scalp acupuncture guidance for identifying the optimal site for transcranial electrical stimulation of the hand.

Transcranial electrical stimulation (tES) often targets the EEG-guided C3/C4 area that may not accurately represent M1 for hand muscles. This study aimed to determine if the neuroanatomy-based scalp acupuncture-guided site (AC) was a more effective spot than the C3 site for neuromodulation. Fifteen healthy subjects received one 20-minute session of high-definition transcranial alternating current stimulation (HD-tACS) intervention (20 Hz at 2 mA) at the AC or C3 sites randomly with a 1-week washout period. Subjects performed ball-squeezing exercises with the dominant hand during the HD-tACS intervention. The AC site was indiscernible from the finger flexor hotspot detected by TMS. At the baseline, the MEP amplitude from finger flexors was greater with less variability at the AC site than at the C3 site. HD-tACS intervention at the AC site significantly increased the MEP amplitude. However, no significant changes were observed after tACS was applied to the C3 site. Our results provide evidence that HD-tACS at the AC site produces better neuromodulation effects on the flexor digitorum superficialis (FDS) muscle compared to the C3 site. The AC localization approach can be used for future tES studies.