Characterizing the Effects of Adding Virtual and Augmented Reality in Robot-Assisted Training.

Extended reality (XR) technology combines physical reality with computer synthetic virtuality to deliver immersive experience to users. Virtual reality (VR) and augmented reality (AR) are two subdomains within XR with different immersion levels. Both of these have the potential to be combined with robot-assisted training protocols to maximize postural control improvement. In this study, we conducted a randomized control experiment with sixty-three healthy subjects to compare the effectiveness of robot-assisted posture training combined with VR or AR against robotic training alone. A robotic Trunk Support Trainer (TruST) was employed to deliver assistive force at the trunk as subjects moved beyond the stability limits during training. Our results showed that both VR and AR significantly enhanced the training outcomes of the TruST intervention. However, the VR group experienced higher simulator sickness compared to the AR group, suggesting that AR is better suited for sitting posture training in conjunction with TruST intervention. Our findings highlight the added value of XR to robot-assisted training and provide novel insights into the differences between AR and VR when integrated into a robotic training protocol. In addition, we developed a custom XR application that suited well for TruST intervention requirements. Our approach can be extended to other studies to develop novel XR-enhanced robotic training platforms.

Infant Sitting and Multi-Directional Reaching Skill.

We tested twenty-one 6- to 10-month-old infants with a wide range of sitting experience in forward and rightward reaching during unsupported sitting on the floor. Sessions were video-recorded for further behavioral and machine learning-based kinematic analyses. All infants, including novice sitters, successfully touched and grasped toys in both directions. Infant falls, hand support, and base of support changes were rare. Infants with more sitting experience showed better upright posture than novice sitters. However, we found no differences in trunk displacement or reaching kinematics between directions or across sitting experience. Thus, multi-directional reaching is functional in both novice and experienced infant sitters. We suggest that trunk and arm stability in sagittal and frontal planes is integral to learning to sit.

Training Postural Balance Control with Pelvic Force Field at the Boundary of Stability.

This study characterizes the effects of a postural training program on balance and muscle control strategies in a virtual reality (VR) environment. The Robotic Upright Stand Trainer (RobUST), which applies perturbative forces on the trunk and assistive forces on the pelvis, was used to deliver perturbation-based balance training (PBT) in a sample of 10 healthy participants. The VR task consisted of catching, aiming, and throwing a ball at a target. All participants received trunk perturbations during the VR task with forces tailored to the participant's maximum tolerance. A subgroup of these participants additionally received assistive forces at the pelvis during training. Postural kinematics were calculated before and after RobUST training, including (i) maximum perturbation force tolerated, (ii) center of pressure (COP) and pelvic excursions, (iii) postural muscle activations (EMG), and (iv) postural control strategies (the ankle and hip strategies). We observed an improvement in the maximum perturbation force and postural stability area in both groups and decreases in muscle activity. The behavior of the two groups differed for perturbations in the posterior direction where the unassisted group moved towards greater use of the hip strategy. In addition, the assisted group changed towards a lower margin of stability and higher pelvic excursion. We show that training with force assistance leads to a reactive balance strategy that permits pelvic excursion but that is efficient at restoring balance from displaced positions while training without assistance leads to reactive balance strategies that restrain pelvic excursion. Patient populations can benefit from a platform that encourages greater use of their range of motion.

Standing Reactive Postural Responses of Lower Limbs With and Without Self-Balance Assistance in Individuals With Spinal Cord Injury Receiving Epidural Stimulation.

Spinal cord epidural stimulation can promote the recovery of motor function in individuals with severe spinal cord injury (SCI) by enabling the spinal circuitry to interpret sensory information and generate related neuromuscular responses. This approach enables the spinal cord to generate lower limb extension patterns during weight bearing, allowing individuals with SCI to achieve upright standing. We have shown that the human spinal cord can generate some standing postural responses during self-initiated body weight shifting. In this study, we investigated the ability of individuals with motor complete SCI receiving epidural stimulation to generate standing reactive postural responses after external perturbations were applied at the trunk. A cable-driven robotic device was used to provide constant assistance for pelvic control and to deliver precise trunk perturbations while participants used their hands to grasp onto handlebars for self-balance support (hands-on) as well as when participants were without support (free-hands). Five individuals with motor complete SCI receiving lumbosacral spinal cord epidural stimulation parameters specific for standing (Stand-scES) participated in this study. Trunk perturbations (average magnitude: 17 ± 3% body weight) were delivered randomly in the four cardinal directions. Participants attempted to control each perturbation such that upright standing was maintained and no additional external assistance was needed. Lower limb postural responses were generally more frequent, larger in magnitude, and appropriately modulated during the free-hands condition. This was associated with trunk displacement and lower limb loading modulation that were larger in the free-hands condition. Further, we observed discernible lower limb muscle synergies that were similar between the two perturbed standing conditions. These findings suggest that the human spinal circuitry involved in postural control retains the ability to generate meaningful lower limb postural responses after SCI when its excitability is properly modulated. Moreover, lower limb postural responses appear enhanced by a standing environment without upper limb stabilization that promotes afferent inputs associated with a larger modulation of ground reaction forces and trunk kinematics. These findings should be considered when developing future experimental frameworks aimed at studying upright postural control and activity-based recovery training protocols aimed at promoting neural plasticity and sensory-motor recovery.

Study protocol for a randomised controlled trial to determine the efficacy of an intensive seated postural intervention delivered with robotic and rigid trunk support systems.

INTRODUCTION: Children with cerebral palsy (CP) classified as gross motor function classification system (GMFCS) levels III-IV demonstrate impaired sitting and reaching control abilities that hamper their overall functional performance. Yet, efficacious interventions for improving sitting-related activities are scarce. We recently designed a motor learning-based intervention delivered with a robotic Trunk-Support-Trainer (TruST-intervention), in which we apply force field technology to individualise sitting balance support. We propose a randomised controlled trial to test the efficacy of the motor intervention delivered with robotic TruST compared with a static trunk support system. METHODS AND ANALYSIS: We will recruit 82 participants with CP, GMFCS III-IV, and aged 6-17 years. Randomisation using concealed allocation to either the TruST-support or static trunk-support intervention will be conducted using opaque-sealed envelopes prepared by someone unrelated to the study. We will apply an intention-to-treat protocol. The interventions will consist of 2 hours/sessions, 3/week, for 4 weeks. Participants will start both interventions with pelvic strapping. In the TruST-intervention, postural task progression will be implemented by a progressive increase of the force field boundaries and then by removing the pelvic straps. In the static trunk support-intervention, we will progressively lower the trunk support and remove pelvic strapping. Outcomes will be assessed at baseline, training midpoint, 1-week postintervention, and 3-month follow-up. Primary outcomes will include the modified functional reach test, a kinematic evaluation of sitting workspace, and the Box and Block test. Secondary outcomes will include The Segmental Assessment of Trunk Control test, Seated Postural & Reaching Control test, Gross Motor Function Measure-Item Set, Canadian Occupational Performance Outcome, The Participation and Environment Measure and Youth, and postural and reaching kinematics. ETHICS AND DISSEMINATION: The study was approved by the Columbia University Institutional Review Board (AAAS7804). This study is funded by the National Institutes of Health (1R01HD101903-01) and is registered at clinicaltrials.gov. TRIAL REGISTRATION NUMBER: NCT04897347; clinicaltrials.gov.

Effects of Boundary-Based Assist-as-Needed Force Field on Lower Limb Muscle Synergies During Standing Posture Training.

The boundary-based assist-as-needed (BAAN) force field is widely used in robotic rehabilitation and has shown promising results in improving trunk control and postural stability. However, the fundamental understanding of how the BAAN force field affects the neuromuscular control remains unclear. In this study, we investigate how the BAAN force field impacts muscle synergy in the lower limbs during standing posture training. We integrated virtual reality (VR) into a cable-driven Robotic Upright Stand Trainer (RobUST) to define a complex standing task that requires both reactive and voluntary dynamic postural control. Ten healthy subjects were randomly assigned to two groups. Each subject performed 100 trials of the standing task with or without assistance from the BAAN force field provided by RobUST. The BAAN force field significantly improved balance control and motor task performance. Our results also indicate that the BAAN force field reduced the total number of lower limb muscle synergies while concurrently increasing the synergy density (i.e., number of muscles recruited in each synergy) during both reactive and voluntary dynamic posture training. This pilot study provides fundamental insights into understanding the neuromuscular basis of the BAAN robotic rehabilitation strategy and its potential for clinical applications. In addition, we expanded the repertoire of training with RobUST that integrates both perturbation training and goal-oriented functional motor training within a single task. This approach can be extended to other rehabilitation robots and training approaches with them.

Dynamic sitting in infants: Limits of stability.

BACKGROUND: Limits of stability-defined by the maximum distances a person is willing to reach without falling or changing the base of support-are measures of dynamic balance. RESEARCH QUESTION: What are infants' sitting stability limits in the forward and right directions? METHODS: Twenty-one 6- to 10-month old infants participated in this cross-sectional study. To incentivize infants to reach beyond arm's length, caregivers began by holding a toy close to their infants at shoulder height. Caregivers then moved the toy farther away as infants tried to reach for it until infants lost balance, placed their hands on the floor, or transitioned out of sitting. All sessions were conducted via Zoom™ and video-recorded for further analyses using DeepLabCut for 2D pose estimation and Datavyu to determine timings of the reach and to code infants' postural behaviors. RESULTS: Infants' trunk excursions in the anterior-posterior plane (for forward reaches) and medio-lateral plane (for rightward reaches) represented their stability limits. Most infants ended the reach by returning to their original sitting position; however, infants with higher Alberta Infant Motor Scale (AIMS) scores transitioned out of sitting and infants with lower AIMS scores sometimes fell (mostly during rightward reaching). Trunk excursions were correlated with months of sitting experience. Rightward trunk excursions were also correlated with AIMS scores and age. Overall, infants' trunk excursions were larger in the forward than in the right direction, and such discrepancy was consistent across infants. Lastly, the more often infants adopted movement strategies with their legs (e.g., bending the knees), the greater the trunk excursion they attained. SIGNIFICANCE: Sitting control entails learning to perceive the boundaries of stability limits and acquiring anticipatory postures to suit the needs of the task. Tests and interventions that target sitting stability limits could be beneficial for infants with or at risk of motor delays.

SARN: Shifted Attention Regression Network for 3D Hand Pose Estimation.

Hand pose estimation (HPE) plays an important role during the functional assessment of the hand and in potential rehabilitation. It is a challenge to predict the pose of the hand conveniently and accurately during functional tasks, and this limits the application of HPE. In this paper, we propose a novel architecture of a shifted attention regression network (SARN) to perform HPE. Given a depth image, SARN first predicts the spatial relationships between points in the depth image and a group of hand keypoints that determine the pose of the hand. Then, SARN uses these spatial relationships to infer the 3D position of each hand keypoint. To verify the effectiveness of the proposed method, we conducted experiments on three open-source datasets of 3D hand poses: NYU, ICVL, and MSRA. The proposed method achieved state-of-the-art performance with 7.32 mm, 5.91 mm, and 7.17 mm of mean error at the hand keypoints, i.e., mean Euclidean distance between the predicted and ground-truth hand keypoint positions. Additionally, to test the feasibility of SARN in hand movement recognition, a hand movement dataset of 26K depth images from 17 healthy subjects was constructed based on the finger tapping test, an important component of neurological exams administered to Parkinson's patients. Each image was annotated with the tips of the index finger and the thumb. For this dataset, the proposed method achieved a mean error of 2.99 mm at the hand keypoints and comparable performance on three task-specific metrics: the distance, velocity, and acceleration of the relative movement of the two fingertips. Results on the open-source datasets demonstrated the effectiveness of the proposed method, and results on our finger tapping dataset validated its potential for applications in functional task characterization.

A Deep-Learning Based Real-Time Prediction of Seated Postural Limits and Its Application in Trunk Rehabilitation.

Seated postural limit defines the boundary of a region such that for any excursions made outside this boundary a subject cannot return the trunk to the neutral position without additional external support. The seated postural limits can be used as a reference to provide assistive support to the torso by the Trunk Support Trainer (TruST). However, fixed boundary representations of seated postural limits are inadequate to capture dynamically changing seated postural limits during training. In this study, we propose a conceptual model of dynamic boundary of the trunk center by assigning a vector that tracks the postural-goal direction and trunk movement amplitude during a sitting task. We experimented with 20 healthy subjects. The results support our hypothesis that TruST intervention with an assist-as-needed force controller based on dynamic boundary representation could achieve more significant sitting postural control improvements than a fixed boundary representation. The second contribution of this paper is that we provide an effective approach to embed deep learning into TruST's real-time controller design. We have compiled a 3D trunk movement dataset which is currently the largest in the literature. We designed a loss function capable of solving the gate-controlled regression problem. We have proposed a novel deep-learning roadmap for the exploration study. Following the roadmap, we developed a deep learning architecture, modified the widely used Inception module, and then obtained a deep learning model capable of accurately predicting the dynamic boundary in real-time. We believe that this approach can be extended to other rehabilitation robots towards designing intelligent dynamic boundary-based assist-as-needed controllers.

Dual-Motor-Task of Catching and Throwing a Ball During Overground Walking in Virtual Reality.

Virtual Reality is a versatile platform to study human behavior in simulated environments and to develop interventions for functional rehabilitation. In this work, we designed a dual-task paradigm in a virtual environment where both tasks demand motor skills. Twenty-one healthy adults (mean age: 24.1 years) participated in this study. The experiment involved three conditions - normal overground walking, catch and throw a ball while standing, and catch and throw a ball while walking overground -all in the virtual environment. We investigated the dual-task gait characteristics and their correlations with outcomes from cognitive assessments. Results show that subjects walk conservatively with smaller stride lengths, larger stride widths and stride time while catching and throwing. However, they are able to throw the balls more accurately at the target and achieve higher scores. During the dual-task throw, we observed that the participants threw more balls during the stance phase of the gait when the foot was in the terminal stance and pre-swing region. During this region, the body has forward momentum. In addition, the changes in gait characteristics during dual-task throw correlate well with outcome measures in standardized cognitive tests. This study provides a new and engaging paradigm to analyze dual-motor-task cost in a virtual reality environment and it can be used as a basis to compare strategies adopted by different population groups with healthy young adults to execute coordinated motor tasks.