Empowering High-Level Spinal Cord Injury Patients in Daily Tasks With a Hybrid Gaze and FEMG-Controlled Assistive Robotic System.

Individuals with high-level spinal cord injuries often face significant challenges in performing essential daily tasks due to their motor impairments. Consequently, the development of reliable, hands-free human-computer interfaces (HCI) for assistive devices is vital for enhancing their quality of life. However, existing methods, including eye-tracking and facial electromyogram (FEMG) control, have demonstrated limitations in stability and efficiency. To address these shortcomings, this paper presents an innovative hybrid control system that seamlessly integrates gaze and FEMG signals. When deployed as a hybrid HCI, this system has been successfully used to assist individuals with high-level spinal cord injuries in performing activities of daily living (ADLs), including tasks like eating, pouring water, and pick-and-place. Importantly, our experimental results confirm that our hybrid control method expedites the performance in pick-place tasks, achieving an average completion time of 34.3 s, which denotes a 28.8% and 21.8% improvement over pure gaze-based control and pure FEMG-based control, respectively. With practice, participants experienced up to a 44% efficiency improvement using the hybrid control method. This state-of-the-art system offers a highly precise and reliable intention interface, suitable for daily use by individuals with high-level spinal cord injuries, ultimately enhancing their quality of life and independence.

Biomechanical Consequences of Walking With the Suspended Backpacks.

OBJECTIVE: This article aimed to investigate the biomechanical mechanisms underlying the energetic advantages of the suspended backpacks during load carriage. METHODS: In this study, we examined eight adults walking with a 15 kg load at 5 km/h with a designed suspended backpack, in which the load could be switched to locked and suspended with four combinations of stiffness. Mechanical work and metabolic cost were measured during load carriage. RESULTS: The results showed that the suspended backpacks led to an average reduction of 23.35% in positive work, 24.77% in negative work, and a 12.51% decrease in metabolic cost across all suspended load conditions. Notably, the decreased mechanical work predominantly occurred during single support (averaging 84.19% and 71.16% for positive and negative work, respectively), rather than during double support. CONCLUSION: Walking with the suspended backpack induced a phase shift between body movement and load movement, altering the human-load interaction. This adjustment caused the body and load to move against each other, resulting in flatter trajectories of the human-load system center of mass (COM) velocities and corresponding profiles in ground reaction forces (GRFs), along with reduced vertical excursions of the trunk. Consequently, this interplay led to flatter trajectories in mechanical work rate and reduced mechanical work, ultimately contributing to the observed reduction in energetic expenditure. SIGNIFICANCE: Understanding these mechanisms is essential for the development of more effective load-carrying devices and strategies in various applications, particularly for enhancing walking abilities during load carriage.

A Probability Fusion Approach for Foot Placement Prediction in Complex Terrains.

Prediction of foot placement presents great potential in better assisting the walking of people with lower-limb disability in daily terrains. Previous researches mainly focus on foot placement prediction in level ground walking, however these methods cannot be applied to daily complex terrains including ramps, stairs, and level ground with obstacles. To predict foot placement in complex terrains, this paper presents a probability fusion approach for foot placement prediction in complex terrains which consists of two parts: model training and foot placement prediction. In the first part, a deep learning model is trained on augmented data to predict the probability distribution of preliminary foot placement. In the second part, environmental information and human walking constraints are used to calculate the feasible area, and finally the feasible area is fused with the probability distribution of preliminary foot placement to predict the foot placement in complex terrains. The proposed method can predict the foot placement of next step in complex terrains when heel-off is detected. Experiments (including structured terrains experiments and complex terrains experiments) show that the root mean square error (RMSE) of prediction is 8.19 ± 1.20 cm, which is less than 8% of the average stride length, and the landing feasible area accuracy (LFAA) of prediction is 95.11 ± 3.09%. Comparing with existing foot placement prediction studies, the method proposed in this paper achieves faster and more accurate prediction in complex terrains.

Neural Correlation of EEG and Eye Movement in Natural Grasping Intention Estimation.

Decoding the user's natural grasp intent enhances the application of wearable robots, improving the daily lives of individuals with disabilities. Electroencephalogram (EEG) and eye movements are two natural representations when users generate grasp intent in their minds, with current studies decoding human intent by fusing EEG and eye movement signals. However, the neural correlation between these two signals remains unclear. Thus, this paper aims to explore the consistency between EEG and eye movement in natural grasping intention estimation. Specifically, six grasp intent pairs are decoded by combining feature vectors and utilizing the optimal classifier. Extensive experimental results indicate that the coupling between the EEG and eye movements intent patterns remains intact when the user generates a natural grasp intent, and concurrently, the EEG pattern is consistent with the eye movements pattern across the task pairs. Moreover, the findings reveal a solid connection between EEG and eye movements even when taking into account cortical EEG (originating from the visual cortex or motor cortex) and the presence of a suboptimal classifier. Overall, this work uncovers the coupling correlation between EEG and eye movements and provides a reference for intention estimation.

An Intelligent Rehabilitation Assessment Method for Stroke Patients Based on Lower Limb Exoskeleton Robot.

The 6-min walk distance (6MWD) and the Fugl-Meyer assessment lower-limb subscale (FMA-LE) of the stroke patients provide the critical evaluation standards for the effect of training and guidance of the training programs. However, gait assessment for stroke patients typically relies on manual observation and table scoring, which raises concerns about wasted manpower and subjective observation results. To address this issue, this paper proposes an intelligent rehabilitation assessment method (IRAM) for rehabilitation assessment of the stroke patients based on sensor data of the lower limb exoskeleton robot. Firstly, the feature parameters of the patient were collected, including age, height, and duration, etc. The sensor data of the exoskeleton robot were also collected, including joint angle, joint velocity, and joint torque, etc. Secondly, a gait feature model was constructed to deduce the walking gait parameters of the patient according to the sensor data of the exoskeleton, including the support phase to swing phase ratio, step length and leg lift height of the patient, etc. Then, the 6MWD and FMA-LE values were collected by traditional methods, feature parameters, gait parameters and human-machine interaction parameters (joint torque) of the patient were adopted to train the rehabilitation assessment model. Finally, the assessment model was trained by a machine-learning based algorithm. The new stroke patients' the 6MWD and FMA-LE values can be predicted by the trained model. The experimental results present that the prediction accuracy for the 6MWD and FMA-LE values reach to 85.19% and 92.66%, respectively.

Fully Automatic Dual-Probe Lung Ultrasound Scanning Robot for Screening Triage.

Two-dimensional lung ultrasound (LUS) has widely emerged as a rapid and noninvasive imaging tool for the detection and diagnosis of coronavirus disease 2019 (COVID-19). However, image differences will be magnified due to changes in ultrasound (US) imaging experience, such as US probe attitude control and force control, which will directly affect the diagnosis results. In addition, the risk of virus transmission between sonographer and patients is increased due to frequent physical contact. In this study, a fully automatic dual-probe US scanning robot for the acquisition of LUS images is proposed and developed. Furthermore, the trajectory was optimized based on the velocity look-ahead strategy, the stability of contact force of the system and the scanning efficiency were improved by 24.13% and 29.46%, respectively. Also, the control ability of the contact force of robotic automatic scanning was 34.14 times higher than that of traditional manual scanning, which significantly improves the smoothness of scanning. Importantly, there was no significant difference in image quality obtained by robotic automatic scanning and manual scanning. Furthermore, the scanning time for a single person is less than 4 min, which greatly improves the efficiency of screening triage of group COVID-19 diagnosis and suspected patients and reduces the risk of virus exposure and spread.

A Supervised-Reinforced Successive Training Framework for a Fuzzy Inference System and Its Application in Robotic Odor Source Searching.

Fuzzy inference systems have been widely applied in robotic control. Previous studies proposed various methods to tune the fuzzy rules and the parameters of the membership functions (MFs). Training the systems with only supervised learning requires a large amount of input-output data, and the performance of the trained system is confined by that of the target system. Training the systems with only reinforcement learning (RL) does not require prior knowledge but is time-consuming, and the initialization of the system remains a problem. In this paper, a supervised-reinforced successive training framework is proposed for a multi-continuous-output fuzzy inference system (MCOFIS). The parameters of the fuzzy inference system are first tuned by a limited number of input-output data from an existing controller with supervised training and then are utilized to initialize the system in the reinforcement training stage. The proposed framework is applied in a robotic odor source searching task and the evaluation results demonstrate that the performance of the fuzzy inference system trained by the successive framework is superior to the systems trained by only supervised learning or RL. The system trained by the proposed framework can achieve around a 10% higher success rate compared to the systems trained by only supervised learning or RL.

Unsupervised Sim-to-Real Adaptation for Environmental Recognition in Assistive Walking.

Powered lower-limb prostheses with vision sensors are expected to restore amputees' mobility in various environments with supervised learning-based environmental recognition. Due to the sim-to-real gap, such as real-world unstructured terrains and the perspective and performance limitations of vision sensor, simulated data cannot meet the requirement for supervised learning. To mitigate this gap, this paper presents an unsupervised sim-to-real adaptation method to accurately classify five common real-world (level ground, stair ascent, stair descent, ramp ascent and ramp descent) and assist amputee's terrain-adaptive locomotion. In this study, augmented simulated environments are generated from a virtual camera perspective to better simulate the real world. Then, unsupervised domain adaptation is incorporated to train the proposed adaptation network consisting of a feature extractor and two classifiers is trained on simulated data and unlabeled real-world data to minimize domain shift between source domain (simulation) and target domain (real world). To interpret the classification mechanism visually, essential features of different terrains extracted by the network are visualized. The classification results in walking experiments indicate that the average accuracy on eight subjects reaches (98.06% ± 0.71 %) and (95.91% ± 1.09 %) in indoor and outdoor environments respectively, which is close to the result of supervised learning using both type of labeled data (98.37% and 97.05%). The promising results demonstrate that the proposed method is expected to realize accurate real-world environmental classification and successful sim-to-real transfer.

Gait Phase Subdivision and Leg Stiffness Estimation During Stair Climbing.

Leg stiffness is considered a prevalent parameter used in data analysis of leg locomotion during different gaits, such as walking, running, and hopping. Quantification of the change in support leg stiffness during stair ascent and descent will enhance our understanding of complex stair climbing gait dynamics. The purpose of this study is to investigate a methodology to estimate leg stiffness during stair climbing and subdivide the stair climbing gait cycle. Leg stiffness was determined as the ratio of changes in ground reaction force in the direction of the support leg Fl (leg force) to the respective changes in length Ll during the entire stance phase. Eight subjects ascended and descended an instrumented staircase at different cadences. In this study, the changes of leg force and length (force-length curve) are described as the leg stiffness curve, the slope of which represents the normalized stiffness during stair climbing. The stair ascent and descent gait cycles were subdivided based on the negative and positive work fluctuations of the center-of-mass (CoM) work rate curve and the characteristics of leg stiffness. We found that the leg stiffness curve consists of several segments in which the force-length relationship was similarly linear and the stiffness value was relatively constant; the phase divided by the leg stiffness curve corresponds to the phase divided by the CoM work rate curve. The results of this study may guide biomimetic control strategies for a wearable lower-extremity robot for the entire stance phase during stair climbing.

A Probability Distribution Model-Based Approach for Foot Placement Prediction in the Early Swing Phase With a Wearable IMU Sensor.

Predicting the next foot placement of humans during walking can help improve compliant interactions between humans and walking aid robots. Previous studies have focused on foot placement estimation with wearable inertial sensors after heel-strike, but few have predicted foot placements in advance during the early swing phase. In this study, a Bayesian inference-based foot placement prediction approach was proposed. Possible foot placements were modeled as a probability distribution grid map. With selected foot motion feature events detected sequentially in the early swing phase, the foot placement probability map could be updated iteratively using the feature models we built. The weighted center of the probability distribution was regarded as the predicted foot placement. Prediction errors were evaluated with collected walking data sets. When testing with the data from inertial measurement units, the prediction errors were (5.46 cm ± 10.89 cm, -0.83 cm ± 10.56 cm) for cross-velocity walking data and (-4.99 cm ± 12.31 cm, -11.27 cm ± 7.74 cm) for cross-subject-cross-velocity walking data. The results were comparable to previous works yet the prediction could be made earlier. For the subject who walked with more stable gaits, the prediction error can be further decreased. The proposed foot placement prediction approach can be utilized to help walking aid robots adjust their pose before each heel-strike event during walking, which will make human-robot interactions more compliant. This study is also expected to inspire additional probabilistic gait analysis works.

A Plane Extraction Approach in Inverse Depth Images Based on Region-Growing.

Planar surfaces are prevalent components of man-made indoor scenes, and plane extraction plays a vital role in practical applications of computer vision and robotics, such as scene understanding, and mobile manipulation. Nowadays, most plane extraction methods are based on reconstruction of the scene. In this paper, plane representation is formulated in inverse-depth images. Based on this representation, we explored the potential to extract planes in images directly. A fast plane extraction approach, which employs the region growing algorithm in inverse-depth images, is presented. This approach consists of two main components: seeding, and region growing. In the seeding component, seeds are carefully selected locally in grid cells to improve exploration efficiency. After seeding, each seed begins to grow into a continuous plane in succession. Both greedy policy and a normal coherence check are employed to find boundaries accurately. During growth, neighbor coplanar planes are checked and merged to overcome the over-segmentation problem. Through experiments on public datasets and generated saw-tooth images, the proposed approach achieves 80.2% CDR (Correct Detection Rate) on the ABW SegComp Dataset, which has proven that it has comparable performance with the state-of-the-art. The proposed approach runs at 5 Hz on typical 680 × 480 images, which has shown its potential in real-time practical applications in computer vision and robotics with further improvement.

A model to predict ground reaction force for elastically-suspended backpacks.

BACKGROUND: Leg muscle fatigue is the most important factor that affects walking endurance. Considering the legs act as actuators in alternate contact with the ground during walking, the ground reaction force (GRF) of each leg can indirectly reflect the strength of leg muscles. However, it is not clear how the elastically-suspended backpack (ESB) affects GRF of each leg during human level walking. RESEARCH QUESTION: How is ESB related in GRF of each leg during walking, and how do multiple variables (stiffness and damping of ESB, load mass, walking speed) affect GRF? METHODS: An extended bipedal walking model (EBW) with a spring-mass-damping system was proposed to predict the GRF of each leg. In order to evaluate the prediction effect of the model, seven healthy subjects were recruited to attend the experiments using our backpack prototype and the GRFs data was compared. Each subject walked under 12 conditions (load states: locked or unlocked, walking speed: 3.6 km/h, 4.0 km/h, 4.5 km/h, 5.0 km/h, 5.5 km/h, 6.0 km/h). RESULTS: Results showed that the model could quantitatively predict experimental GRFs over the whole gait cycle (R2≥0.9628) and the characteristic forces (two peak forces and one trough force) were close to the experimental data (average predicted accuracy 93.7 %). The model can reflect relationships between variables and GRF. The relationships showed that an apparent tradeoff exists among the three characteristic forces, and the ESB can produce positive or negative effect under different variables. SIGNIFICANCE: This work could help us understand the experimental GRF phenomena, especially the contradictory experimental phenomenon caused by the different parameters. It could also help designers optimize structural parameters of ESB for excellent effects on human. The ESBs with excellent performance can be wildly used in military and tourism.

An Occlusion-Aware Framework for Real-Time 3D Pose Tracking.

Random forest-based methods for 3D temporal tracking over an image sequence have gained increasing prominence in recent years. They do not require object's texture and only use the raw depth images and previous pose as input, which makes them especially suitable for textureless objects. These methods learn a built-in occlusion handling from predetermined occlusion patterns, which are not always able to model the real case. Besides, the input of random forest is mixed with more and more outliers as the occlusion deepens. In this paper, we propose an occlusion-aware framework capable of real-time and robust 3D pose tracking from RGB-D images. To this end, the proposed framework is anchored in the random forest-based learning strategy, referred to as RFtracker. We aim to enhance its performance from two aspects: integrated local refinement of random forest on one side, and online rendering based occlusion handling on the other. In order to eliminate the inconsistency between learning and prediction of RFtracker, a local refinement step is embedded to guide random forest towards the optimal regression. Furthermore, we present an online rendering-based occlusion handling to improve the robustness against dynamic occlusion. Meanwhile, a lightweight convolutional neural network-based motion-compensated (CMC) module is designed to cope with fast motion and inevitable physical delay caused by imaging frequency and data transmission. Finally, experiments show that our proposed framework can cope better with heavily-occluded scenes than RFtracker and preserve the real-time performance.