Capturing Upper Body Kinematics and Localization with Low-Cost Sensors for Rehabilitation Applications.

For upper extremity rehabilitation, quantitative measurements of a person's capabilities during activities of daily living could provide useful information for therapists, including in telemedicine scenarios. Specifically, measurements of a person's upper body kinematics could give information about which arm motions or movement features are in need of additional therapy, and their location within the home could give context to these motions. To that end, we present a new algorithm for identifying a person's location in a region of interest based on a Bluetooth received signal strength (RSS) and present an experimental evaluation of this and a different Bluetooth RSS-based localization algorithm via fingerprinting. We further present algorithms for and experimental results of inferring the complete upper body kinematics based on three standalone inertial measurement unit (IMU) sensors mounted on the wrists and pelvis. Our experimental results for localization find the target location with a mean square error of 1.78 m. Our kinematics reconstruction algorithms gave lower errors with the pelvis sensor mounted on the person's back and with individual calibrations for each test. With three standalone IMUs, the mean angular error for all of the upper body segment orientations was close to 21 degrees, and the estimated elbow and shoulder angles had mean errors of less than 4 degrees.

Smart Skin: Vision-Based Soft Pressure Sensing System for In-Home Hand Rehabilitation.

We introduce a novel in-home hand rehabilitation system for monitoring hand motions and assessing grip forces of stroke patients. The overall system is composed of a sensing device and a computer vision system. The sensing device is a lightweight cylindrical object for easy grip and manipulation, which is covered by a passive sensing layer called "Smart Skin." The Smart Skin is fabricated using soft silicone elastomer, which contains embedded microchannels partially filled with colored fluid. When the Smart Skin is compressed by grip forces, the colored fluid rises and fills in the top surface display area. Then, the computer vision system captures the image of the display area through a red-green-blue camera, detects the length change of the liquid through image processing, and eventually maps the liquid length to the calibrated force for estimating the gripping force. The passive sensing mechanism of the proposed Smart Skin device works in conjunction with a single camera setup, making the system simple and easy to use, while also requiring minimum maintenance effort. Our system, on one hand, aims to support home-based rehabilitation therapy with minimal or no supervision by recording the training process and the force data, which can be automatically conveyed to physical therapists. In contrast, the therapists can also remotely instruct the patients with their training prescriptions through online videos. This study first describes the design, fabrication, and calibration of the Smart Skin, and the algorithm for image processing, and then presents experimental results from the integrated system. The Smart Skin prototype shows a relatively linear relationship between the applied force and the length change of the liquid in the range of 0-35 N. The computer vision system shows the estimation error <4% and a relatively high stability in estimation under different hand motions.

Automated Movement Assessment in Stroke Rehabilitation.

We are developing a system for long term Semi-Automated Rehabilitation At the Home (SARAH) that relies on low-cost and unobtrusive video-based sensing. We present a cyber-human methodology used by the SARAH system for automated assessment of upper extremity stroke rehabilitation at the home. We propose a hierarchical model for automatically segmenting stroke survivor's movements and generating training task performance assessment scores during rehabilitation. The hierarchical model fuses expert therapist knowledge-based approaches with data-driven techniques. The expert knowledge is more observable in the higher layers of the hierarchy (task and segment) and therefore more accessible to algorithms incorporating high level constraints relating to activity structure (i.e., type and order of segments per task). We utilize an HMM and a Decision Tree model to connect these high level priors to data driven analysis. The lower layers (RGB images and raw kinematics) need to be addressed primarily through data driven techniques. We use a transformer based architecture operating on low-level action features (tracking of individual body joints and objects) and a Multi-Stage Temporal Convolutional Network(MS-TCN) operating on raw RGB images. We develop a sequence combining these complimentary algorithms effectively, thus encoding the information from different layers of the movement hierarchy. Through this combination, we produce a robust segmentation and task assessment results on noisy, variable and limited data, which is characteristic of low cost video capture of rehabilitation at the home. Our proposed approach achieves 85% accuracy in per-frame labeling, 99% accuracy in segment classification and 93% accuracy in task completion assessment. Although the methodology proposed in this paper applies to upper extremity rehabilitation using the SARAH system, it can potentially be used, with minor alterations, to assist automation in many other movement rehabilitation contexts (i.e., lower extremity training for neurological accidents).

Component-Level Tuning of Kinematic Features From Composite Therapist Impressions of Movement Quality.

In this paper, we propose a general framework for tuning component-level kinematic features using therapists' overall impressions of movement quality, in the context of a home-based adaptive mixed reality rehabilitation (HAMRR) system. We propose a linear combination of nonlinear kinematic features to model wrist movement, and propose an approach to learn feature thresholds and weights using high-level labels of overall movement quality provided by a therapist. The kinematic features are chosen such that they correlate with the quality of wrist movements to clinical assessment scores. Further, the proposed features are designed to be reliably extracted from an inexpensive and portable motion capture system using a single reflective marker on the wrist. Using a dataset collected from ten stroke survivors, we demonstrate that the framework can be reliably used for movement quality assessment in HAMRR systems. The system is currently being deployed for large-scale evaluations, and will represent an increasingly important application area of motion capture and activity analysis.

Interdisciplinary concepts for design and implementation of mixed reality interactive neurorehabilitation systems for stroke.

Interactive neurorehabilitation (INR) systems provide therapy that can evaluate and deliver feedback on a patient's movement computationally. There are currently many approaches to INR design and implementation, without a clear indication of which methods to utilize best. This article presents key interactive computing, motor learning, and media arts concepts utilized by an interdisciplinary group to develop adaptive, mixed reality INR systems for upper extremity therapy of patients with stroke. Two INR systems are used as examples to show how the concepts can be applied within: (1) a small-scale INR clinical study that achieved integrated improvement of movement quality and functionality through continuously supervised therapy and (2) a pilot study that achieved improvement of clinical scores with minimal supervision. The notion is proposed that some of the successful approaches developed and tested within these systems can form the basis of a scalable design methodology for other INR systems. A coherent approach to INR design is needed to facilitate the use of the systems by physical therapists, increase the number of successful INR studies, and generate rich clinical data that can inform the development of best practices for use of INR in physical therapy.

Decision support for stroke rehabilitation therapy via describable attribute-based decision trees.

This paper proposes a computational framework for movement quality assessment using a decision tree model that can potentially assist a physical therapist in a telerehabilitation context. Using a dataset of key kinematic attributes collected from eight stroke survivors, we demonstrate that the framework can be reliably used for movement quality assessment of a reach-to-grasp cone task, an activity commonly used in upper extremity stroke rehabilitation therapy. The proposed framework is capable of providing movement quality scores that are highly correlated to the ratings provided by therapists, who used a custom rating rubric created by rehabilitation experts. Our hypothesis is that a decision tree model could be easily utilized by therapists as a potential assistive tool, especially in evaluating movement quality on a large-scale dataset collected during unsupervised rehabilitation (e.g., training at the home), thereby reducing the time and cost of rehabilitation treatment.

Adaptive mixed reality rehabilitation improves quality of reaching movements more than traditional reaching therapy following stroke.

BACKGROUND: Adaptive mixed reality rehabilitation (AMRR) is a novel integration of motion capture technology and high-level media computing that provides precise kinematic measurements and engaging multimodal feedback for self-assessment during a therapeutic task. OBJECTIVE: We describe the first proof-of-concept study to compare outcomes of AMRR and traditional upper-extremity physical therapy. METHODS: Two groups of participants with chronic stroke received either a month of AMRR therapy (n = 11) or matched dosing of traditional repetitive task therapy (n = 10). Participants were right handed, between 35 and 85 years old, and could independently reach to and at least partially grasp an object in front of them. Upper-extremity clinical scale scores and kinematic performances were measured before and after treatment. RESULTS: Both groups showed increased function after therapy, demonstrated by statistically significant improvements in Wolf Motor Function Test and upper-extremity Fugl-Meyer Assessment (FMA) scores, with the traditional therapy group improving significantly more on the FMA. However, only participants who received AMRR therapy showed a consistent improvement in kinematic measurements, both for the trained task of reaching to grasp a cone and the untrained task of reaching to push a lighted button. CONCLUSIONS: AMRR may be useful in improving both functionality and the kinematics of reaching. Further study is needed to determine if AMRR therapy induces long-term changes in movement quality that foster better functional recovery.

Exploring the bases for a mixed reality stroke rehabilitation system, Part II: design of interactive feedback for upper limb rehabilitation.

BACKGROUND: Few existing interactive rehabilitation systems can effectively communicate multiple aspects of movement performance simultaneously, in a manner that appropriately adapts across various training scenarios. In order to address the need for such systems within stroke rehabilitation training, a unified approach for designing interactive systems for upper limb rehabilitation of stroke survivors has been developed and applied for the implementation of an Adaptive Mixed Reality Rehabilitation (AMRR) System. RESULTS: The AMRR system provides computational evaluation and multimedia feedback for the upper limb rehabilitation of stroke survivors. A participant's movements are tracked by motion capture technology and evaluated by computational means. The resulting data are used to generate interactive media-based feedback that communicates to the participant detailed, intuitive evaluations of his performance. This article describes how the AMRR system's interactive feedback is designed to address specific movement challenges faced by stroke survivors. Multimedia examples are provided to illustrate each feedback component. Supportive data are provided for three participants of varying impairment levels to demonstrate the system's ability to train both targeted and integrated aspects of movement. CONCLUSIONS: The AMRR system supports training of multiple movement aspects together or in isolation, within adaptable sequences, through cohesive feedback that is based on formalized compositional design principles. From preliminary analysis of the data, we infer that the system's ability to train multiple foci together or in isolation in adaptable sequences, utilizing appropriately designed feedback, can lead to functional improvement. The evaluation and feedback frameworks established within the AMRR system will be applied to the development of a novel home-based system to provide an engaging yet low-cost extension of training for longer periods of time.

Exploring the bases for a mixed reality stroke rehabilitation system, part I: a unified approach for representing action, quantitative evaluation, and interactive feedback.

BACKGROUND: Although principles based in motor learning, rehabilitation, and human-computer interfaces can guide the design of effective interactive systems for rehabilitation, a unified approach that connects these key principles into an integrated design, and can form a methodology that can be generalized to interactive stroke rehabilitation, is presently unavailable. RESULTS: This paper integrates phenomenological approaches to interaction and embodied knowledge with rehabilitation practices and theories to achieve the basis for a methodology that can support effective adaptive, interactive rehabilitation. Our resulting methodology provides guidelines for the development of an action representation, quantification of action, and the design of interactive feedback. As Part I of a two-part series, this paper presents key principles of the unified approach. Part II then describes the application of this approach within the implementation of the Adaptive Mixed Reality Rehabilitation (AMRR) system for stroke rehabilitation. CONCLUSIONS: The accompanying principles for composing novel mixed reality environments for stroke rehabilitation can advance the design and implementation of effective mixed reality systems for the clinical setting, and ultimately be adapted for home-based application. They furthermore can be applied to other rehabilitation needs beyond stroke.

A novel adaptive mixed reality system for stroke rehabilitation: principles, proof of concept, and preliminary application in 2 patients.

This article presents the principles of an adaptive mixed reality rehabilitation (AMRR) system, as well as the training process and results from 2 stroke survivors who received AMRR therapy, to illustrate how the system can be used in the clinic. The AMRR system integrates traditional rehabilitation practices with state-of-the-art computational and motion capture technologies to create an engaging environment to train reaching movements. The system provides real-time, intuitive, and integrated audio and visual feedback (based on detailed kinematic data) representative of goal accomplishment, activity performance, and body function during a reaching task. The AMRR system also provides a quantitative kinematic evaluation that measures the deviation of the stroke survivor's movement from an idealized, unimpaired movement. The therapist, using the quantitative measure and knowledge and observations, can adapt the feedback and physical environment of the AMRR system throughout therapy to address each participant's individual impairments and progress. Individualized training plans, kinematic improvements measured over the entire therapy period, and the changes in relevant clinical scales and kinematic movement attributes before and after the month-long therapy are presented for 2 participants. The substantial improvements made by both participants after AMRR therapy demonstrate that this system has the potential to considerably enhance the recovery of stroke survivors with varying impairments for both kinematic improvements and functional ability.

Design of a home-based adaptive mixed reality rehabilitation system for stroke survivors.

This paper presents the design of a home-based adaptive mixed reality system (HAMRR) for upper extremity stroke rehabilitation. The goal of HAMRR is to help restore motor function to chronic stroke survivors by providing an engaging long-term reaching task therapy at home. The system uses an intelligent adaptation scheme to create a continuously challenging and unique multi-year therapy experience. The therapy is overseen by a physical therapist, but day-to-day use of the system can be independently set up and completed by a stroke survivor. The HAMMR system tracks movement of the wrist and torso and provides real-time, post-trial, and post-set feedback to encourage the stroke survivor to self-assess his or her movement and engage in active learning of new movement strategies. The HAMRR system consists of a custom table, chair, and media center, and is designed to easily integrate into any home.

Interactive rehabilitation and dynamical analysis of scalp EEG.

Electroencephalography (EEG) has been used for decades to measure the brain's electrical activity. Planning and performing a complex movement (e.g., reaching and grasping) requires the coordination of muscles by electrical activity that can be recorded with scalp EEG from relevant regions of the cortex. Prior studies, utilizing motion capture and kinematic measures, have shown that an augmented reality feedback system for rehabilitation of stroke patients can help patients develop new motor plans and perform reaching tasks more accurately. Historically, traditional signal analysis techniques have been utilized to quantify changes in EEG when subjects perform common, simple movements. These techniques have included measures of event-related potentials in the time and frequency domains (e.g., energy and coherence measures). In this study, a more advanced, nonlinear, analysis technique, mutual information (MI), is applied to the EEG to capture the dynamics of functional connections between brain sites. In particular, the cortical activity that results from the planning and execution of novel reach trajectories by normal subjects in an augmented reality system was quantified by using statistically significant MI interactions between brain sites over time. The results show that, during the preparation for as well as the execution of a reach, the functional connectivity of the brain changes in a consistent manner over time, in terms of both the number and strength of cortical connections. A similar analysis of EEG from stroke patients may provide new insights into the functional deficiencies developed in the brain after stroke, and contribute to evaluation, and possibly the design, of novel therapeutic schemes within the framework of rehabilitation and BMI (brain machine interface).

Utilizing media arts principles for developing effective interactive neurorehabilitation systems.

This paper discusses how interactive neurorehabilitation systems can increase their effectiveness through systematic integration of media arts principles and practice. Media arts expertise can foster the development of complex yet intuitive extrinsic feedback displays that match the inherent complexity and intuitive nature of motor learning. Abstract, arts-based feedback displays can be powerful metaphors that provide re-contextualization, engagement and appropriate reward mechanisms for mature adults. Such virtual feedback displays must be seamlessly integrated with physical components to produce mixed reality training environments that promote active, generalizable learning. The proposed approaches are illustrated through examples from mixed reality rehabilitation systems developed by our team.

A computational framework for constructing interactive feedback for assisting motor learning.

New motion capture technologies are allowing detailed, precise and complete monitoring of movement through real-time kinematic analysis. However, a clinically relevant understanding of movement impairment through kinematic analysis requires the development of computational models that integrate clinical expertise in the weighing of the kinematic parameters. The resulting kinematics based measures of movement impairment would further need to be integrated with existing clinical measures of activity disability. This is a challenging process requiring computational solutions that can extract correlations within and between three diverse data sets: human driven assessment of body function, kinematic based assessment of movement impairment and human driven assessment of activity. We propose to identify and characterize different sensorimotor control strategies used by normal individuals and by hemiparetic stroke survivors acquiring a skilled motor task. We will use novel quantitative approaches to further our understanding of how human motor function is coupled to multiple and simultaneous modes of feedback. The experiments rely on a novel interactive tasks environment developed by our team in which subjects are provided with rich auditory and visual feedback of movement variables to drive motor learning. Our proposed research will result in a computational framework for applying virtual information to assist motor learning for complex tasks that require coupling of proprioception, vision audio and haptic cues. We shall use the framework to devise a computational tool to assist with therapy of stroke survivors. This tool will utilize extracted relationships in a pre-clinical setting to generate effective and customized rehabilitation strategies.

A low cost, adaptive mixed reality system for home-based stroke rehabilitation.

This paper presents a novel, low-cost, real-time adaptive multimedia environment for home-based upper extremity rehabilitation of stroke survivors. The primary goal of this system is to provide an interactive tool with which the stroke survivor can sustain gains achieved within the clinical phase of therapy and increase the opportunity for functional recovery. This home-based mediated system has low cost sensing, off the shelf components for the auditory and visual feedback, and remote monitoring capability. The system is designed to continue active learning by reducing dependency on real-time feedback and focusing on summary feedback after a single task and sequences of tasks. To increase system effectiveness through customization, we use data from the training strategy developed by the therapist at the clinic for each stroke survivor to drive automated system adaptation at the home. The adaptation includes changing training focus, selecting proper feedback coupling both in real-time and in summary, and constructing appropriate dialogues with the stroke survivor to promote more efficient use of the system. This system also allows the therapist to review participant's progress and adjust the training strategy weekly.

Mixed reality rehabilitation for stroke survivors promotes generalized motor improvements.

This paper presents results from a clinical study of stroke survivors using an adaptive, mixed-reality rehabilitation (AMRR) system for reach and grasp therapy. The AMRR therapy provides audio and visual feedback on the therapy task, based on detailed motion capture, that places the movement in an abstract, artistic context. This type of environment promotes the generalizability of movement strategies, which is shown through kinematic improvements on an untrained reaching task and higher clinical scale scores, in addition to kinematic improvements in the trained task.

An adaptive mixed reality training system for stroke rehabilitation.

This paper presents a novel mixed reality rehabilitation system used to help improve the reaching movements of people who have hemiparesis from stroke. The system provides real-time, multimodal, customizable, and adaptive feedback generated from the movement patterns of the subject's affected arm and torso during reaching to grasp. The feedback is provided via innovative visual and musical forms that present a stimulating, enriched environment in which to train the subjects and promote multimodal sensory-motor integration. A pilot study was conducted to test the system function, adaptation protocol and its feasibility for stroke rehabilitation. Three chronic stroke survivors underwent training using our system for six 75-min sessions over two weeks. After this relatively short time, all three subjects showed significant improvements in the movement parameters that were targeted during training. Improvements included faster and smoother reaches, increased joint coordination and reduced compensatory use of the torso and shoulder. The system was accepted by the subjects and shows promise as a useful tool for physical and occupational therapists to enhance stroke rehabilitation.

A portable, low-cost assessment device for reaching times.

We present the Systematic Test for Assessing Reaching Times (START), an interactive device for measuring reaction and movement times. START uses randomly lighted buttons to elicit reaches to a variety of target locations with a standardized setup. A magnetized wristband and reed switch monitor the hand rest position while capacitive touch sensors embedded in the buttons sense successful target reaches. A microcontroller measures reaching times at tens of milliseconds resolution. START can easily translate to the clinic because it is portable, easy to use and inexpensive to construct. We designed START for use with stroke patients and are using START to assess the progress of upper extremity rehabilitation of stroke survivors. It can be modified to fit a number of other motor control or cognitive tasks.

Interactive multimodal biofeedback for task-oriented neural rehabilitation.

Previous studies have suggested that task-oriented biofeedback training may be effective for functional motor improvement. The purpose of this project was to design an interactive, multimodal biofeedback system for the task-oriented training of goal-directed reaching. The central controller, based on a user context model, identifies the state of task performance using multisensing data and provides augmented feedback, through interactive 3D graphics and music, to encourage the patients' self-regulation and performance of the task. The design allows stroke patients to train with functional tasks, and receive real-time performance evaluation through successful processing of multimodal sensory feedback. In addition, the environment and training task is customizable. Overall, the system delivers an engaging training experience. Preliminary results of a pilot study involving stroke patients demonstrate the potential of the system to improve patients' reaching performance.