Development of a Vision-Guided Shared-Control System for Assistive Robotic Manipulators.

Assistive robotic manipulators (ARMs) provide a potential solution to mitigating the difficulties and lost independence associated with manipulation deficits in individuals with upper-limb impairments. However, achieving efficient control of an ARM can be a challenge due to the multiple degrees of freedom (DoFs) of an ARM that need to be controlled. This study describes the development of a vision-guided shared-control (VGS) system and how it is applied to a multi-step drinking task. The VGS control allows the user to control the gross motion of the ARM via teleoperation and commands the ARM to autonomously perform fine manipulation. A bench-top test of the autonomous actions showed that success rates for different subtasks ranged from 80% to 100%. An evaluation with three test pilots showed that the overall task performance, in terms of success rate, task completion time, and joystick mode-switch frequency, was better with VGS than with teleoperation. Similar trends were observed with a case participant with a spinal cord injury. While his performance was better and he perceived a smaller workload with VGS, his perceived usability for VGS and teleoperation was similar. More work is needed to further improve and test VGS on participants with disabilities.

Reliability of 3D Depth Motion Sensors for Capturing Upper Body Motions and Assessing the Quality of Wheelchair Transfers.

Wheelchair users must use proper technique when performing sitting-pivot-transfers (SPTs) to prevent upper extremity pain and discomfort. Current methods to analyze the quality of SPTs include the TransKinect, a combination of machine learning (ML) models, and the Transfer Assessment Instrument (TAI), to automatically score the quality of a transfer using Microsoft Kinect V2. With the discontinuation of the V2, there is a necessity to determine the compatibility of other commercial sensors. The Intel RealSense D435 and the Microsoft Kinect Azure were compared against the V2 for inter- and intra-sensor reliability. A secondary analysis with the Azure was also performed to analyze its performance with the existing ML models used to predict transfer quality. The intra- and inter-sensor reliability was higher for the Azure and V2 (n = 7; ICC = 0.63 to 0.92) than the RealSense and V2 (n = 30; ICC = 0.13 to 0.7) for four key features. Additionally, the V2 and the Azure both showed high agreement with each other on the ML outcomes but not against a ground truth. Therefore, the ML models may need to be retrained ideally with the Azure, as it was found to be a more reliable and robust sensor for tracking wheelchair transfers in comparison to the V2.

Usability Evaluation of a Novel Robotic Power Wheelchair for Indoor and Outdoor Navigation.

OBJECTIVE: To compare the Mobility Enhancement roBotic (MEBot) wheelchair's capabilities with commercial electric-powered wheelchairs (EPWs) by performing a systematic usability evaluation. DESIGN: Usability in effectiveness, efficacy, and satisfaction was evaluated using quantitative measures. A semistructured interview was employed to gather feedback about the users' interaction with MEBot. SETTING: Laboratory testing of EPW driving performance with 2 devices in a controlled setting simulating common EPW driving tasks. PARTICIPANTS: A convenience sample of expert EPW users (N=12; 9 men, 3 women) with an average age of 54.7±10.9 years and 16.3± 8.1 years of EPW driving experience. INTERVENTIONS: Not applicable. MAIN OUTCOME MEASURES: Powered mobility clinical driving assessment (PMCDA), Satisfaction Questionnaire, National Aeronautics and Space Administration's Task Load Index. RESULTS: Participants were able to perform significantly higher number of tasks (P=.004), with significantly higher scores in both the adequacy-efficacy (P=.005) and the safety (P=.005) domains of the PMCDA while using MEBot over curbs and cross-slopes. However, participants reported significantly higher mental demand (P=.005) while using MEBot to navigate curbs and cross-slopes due to MEBot's complexity to perform its mobility applications which increased user's cognitive demands. CONCLUSIONS: Overall, this usability evaluation demonstrated that MEBot is a promising EPW device to use indoors and outdoors with architectural barriers such as curbs and cross-slopes. Current design limitations were highlighted with recommendations for further improvement.

Smart speakers and skill use: what do we know?

Personal smart technologies are becoming increasingly interwoven into everyday life, yet the usability and usefulness for some of these off-the-shelf technologies for persons with disabilities has yet to be determined. Smart speakers with both their native and downloadable functionalities (skills) have great potential to support individuals with disabilities through communication functionalities, smart home integrations, and more. However, the potential for usefulness does not always translate to how something is actually perceived or used in the real-world. Therefore, the objective of this qualitative study was to gather insight from individuals with disabilities on their experiences with smart speakers and smart speaker skills. Participant feedback highlighted several primary themes: (1) external factors that might influence extent of, and advancement in smart speaker use, (2) Smart speaker use barriers, (3) Smart speaker use facilitators, and (4) Smart speaker uses specific to individuals with disabilities. Continued research and development is needed to help ensure that commercially available technologies are designed with universal design principles that will ensure accessibility for all potential users.

Smart speaker skills are being used for smart home management by persons with disabilities, but these functionalities are not without barriers.Persons with various disabilities should be involved in the development and translation of smart speaker functions that are intended to, or that could, support the needs of this population.

TransKinect: a computer vision and machine learning clinical decision support system for automatic independent wheelchair transfer technique assessment.

BACKGROUND: Physical and occupational therapists provide routine care for manual wheelchair users and are responsible for training and assessing the quality of transfers. These transfers can produce large loads on the upper extremity joints if improper sitting-pivot-technique is used. Methods to assess quality of transfers include the Transfer Assessment Instrument, a clinically validated tool derived from quantitative biomechanical features; however, adoption of this tool is low due to the complex usage requirements and speed of typical transfers. OBJECTIVE: The objective of this study is to develop and validate a computer vison and machine learning solution to better implement the Transfer Assessment Instrument in clinical settings. METHODS: The prototype system, TransKinect, consists of an infrared depth sensor and a custom software application; usability testing was carried out with fifteen therapists who performed two transfer assessments with the TransKinect. Proficiency in using features, usability, acceptability and satisfaction were analysed with validated surveys and themes were extracted from the qualitative feedback. RESULTS: The therapists were able to successfully complete the transfer quality assessments with 86.7 ± 5.4% proficiency. Total scores for System Usability Scale (77.6 ± 14.7%) and Questionnaire for User Interface Satisfaction (83.5 ± 8.7%) indicated that the system was usable and satisfactory. Qualitative feedback indicated that TransKinect was user-friendly, easy to learn, and had high potential. DISCUSSION: The results support TransKinect as a potential clinical decision support system for therapists for the comprehensive assessment of independent transfer technique. Future research is needed to investigate the utility and acceptance of TransKinect in real clinical environments. Implications for RehabilitationMachine learning and computer vision can be used to analyze transfer techniqueTransKinect is a usable and user-friendly means for therapists to automate analysisSummary reports and videos of transfers show high potential for clinical useAdoption of TransKinect can increase quality of care for manual wheelchair users.

Functional assessment and performance evaluation for assistive robotic manipulators: Literature review.

CONTEXT: The user interface development of assistive robotic manipulators can be traced back to the 1960s. Studies include kinematic designs, cost-efficiency, user experience involvements, and performance evaluation. This paper is to review studies conducted with clinical trials using activities of daily living (ADLs) tasks to evaluate performance categorized using the International Classification of Functioning, Disability, and Health (ICF) frameworks, in order to give the scope of current research and provide suggestions for future studies. METHODS: We conducted a literature search of assistive robotic manipulators from 1970 to 2012 in PubMed, Google Scholar, and University of Pittsburgh Library System - PITTCat. RESULTS: Twenty relevant studies were identified. CONCLUSION: Studies were separated into two broad categories: user task preferences and user-interface performance measurements of commercialized and developing assistive robotic manipulators. The outcome measures and ICF codes associated with the performance evaluations are reported. Suggestions for the future studies include (1) standardized ADL tasks for the quantitative and qualitative evaluation of task efficiency and performance to build comparable measures between research groups, (2) studies relevant to the tasks from user priority lists and ICF codes, and (3) appropriate clinical functional assessment tests with consideration of constraints in assistive robotic manipulator user interfaces. In addition, these outcome measures will help physicians and therapists build standardized tools while prescribing and assessing assistive robotic manipulators.

Development of an advanced mobile base for personal mobility and manipulation appliance generation II robotic wheelchair.

BACKGROUND: This paper describes the development of a mobile base for the Personal Mobility and Manipulation Appliance Generation II (PerMMA Gen II robotic wheelchair), an obstacle-climbing wheelchair able to move in structured and unstructured environments, and to climb over curbs as high as 8 inches. The mechanical, electrical, and software systems of the mobile base are presented in detail, and similar devices such as the iBOT mobility system, TopChair, and 6X6 Explorer are described. FINDINGS: The mobile base of PerMMA Gen II has two operating modes: "advanced driving mode" on flat and uneven terrain, and "automatic climbing mode" during stair climbing. The different operating modes are triggered either by local and dynamic conditions or by external commands from users. A step-climbing sequence, up to 0.2 m, is under development and to be evaluated via simulation. The mathematical model of the mobile base is introduced. A feedback and a feed-forward controller have been developed to maintain the posture of the passenger when driving over uneven surfaces or slopes. The effectiveness of the controller has been evaluated by simulation using the open dynamics engine tool. CONCLUSION: Future work for PerMMA Gen II mobile base is implementation of the simulation and control on a real system and evaluation of the system via further experimental tests.

Classification of wheelchair pressure relief maneuvers using changes in center of pressure and weight on the seat.

BACKGROUND: Pressure injuries from prolonged sitting are a significant problem for wheelchair users incurring high costs in healthcare expenditures and reducing quality-of-life. There is a need to improve pressure relief training and adherence in a variety of settings. OBJECTIVE: To identify effective common wheelchair pressure relief (PR) manoeuvres based on changes to users' seated centre of pressure (CoP) and seated weight. PARTICIPANTS: 20 individuals who use manual wheelchairs as their primary means of mobility. METHODS: Participants performed 5 types of PR including seated push-ups, leftward, rightward, forward, and backward leans-while sitting in a wheelchair equipped with a custom instrumented seat pan support. Data were analysed using both clustering and decision tree approaches to identify types of PR. RESULTS: Both clustering and decision tree approaches were able to identify and classify PR though neither could accurately distinguish between forward and backward PR. CONCLUSION: Changes in the centre of pressure and the total weight on the wheelchair's seat can be used to automatically characterise type, amplitude and duration of pressure relief manoeuvres. Building such a classification and quality assessment scheme into an algorithm could enable a virtual coaching system to track users' pressure relief behaviour and make suggestions to improve adherence with clinical recommendations.IMPLICATIONS FOR REHABILITATIONMultiple bending beam load cells can be used to measure wheelchair users' seated centre of pressure independent of type of cushion used.Both cluster analysis and decision tree algorithms can classify commonly practiced pressure reliefs by measuring changes to the centre of pressure and total weight on the wheelchair's seat.The combination of force sensing for centre of pressure determination and either algorithm could serve as the basis for an application to coach wheelchair users to do effective pressure reliefs.

Anthropomorphic model rigid loading indenter with embedded sensor development for wheelchair cushion standard testing.

Develop an anthropomorphic model cushion rigid loading indenter with embedded sensors (AMCRLI-ES) to assess compression and shear forces at key locations such as trochanters and ischial tuberosities. The sensor design was optimized using finite element analysis. The AMCRLI-ES was designed with the same dimensions as specified in ISO 16840-2 tests. The AMCRLI-ES is divided into eight independent sections, and each section consists of one 3-axis load cell sensor to measure compression and shear forces normal to the compression direction. Six commercial cushions were tested using the AMCRLI-ES with standard ISO 16840-2 testing procedures. Statistical differences were found for energy dissipation between cushions. Statistical differences (p < 0.001) were found in all stiffness values. Test results showed that energy dissipation (ED) was correlated with hysteresis at 500 N with moderate to high Pearson product correlation r = -0.537, p = 0.022. The hysteresis at 250 N did not show a statistical correlation with ED. The AMCRLI-ES demonstrated the ability to measure compression and shear forces at key locations on the cushion including the thigh, trochanter, ischial tuberosity, and sacral area. It provides in-depth information about how the weight was distributed on the cushions.

Participatory action design and engineering of a manual wheelchair virtual coach including in-home and community usage.

BACKGROUND: Current clinical practice guidelines (CPG) recommend periodic pressure redistribution (PR) to alter sitting pressure and reduce the risk of developing pressure injuries (PI). Individuals who have strength and trunk stability are asked to perform PR such as wheelies, leaning laterally, and forward-leaning to minimize the duration of pressure acting on the same region of the body. OBJECTIVE: Our long-term objective is to build upon previous research and development to create a more effective device for improving PR training and adherence to CPG among manual wheelchair users (MWU). Through this study, we employed a participatory action design and engineering (PADE) approach in developing the hardware and user interface to increase the likelihood of eventually yielding a device effective for both MWU and clinicians. PARTICIPANTS: Focus Groups: Ten clinicians - 6 physical therapists, 3 occupational therapists, and one registered nurse, and 10 MWU with spinal cord injuries (SCI) who reported using their wheelchairs 40-80 h per week. Five-Day Assessment: Five male MWU with SCI who had been wheelchair users for 18.5 ± 16.2 years. Four-Week Investigation: The 7 participants with SCI were 5 males and 2 females, who had been wheelchair users for 24.7 ± 17.0 years. METHODS: A PADE approach was used to improve upon a manual wheelchair virtual coaching system for people with SCI. The system comprises a seat support instrumented with force sensors, software algorithms to detect PR, and a smart phone app for user interface. The methods included three stages: multiple focus groups, a five-day evaluation phase with participants using their own wheelchairs in their homes and communities, and a 4-week assessment with improvements made based on the 5-day results by users with their own wheelchairs in their homes and communities. RESULTS: The focus groups yielded guidance for ergonomics, user interface, charging frequency, and key dimensions and mass. The 5-day study identified mechanical, electrical, and connectivity challenges, which were resolved before the 4-week study. The 4-week trial suggested that participants performed PR less frequently than clinically recommended and provided an indication of the types of maneuvers that they performed. CONCLUSION: A prototype manual wheelchair virtual coaching system was developed using a PADE process. The system was able to detect and record PR in home and community environments. Following improvements identified in this study, a future version will be tested with additional users to determine whether it can improve adherence to PR guidance.

Automating the Clinical Assessment of Independent Wheelchair Sitting Pivot Transfer Techniques.

BACKGROUND: Using proper transfer technique can help to reduce forces and prevent secondary injuries. However, current assessment tools rely on the ability to subjectively identify harmful movement patterns. OBJECTIVES: The purpose of the study was to determine the accuracy of using a low-cost markerless motion capture camera and machine learning methods to evaluate the quality of independent wheelchair sitting pivot transfers. We hypothesized that the algorithms would be able to discern proper (low risk) and improper (high risk) wheelchair transfer techniques in accordance with component items on the Transfer Assessment Instrument (TAI). METHODS: Transfer motions of 91 full-time wheelchair users were recorded and used to develop machine learning classifiers that could be used to discern proper from improper technique. The data were labeled using the TAI item scores. Eleven out of 18 TAI items were evaluated by the classifiers. Motion variables from the Kinect were inputted as the features. Random forests and k-nearest neighbors algorithms were chosen as the classifiers. Eighty percent of the data were used for model training and hyperparameter turning. The validation process was performed using 20% of the data as the test set. RESULTS: The area under the receiver operating characteristic curve of the test set for each item was over 0.79. After adjusting the decision threshold, the precisions of the models were over 0.87, and the model accuracies were over 71%. CONCLUSION: The results show promise for the objective assessment of the transfer technique using a low cost camera and machine learning classifiers.

Usability evaluation of attitude control for a robotic wheelchair for tip mitigation in outdoor environments.

Tips and falls are the most prominent causes of wheelchair accidents that occur when driving on uneven terrains and less accessible environments. The Mobility Enhancement Robotic Wheelchair (MEBot) was designed to improve the stability of Electric Powered Wheelchairs (EPW) when driving over these environments. MEBot offers six independently height-adjustable wheels to control attitude of its seat over uneven and angled terrains. Its attitude control application uses an inertial measurement unit to detect seat angles changes to adjust each wheel-height accordingly. MEBot was compared to commercial EPWs in terms of EPW performance (seat angle changes and response time) and participant perception (satisfaction and task-load demand) towards each device. Ten participants drove their own EPW and MEBot for five trials each through driving tasks that replicated outdoor environments. Results showed less change in the pitch angle when driving up and down a 10° slope using MEBot (5.6 ± 1.6°, 6.6 ± 0.5°) compared to the participants' own EPW (14.6 ± 2.6°, 12.1 ± 2.6°). However, MEBot required 7.8 ± 3.0 s to self-adjust to the minimum angle when driving over the tasks. Participants reported no difference in satisfaction and task load demand between EPWs due to similarities in comfort and ease-of-use. Improving the speed and efficiency of MEBot's attitude control application will be addressed in future work based upon participants' feedback.

Design and operation verification of an automated pressure mapping and modulating seat cushion for pressure ulcer prevention.

A sensorized air cell-based seat cushion system was developed to address the issues of loading magnitude and duration at a sitting interface to aid in reducing risk of sitting acquired pressure ulcers. This system is capable of pressure mapping, redistribution, and offloading which were verified using an anthropomorphic model and a human subject. The system is comprised of an air cell array cushion, a pneumatic control unit, and a graphical user interface. ISO load deflection testing confirmed that the cushion's loading response is comparable to commercial air cell-based seat cushions. Testing demonstrated that the internal pressure of the air cells are indicative of interface pressure and can be used as input to pressure modulating algorithms. Uniform pressure distribution was achieved through automated pressure redistribution algorithm implementation where the immersion of a subject into the seat cushion increased and interface pressure decreased. High pressure point identification and automatic offloading were performed in which newly created high pressure points were addressed using subsequent redistribution. Pressure mapping enabled offloading and redistribution can objectively manage the effects of loading magnitude and duration at the sitting interface.

Performance evaluation of 3D vision-based semi-autonomous control method for assistive robotic manipulator.

We developed a 3D vision-based semi-autonomous control interface for assistive robotic manipulators. It was implemented based on one of the most popular commercially available assistive robotic manipulator combined with a low-cost depth-sensing camera mounted on the robot base. To perform a manipulation task with the 3D vision-based semi-autonomous control interface, a user starts operating with a manual control method available to him/her. When detecting objects within a set range, the control interface automatically stops the robot, and provides the user with possible manipulation options through audible text output, based on the detected object characteristics. Then, the system waits until the user states a voice command. Once the user command is given, the control interface drives the robot autonomously until the given command is completed. In the empirical evaluations conducted with human subjects from two different groups, it was shown that the semi-autonomous control can be used as an alternative control method to enable individuals with impaired motor control to more efficiently operate the robot arms by facilitating their fine motion control. The advantage of semi-autonomous control was not so obvious for the simple tasks. But, for the relatively complex real-life tasks, the 3D vision-based semi-autonomous control showed significantly faster performance. Implications for Rehabilitation A 3D vision-based semi-autonomous control interface will improve clinical practice by providing an alternative control method that is less demanding physically as well cognitively. A 3D vision-based semi-autonomous control provides the user with task specific intelligent semiautonomous manipulation assistances. A 3D vision-based semi-autonomous control gives the user the feeling that he or she is still in control at any moment. A 3D vision-based semi-autonomous control is compatible with different types of new and existing manual control methods for ARMs.

Task-Oriented Performance Evaluation for Assistive Robotic Manipulators: A Pilot Study.

OBJECTIVE: The objectives of this study were to evaluate the performance of commercially available assistive robotic manipulators (ARMs) user interfaces and to investigate the concurrent validity and sensitivity to change with task-oriented performance evaluation tools (TO-PETs) for ARMs. DESIGN: This was a nonblinded randomized controlled study with power-wheelchair users with upper-extremity impairments (N = 10). Participants were trained to use 2 ARMs with their respective original user interfaces (keypad and joystick) and evaluated the performance using TO-PET and the adapted Wolf Motor Function Test (WMFT-ARM). Task completion time, ISO 9241-9 throughput, trajectory parameters, NASA-TLX, and questionnaires were the main outcome measurements. Concurrent validity and sensitivity were evaluated. RESULTS: Statistical differences were found in ISO 9241-9 throughput between the 2 user interfaces for the single motion tasks and WMFT-ARM. However, there was no statistical difference found on the self-reported perceived workload and ease of use. Moderate to high correlation was found between the TO-PET and WMFT-ARM (P < 0.001). The TO-PET demonstrated higher Cohen d (0.910-1.085) than the WMFT-ARM. CONCLUSIONS: The findings of this study provide a preliminary comparison between 2 commercial ARMs with their different user interfaces among novice ARM users. Recommendations for training and evaluation were revealed.

Performance Evaluation of a Mobile Touchscreen Interface for Assistive Robotic Manipulators: A Pilot Study.

Background: Assistive robotic manipulators (ARMs) have been developed to provide enhanced assistance and independence in performance of daily activities among people with spinal cord injury when a caregiver is not on site. However, the current commercial ARM user interfaces (UIs) may be difficult to learn and control. A touchscreen mobile UI was developed to overcome these challenges. Objective: The object of this study was to evaluate the performance between 2 ARM UIs, touchscreen and the original joystick, using an ARM evaluation tool (ARMET). Methods: This is a pilot study of people with upper extremity impairments (N = 8). Participants were trained on 2 UIs, and then they chose one to use when performing 3 tasks on the ARMET: flipping a toggle switch, pushing down a door handle, and turning a knob. Task completion time, mean velocity, and open interviews were the main outcome measurements. Results: Among 8 novice participants, 7 chose the touchscreen UI and 1 chose the joystick UI. All participants could complete the ARMET tasks independently. Use of the touchscreen UI resulted in enhanced ARMET performance (higher mean moving speed and faster task completion). Conclusions: Mobile ARM UIs demonstrated easier learning experience, less physical effort, and better ARMET performance. The improved performance, the accessibility, and lower physical effort suggested that the touchscreen UI might be an efficient tool for the ARM users.

Design and evaluation of a seat orientation controller during uneven terrain driving.

Electric powered wheelchairs (EPWs) are essential devices for people with disabilities as aids for mobility and quality of life improvement. However, the design of currently available common EPWs is still limited and makes it challenging for the users to drive in both indoor and outdoor environments such as uneven surfaces, steep hills, or cross slopes, making EPWs susceptible to loss of stability and at risk for falls. An alternative wheel-legged robotic wheelchair, "MEBot", was designed to improve the safety and mobility of EPW users in both indoor and outdoor environments. MEBot is able to elevate its six wheels using pneumatic actuators, as well to detect changes in the seat angle using a gyroscope and accelerometer. This capability enables MEBot to provide sensing for a dynamic self-leveling seat application that can maintain the center of mass within the boundaries of the wheelchair, thereby, improving EPW safety. To verify the effectiveness of the application, MEBot was tested on a motion platform with six degrees of freedom to simulate different slopes that could be experienced by the EPW in outdoor environments. The results demonstrate the robustness of the application to maintain the wheelchair seat in a horizontal reference against changes in the ground angle.

Stability analysis of electrical powered wheelchair-mounted robotic-assisted transfer device.

The ability of people with disabilities to live in their homes and communities with maximal independence often hinges, at least in part, on their ability to transfer or be transferred by an assistant. Because of limited resources and the expense of personal care, robotic transfer assistance devices will likely be in great demand. An easy-to-use system for assisting with transfers, attachable to electrical powered wheelchairs (EPWs) and readily transportable, could have a significant positive effect on the quality of life of people with disabilities. We investigated the stability of our newly developed Strong Arm, which is attached and integrated with an EPW to assist with transfers. The stability of the system was analyzed and verified by experiments applying different loads and using different system configurations. The model predicted the distributions of the system's center of mass very well compared with the experimental results. When real transfers were conducted with 50 and 75 kg loads and an 83.25 kg dummy, the current Strong Arm could transfer all weights safely without tip-over. Our modeling accurately predicts the stability of the system and is suitable for developing better control algorithms to enhance the safety of the device.

Autonomous function of wheelchair-mounted robotic manipulators to perform daily activities.

Autonomous functions for wheelchair-mounted robotic manipulators (WMRMs) allow a user to focus more on the outcome from the task - for example, eating or drinking, instead of moving robot joints through user interfaces. In this paper, we introduce a novel personal assistive robotic system based on a position-based visual servoing (PBVS) approach. The system was evaluated with a complete drinking task, which included recognizing the location of the drink, picking up the drink from a start location, conveying the drink to the proximity of the user's mouth without spilling, and placing the drink back on the table. For a drink located in front of the wheelchair, the success rate was nearly 100%. Overall, the total time of completing drinking task is within 40 seconds.