A multifactorial fall risk assessment system for older people utilizing a low-cost, markerless Microsoft Kinect.

Falls among older people are a major health concern. This study aims to develop a multifactorial fall risk assessment system for older people using a low-cost, markerless Microsoft Kinect. A Kinect-based test battery was designed to comprehensively assess major fall risk factors. A follow-up experiment was conducted with 102 older participants to assess their fall risks. Participants were divided into high and low fall risk groups based on their prospective falls over a 6-month period. Results showed that the high fall risk group performed significantly worse on the Kinect-based test battery. The developed random forest classification model achieved an average classification accuracy of 84.7%. In addition, the individual's performance was computed as the percentile value of a normative database to visualise deficiencies and targets for intervention. These findings indicate that the developed system can not only screen out 'at risk' older individuals with good accuracy, but also identify potential fall risk factors for effective fall intervention.Practitioner summary: Falls are the leading cause of injuries in older people. We newly developed a multifactorial fall risk assessment system for older people utilising a low-cost, markerless Kinect. Results showed that the developed system can screen out 'at risk' individuals and identify potential risk factors for effective fall intervention.

A Simple Method to Optimally Select Upper-Limb Joint Angle Trajectories from Two Kinect Sensors during the Twisting Task for Posture Analysis.

A trunk-twisting posture is strongly associated with physical discomfort. Measurement of joint kinematics to assess physical exposure to injuries is important. However, using a single Kinect sensor to track the upper-limb joint angle trajectories during twisting tasks in the workplace is challenging due to sensor view occlusions. This study provides and validates a simple method to optimally select the upper-limb joint angle data from two Kinect sensors at different viewing angles during the twisting task, so the errors of trajectory estimation can be improved. Twelve healthy participants performed a rightward twisting task. The tracking errors of the upper-limb joint angle trajectories of two Kinect sensors during the twisting task were estimated based on concurrent data collected using a conventional motion tracking system. The error values were applied to generate the error trendlines of two Kinect sensors using third-order polynomial regressions. The intersections between two error trendlines were used to define the optimal data selection points for data integration. The finding indicates that integrating the outputs from two Kinect sensor datasets using the proposed method can be more robust than using a single sensor for upper-limb joint angle trajectory estimations during the twisting task.

Simple action for depression detection: using kinect-recorded human kinematic skeletal data.

BACKGROUND: Depression, a common worldwide mental disorder, which brings huge challenges to family and social burden around the world is different from fluctuant emotion and psychological pressure in their daily life. Although body signs have been shown to present manifestations of depression in general, few researches focus on whole body kinematic cues with the help of machine learning methods to aid depression recognition. Using the Kinect V2 device to record participants' simple kinematic skeleton data of the participant's body joints, the presented spatial features and low-level features is directly extracted from the record original Kinect-3D coordinates. This research aimed to constructed machine learning model with the preprocessed data importing, which could be used for depression automatic classification. METHODS: Considering some patients' conditions and current status and refer to psychiatrists' advices, simple and significant designed stimulus task will lead human skeleton data collection job. With original Kinect skeleton data extracting and preprocessing, the proposed experiment demonstrated four strong machine learning tools: Support Vector Machine, Logistic Regression, Random Forest and Gradient Boosting. Using the precision, recall, sensitivity, specificity, roc-curve, confusion matrix et.al, indicators were calculated as the measurement of methods, which were commonly used to evaluate classification methodologies. RESULTS: Across screened 64 pairs with age and gender totally matching in depression and control group, and Gradient Boosting achieved the best performance with the prediction accuracy of 76.92%. Sorted by female (54.69%) and male for the gender-based depression recognition, we applied best performance classifier Gradient Boosting got prediction accuracy of 66.67% in the male group, and 71.73% in the female group. Utilizing the best model Gradient Boosting for age-based classification, prediction accuracy got 76.92% in the older group (age >40, 50% of total) and 53.85% accuracy in the younger group (age <= 40). CONCLUSION: The depression and non-depression individuals can be well classified by computational models using Kinect captured skeletal data. The Gradient Boosting, an excellent machine learning tool, get the performance in the four methods we demonstrated. Meanwhile, in the gender-based depression classification also gets reasonable accuracy. In particular, the recognition results of the old group are significantly better than that of the young group. All these findings suggest that kinematic skeletal data based depression recognition can be applied as an effective tool for assisting in depression analysis.

The Accuracy of the Microsoft Kinect V2 Sensor for Human Gait Analysis. A Different Approach for Comparison with the Ground Truth.

Several studies have examined the accuracy of the Kinect V2 sensor during gait analysis. Usually the data retrieved by the Kinect V2 sensor are compared with the ground truth of certified systems using a Euclidean comparison. Due to the Kinect V2 sensor latency, the application of a uniform temporal alignment is not adequate to compare the signals. On that basis, the purpose of this study was to explore the abilities of the dynamic time warping (DTW) algorithm to compensate for sensor latency (3 samples or 90 ms) and develop a proper accuracy estimation. During the experimental stage, six iterations were performed using the a dual Kinect V2 system. The walking tests were developed at a self-selected speed. The sensor accuracy for Euclidean matching was consistent with that reported in previous studies. After latency compensation, the sensor accuracy demonstrated considerably lower error rates for all joints. This demonstrated that the accuracy was underestimated due to the use of inappropriate comparison techniques. On the contrary, DTW is a potential method that compensates for the sensor latency, and works sufficiently in comparison with certified systems.

A Game-Based Rehabilitation System for Upper-Limb Cerebral Palsy: A Feasibility Study.

Game-based rehabilitation systems provide an effective tool to engage cerebral palsy patients in physical exercises within an exciting and entertaining environment. A crucial factor to ensure the effectiveness of game-based rehabilitation systems is to assess the correctness of the movements performed by the patient during the game-playing sessions. In this study, we propose a game-based rehabilitation system for upper-limb cerebral palsy that includes three game-based exercises and a computerized assessment method. The game-based exercises aim to engage the participant in shoulder flexion, shoulder horizontal abduction/adduction, and shoulder adduction physical exercises that target the right arm. Human interaction with the game-based rehabilitation system is achieved using a Kinect sensor that tracks the skeleton joints of the participant. The computerized assessment method aims to assess the correctness of the right arm movements during each game-playing session by analyzing the tracking data acquired by the Kinect sensor. To evaluate the performance of the computerized assessment method, two groups of participants volunteered to participate in the game-based exercises. The first group included six cerebral palsy children and the second group included twenty typically developing subjects. For every participant, the computerized assessment method was employed to assess the correctness of the right arm movements in each game-playing session and these computer-based assessments were compared with matching gold standard evaluations provided by an experienced physiotherapist. The results reported in this study suggest the feasibility of employing the computerized assessment method to evaluate the correctness of the right arm movements during the game-playing sessions.

Improving Cognitive Visual-Motor Abilities in Individuals with Down Syndrome.

Down syndrome causes a reduction in cognitive abilities, with visual-motor skills being particularly affected. In this work, we have focused on this skill in order to stimulate better learning. The proposal relies on stimulating the cognitive visual-motor skills of individuals with Down Syndrome (DS) using exercises with a gestural interaction platform based on the KINECT sensor named TANGO:H, the goal being to improve them. To validate the proposal, an experimental single-case study method was designed using two groups: a control group and an experimental one, with similar cognitive ages. Didactic exercises were provided to the experimental group using visual cognitive stimulation. These exercises were created on the TANGO:H Designer, a platform that was designed for gestural interaction using the KINECT sensor. As a result, TANGO:H allows for visual-motor cognitive stimulation through the movement of hands, arms, feet and head. The "Illinois Test of Psycholinguistic Abilities (ITPA)" was applied to both groups as a pre-test and post-test in its four reference sections: visual comprehension, visual-motor sequential memory, visual association, and visual integration. Two checks were made, one using the longitudinal comparison of the pre-test/post-test of the experimental group, and another that relied on comparing the difference of the means of the pre-test/post-test. We also used an observational methodology for the working sessions from the experimental group. Although the statistical results do not show significant differences between the two groups, the results of the observations exhibited an improvement in visual-motor cognitive skills.

An Augmented Reality Based Human-Robot Interaction Interface Using Kalman Filter Sensor Fusion.

In this paper, the application of Augmented Reality (AR) for the control and adjustment of robots has been developed, with the aim of making interaction and adjustment of robots easier and more accurate from a remote location. A LeapMotion sensor based controller has been investigated to track the movement of the operator hands. The data from the controller allows gestures and the position of the hand palm's central point to be detected and tracked. A Kinect V2 camera is able to measure the corresponding motion velocities in x, y, z directions after our investigated post-processing algorithm is fulfilled. Unreal Engine 4 is used to create an AR environment for the user to monitor the control process immersively. Kalman filtering (KF) algorithm is employed to fuse the position signals from the LeapMotion sensor with the velocity signals from the Kinect camera sensor, respectively. The fused/optimal data are sent to teleoperate a Baxter robot in real-time by User Datagram Protocol (UDP). Several experiments have been conducted to test the validation of the proposed method.

Graph Cut-Based Human Body Segmentation in Color Images Using Skeleton Information from the Depth Sensor.

Segmentation of human bodies in images is useful for a variety of applications, including background substitution, human activity recognition, security, and video surveillance applications. However, human body segmentation has been a challenging problem, due to the complicated shape and motion of a non-rigid human body. Meanwhile, depth sensors with advanced pattern recognition algorithms provide human body skeletons in real time with reasonable accuracy. In this study, we propose an algorithm that projects the human body skeleton from a depth image to a color image, where the human body region is segmented in the color image by using the projected skeleton as a segmentation cue. Experimental results using the Kinect sensor demonstrate that the proposed method provides high quality segmentation results and outperforms the conventional methods.

Investigating 2-D and 3-D Proximal Remote Sensing Techniques for Vineyard Yield Estimation.

Vineyard yield estimation provides the winegrower with insightful information regarding the expected yield, facilitating managerial decisions to achieve maximum quantity and quality and assisting the winery with logistics. The use of proximal remote sensing technology and techniques for yield estimation has produced limited success within viticulture. In this study, 2-D RGB and 3-D RGB-D (Kinect sensor) imagery were investigated for yield estimation in a vertical shoot positioned (VSP) vineyard. Three experiments were implemented, including two measurement levels and two canopy treatments. The RGB imagery (bunch- and plant-level) underwent image segmentation before the fruit area was estimated using a calibrated pixel area. RGB-D imagery captured at bunch-level (mesh) and plant-level (point cloud) was reconstructed for fruit volume estimation. The RGB and RGB-D measurements utilised cross-validation to determine fruit mass, which was subsequently used for yield estimation. Experiment one's (laboratory conditions) bunch-level results achieved a high yield estimation agreement with RGB-D imagery (r2 = 0.950), which outperformed RGB imagery (r2 = 0.889). Both RGB and RGB-D performed similarly in experiment two (bunch-level), while RGB outperformed RGB-D in experiment three (plant-level). The RGB-D sensor (Kinect) is suited to ideal laboratory conditions, while the robust RGB methodology is suitable for both laboratory and in-situ yield estimation.

Emotion Recognition from Skeletal Movements.

Automatic emotion recognition has become an important trend in many artificial intelligence (AI) based applications and has been widely explored in recent years. Most research in the area of automated emotion recognition is based on facial expressions or speech signals. Although the influence of the emotional state on body movements is undeniable, this source of expression is still underestimated in automatic analysis. In this paper, we propose a novel method to recognise seven basic emotional states-namely, happy, sad, surprise, fear, anger, disgust and neutral-utilising body movement. We analyse motion capture data under seven basic emotional states recorded by professional actor/actresses using Microsoft Kinect v2 sensor. We propose a new representation of affective movements, based on sequences of body joints. The proposed algorithm creates a sequential model of affective movement based on low level features inferred from the spacial location and the orientation of joints within the tracked skeleton. In the experimental results, different deep neural networks were employed and compared to recognise the emotional state of the acquired motion sequences. The experimental results conducted in this work show the feasibility of automatic emotion recognition from sequences of body gestures, which can serve as an additional source of information in multimodal emotion recognition.

Automated functional upper limb evaluation of patients with Friedreich ataxia using serious games rehabilitation exercises.

BACKGROUND: Friedreich ataxia (FRDA) is a disease with neurological and systemic involvement. Clinical assessment tools commonly used for FRDA become less effective in evaluating decay in patients with advanced FRDA, particularly when they are in a wheelchair. Further motor worsening mainly impairs upper limb function. In this study, we tested if serious games (SG) developed for rehabilitation can be used as an assessment tool for upper limb function even in patients with advanced FRDA. METHODS: A specific SG has been developed for physical rehabilitation of patients suffering from neurologic diseases. The use of this SG, coupled with Kinect sensor, has been validated to perform functional evaluation of the upper limbs with healthy subjects across lifespan. Twenty-seven FRDA patients were included in the study. Patients were invited to perform upper limb rehabilitation exercises embedded in SG. Motions were recorded by the Kinect and clinically relevant parameters were extracted from the collected motions. We tested if the existence of correlations between the scores from the serious games and the severity of the disease using clinical assessment tools commonly used for FRDA. Results of patients were compared with a group a healthy subjects of similar age. RESULTS: Very highly significant differences were found for time required to perform the exercise (increase of 76%, t(68) = 7.22, P < 0.001) and for accuracy (decrease of 6%, t(68) = - 3.69, P < 0.001) between patients and healthy subjects. Concerning the patients significant correlations were found between age and time (R = 0.65, p = 0.015), accuracy (R = - 0.75, p = 0.004) and the total displacement of upper limbs. (R = 0.55, p = 0.031). Statistically significant correlations were found between the age of diagnosis and speed related parameters. CONCLUSIONS: The results of this study indicate that SG reliably captures motor impairment of FRDA patients due to cerebellar and pyramidal involvement. Results also show that functional evaluation of FRDA patients can be performed during rehabilitation therapy embedded in games with the patient seated in a wheelchair. TRIAL REGISTRATION: The study was approved as a component of the EFACTS study ( Clinicaltrials.gov identifier NCT02069509 , registered May 2010) by the local institutional Ethics Committee (ref. P2010/132).

Gait Type Analysis Using Dynamic Bayesian Networks.

This paper focuses on gait abnormality type identification-specifically, recognizing antalgic gait. Through experimentation, we demonstrate that detecting an individual's gait type is a viable biometric that can be used along with other common biometrics for applications such as forensics. To classify gait, the gait data is represented by coordinates that reflect the body joint coordinates obtained using a Microsoft Kinect v2 system. Features such as cadence, stride length, and other various joint angles are extracted from the input data. Using approaches such as the dynamic Bayesian network, the obtained features are used to model as well as perform gait type classification. The proposed approach is compared with other classification techniques and experimental results reveal that it is capable of obtaining a 88.68% recognition rate. The results illustrate the potential of using a dynamic Bayesian network for gait abnormality classification.

Improving Video Segmentation by Fusing Depth Cues and the Visual Background Extractor (ViBe) Algorithm.

Depth-sensing technology has led to broad applications of inexpensive depth cameras that can capture human motion and scenes in three-dimensional space. Background subtraction algorithms can be improved by fusing color and depth cues, thereby allowing many issues encountered in classical color segmentation to be solved. In this paper, we propose a new fusion method that combines depth and color information for foreground segmentation based on an advanced color-based algorithm. First, a background model and a depth model are developed. Then, based on these models, we propose a new updating strategy that can eliminate ghosting and black shadows almost completely. Extensive experiments have been performed to compare the proposed algorithm with other, conventional RGB-D (Red-Green-Blue and Depth) algorithms. The experimental results suggest that our method extracts foregrounds with higher effectiveness and efficiency.

An Objective Balance Error Scoring System for Sideline Concussion Evaluation Using Duplex Kinect Sensors.

Sports-related concussion is a common sports injury that might induce potential long-term consequences without early diagnosis and intervention in the field. However, there are few options of such sensor systems available. The aim of the study is to propose and validate an automated concussion administration and scoring approach, which is objective, affordable and capable of detecting all balance errors required by the balance error scoring system (BESS) protocol in the field condition. Our approach is first to capture human body skeleton positions using two Microsoft Kinect sensors in the proposed configuration and merge the data by a custom-made algorithm to remove the self-occlusion of limbs. The standing balance errors according to BESS protocol were further measured and accessed automatically by the proposed algorithm. Simultaneously, the BESS test was filmed for scoring by an experienced rater. Two results were compared using Pearson coefficient r, obtaining an excellent consistency (r = 0.93, p < 0.05). In addition, BESS test-retest was performed after seven days and compared using intraclass correlation coefficients (ICC), showing a good test-retest reliability (ICC = 0.81, p < 0.01). The proposed approach could be an alternative of objective tools to assess postural stability for sideline sports concussion diagnosis.

Influence of Wind Speed on RGB-D Images in Tree Plantations.

Weather conditions can affect sensors' readings when sampling outdoors. Although sensors are usually set up covering a wide range of conditions, their operational range must be established. In recent years, depth cameras have been shown as a promising tool for plant phenotyping and other related uses. However, the use of these devices is still challenged by prevailing field conditions. Although the influence of lighting conditions on the performance of these cameras has already been established, the effect of wind is still unknown. This study establishes the associated errors when modeling some tree characteristics at different wind speeds. A system using a Kinect v2 sensor and a custom software was tested from null wind speed up to 10 m·s-1. Two tree species with contrasting architecture, poplars and plums, were used as model plants. The results showed different responses depending on tree species and wind speed. Estimations of Leaf Area (LA) and tree volume were generally more consistent at high wind speeds in plum trees. Poplars were particularly affected by wind speeds higher than 5 m·s-1. On the contrary, height measurements were more consistent for poplars than for plum trees. These results show that the use of depth cameras for tree characterization must take into consideration wind conditions in the field. In general, 5 m·s-1 (18 km·h-1) could be established as a conservative limit for good estimations.

Real Time Apnoea Monitoring of Children Using the Microsoft Kinect Sensor: A Pilot Study.

The objective of this study was to design a non-invasive system for the observation of respiratory rates and detection of apnoea using analysis of real time image sequences captured in any given sleep position and under any light conditions (even in dark environments). A Microsoft Kinect sensor was used to visualize the variations in the thorax and abdomen from the respiratory rhythm. These variations were magnified, analyzed and detected at a distance of 2.5 m from the subject. A modified motion magnification system and frame subtraction technique were used to identify breathing movements by detecting rapid motion areas in the magnified frame sequences. The experimental results on a set of video data from five subjects (3 h for each subject) showed that our monitoring system can accurately measure respiratory rate and therefore detect apnoea in infants and young children. The proposed system is feasible, accurate, safe and low computational complexity, making it an efficient alternative for non-contact home sleep monitoring systems and advancing health care applications.

Are We Ready to Build a System for Assisting Blind People in Tactile Exploration of Bas-Reliefs?

Nowadays, the creation of methodologies and tools for facilitating the 3D reproduction of artworks and, contextually, to make their exploration possible and more meaningful for blind users is becoming increasingly relevant in society. Accordingly, the creation of integrated systems including both tactile media (e.g., bas-reliefs) and interfaces capable of providing the users with an experience cognitively comparable to the one originally envisioned by the artist, may be considered the next step for enhancing artworks exploration. In light of this, the present work provides a description of a first-attempt system designed to aid blind people (BP) in the tactile exploration of bas-reliefs. In detail, consistent hardware layout, comprising a hand-tracking system based on Kinect(®) sensor and an audio device, together with a number of methodologies, algorithms and information related to physical design are proposed. Moreover, according to experimental test on the developed system related to the device position, some design alternatives are suggested so as to discuss pros and cons.

Three-Dimensional Object Recognition and Registration for Robotic Grasping Systems Using a Modified Viewpoint Feature Histogram.

This paper presents a novel 3D feature descriptor for object recognition and to identify poses when there are six-degrees-of-freedom for mobile manipulation and grasping applications. Firstly, a Microsoft Kinect sensor is used to capture 3D point cloud data. A viewpoint feature histogram (VFH) descriptor for the 3D point cloud data then encodes the geometry and viewpoint, so an object can be simultaneously recognized and registered in a stable pose and the information is stored in a database. The VFH is robust to a large degree of surface noise and missing depth information so it is reliable for stereo data. However, the pose estimation for an object fails when the object is placed symmetrically to the viewpoint. To overcome this problem, this study proposes a modified viewpoint feature histogram (MVFH) descriptor that consists of two parts: a surface shape component that comprises an extended fast point feature histogram and an extended viewpoint direction component. The MVFH descriptor characterizes an object's pose and enhances the system's ability to identify objects with mirrored poses. Finally, the refined pose is further estimated using an iterative closest point when the object has been recognized and the pose roughly estimated by the MVFH descriptor and it has been registered on a database. The estimation results demonstrate that the MVFH feature descriptor allows more accurate pose estimation. The experiments also show that the proposed method can be applied in vision-guided robotic grasping systems.

Head Pose Estimation on Top of Haar-Like Face Detection: A Study Using the Kinect Sensor.

Head pose estimation is a crucial initial task for human face analysis, which is employed in several computer vision systems, such as: facial expression recognition, head gesture recognition, yawn detection, etc. In this work, we propose a frame-based approach to estimate the head pose on top of the Viola and Jones (VJ) Haar-like face detector. Several appearance and depth-based feature types are employed for the pose estimation, where comparisons between them in terms of accuracy and speed are presented. It is clearly shown through this work that using the depth data, we improve the accuracy of the head pose estimation. Additionally, we can spot positive detections, faces in profile views detected by the frontal model, that are wrongly cropped due to background disturbances. We introduce a new depth-based feature descriptor that provides competitive estimation results with a lower computation time. Evaluation on a benchmark Kinect database shows that the histogram of oriented gradients and the developed depth-based features are more distinctive for the head pose estimation, where they compare favorably to the current state-of-the-art approaches. Using a concatenation of the aforementioned feature types, we achieved a head pose estimation with average errors not exceeding 5:1; 4:6; 4:2 for pitch, yaw and roll angles, respectively.

Spinal alignment measurement with Kinect sensor is valid for thoracic kyphosis but not for lumbar lordosis.

BACKGROUND: Spinal alignment evaluation is commonly performed in the clinical setting during rehabilitation. However, there is no simple method for its quantitative measurement. Recently, the depth cameras in Kinect sensors have been employed in various commercial and research projects in the healthcare field. We hypothesized that the time-of-flight technology of the Kinect sensor could be applied to quantitatively evaluate spinal alignment. The purpose of this study was to develop a simple and noninvasive evaluation for spinal alignment using the Kinect sensor and to investigate its validity. METHODS: Twenty-four healthy men participated in the study. Measurement outcomes were the thoracic kyphosis and lumbar lordosis angles in the standing position, using a Spinal Mouse, the validity of which has been previously reported, and the Kinect sensor. In the measurement by the Kinect sensor, a program was created to obtain the three-dimensional coordinates of each point within an area marked on the monitor, and the sums of the angles at each vertebral level were calculated for the thoracic and lumbar areas. Pearson's correlation coefficient was used to analyze the relationship between the Kinect sensor and Spinal Mouse measurements of thoracic kyphosis and lumbar lordosis angles. RESULTS: There was a significant positive and moderate correlation between the thoracic kyphosis measurements taken by each device. Contrarily, there was no significant correlation in the lordosis angle between measurements using the Kinect sensor and Spinal Mouse. CONCLUSIONS: Our results demonstrated the validity of measuring the thoracic kyphosis angle using the Kinect sensor. This indicates that the depth camera in the Kinect sensor is able to perform accurate thoracic alignment measurements quickly and noninvasively.