Hybrid Deep Neural Network Framework Combining Skeleton and Gait Features for Pathological Gait Recognition.

Human skeleton data obtained using a depth camera have been used for pathological gait recognition to support doctor or physician diagnosis decisions. Most studies for skeleton-based pathological gait recognition have used either raw skeleton sequences directly or gait features, such as gait parameters and joint angles, extracted from raw skeleton sequences. We hypothesize that using skeleton, joint angles, and gait parameters together can improve recognition performance. This study aims to develop a deep neural network model that effectively combines different types of input data. We propose a hybrid deep neural network framework composed of a graph convolutional network, recurrent neural network, and artificial neural network to effectively encode skeleton sequences, joint angle sequences, and gait parameters, respectively. The features extracted from three different input data types are fused and fed into the final classification layer. We evaluate the proposed model on two different skeleton datasets (a simulated pathological gait dataset and a vestibular disorder gait dataset) that were collected using an Azure Kinect. The proposed model, with multiple types of input, improved the pathological gait recognition performance compared to single input models on both datasets. Furthermore, it achieved the best performance among the state-of-the-art models for skeleton-based action recognition.

Markerless 3D Skeleton Tracking Algorithm by Merging Multiple Inaccurate Skeleton Data from Multiple RGB-D Sensors.

Skeleton data, which is often used in the HCI field, is a data structure that can efficiently express human poses and gestures because it consists of 3D positions of joints. The advancement of RGB-D sensors, such as Kinect sensors, enabled the easy capture of skeleton data from depth or RGB images. However, when tracking a target with a single sensor, there is an occlusion problem causing the quality of invisible joints to be randomly degraded. As a result, multiple sensors should be used to reliably track a target in all directions over a wide range. In this paper, we proposed a new method for combining multiple inaccurate skeleton data sets obtained from multiple sensors that capture a target from different angles into a single accurate skeleton data. The proposed algorithm uses density-based spatial clustering of applications with noise (DBSCAN) to prevent noise-added inaccurate joint candidates from participating in the merging process. After merging with the inlier candidates, we used Kalman filter to denoise the tremble error of the joint's movement. We evaluated the proposed algorithm's performance using the best view as the ground truth. In addition, the results of different sizes for the DBSCAN searching area were analyzed. By applying the proposed algorithm, the joint position accuracy of the merged skeleton improved as the number of sensors increased. Furthermore, highest performance was shown when the searching area of DBSCAN was 10 cm.

Deep-Learning-Based ADHD Classification Using Children's Skeleton Data Acquired through the ADHD Screening Game.

The identification of attention deficit hyperactivity disorder (ADHD) in children, which is increasing every year worldwide, is very important for early diagnosis and treatment. However, since ADHD is not a simple disease that can be diagnosed with a simple test, doctors require a large period of time and substantial effort for accurate diagnosis and treatment. Currently, ADHD classification studies using various datasets and machine learning or deep learning algorithms are actively being conducted for the screening diagnosis of ADHD. However, there has been no study of ADHD classification using only skeleton data. It was hypothesized that the main symptoms of ADHD, such as distraction, hyperactivity, and impulsivity, could be differentiated through skeleton data. Thus, we devised a game system for the screening and diagnosis of children's ADHD and acquired children's skeleton data using five Azure Kinect units equipped with depth sensors, while the game was being played. The game for screening diagnosis involves a robot first travelling on a specific path, after which the child must remember the path the robot took and then follow it. The skeleton data used in this study were divided into two categories: standby data, obtained when a child waits while the robot demonstrates the path; and game data, obtained when a child plays the game. The acquired data were classified using the RNN series of GRU, RNN, and LSTM algorithms; a bidirectional layer; and a weighted cross-entropy loss function. Among these, an LSTM algorithm using a bidirectional layer and a weighted cross-entropy loss function obtained a classification accuracy of 97.82%.

Deep Learning-Based ADHD and ADHD-RISK Classification Technology through the Recognition of Children's Abnormal Behaviors during the Robot-Led ADHD Screening Game.

Although attention deficit hyperactivity disorder (ADHD) in children is rising worldwide, fewer studies have focused on screening than on the treatment of ADHD. Most previous similar ADHD classification studies classified only ADHD and normal classes. However, medical professionals believe that better distinguishing the ADHD-RISK class will assist them socially and medically. We created a projection-based game in which we can see stimuli and responses to better understand children's abnormal behavior. The developed screening game is divided into 11 stages. Children play five games. Each game is divided into waiting and game stages; thus, 10 stages are created, and the additional waiting stage includes an explanation stage where the robot waits while explaining the first game. Herein, we classified normal, ADHD-RISK, and ADHD using skeleton data obtained through games for ADHD screening of children and a bidirectional long short-term memory-based deep learning model. We verified the importance of each stage by passing the feature for each stage through the channel attention layer. Consequently, the final classification accuracy of the three classes was 98.15% using bi-directional LSTM with channel attention model. Additionally, the attention scores obtained through the channel attention layer indicated that the data in the latter part of the game are heavily involved in learning the ADHD-RISK case. These results imply that for ADHD-RISK, the game is repeated, and children's attention decreases as they progress to the second half.

Abnormal Gait Recognition Using 3D Joint information of Multiple Kinects System and RNN-LSTM.

Gait is an important indicator for specific diseases. Abnormal gait patterns are caused by various factors such as physical, neurological, and sensory problems. If it is possible to recognize abnormal gait patterns in the early stage of the related disease, patients can receive proper treatment early and prevent secondary accidents such as falls caused by unbalanced gait. In this paper, we propose a gait recognition system that can recognize 5 abnormal gait patterns. Our system using 3D joint information obtained by using multiple Kinect v2 sensors and RNN-LSTM. In particular, abnormal gaits caused by physical problems such as injury, weakness of muscle, and joint problems are targeted for recognition. The purpose of this paper is to find optimal condition for gait recognition when using the multiple Kinect v2 sensors. Experiments were conducted by comparing the test accuracies on 14 combinations of human joint. Through this experiment, we selected optimal joints to show outstanding results so that our gait recognition model performs optimally. Results show that Ankles, Wrists, and the Head are the most influential joints on RNN-LSTM model. We applied 25-joint information of the human body to recognize gait patterns and achieved an accuracy over 97%.