Autoencoder and Partially Impossible Reconstruction Losses.

The generally unsupervised nature of autoencoder models implies that the main training metric is formulated as the error between input images and their corresponding reconstructions. Different reconstruction loss variations and latent space regularizations have been shown to improve model performances depending on the tasks to solve and to induce new desirable properties such as disentanglement. Nevertheless, measuring the success in, or enforcing properties by, the input pixel space is a challenging endeavour. In this work, we want to make use of the available data more efficiently and provide design choices to be considered in the recording or generation of future datasets to implicitly induce desirable properties during training. To this end, we propose a new sampling technique which matches semantically important parts of the image while randomizing the other parts, leading to salient feature extraction and a neglection of unimportant details. The proposed method can be combined with any existing reconstruction loss and the performance gain is superior to the triplet loss. We analyse the resulting properties on various datasets and show improvements on several computer vision tasks: illumination and unwanted features can be normalized or smoothed out and shadows are removed such that classification or other tasks work more reliably; a better invariances with respect to unwanted features is induced; the generalization capacities from synthetic to real images is improved, such that more of the semantics are preserved; uncertainty estimation is superior to Monte Carlo Dropout and an ensemble of models, particularly for datasets of higher visual complexity. Finally, classification accuracy by means of simple linear classifiers in the latent space is improved compared to the triplet loss. For each task, the improvements are highlighted on several datasets commonly used by the research community, as well as in automotive applications.

Classification and Automated Interpretation of Spinal Posture Data Using a Pathology-Independent Classifier and Explainable Artificial Intelligence (XAI).

Clinical classification models are mostly pathology-dependent and, thus, are only able to detect pathologies they have been trained for. Research is needed regarding pathology-independent classifiers and their interpretation. Hence, our aim is to develop a pathology-independent classifier that provides prediction probabilities and explanations of the classification decisions. Spinal posture data of healthy subjects and various pathologies (back pain, spinal fusion, osteoarthritis), as well as synthetic data, were used for modeling. A one-class support vector machine was used as a pathology-independent classifier. The outputs were transformed into a probability distribution according to Platt's method. Interpretation was performed using the explainable artificial intelligence tool Local Interpretable Model-Agnostic Explanations. The results were compared with those obtained by commonly used binary classification approaches. The best classification results were obtained for subjects with a spinal fusion. Subjects with back pain were especially challenging to distinguish from the healthy reference group. The proposed method proved useful for the interpretation of the predictions. No clear inferiority of the proposed approach compared to commonly used binary classifiers was demonstrated. The application of dynamic spinal data seems important for future works. The proposed approach could be useful to provide an objective orientation and to individually adapt and monitor therapy measures pre- and post-operatively.

Interpretability of Input Representations for Gait Classification in Patients after Total Hip Arthroplasty.

Many machine learning models show black box characteristics and, therefore, a lack of transparency, interpretability, and trustworthiness. This strongly limits their practical application in clinical contexts. For overcoming these limitations, Explainable Artificial Intelligence (XAI) has shown promising results. The current study examined the influence of different input representations on a trained model's accuracy, interpretability, as well as clinical relevancy using XAI methods. The gait of 27 healthy subjects and 20 subjects after total hip arthroplasty (THA) was recorded with an inertial measurement unit (IMU)-based system. Three different input representations were used for classification. Local Interpretable Model-Agnostic Explanations (LIME) was used for model interpretation. The best accuracy was achieved with automatically extracted features (mean accuracy Macc = 100%), followed by features based on simple descriptive statistics (Macc = 97.38%) and waveform data (Macc = 95.88%). Globally seen, sagittal movement of the hip, knee, and pelvis as well as transversal movement of the ankle were especially important for this specific classification task. The current work shows that the type of input representation crucially determines interpretability as well as clinical relevance. A combined approach using different forms of representations seems advantageous. The results might assist physicians and therapists finding and addressing individual pathologic gait patterns.

Force Shadows: An Online Method to Estimate and Distribute Vertical Ground Reaction Forces from Kinematic Data.

Kinetic models of human motion rely on boundary conditions which are defined by the interaction of the body with its environment. In the simplest case, this interaction is limited to the foot contact with the ground and is given by the so called ground reaction force (GRF). A major challenge in the reconstruction of GRF from kinematic data is the double support phase, referring to the state with multiple ground contacts. In this case, the GRF prediction is not well defined. In this work we present an approach to reconstruct and distribute vertical GRF (vGRF) to each foot separately, using only kinematic data. We propose the biomechanically inspired force shadow method (FSM) to obtain a unique solution for any contact phase, including double support, of an arbitrary motion. We create a kinematic based function, model an anatomical foot shape and mimic the effect of hip muscle activations. We compare our estimations with the measurements of a Zebris pressure plate and obtain correlations of 0.39≤r≤0.94 for double support motions and 0.83≤r≤0.87 for a walking motion. The presented data is based on inertial human motion capture, showing the applicability for scenarios outside the laboratory. The proposed approach has low computational complexity and allows for online vGRF estimation.

Towards an Inertial Sensor-Based Wearable Feedback System for Patients after Total Hip Arthroplasty: Validity and Applicability for Gait Classification with Gait Kinematics-Based Features.

Patients after total hip arthroplasty (THA) suffer from lingering musculoskeletal restrictions. Three-dimensional (3D) gait analysis in combination with machine-learning approaches is used to detect these impairments. In this work, features from the 3D gait kinematics, spatio temporal parameters (Set 1) and joint angles (Set 2), of an inertial sensor (IMU) system are proposed as an input for a support vector machine (SVM) model, to differentiate impaired and non-impaired gait. The features were divided into two subsets. The IMU-based features were validated against an optical motion capture (OMC) system by means of 20 patients after THA and a healthy control group of 24 subjects. Then the SVM model was trained on both subsets. The validation of the IMU system-based kinematic features revealed root mean squared errors in the joint kinematics from 0.24° to 1.25°. The validity of the spatio-temporal gait parameters (STP) revealed a similarly high accuracy. The SVM models based on IMU data showed an accuracy of 87.2% (Set 1) and 97.0% (Set 2). The current work presents valid IMU-based features, employed in an SVM model for the classification of the gait of patients after THA and a healthy control. The study reveals that the features of Set 2 are more significant concerning the classification problem. The present IMU system proves its potential to provide accurate features for the incorporation in a mobile gait-feedback system for patients after THA.

IMU-to-Segment Assignment and Orientation Alignment for the Lower Body Using Deep Learning.

Human body motion analysis based on wearable inertial measurement units (IMUs) receives a lot of attention from both the research community and the and industrial community. This is due to the significant role in, for instance, mobile health systems, sports and human computer interaction. In sensor based activity recognition, one of the major issues for obtaining reliable results is the sensor placement/assignment on the body. For inertial motion capture (joint kinematics estimation) and analysis, the IMU-to-segment (I2S) assignment and alignment are central issues to obtain biomechanical joint angles. Existing approaches for I2S assignment usually rely on hand crafted features and shallow classification approaches (e.g., support vector machines), with no agreement regarding the most suitable features for the assignment task. Moreover, estimating the complete orientation alignment of an IMU relative to the segment it is attached to using a machine learning approach has not been shown in literature so far. This is likely due to the high amount of training data that have to be recorded to suitably represent possible IMU alignment variations. In this work, we propose online approaches for solving the assignment and alignment tasks for an arbitrary amount of IMUs with respect to a biomechanical lower body model using a deep learning architecture and windows of 128 gyroscope and accelerometer data samples. For this, we combine convolutional neural networks (CNNs) for local filter learning with long-short-term memory (LSTM) recurrent networks as well as generalized recurrent units (GRUs) for learning time dynamic features. The assignment task is casted as a classification problem, while the alignment task is casted as a regression problem. In this framework, we demonstrate the feasibility of augmenting a limited amount of real IMU training data with simulated alignment variations and IMU data for improving the recognition/estimation accuracies. With the proposed approaches and final models we achieved 98.57% average accuracy over all segments for the I2S assignment task (100% when excluding left/right switches) and an average median angle error over all segments and axes of 2 . 91 for the I2S alignment task.

Validity, Test-Retest Reliability and Long-Term Stability of Magnetometer Free Inertial Sensor Based 3D Joint Kinematics.

The present study investigates an algorithm for the calculation of 3D joint angles based on inertial measurement units (IMUs), omitting magnetometer data. Validity, test-retest reliability, and long-term stability are evaluated in reference to an optical motion capture (OMC) system. Twenty-eight healthy subjects performed a 6 min walk test. Three-dimensional joint kinematics of the lower extremity was recorded simultaneously by means of seven IMUs and an OptiTrack OMC system. To evaluate the performance, the root mean squared error (RMSE), mean range of motion error (ROME), coefficient of multiple correlations (CMC), Bland-Altman (BA) analysis, and intraclass correlation coefficient (ICC) were calculated. For all joints, the RMSE was lower than 2.40°, and the ROME was lower than 1.60°. The CMC revealed good to excellent waveform similarity. Reliability was moderate to excellent with ICC values of 0.52⁻0.99 for all joints. Error measures did not increase over time. When considering soft tissue artefacts, RMSE and ROME increased by an average of 2.2° ± 1.5° and 2.9° ± 1.7°. This study revealed an excellent correspondence of a magnetometer-free IMU system with an OMC system when excluding soft tissue artefacts.

Towards Inertial Sensor Based Mobile Gait Analysis: Event-Detection and Spatio-Temporal Parameters.

The aim of this study was to assess the validity and test-retest reliability of an inertial measurement unit (IMU) system for gait analysis. Twenty-four healthy subjects conducted a 6-min walking test and were instrumented with seven IMUs and retroreflective markers. A kinematic approach was used to estimate the initial and terminal contact events in real-time. Based on these events twelve spatio-temporal parameters (STP) were calculated. A marker based optical motion capture (OMC) system provided the reference. Event-detection rate was about 99%. Detection offset was below 0.017 s. Relative root mean square error (RMSE) ranged from 0.90% to 4.40% for most parameters. However, the parameters that require spatial information of both feet showed higher errors. Step length showed a relative RMSE of 6.69%. Step width and swing width revealed the highest relative RMSE (34.34% and 35.20%). Test-retest results ranged from 0.67 to 0.92, except for the step width (0.25). Summarizing, it appears that the parameters describing the lateral distance between the feet need further improvement. However, the results of the validity and reliability of the IMU system encourage its validation in clinical settings as well as further research.

On Inertial Body Tracking in the Presence of Model Calibration Errors.

In inertial body tracking, the human body is commonly represented as a biomechanical model consisting of rigid segments with known lengths and connecting joints. The model state is then estimated via sensor fusion methods based on data from attached inertial measurement units (IMUs). This requires the relative poses of the IMUs w.r.t. the segments-the IMU-to-segment calibrations, subsequently called I2S calibrations-to be known. Since calibration methods based on static poses, movements and manual measurements are still the most widely used, potentially large human-induced calibration errors have to be expected. This work compares three newly developed/adapted extended Kalman filter (EKF) and optimization-based sensor fusion methods with an existing EKF-based method w.r.t. their segment orientation estimation accuracy in the presence of model calibration errors with and without using magnetometer information. While the existing EKF-based method uses a segment-centered kinematic chain biomechanical model and a constant angular acceleration motion model, the newly developed/adapted methods are all based on a free segments model, where each segment is represented with six degrees of freedom in the global frame. Moreover, these methods differ in the assumed motion model (constant angular acceleration, constant angular velocity, inertial data as control input), the state representation (segment-centered, IMU-centered) and the estimation method (EKF, sliding window optimization). In addition to the free segments representation, the optimization-based method also represents each IMU with six degrees of freedom in the global frame. In the evaluation on simulated and real data from a three segment model (an arm), the optimization-based method showed the smallest mean errors, standard deviations and maximum errors throughout all tests. It also showed the lowest dependency on magnetometer information and motion agility. Moreover, it was insensitive w.r.t. I2S position and segment length errors in the tested ranges. Errors in the I2S orientations were, however, linearly propagated into the estimated segment orientations. In the absence of magnetic disturbances, severe model calibration errors and fast motion changes, the newly developed IMU centered EKF-based method yielded comparable results with lower computational complexity.

JointTracker: Real-time inertial kinematic chain tracking with joint position estimation.

In-field human motion capture (HMC) is drawing increasing attention due to the multitude of application areas. Plenty of research is currently invested in camera-based (markerless) HMC, with the advantage of no infrastructure being required on the body, and additional context information being available from the surroundings. However, the inherent drawbacks of camera-based approaches are the limited field of view and occlusions. In contrast, inertial HMC (IHMC) does not suffer from occlusions, thus being a promising approach for capturing human motion outside the laboratory. However, one major challenge of such methods is the necessity of spatial registration. Typically, during a predefined calibration sequence, the orientation and location of each inertial sensor are registered with respect to the underlying skeleton model. This work contributes to calibration-free IHMC, as it proposes a recursive estimator for the simultaneous online estimation of all sensor poses and joint positions of a kinematic chain model like the human skeleton. The full derivation from an optimization objective is provided. The approach can directly be applied to a synchronized data stream from a body-mounted inertial sensor network. Successful evaluations are demonstrated on noisy simulated data from a three-link chain, real lower-body walking data from 25 young, healthy persons, and walking data captured from a humanoid robot. The estimated and derived quantities, global and relative sensor orientations, joint positions, and segment lengths can be exploited for human motion analysis and anthropometric measurements, as well as in the context of hybrid markerless visual-inertial HMC.

Machine learning techniques demonstrating individual movement patterns of the vertebral column: the fingerprint of spinal motion.

Surface topography systems enable the capture of spinal dynamic movement; however, it is unclear whether vertebral dynamics are unique enough to identify individuals. Therefore, in this study, we investigated whether the identification of individuals is possible based on dynamic spinal data. Three different data representations were compared (automated extracted features using contrastive loss and triplet loss functions, as well as simple descriptive statistics). High accuracies indicated the possible existence of a personal spinal 'fingerprint', therefore enabling subject recognition. The present work forms the basis for an objective comparison of subjects and the transfer of the method to clinical use cases.

Automated detection and explainability of pathological gait patterns using a one-class support vector machine trained on inertial measurement unit based gait data.

BACKGROUND: Machine learning approaches for the classification of pathological gait based on kinematic data, e.g. derived from inertial sensors, are commonly used in terms of a multi-class classification problem. However, there is a lack of research regarding one-class classifiers that are independent of certain pathologies. Therefore, it was the aim of this work to design a one-class classifier based on healthy norm-data that provides not only a prediction probability but rather an explanation of the classification decision, increasing the acceptance of this machine learning approach. METHODS: The inertial sensor based gait kinematics of 25 healthy subjects was employed to train a one-class support vector machine. 25 healthy subjects, 20 patients after total hip arthroplasty and one transfemoral amputee served to validate the classifier. Prediction probabilities and feature importance scores were estimated for each subject. FINDINGS: The support vector machine predicted 100% of the patients as outliers from the healthy group. Three healthy subjects were predicted as outliers. The feature importance calculation revealed the hip in the sagittal plane as most relevant feature concerning the group after total hip arthroplasty. For the misclassified healthy subject with the lowest probability score the knee flexion and the pelvis obliquity were identified. INTERPRETATION: The support vector machine seems a useful tool to identify outliers from a healthy norm-group. The feature importance examination proved to provide valuable information on the musculoskeletal status of the subjects. In this combination, the present approach could be employed in various disciplines to identify abnormal gait and suggest subsequent training.

General method for automated feature extraction and selection and its application for gender classification and biomechanical knowledge discovery of sex differences in spinal posture during stance and gait.

Modern technologies enable to capture multiple biomechanical parameters often resulting in relational data. The current work proposes a generally applicable method comprising automated feature extraction, ensemble feature selection and classification to best capture the potentials of the data also for generating new biomechanical knowledge. Its benefits are demonstrated in the concrete biomechanically and medically relevant use case of gender classification based on spinal data for stance and gait. Very good results for accuracy were obtained using gait data. Dynamic movements of the lumbar spine in sagittal and frontal plane and of the pelvis in frontal plane best map gender differences.

Depth camera based statistical shape fitting approach for the creation of an individualized lower body biomechanical model: validity and reliability.

Systems for instrumented motion analysis depend on an accurate biomechanical model. In this work an existing statistical shape fitting approach was adapted to register a parametrized human surface model with annotated anatomical landmarks to a point cloud obtained from one frontal depth camera view. Based on the obtained landmark positions joint centers, segment lengths and segment orientations of the lower body were calculated. The outcome was validated among two groups of healthy and impaired subjects, using a marker-based optical motion capture system. The results reveal a valid and reliable approach for obtaining an individualized lower body biomechanical model with minimum effort.

Flow Fields: Dense Correspondence Fields for Highly Accurate Large Displacement Optical Flow Estimation.

Modern large displacement optical flow algorithms usually use an initialization by either sparse descriptor matching techniques or dense approximate nearest neighbor fields. While the latter have the advantage of being dense, they have the major disadvantage of being very outlier-prone as they are not designed to find the optical flow, but the visually most similar correspondence. In this article we present a dense correspondence field approach that is much less outlier-prone and thus much better suited for optical flow estimation than approximate nearest neighbor fields. Our approach does not require explicit regularization, smoothing (like median filtering) or a new data term. Instead we solely rely on patch matching techniques and a novel multi-scale matching strategy. We also present enhancements for outlier filtering. We show that our approach is better suited for large displacement optical flow estimation than modern descriptor matching techniques. We do so by initializing EpicFlow with our approach instead of their originally used state-of-the-art descriptor matching technique. We significantly outperform the original EpicFlow on MPI-Sintel, KITTI 2012, KITTI 2015 and Middlebury. In this extended article of our former conference publication we further improve our approach in matching accuracy as well as runtime and present more experiments and insights.

Validity of inertial sensor based 3D joint kinematics of static and dynamic sport and physiotherapy specific movements.

3D joint kinematics can provide important information about the quality of movements. Optical motion capture systems (OMC) are considered the gold standard in motion analysis. However, in recent years, inertial measurement units (IMU) have become a promising alternative. The aim of this study was to validate IMU-based 3D joint kinematics of the lower extremities during different movements. Twenty-eight healthy subjects participated in this study. They performed bilateral squats (SQ), single-leg squats (SLS) and countermovement jumps (CMJ). The IMU kinematics was calculated using a recently-described sensor-fusion algorithm. A marker based OMC system served as a reference. Only the technical error based on algorithm performance was considered, incorporating OMC data for the calibration, initialization, and a biomechanical model. To evaluate the validity of IMU-based 3D joint kinematics, root mean squared error (RMSE), range of motion error (ROME), Bland-Altman (BA) analysis as well as the coefficient of multiple correlation (CMC) were calculated. The evaluation was twofold. First, the IMU data was compared to OMC data based on marker clusters; and, second based on skin markers attached to anatomical landmarks. The first evaluation revealed means for RMSE and ROME for all joints and tasks below 3°. The more dynamic task, CMJ, revealed error measures approximately 1° higher than the remaining tasks. Mean CMC values ranged from 0.77 to 1 over all joint angles and all tasks. The second evaluation showed an increase in the RMSE of 2.28°- 2.58° on average for all joints and tasks. Hip flexion revealed the highest average RMSE in all tasks (4.87°- 8.27°). The present study revealed a valid IMU-based approach for the measurement of 3D joint kinematics in functional movements of varying demands. The high validity of the results encourages further development and the extension of the present approach into clinical settings.

A Biofeedback App to Instruct Abdominal Breathing (Breathing-Mentor): Pilot Experiment.

BACKGROUND: Deep and slow abdominal breathing is an important skill for the management of stress and pain. However, despite multiple proofs on the effectiveness of biofeedback, most breathing apps remain limited to pacing specific breathing patterns, without sensor feedback on the actual breathing behavior. OBJECTIVE: To fill this gap, an app named Breathing-Mentor was developed. This app combines effective visualization of the instruction with biofeedback on deep abdominal breathing, based on the mobile phone's accelerometers. The aim of this pilot study was to investigate users' feedback and breathing behavior during initial contact with the app. METHODS: To reveal the possible effects of biofeedback, two versions of the mobile app were developed. Both contained the same visual instruction, but only the full version included additional biofeedback. In total, 40 untrained participants were randomly assigned to one of the two versions of the app. They had to follow the app's instructions as closely as possible for 5 min. RESULTS: The group with additional biofeedback showed an increased signal-to-noise ratio for instructed breathing frequency (0.1 Hz) compared with those using visual instruction without biofeedback (F1,37=4.18; P<.048). During this initial contact with the full version, self-reported relaxation effectivity was, however, lower than the group using visual instruction without biofeedback (t37=-2.36; P=.02), probably owing to increased cognitive workload to follow the instruction. CONCLUSIONS: This study supports the feasibility and usefulness of incorporating biofeedback in the Breathing-Mentor app to train abdominal breathing. Immediate effects on relaxation levels should, however, not be expected for untrained users.