Sensor-Based Human Activity and Behavior Research: Where Advanced Sensing and Recognition Technologies Meet.

Human activity recognition (HAR) and human behavior recognition (HBR) have been playing increasingly important roles in the digital age [...].

Automatic Cognitive Fatigue Detection Using Wearable fNIRS and Machine Learning.

Wearable sensors have increasingly been applied in healthcare to generate data and monitor patients unobtrusively. Their application for Brain-Computer Interfaces (BCI) allows for unobtrusively monitoring one's cognitive state over time. A particular state relevant in multiple domains is cognitive fatigue, which may impact performance and attention, among other capabilities. The monitoring of this state will be applied in real learning settings to detect and advise on effective break periods. In this study, two functional near-infrared spectroscopy (fNIRS) wearable devices were employed to build a BCI to automatically detect the state of cognitive fatigue using machine learning algorithms. An experimental procedure was developed to effectively induce cognitive fatigue that included a close-to-real digital lesson and two standard cognitive tasks: Corsi-Block task and a concentration task. Machine learning models were user-tuned to account for the individual dynamics of each participant, reaching classification accuracy scores of around 70.91 ± 13.67 %. We concluded that, although effective for some subjects, the methodology needs to be individually validated before being applied. Moreover, time on task was not a particularly determining factor for classification, i.e., to induce cognitive fatigue. Further research will include other physiological signals and human-computer interaction variables.

Comparing Handcrafted Features and Deep Neural Representations for Domain Generalization in Human Activity Recognition.

Human Activity Recognition (HAR) has been studied extensively, yet current approaches are not capable of generalizing across different domains (i.e., subjects, devices, or datasets) with acceptable performance. This lack of generalization hinders the applicability of these models in real-world environments. As deep neural networks are becoming increasingly popular in recent work, there is a need for an explicit comparison between handcrafted and deep representations in Out-of-Distribution (OOD) settings. This paper compares both approaches in multiple domains using homogenized public datasets. First, we compare several metrics to validate three different OOD settings. In our main experiments, we then verify that even though deep learning initially outperforms models with handcrafted features, the situation is reversed as the distance from the training distribution increases. These findings support the hypothesis that handcrafted features may generalize better across specific domains.

Respiratory Inductance Plethysmography to Assess Fatigability during Repetitive Work.

Cumulative fatigue during repetitive work is associated with occupational risk and productivity reduction. Usually, subjective measures or muscle activity are used for a cumulative evaluation; however, Industry 4.0 wearables allow overcoming the challenges observed in those methods. Thus, the aim of this study is to analyze alterations in respiratory inductance plethysmography (RIP) to measure the asynchrony between thorax and abdomen walls during repetitive work and its relationship with local fatigue. A total of 22 healthy participants (age: 27.0 ± 8.3 yrs; height: 1.72 ± 0.09 m; mass: 63.4 ± 12.9 kg) were recruited to perform a task that includes grabbing, moving, and placing a box in an upper and lower shelf. This task was repeated for 10 min in three trials with a fatigue protocol between them. Significant main effects were found from Baseline trial to the Fatigue trials (p < 0.001) for both RIP correlation and phase synchrony. Similar results were found for the activation amplitude of agonist muscle (p < 0.001), and to the muscle acting mainly as a joint stabilizer (p < 0.001). The latter showed a significant effect in predicting both RIP correlation and phase synchronization. Both RIP correlation and phase synchronization can be used for an overall fatigue assessment during repetitive work.

Human Activity Recognition for Indoor Localization Using Smartphone Inertial Sensors.

With the fast increase in the demand for location-based services and the proliferation of smartphones, the topic of indoor localization is attracting great interest. In indoor environments, users' performed activities carry useful semantic information. These activities can then be used by indoor localization systems to confirm users' current relative locations in a building. In this paper, we propose a deep-learning model based on a Convolutional Long Short-Term Memory (ConvLSTM) network to classify human activities within the indoor localization scenario using smartphone inertial sensor data. Results show that the proposed human activity recognition (HAR) model accurately identifies nine types of activities: not moving, walking, running, going up in an elevator, going down in an elevator, walking upstairs, walking downstairs, or going up and down a ramp. Moreover, predicted human activities were integrated within an existing indoor positioning system and evaluated in a multi-story building across several testing routes, with an average positioning error of 2.4 m. The results show that the inclusion of human activity information can reduce the overall localization error of the system and actively contribute to the better identification of floor transitions within a building. The conducted experiments demonstrated promising results and verified the effectiveness of using human activity-related information for indoor localization.

ECG Biometrics Using Deep Learning and Relative Score Threshold Classification.

The field of biometrics is a pattern recognition problem, where the individual traits are coded, registered, and compared with other database records. Due to the difficulties in reproducing Electrocardiograms (ECG), their usage has been emerging in the biometric field for more secure applications. Inspired by the high performance shown by Deep Neural Networks (DNN) and to mitigate the intra-variability challenges displayed by the ECG of each individual, this work proposes two architectures to improve current results in both identification (finding the registered person from a sample) and authentication (prove that the person is whom it claims) processes: Temporal Convolutional Neural Network (TCNN) and Recurrent Neural Network (RNN). Each architecture produces a similarity score, based on the prediction error of the former and the logits given by the last, and fed to the same classifier, the Relative Score Threshold Classifier (RSTC).The robustness and applicability of these architectures were trained and tested on public databases used by literature in this context: Fantasia, MIT-BIH, and CYBHi databases. Results show that overall the TCNN outperforms the RNN achieving almost 100%, 96%, and 90% accuracy, respectively, for identification and 0.0%, 0.1%, and 2.2% equal error rate (EER) for authentication processes. When comparing to previous work, both architectures reached results beyond the state-of-the-art. Nevertheless, the improvement of these techniques, such as enriching training with extra varied data and transfer learning, may provide more robust systems with a reduced time required for validation.

An Infrastructure-Free Magnetic-Based Indoor Positioning System with Deep Learning.

Infrastructure-free Indoor Positioning Systems (IPS) are becoming popular due to their scalability and a wide range of applications. Such systems often rely on deployed Wi-Fi networks. However, their usability may be compromised, either due to scanning restrictions from recent Android versions or the proliferation of 5G technology. This raises the need for new infrastructure-free IPS independent of Wi-Fi networks. In this paper, we propose the use of magnetic field data for IPS, through Deep Neural Networks (DNN). Firstly, a dataset of human indoor trajectories was collected with different smartphones. Afterwards, a magnetic fingerprint was constructed and relevant features were extracted to train a DNN that returns a probability map of a user's location. Finally, two postprocessing methods were applied to obtain the most probable location regions. We asserted the performance of our solution against a test dataset, which produced a Success Rate of around 80%. We believe that these results are competitive for an IPS based on a single sensing source. Moreover, the magnetic field can be used as an additional information layer to increase the robustness and redundancy of current multi-source IPS.

Biosensing and Actuation-Platforms Coupling Body Input-Output Modalities for Affective Technologies.

Research in the use of ubiquitous technologies, tracking systems and wearables within mental health domains is on the rise. In recent years, affective technologies have gained traction and garnered the interest of interdisciplinary fields as the research on such technologies matured. However, while the role of movement and bodily experience to affective experience is well-established, how to best address movement and engagement beyond measuring cues and signals in technology-driven interactions has been unclear. In a joint industry-academia effort, we aim to remodel how affective technologies can help address body and emotional self-awareness. We present an overview of biosignals that have become standard in low-cost physiological monitoring and show how these can be matched with methods and engagements used by interaction designers skilled in designing for bodily engagement and aesthetic experiences. Taking both strands of work together offers unprecedented design opportunities that inspire further research. Through first-person soma design, an approach that draws upon the designer's felt experience and puts the sentient body at the forefront, we outline a comprehensive work for the creation of novel interactions in the form of couplings that combine biosensing and body feedback modalities of relevance to affective health. These couplings lie within the creation of design toolkits that have the potential to render rich embodied interactions to the designer/user. As a result we introduce the concept of "orchestration". By orchestration, we refer to the design of the overall interaction: coupling sensors to actuation of relevance to the affective experience; initiating and closing the interaction; habituating; helping improve on the users' body awareness and engagement with emotional experiences; soothing, calming, or energising, depending on the affective health condition and the intentions of the designer. Through the creation of a range of prototypes and couplings we elicited requirements on broader orchestration mechanisms. First-person soma design lets researchers look afresh at biosignals that, when experienced through the body, are called to reshape affective technologies with novel ways to interpret biodata, feel it, understand it and reflect upon our bodies.

Autonomic nervous system response to remote ischemic conditioning: heart rate variability assessment.

BACKGROUND: Remote ischemic conditioning (RIC) is a procedure applied in a limb for triggering endogenous protective pathways in distant organs, namely brain or heart. The underlying mechanisms of RIC are still not fully understood, and it is hypothesized they are mediated either by humoral factors, immune cells and/or the autonomic nervous system. Herein, heart rate variability (HRV) was used to evaluate the electrophysiological processes occurring in the heart during RIC and, in turn to assess the role of autonomic nervous system. METHODS: Healthy subjects were submitted to RIC protocol and electrocardiography (ECG) was used to evaluate HRV, by assessing the variability of time intervals between two consecutive heart beats. This is a pilot study based on the analysis of 18 ECG from healthy subjects submitted to RIC. HRV was characterized in three domains (time, frequency and non-linear features) that can be correlated with the autonomic nervous system function. RESULTS: RIC procedure increased significantly the non-linear parameter SD2, which is associated with long term HRV. This effect was observed in all subjects and in the senior (> 60 years-old) subset analysis. SD2 increase suggests an activation of both parasympathetic and sympathetic nervous system, namely via fast vagal response (parasympathetic) and the slow sympathetic response to the baroreceptors stimulation. CONCLUSIONS: RIC procedure modulates both parasympathetic and sympathetic autonomic nervous system. Furthermore, this modulation is more pronounced in the senior subset of subjects. Therefore, the autonomic nervous system regulation could be one of the mechanisms for RIC therapeutic effectiveness.

A Semi-Automatic Annotation Approach for Human Activity Recognition.

Modern smartphones and wearables often contain multiple embedded sensors which generate significant amounts of data. This information can be used for body monitoring-based areas such as healthcare, indoor location, user-adaptive recommendations and transportation. The development of Human Activity Recognition (HAR) algorithms involves the collection of a large amount of labelled data which should be annotated by an expert. However, the data annotation process on large datasets is expensive, time consuming and difficult to obtain. The development of a HAR approach which requires low annotation effort and still maintains adequate performance is a relevant challenge. We introduce a Semi-Supervised Active Learning (SSAL) based on Self-Training (ST) approach for Human Activity Recognition to partially automate the annotation process, reducing the annotation effort and the required volume of annotated data to obtain a high performance classifier. Our approach uses a criterion to select the most relevant samples for annotation by the expert and propagate their label to the most confident samples. We present a comprehensive study comparing supervised and unsupervised methods with our approach on two datasets composed of daily living activities. The results showed that it is possible to reduce the required annotated data by more than 89% while still maintaining an accurate model performance.

Fingerprints and Floor Plans Construction for Indoor Localisation Based on Crowdsourcing.

The demand for easily deployable indoor localisation solutions has been growing. Although several systems have been proposed, their limitations regarding the high implementation costs hinder most of them to be widely used. Fingerprinting-based IPS (Indoor Positioning Systems) depend on characteristics pervasively available in buildings. However, such systems require indoor floor plans, which might not be available, as well as environmental fingerprints, that need to be collected through human resources intensive processes. To overcome these limitations, this paper proposes an algorithm for the automatic construction of indoor maps and fingerprints, solely depending on non-annotated crowdsourced data from smartphones. Our system relies on multiple gait-model based filtering techniques for accurate movement quantification in combination with opportunistic sensing observations. After the reconstruction of users' movement with PDR (Pedestrian Dead Reckoning) techniques, Wi-Fi measurements are clustered to partition the trajectories into segments. Similar segments, which belong to the same cluster, are identified using an adaptive approach based on a geomagnetic field distance. Finally, the floor plans are obtained through a data fusion process. Merging the acquired environmental data using the obtained floor plan, fingerprints are aligned to physical locations. Experimental results show that the proposed solution achieved comparable floor plans and fingerprints to those acquired manually, allowing the conclusion that is possible to automate the setup process of infrastructure-free IPS.

Tunable Gas Sensing Gels by Cooperative Assembly.

The cooperative assembly of biopolymers and small molecules can yield functional materials with precisely tunable properties. Here, the fabrication, characterization, and use of multicomponent hybrid gels as selective gas sensors are reported. The gels are composed of liquid crystal droplets self-assembled in the presence of ionic liquids, which further coassemble with biopolymers to form stable matrices. Each individual component can be varied and acts cooperatively to tune gels' structure and function. The unique molecular environment in hybrid gels is explored for supramolecular recognition of volatile compounds. Gels with distinct compositions are used as optical and electrical gas sensors, yielding a combinatorial response conceptually mimicking olfactory biological systems, and tested to distinguish volatile organic compounds and to quantify ethanol in automotive fuel. The gel response is rapid, reversible, and reproducible. These robust, versatile, modular, pliant electro-optical soft materials possess new possibilities in sensing triggered by chemical and physical stimuli.

Biosignals learning and synthesis using deep neural networks.

BACKGROUND: Modeling physiological signals is a complex task both for understanding and synthesize biomedical signals. We propose a deep neural network model that learns and synthesizes biosignals, validated by the morphological equivalence of the original ones. This research could lead the creation of novel algorithms for signal reconstruction in heavily noisy data and source detection in biomedical engineering field. METHOD: The present work explores the gated recurrent units (GRU) employed in the training of respiration (RESP), electromyograms (EMG) and electrocardiograms (ECG). Each signal is pre-processed, segmented and quantized in a specific number of classes, corresponding to the amplitude of each sample and fed to the model, which is composed by an embedded matrix, three GRU blocks and a softmax function. This network is trained by adjusting its internal parameters, acquiring the representation of the abstract notion of the next value based on the previous ones. The simulated signal was generated by forecasting a random value and re-feeding itself. RESULTS AND CONCLUSIONS: The resulting generated signals are similar with the morphological expression of the originals. During the learning process, after a set of iterations, the model starts to grasp the basic morphological characteristics of the signal and later their cyclic characteristics. After training, these models' prediction are closer to the signals that trained them, specially the RESP and ECG. This synthesis mechanism has shown relevant results that inspire the use to characterize signals from other physiological sources.

Ensemble Predictors: Possibilistic Combination of Conformal Predictors for Multivariate Time Series Classification.

In this article we propose a conceptual framework to study ensembles of conformal predictors (CP), that we call Ensemble Predictors (EP). Our approach is inspired by the application of imprecise probabilities in information fusion. Based on the proposed framework, we study, for the first time in the literature, the theoretical properties of CP ensembles in a general setting, by focusing on simple and commonly used possibilistic combination rules. We also illustrate the applicability of the proposed methods in the setting of multivariate time-series classification, showing that these methods provide better performance (in terms of both robustness, conservativeness, accuracy and running time) than both standard classification algorithms and other combination rules proposed in the literature, on a large set of benchmarks from the UCR time series archive.

Predicting post-discharge complications in cardiothoracic surgery: A clinical decision support system to optimize remote patient monitoring resources.

Cardiac surgery patients are highly prone to severe complications post-discharge. Close follow-up through remote patient monitoring can help detect adverse outcomes earlier or prevent them, closing the gap between hospital and home care. However, equipment is limited due to economic and human resource constraints. This issue raises the need for efficient risk estimation to provide clinicians with insights into the potential benefit of remote monitoring for each patient. Standard models, such as the EuroSCORE, predict the mortality risk before the surgery. While these are used and validated in real settings, the models lack information collected during or following the surgery, determinant to predict adverse outcomes occurring further in the future. This paper proposes a Clinical Decision Support System based on Machine Learning to estimate the risk of severe complications within 90 days following cardiothoracic surgery discharge, an innovative objective underexplored in the literature. Health records from a cardiothoracic surgery department regarding 5 045 patients (60.8% male) collected throughout ten years were used to train predictive models. Clinicians' insights contributed to improving data preparation and extending traditional pipeline optimization techniques, addressing medical Artificial Intelligence requirements. Two separate test sets were used to evaluate the generalizability, one derived from a patient-grouped 70/30 split and another including all surgeries from the last available year. The achieved Area Under the Receiver Operating Characteristic curve on these test sets was 69.5% and 65.3%, respectively. Also, additional testing was implemented to simulate a real-world use case considering the weekly distribution of remote patient monitoring resources post-discharge. Compared to the random resource allocation, the selection of patients with respect to the outputs of the proposed model was proven beneficial, as it led to a higher number of high-risk patients receiving remote monitoring equipment.

Occupational health knowledge discovery based on association rules applied to workers' body parts protection: a case study in the automotive industry.

Occupational Health Protection (OHP) is mandatory by law and can be accomplished by considering the participation of others besides occupational physicians. The data shared can originate knowledge that might influence other processes related to occupational risk prevention. In this study, we used Artificial Intelligence (AI) methods to extract patterns among records shared under these circumstances over two years in the automotive industry. Records featuring OHP data against physical working conditions were selected, and a database of 383 profiles was designed. As Occupational Health Protection profiles under study are associated with work functional ability reduction, the body part(s) (n = 14) where it occurred were identified. Association Rules (ARs) coupled with Natural Language Processing techniques were applied to find meaningful hidden relationships and to identify the occurrence of protection profiles being assigned to at least two body parts simultaneously. After filtering ARs using three metrics (support, confidence, and lift), 54 ARs were found. The distribution of simultaneous body parts is presented as being higher in Special projects (n = 5). The results can use in: (i) design a multi-site body parts functional work ability (loss) model; (ii) model the capacity of organizations to retain workers in their working settings and (iii) prevent work-related musculoskeletal symptoms.

Rams, hounds and white boxes: Investigating human-AI collaboration protocols in medical diagnosis.

In this paper, we study human-AI collaboration protocols, a design-oriented construct aimed at establishing and evaluating how humans and AI can collaborate in cognitive tasks. We applied this construct in two user studies involving 12 specialist radiologists (the knee MRI study) and 44 ECG readers of varying expertise (the ECG study), who evaluated 240 and 20 cases, respectively, in different collaboration configurations. We confirm the utility of AI support but find that XAI can be associated with a "white-box paradox", producing a null or detrimental effect. We also find that the order of presentation matters: AI-first protocols are associated with higher diagnostic accuracy than human-first protocols, and with higher accuracy than both humans and AI alone. Our findings identify the best conditions for AI to augment human diagnostic skills, rather than trigger dysfunctional responses and cognitive biases that can undermine decision effectiveness.

Does whole body vibration training affect knee kinematics and neuromuscular control in healthy people?

This study aimed to investigate the effect of whole body vibration (WBV) training on the knee kinematics and neuromuscular control after single-legged drop landings. Surface electromyographic (EMG) activity of the rectus femoris and hamstring muscles and knee and ankle accelerometry signals were acquired from 42 healthy volunteers. Participants performed three pre-test landings and after a recovery period of three minutes, they completed one set of six bouts of WBV each of one minute duration (30 Hz - 4 mm), followed by a single-leg drop landing. After the WBV intervention no significant changes were observed in the kinematic outcomes measured, although the time to stabilise the lower-limb was significantly lower after the vibration training (F(8,41) = 6.55; P < 0.01). EMG analysis showed no significant differences in the amplitude of rectus femoris or hamstring muscles after WBV training, however, significant differences in EMG frequency of the rectus femoris were found before (F(8,41) = 7.595; P < 0.01) and after toe-down (F(8,41) = 4.440; P < 0.001). Finally, no significant changes were observed in knee or ankle acceleration after WBV. Results suggest that WBV can help to acutely enhance knee neuromuscular control, which may have clinical significance and help in the design of rehabilitation programmes.

Gender differences in knee stability in response to whole-body vibration.

The purpose of this study was to determine whether there are kinematic and electromyographic (EMG) differences between men and women in how the knee is controlled during a single-legged drop landing in response to whole-body vibration (WBV). Forty-five healthy volunteers, 30 men (age 22 ± 3 years; weight 76.8 ± 8.8 kg; height 179.0 ± 6.8 cm) and 15 women (age 22 ± 3 years; weight 61.0 ± 7.7 kg; height 161.9 ± 7.2 cm) were recruited for this study. Knee angles, vertical ground reaction forces, and the time to stabilize the knee were assessed after single-legged drop landings from a 30-cm platform. Surface EMG data in rectus femoris (RF) and hamstrings (H) and knee and ankle accelerometry signals were also acquired. The participants performed 3 pretest landings, followed by a 3-minute recovery and then completed 1 minute of WBV (30 Hz to 4 mm). Before vibration, the female subjects had a significantly higher peak vertical force value, knee flexion angles, and greater H preactivity (EMG(RMS) 50 milliseconds before activation) than did the male subjects. In addition, although not significant, the medial-lateral (ML) acceleration in both knee and ankle was also higher in women. After WBV, no significant differences were found for any of the other variables. However, there was a decrease in the RF to H activation ratio during the precontact phase and an increase in the ratio during the postcontact phase just in women, which leads to a decrement in ML acceleration. The gender differences reported in knee stability in response to WBV underline the necessity to perform specific neuromuscular training programs based on WBV together with instruction of the proper technique, which can assist the clinician in the knee injury prevention.

Human-Centered Explainable Artificial Intelligence: Automotive Occupational Health Protection Profiles in Prevention Musculoskeletal Symptoms.

In automotive and industrial settings, occupational physicians are responsible for monitoring workers' health protection profiles. Workers' Functional Work Ability (FWA) status is used to create Occupational Health Protection Profiles (OHPP). This is a novel longitudinal study in comparison with previous research that has predominantly relied on the causality and explainability of human-understandable models for industrial technical teams like ergonomists. The application of artificial intelligence can support the decision-making to go from a worker's Functional Work Ability to explanations by integrating explainability into medical (restriction) and support in contexts of individual, work-related, and organizational risk conditions. A sample of 7857 for the prognosis part of OHPP based on Functional Work Ability in the Portuguese language in the automotive industry was taken from 2019 to 2021. The most suitable regression models to predict the next medical appointment for the workers' body parts protection were the models based on CatBoost regression, with an RMSLE of 0.84 and 1.23 weeks (mean error), respectively. CatBoost algorithm is also used to predict the next body part severity of OHPP. This information can help our understanding of potential risk factors for OHPP and identify warning signs of the early stages of musculoskeletal symptoms and work-related absenteeism.