MM-HAR: Multi-Modal Human Activity Recognition Using Consumer Smartwatch and Earbuds.

Human Activity Recognition (HAR) is one of the important applications of digital health that helps to track fitness or to avoid sedentary behavior by monitoring daily activities. Due to the growing popularity of consumer wearable devices, smartwatches, and earbuds are being widely adopted for HAR applications. However, using just one of the devices may not be sufficient to track all activities properly. This paper proposes a multi-modal approach to HAR by using both buds and watch. Using a large dataset of 44 subjects collected from both in-lab and in-home environments, we demonstrate the limitations of using a single modality as well as the importance of a multi-modal approach. Moreover, we also train and evaluate the performance of five different machine learning classifiers for various combinations of devices such as buds only, watch only, and both. We believe the detailed analyses presented in this paper may serve as a benchmark for the research community to explore and build upon in the future.

Power Optimized Smartwatch-Earbuds Multimodal System for Monitoring Activities of Daily Living.

Activities of daily living is an important entity to monitor for promoting healthy lifestyle for chronic disease patients, children and the healthy population. This paper presents a smartwatch and earbuds inertial sensors based multi-modal power efficient end-to-end mobile system for continuous, passive and accurate detection of broad daily activity classes. We collected various posture, stationary and moving activity data from 40 diverse subjects using earbuds and smartwatch and develop the novel power optimized end-to-end operational system consisting of i) optimized device sampling rates and Bluetooth packet transfer rates, ii) data buffering mechanism, iii) background services, and iv) optimized model size, and demonstrating 93% macro recall score in detecting various activities. Our power optimized solution uses 80%, 40% and 33.33% less battery power for the smartphone, smartwatch, and earbuds respectively, compared to a power agnostic system with an estimated continuous no-charging run time of 50 hours, 16.67 hours, and 25 hours for the smartphone, smartwatch, and earbuds respectively.Clinical relevance- The end-to-end power optimized activity detection system presented in this paper will assist practicing clinicians toward treatment of various chronic disease patients (e.g. diabetes, hypertension, heart disease and obesity) by long-term, continuous monitoring of their lifestyle and sedentary behavior.

Feature Augmented Hybrid CNN for Stress Recognition Using Wrist-based Photoplethysmography Sensor.

Stress is a physiological state that hampers mental health and has serious consequences to physical health. More-over, the COVID-19 pandemic has increased stress levels among people across the globe. Therefore, continuous monitoring and detection of stress are necessary. The recent advances in wearable devices have allowed the monitoring of several physiological signals related to stress. Among them, wrist-worn wearable devices like smartwatches are most popular due to their convenient usage. And the photoplethysmography (PPG) sensor is the most prevalent sensor in almost all consumer-grade wrist-worn smartwatches. Therefore, this paper focuses on using a wrist-based PPG sensor that collects Blood Volume Pulse (BVP) signals to detect stress which may be applicable for consumer-grade wristwatches. Moreover, state-of-the-art works have used either classical machine learning algorithms to detect stress using hand-crafted features or have used deep learning algorithms like Convolutional Neural Network (CNN) which automatically extracts features. This paper proposes a novel hybrid CNN (H-CNN) classifier that uses both the hand-crafted features and the automatically extracted features by CNN to detect stress using the BVP signal. Evaluation on the benchmark WESAD dataset shows that, for 3-class classification (Baseline vs. Stress vs. Amusement), our proposed H-CNN outperforms traditional classifiers and normal CNN by ≈5% and ≈7% accuracy, and ≈10% and ≈7% macro F1 score, respectively. Also for 2-class classification (Stress vs. Non-stress), our proposed H-CNN outperforms traditional classifiers and normal CNN by ≈3% and ≈5% accuracy, and ≈3% and ≈7% macro F1score, respectively.

Feature Augmented Hybrid CNN for Stress Recognition Using Wrist-based Photoplethysmography Sensor.

Stress is a physiological state that hampers mental health and has serious consequences to physical health. Moreover, the COVID-19 pandemic has increased stress levels among people across the globe. Therefore, continuous monitoring and detection of stress are necessary. The recent advances in wearable devices have allowed the monitoring of several physiological signals related to stress. Among them, wrist-worn wearable devices like smartwatches are most popular due to their convenient usage. And the photoplethysmography (PPG) sensor is the most prevalent sensor in almost all consumer-grade wrist-worn smartwatches. Therefore, this paper focuses on using a wrist-based PPG sensor that collects Blood Volume Pulse (BVP) signals to detect stress which may be applicable for consumer-grade wristwatches. Moreover, state-of-the-art works have used either classical machine learning algorithms to detect stress using hand-crafted features or have used deep learning algorithms like Convolutional Neural Network (CNN) which automatically extracts features. This paper proposes a novel hybrid CNN (H-CNN) classifier that uses both the hand-crafted features and the automatically extracted features by CNN to detect stress using the BVP signal. Evaluation on the benchmark WESAD dataset shows that, for 3-class classification (Baseline vs. Stress vs. Amusement), our proposed H-CNN outperforms traditional classifiers and normal CNN by 5% and 7% accuracy, and 10% and 7% macro F1 score, respectively. Also for 2-class classification (Stress vs. Non-stress), our proposed H-CNN outperforms traditional classifiers and normal CNN by 3% and ~5% accuracy, and ~3% and ~7% macro F1 score, respectively.

Wireless Qi-powered, Multinodal and Multisensory Body Area Network for Mobile Health.

Wireless, battery-free Body Area Networks (BAN) enable reliable long-term health monitoring with minimal intervention, and have the potential to transform patient care via mobile health monitoring. Current approaches for achieving such battery-free networks are limited in the number, capability, and positioning of sensing nodes-this is related to constraints in power supply, data rate, and working distance requirements between the wireless power source and sensing nodes. Here, we investigate a Qi-based, near-field power transfer scheme that can effectively drive wireless, battery-free, multi-node and multi-sensor BAN over long distances. This consists of a single Qi power source (such as a cellphone), a detached/untethered Passive Intermediate Relay (PIR) (facilitates power transfer from a central Qi source to multiple nodes on the body), and finally individual/detached sensing nodes placed throughout the body. Alongside this power scheme we implement the star network topology of a Gazell protocol to enable the continuous connection of one host to many sensing nodes while minimizing data loss over long temporal periods. The high-power transmission capabilities of Qi enables wireless support for a multitude of sensors (up to 12), and sensing nodes (up to 6) with a single transmitter at long distances (60 cm) and a sample rate of 20 Hz. This scheme is studied both in-vitro and in-vivo on the body.

Energy-efficient Real-time Myocardial Infarction Detection on Wearable Devices.

Myocardial Infarction (MI) is a fatal heart disease that is a leading cause of death. The silent and recurrent nature of MI requires real-time monitoring on a daily basis through wearable devices. Real-time MI detection on wearable devices requires a fast and energy-efficient solution to enable long term monitoring. In this paper, we propose an MI detection methodology using Binary Convolutional Neural Network (BCNN) that is fast, energy-efficient and outperforms the state-of-the- art work on wearable devices. We validate the performance of our methodology on the well known PTB diagnostic ECG database from PhysioNet. Evaluation on real hardware shows that our BCNN is faster and achieves up to 12x energy efficiency compared to the state-of-the-art work.