The State-of-the-Art Sensing Techniques in Human Activity Recognition: A Survey.

Human activity recognition (HAR) has become an intensive research topic in the past decade because of the pervasive user scenarios and the overwhelming development of advanced algorithms and novel sensing approaches. Previous HAR-related sensing surveys were primarily focused on either a specific branch such as wearable sensing and video-based sensing or a full-stack presentation of both sensing and data processing techniques, resulting in weak focus on HAR-related sensing techniques. This work tries to present a thorough, in-depth survey on the state-of-the-art sensing modalities in HAR tasks to supply a solid understanding of the variant sensing principles for younger researchers of the community. First, we categorized the HAR-related sensing modalities into five classes: mechanical kinematic sensing, field-based sensing, wave-based sensing, physiological sensing, and hybrid/others. Specific sensing modalities are then presented in each category, and a thorough description of the sensing tricks and the latest related works were given. We also discussed the strengths and weaknesses of each modality across the categorization so that newcomers could have a better overview of the characteristics of each sensing modality for HAR tasks and choose the proper approaches for their specific application. Finally, we summarized the presented sensing techniques with a comparison concerning selected performance metrics and proposed a few outlooks on the future sensing techniques used for HAR tasks.

Classifying Muscle States with One-Dimensional Radio-Frequency Signals from Single Element Ultrasound Transducers.

The reliable assessment of muscle states, such as contracted muscles vs. non-contracted muscles or relaxed muscles vs. fatigue muscles, is crucial in many sports and rehabilitation scenarios, such as the assessment of therapeutic measures. The goal of this work was to deploy machine learning (ML) models based on one-dimensional (1-D) sonomyography (SMG) signals to facilitate low-cost and wearable ultrasound devices. One-dimensional SMG is a non-invasive technique using 1-D ultrasound radio-frequency signals to measure muscle states and has the advantage of being able to acquire information from deep soft tissue layers. To mimic real-life scenarios, we did not emphasize the acquisition of particularly distinct signals. The ML models exploited muscle contraction signals of eight volunteers and muscle fatigue signals of 21 volunteers. We evaluated them with different schemes on a variety of data types, such as unprocessed or processed raw signals and found that comparatively simple ML models, such as Support Vector Machines or Logistic Regression, yielded the best performance w.r.t. accuracy and evaluation time. We conclude that our framework for muscle contraction and muscle fatigue classifications is very well-suited to facilitate low-cost and wearable devices based on ML models using 1-D SMG.

Using Privacy Respecting Sound Analysis to Improve Bluetooth Based Proximity Detection for COVID-19 Exposure Tracing and Social Distancing.

We propose to use ambient sound as a privacy-aware source of information for COVID-19-related social distance monitoring and contact tracing. The aim is to complement currently dominant Bluetooth Low Energy Received Signal Strength Indicator (BLE RSSI) approaches. These often struggle with the complexity of Radio Frequency (RF) signal attenuation, which is strongly influenced by specific surrounding characteristics. This in turn renders the relationship between signal strength and the distance between transmitter and receiver highly non-deterministic. We analyze spatio-temporal variations in what we call "ambient sound fingerprints". We leverage the fact that ambient sound received by a mobile device is a superposition of sounds from sources at many different locations in the environment. Such a superposition is determined by the relative position of those sources with respect to the receiver. We present a method for using the above general idea to classify proximity between pairs of users based on Kullback-Leibler distance between sound intensity histograms. The method is based on intensity analysis only, and does not require the collection of any privacy sensitive signals. Further, we show how this information can be fused with BLE RSSI features using adaptive weighted voting. We also take into account that sound is not available in all windows. Our approach is evaluated in elaborate experiments in real-world settings. The results show that both Bluetooth and sound can be used to differentiate users within and out of critical distance (1.5 m) with high accuracies of 77% and 80% respectively. Their fusion, however, improves this to 86%, making evident the merit of augmenting BLE RSSI with sound. We conclude by discussing strengths and limitations of our approach and highlighting directions for future work.

Induced Magnetic Field-Based Indoor Positioning System for Underwater Environments.

Autonomous underwater vehicles (AUV) are seen as an emerging technology for maritime exploration but are still restricted by the availability of short range, accurate positioning methods necessary, e.g., when docking remote assets. Typical techniques used for high-accuracy positioning in indoor use case scenarios, such as systems using ultra-wide band radio signals (UWB), cannot be applied for underwater positioning because of the quick absorption of the positioning medium caused by the water. Acoustic and optic solutions for underwater positioning also face known problems, such as the multi-path effects, high propagation delay (acoustics), and environmental dependency. This paper presents an oscillating magnetic field-based indoor and underwater positioning system. Unlike those radio wave-based positioning modalities, the magnetic approach generates a bubble-formed magnetic field that will not be deformed by the environmental variation because of the very similar permeability of water and air. The proposed system achieves an underwater positioning mean accuracy of 13.3 cm in 2D and 19.0 cm in 3D with the multi-lateration positioning method and concludes the potential of the magnetic field-based positioning technique for underwater applications. A similar accuracy was also achieved for various indoor environments that were used to test the influence of cluttered environment and of cross environment. The low cost and power consumption system is scalable for extensive coverage area and could plug-and-play without pre-calibration.

Give more data, awareness and control to individual citizens, and they will help COVID-19 containment.

The rapid dynamics of COVID-19 calls for quick and effective tracking of virus transmission chains and early detection of outbreaks, especially in the "phase 2" of the pandemic, when lockdown and other restriction measures are progressively withdrawn, in order to avoid or minimize contagion resurgence. For this purpose, contact-tracing apps are being proposed for large scale adoption by many countries. A centralized approach, where data sensed by the app are all sent to a nation-wide server, raises concerns about citizens' privacy and needlessly strong digital surveillance, thus alerting us to the need to minimize personal data collection and avoiding location tracking. We advocate the conceptual advantage of a decentralized approach, where both contact and location data are collected exclusively in individual citizens' "personal data stores", to be shared separately and selectively (e.g., with a backend system, but possibly also with other citizens), voluntarily, only when the citizen has tested positive for COVID-19, and with a privacy preserving level of granularity. This approach better protects the personal sphere of citizens and affords multiple benefits: it allows for detailed information gathering for infected people in a privacy-preserving fashion; and, in turn this enables both contact tracing, and, the early detection of outbreak hotspots on more finely-granulated geographic scale. The decentralized approach is also scalable to large populations, in that only the data of positive patients need be handled at a central level. Our recommendation is two-fold. First to extend existing decentralized architectures with a light touch, in order to manage the collection of location data locally on the device, and allow the user to share spatio-temporal aggregates-if and when they want and for specific aims-with health authorities, for instance. Second, we favour a longer-term pursuit of realizing a Personal Data Store vision, giving users the opportunity to contribute to collective good in the measure they want, enhancing self-awareness, and cultivating collective efforts for rebuilding society.

CEGAN: Classification Enhancement Generative Adversarial Networks for unraveling data imbalance problems.

The data imbalance problem in classification is a frequent but challenging task. In real-world datasets, numerous class distributions are imbalanced and the classification result under such condition reveals extreme bias in the majority data class. Recently, the potential of GAN as a data augmentation method on minority data has been studied. In this paper, we propose a classification enhancement generative adversarial networks (CEGAN) to enhance the quality of generated synthetic minority data and more importantly, to improve the prediction accuracy in data imbalanced condition. In addition, we propose an ambiguity reduction method using the generated synthetic minority data for the case of multiple similar classes that are degenerating the classification accuracy. The proposed method is demonstrated with five benchmark datasets. The results indicate that approximating the real data distribution using CEGAN improves the classification performance significantly in data imbalanced conditions compared with various standard data augmentation methods.

Facial Muscle Activity Recognition with Reconfigurable Differential Stethoscope-Microphones.

Many human activities and states are related to the facial muscles' actions: from the expression of emotions, stress, and non-verbal communication through health-related actions, such as coughing and sneezing to nutrition and drinking. In this work, we describe, in detail, the design and evaluation of a wearable system for facial muscle activity monitoring based on a re-configurable differential array of stethoscope-microphones. In our system, six stethoscopes are placed at locations that could easily be integrated into the frame of smart glasses. The paper describes the detailed hardware design and selection and adaptation of appropriate signal processing and machine learning methods. For the evaluation, we asked eight participants to imitate a set of facial actions, such as expressions of happiness, anger, surprise, sadness, upset, and disgust, and gestures, like kissing, winkling, sticking the tongue out, and taking a pill. An evaluation of a complete data set of 2640 events with 66% training and a 33% testing rate has been performed. Although we encountered high variability of the volunteers' expressions, our approach shows a recall = 55%, precision = 56%, and f1-score of 54% for the user-independent scenario(9% chance-level). On a user-dependent basis, our worst result has an f1-score = 60% and best result with f1-score = 89%. Having a recall ≥60% for expressions like happiness, anger, kissing, sticking the tongue out, and neutral(Null-class).

Social Distance Monitor with a Wearable Magnetic Field Proximity Sensor.

Social distancing and contact/exposure tracing are accepted to be critical strategies in the fight against the COVID-19 epidemic. They are both closely connected to the ability to reliably establish the degree of proximity between people in real-world environments. We proposed, implemented, and evaluated a wearable proximity sensing system based on an oscillating magnetic field that overcomes many of the weaknesses of the current state of the art Bluetooth based proximity detection. In this paper, we first described the underlying physical principle, proposed a protocol for the identification and coordination of the transmitter (which is compatible with the current smartphone-based exposure tracing protocols). Subsequently, the system architecture and implementation were described, finally an elaborate characterization and evaluation of the performance (both in systematic lab experiments and in real-world settings) were performed. Our work demonstrated that the proposed system is much more reliable than the widely-used Bluetooth-based approach, particularly when it comes to distinguishing between distances above and below the 2.0 m threshold due to the magnetic field's physical properties.

Accurate Spirometry with Integrated Barometric Sensors in Face-Worn Garments.

Cardiorespiratory (CR) signals are crucial vital signs for fitness condition tracking, medical diagnosis, and athlete performance evaluation. Monitoring such signals in real-life settings is among the most widespread applications of wearable computing. We investigate how miniaturized barometers can be used to perform accurate spirometry in a wearable system that is built on off-the-shelf training masks often used by athletes as a training aid. We perform an evaluation where differential barometric pressure sensors are compared concurrently with a digital spirometer, during an experimental setting of clinical forced vital capacity (FVC) test procedures with 20 participants. The relationship between the two instruments is derived by mathematical modeling first, then by various regression methods from experiment data. The results show that the error of FVC vital values between the two instruments can be as low as 2∼3%. Beyond clinical tests, the method can also measure continuous tidal breathing air volumes with a 1∼3% error margin. Overall, we conclude that barometers with millimeter footprints embedded in face mask apparel can perform similarly to a digital spirometer to monitor breathing airflow and volume in pulmonary function tests.

Expressure: Detect Expressions Related to Emotional and Cognitive Activities Using Forehead Textile Pressure Mechanomyography.

We investigate how pressure-sensitive smart textiles, in the form of a headband, can detect changes in facial expressions that are indicative of emotions and cognitive activities. Specifically, we present the Expressure system that performs surface pressure mechanomyography on the forehead using an array of textile pressure sensors that is not dependent on specific placement or attachment to the skin. Our approach is evaluated in systematic psychological experiments. First, through a mimicking expression experiment with 20 participants, we demonstrate the system's ability to detect well-defined facial expressions. We achieved accuracies of 0.824 to classify among three eyebrow movements (0.333 chance-level) and 0.381 among seven full-face expressions (0.143 chance-level). A second experiment was conducted with 20 participants to induce cognitive loads with N-back tasks. Statistical analysis has shown significant correlations between the Expressure features on a fine time granularity and the cognitive activity. The results have also shown significant correlations between the Expressure features and the N-back score. From the 10 most facially expressive participants, our approach can predict whether the N-back score is above or below the average with 0.767 accuracy.

Participatory Disease Surveillance Systems: Ethical Framework.

Advances in information technology are changing public health at an unprecedented rate. Participatory surveillance systems are contributing to public health by actively engaging digital (eg, Web-based) communities of volunteer citizens to report symptoms and other pertinent information on public health threats and also by empowering individuals to promptly respond to them. However, this digital model raises ethical issues on top of those inherent in traditional forms of public health surveillance. Research ethics are undergoing significant changes in the digital era where not only participants' physical and psychological well-being but also the protection of their sensitive data have to be considered. In this paper, the digital platform of Influenzanet is used as a case study to illustrate those ethical challenges posed to participatory surveillance systems using digital platforms and mobile apps. These ethical challenges include the implementation of electronic consent, the protection of participants' privacy, the promotion of justice, and the need for interdisciplinary capacity building of research ethics committees. On the basis of our analysis, we propose a framework to regulate and strengthen ethical approaches in the field of digital public health surveillance.

Textile Pressure Mapping Sensor for Emotional Touch Detection in Human-Robot Interaction.

In this paper, we developed a fully textile sensing fabric for tactile touch sensing as the robot skin to detect human-robot interactions. The sensor covers a 20-by-20 cm 2 area with 400 sensitive points and samples at 50 Hz per point. We defined seven gestures which are inspired by the social and emotional interactions of typical people to people or pet scenarios. We conducted two groups of mutually blinded experiments, involving 29 participants in total. The data processing algorithm first reduces the spatial complexity to frame descriptors, and temporal features are calculated through basic statistical representations and wavelet analysis. Various classifiers are evaluated and the feature calculation algorithms are analyzed in details to determine each stage and segments' contribution. The best performing feature-classifier combination can recognize the gestures with a 93 . 3 % accuracy from a known group of participants, and 89 . 1 % from strangers.

Smartphone-based recognition of states and state changes in bipolar disorder patients.

Today's health care is difficult to imagine without the possibility to objectively measure various physiological parameters related to patients' symptoms (from temperature through blood pressure to complex tomographic procedures). Psychiatric care remains a notable exception that heavily relies on patient interviews and self-assessment. This is due to the fact that mental illnesses manifest themselves mainly in the way patients behave throughout their daily life and, until recently there were no "behavior measurement devices." This is now changing with the progress in wearable activity recognition and sensor enabled smartphones. In this paper, we introduce a system, which, based on smartphone-sensing is able to recognize depressive and manic states and detect state changes of patients suffering from bipolar disorder. Drawing upon a real-life dataset of ten patients, recorded over a time period of 12 weeks (in total over 800 days of data tracing 17 state changes) by four different sensing modalities, we could extract features corresponding to all disease-relevant aspects in behavior. Using these features, we gain recognition accuracies of 76% by fusing all sensor modalities and state change detection precision and recall of over 97%. This paper furthermore outlines the applicability of this system in the physician-patient relations in order to facilitate the life and treatment of bipolar patients.

Design and construction of the high-speed optoelectronic memory system demonstrator.

The high-speed optoelectronic memory system project is concerned with the reduction of latency within multiprocessor computer systems (a key problem) by the use of optoelectronics and associated packaging technologies. System demonstrators have been constructed to enable the evaluation of the technologies in terms of manufacturability. The system combines fiber, free space, and planar integrated optical waveguide technologies to augment the electronic memory and the processor components. Modeling and simulation techniques were developed toward the analysis and design of board-integrated waveguide transmission characteristics and optical interfacing. We describe the fabrication, assembly, and simulation of the major components within the system.

Activity recognition of assembly tasks using body-worn microphones and accelerometers.

In order to provide relevant information to mobile users, such as workers engaging in the manual tasks of maintenance and assembly, a wearable computer requires information about the user's specific activities. This work focuses on the recognition of activities that are characterized by a hand motion and an accompanying sound. Suitable activities can be found in assembly and maintenance work. Here, we provide an initial exploration into the problem domain of continuous activity recognition using on-body sensing. We use a mock "wood workshop" assembly task to ground our investigation. We describe a method for the continuous recognition of activities (sawing, hammering, filing, drilling, grinding, sanding, opening a drawer, tightening a vise, and turning a screwdriver) using microphones and three-axis accelerometers mounted at two positions on the user's arms. Potentially "interesting" activities are segmented from continuous streams of data using an analysis of the sound intensity detected at the two different locations. Activity classification is then performed on these detected segments using linear discriminant analysis (LDA) on the sound channel and hidden Markov models (HMMs) on the acceleration data. Four different methods at classifier fusion are compared for improving these classifications. Using user-dependent training, we obtain continuous average recall and precision rates (for positive activities) of 78 percent and 74 percent, respectively. Using user-independent training (leave-one-out across five users), we obtain recall rates of 66 percent and precision rates of 63 percent. In isolation, these activities were recognized with accuracies of 98 percent, 87 percent, and 95 percent for the user-dependent, user-independent, and user-adapted cases, respectively.

Motion aware clothing for the personal health assistant.

This paper sketches the vision and first results of a 'Personal Health Assistant' PHA, opening up new vistas in patient centred healthcare. The PHA is comprised of a wearable sensing and communicating system, seamlessly embedded in daily clothing. Several on-body sensors monitor the biometric and contextual status of the wearer continuously. The embedded computer fuses the vital and physiological data with activity patterns of the wearer and with the social environment; based on these data the on-body computer generates the 'Life Balance Factor' LBF as an individual feedback to the user and to the surroundings afford-ing effective disease prevention, management and rehabilitation, the last also involving telemedicine. The state-of-the-art enabling technologies: smart textile technology and miniaturization of electronics combined with wireless communication, along with recent developments in wearable computing are presented and assessed in the context of multiparameter health monitoring.

AMON: a wearable multiparameter medical monitoring and alert system.

This paper describes an advanced care and alert portable telemedical monitor (AMON), a wearable medical monitoring and alert system targeting high-risk cardiac/respiratory patients. The system includes continuous collection and evaluation of multiple vital signs, intelligent multiparameter medical emergency detection, and a cellular connection to a medical center. By integrating the whole system in an unobtrusive, wrist-worn enclosure and applying aggressive low-power design techniques, continuous long-term monitoring can be performed without interfering with the patients' everyday activities and without restricting their mobility. In the first two and a half years of this EU IST sponsored project, the AMON consortium has designed, implemented, and tested the described wrist-worn device, a communication link, and a comprehensive medical center software package. The performance of the system has been validated by a medical study with a set of 33 subjects. The paper describes the main concepts behind the AMON system and presents details of the individual subsystems and solutions as well as the results of the medical validation.

Multiple imaging technique for extending depth of focus in retinal displays.

Limited depth of focus is among the main problems of todays see-through head-mounted displays. In this paper we propose and evaluate a new solution to this problem: the use of the coherent multiple imaging technique in a retinal projection display by incorporating an appropriate phase-only mask. The evaluation is based on a schematic eye model and on the partial coherence simulation tool SPLAT which allows us to calculate the projected retinal images of a text target. Objective image quality criteria demonstrate that this approach is promising provided that partially coherent illumination light is used. In this case, psychometric measurements reveal that the depth of focus for reading text can be extended by a factor of up to 3.2. For fully coherent and incoherent illumination, however, the retinal images suffer from structural and contrast degradation effects, respectively.