EEG Amplitude Modulation Analysis across Mental Tasks: Towards Improved Active BCIs.

Brain-computer interface (BCI) technology has emerged as an influential communication tool with extensive applications across numerous fields, including entertainment, marketing, mental state monitoring, and particularly medical neurorehabilitation. Despite its immense potential, the reliability of BCI systems is challenged by the intricacies of data collection, environmental factors, and noisy interferences, making the interpretation of high-dimensional electroencephalogram (EEG) data a pressing issue. While the current trends in research have leant towards improving classification using deep learning-based models, our study proposes the use of new features based on EEG amplitude modulation (AM) dynamics. Experiments on an active BCI dataset comprised seven mental tasks to show the importance of the proposed features, as well as their complementarity to conventional power spectral features. Through combining the seven mental tasks, 21 binary classification tests were explored. In 17 of these 21 tests, the addition of the proposed features significantly improved classifier performance relative to using power spectral density (PSD) features only. Specifically, the average kappa score for these classifications increased from 0.57 to 0.62 using the combined feature set. An examination of the top-selected features showed the predominance of the AM-based measures, comprising over 77% of the top-ranked features. We conclude this paper with an in-depth analysis of these top-ranked features and discuss their potential for use in neurophysiology.

Automated Multi-Wavelength Quality Assessment of Photoplethysmography Signals Using Modulation Spectrum Shape Features.

Photoplethysmography (PPG) is used to measure blood volume changes in the microvascular bed of tissue. Information about these changes along time can be used for estimation of various physiological parameters, such as heart rate variability, arterial stiffness, and blood pressure, to name a few. As a result, PPG has become a popular biological modality and is widely used in wearable health devices. However, accurate measurement of various physiological parameters requires good-quality PPG signals. Therefore, various signal quality indexes (SQIs) for PPG signals have been proposed. These metrics have usually been based on statistical, frequency, and/or template analyses. The modulation spectrogram representation, however, captures the second-order periodicities of a signal and has been shown to provide useful quality cues for electrocardiograms and speech signals. In this work, we propose a new PPG quality metric based on properties of the modulation spectrum. The proposed metric is tested using data collected from subjects while they performed various activity tasks contaminating the PPG signals. Experiments on this multi-wavelength PPG dataset show the combination of proposed and benchmark measures significantly outperforming several benchmark SQIs with improvements of 21.3% BACC (balanced accuracy) for green, 21.6% BACC for red, and 19.0% BACC for infrared wavelengths, respectively, for PPG quality detection tasks. The proposed metrics also generalize for cross-wavelength PPG quality detection tasks.

Cross-Language Speech Emotion Recognition Using Bag-of-Word Representations, Domain Adaptation, and Data Augmentation.

To date, several methods have been explored for the challenging task of cross-language speech emotion recognition, including the bag-of-words (BoW) methodology for feature processing, domain adaptation for feature distribution "normalization", and data augmentation to make machine learning algorithms more robust across testing conditions. Their combined use, however, has yet to be explored. In this paper, we aim to fill this gap and compare the benefits achieved by combining different domain adaptation strategies with the BoW method, as well as with data augmentation. Moreover, while domain adaptation strategies, such as the correlation alignment (CORAL) method, require knowledge of the test data language, we propose a variant that we term N-CORAL, in which test languages (in our case, Chinese) are mapped to a common distribution in an unsupervised manner. Experiments with German, French, and Hungarian language datasets were performed, and the proposed N-CORAL method, combined with BoW and data augmentation, was shown to achieve the best arousal and valence prediction accuracy, highlighting the usefulness of the proposed method for "in the wild" speech emotion recognition. In fact, N-CORAL combined with BoW was shown to provide robustness across languages, whereas data augmentation provided additional robustness against cross-corpus nuance factors.

Modulation Spectral Signal Representation for Quality Measurement and Enhancement of Wearable Device Data: A Technical Note.

Wearable devices are burgeoning, and applications across numerous verticals are emerging, including human performance monitoring, at-home patient monitoring, and health tracking, to name a few. Off-the-shelf wearables have been developed with focus on portability, usability, and low-cost. As such, when deployed in highly ecological settings, wearable data can be corrupted by artifacts and by missing data, thus severely hampering performance. In this technical note, we overview a signal processing representation called the modulation spectrum. The representation quantifies the rate-of-change of different spectral magnitude components and is shown to separate signal from noise, thus allowing for improved quality measurement, quality enhancement, and noise-robust feature extraction, as well as for disease characterization. We provide an overview of numerous applications developed by the authors over the last decade spanning different wearable modalities and list the results obtained from experimental results alongside comparisons with various state-of-the-art benchmark methods. Open-source software is showcased with the hope that new applications can be developed. We conclude with a discussion on possible future research directions, such as context awareness, signal compression, and improved input representations for deep learning algorithms.

Virtual reality and non-invasive brain stimulation for rehabilitation applications: a systematic review.

The present article reports the results of a systematic review on the potential benefits of the combined use of virtual reality (VR) and non-invasive brain stimulation (NIBS) as a novel approach for rehabilitation. VR and NIBS are two rehabilitation techniques that have been consistently explored by health professionals, and in recent years there is strong evidence of the therapeutic benefits of their combined use. In this work, we reviewed research articles that report the combined use of VR and two common NIBS techniques, namely transcranial direct current stimulation (tDCS) and transcranial magnetic stimulation (TMS). Relevant queries to six major bibliographic databases were performed to retrieve original research articles that reported the use of the combination VR-NIBS for rehabilitation applications. A total of 16 articles were identified and reviewed. The reviewed studies have significant differences in the goals, materials, methods, and outcomes. These differences are likely caused by the lack of guidelines and best practices on how to combine VR and NIBS techniques. Five therapeutic applications were identified: stroke, neuropathic pain, cerebral palsy, phobia and post-traumatic stress disorder, and multiple sclerosis rehabilitation. The majority of the reviewed studies reported positive effects of the use of VR-NIBS. However, further research is still needed to validate existing results on larger sample sizes and across different clinical conditions. For these reasons, in this review recommendations for future studies exploring the combined use of VR and NIBS are presented to facilitate the comparison among works.

Multimodal Human and Environmental Sensing for Longitudinal Behavioral Studies in Naturalistic Settings: Framework for Sensor Selection, Deployment, and Management.

BACKGROUND: Recent advances in mobile technologies for sensing human biosignals are empowering researchers to collect real-world data outside of the laboratory, in natural settings where participants can perform their daily activities with minimal disruption. These new sensing opportunities usher a host of challenges and constraints for both researchers and participants. OBJECTIVE: This viewpoint paper aims to provide a comprehensive guide to aid research teams in the selection and management of sensors before beginning and while conducting human behavior studies in the wild. The guide aims to help researchers achieve satisfactory participant compliance and minimize the number of unexpected procedural outcomes. METHODS: This paper presents a collection of challenges, consideration criteria, and potential solutions for enabling researchers to select and manage appropriate sensors for their research studies. It explains a general data collection framework suitable for use with modern consumer sensors, enabling researchers to address many of the described challenges. In addition, it provides a description of the criteria affecting sensor selection, management, and integration that researchers should consider before beginning human behavior studies involving sensors. On the basis of a survey conducted in mid-2018, this paper further illustrates an organized snapshot of consumer-grade human sensing technologies that can be used for human behavior research in natural settings. RESULTS: The research team applied the collection of methods and criteria to a case study aimed at predicting the well-being of nurses and other staff in a hospital. Average daily compliance for sensor usage measured by the presence of data exceeding half the total possible hours each day was about 65%, yielding over 355,000 hours of usable sensor data across 212 participants. A total of 6 notable unexpected events occurred during the data collection period, all of which had minimal impact on the research project. CONCLUSIONS: The satisfactory compliance rates and minimal impact of unexpected events during the case study suggest that the challenges, criteria, methods, and mitigation strategies presented as a guide for researchers are helpful for sensor selection and management in longitudinal human behavior studies in the wild.

Multi-Scale Heart Beat Entropy Measures for Mental Workload Assessment of Ambulant Users.

Mental workload assessment is crucial in many real life applications which require constant attention and where imbalance of mental workload resources may cause safety hazards. As such, mental workload and its relationship with heart rate variability (HRV) have been well studied in the literature. However, the majority of the developed models have assumed individuals are not ambulant, thus bypassing the issue of movement-related electrocardiography (ECG) artifacts and changing heart beat dynamics due to physical activity. In this work, multi-scale features for mental workload assessment of ambulatory users is explored. ECG data was sampled from users while they performed different types and levels of physical activity while performing the multi-attribute test battery (MATB-II) task at varying difficulty levels. Proposed features are shown to outperform benchmark ones and further exhibit complementarity when used in combination. Indeed, results show gains over the benchmark HRV measures of 24.41 % in accuracy and of 27.97 % in F1 score can be achieved even at high activity levels.

In-ear microphone speech quality enhancement via adaptive filtering and artificial bandwidth extension.

Bone and tissue conducted speech has been used in noisy environments to provide a relatively high signal-to-noise ratio signal. However, the limited bandwidth of bone and tissue conducted speech degrades the quality of the speech signal. Moreover in very noisy conditions, bandwidth extension of the bone and tissue conducted speech becomes problematic. In this paper, speech generated from bone and tissue conduction captured using an in-ear microphone is enhanced using adaptive filtering and a non-linear bandwidth extension method. Objective and subjective tests are used to evaluate the performance of the proposed techniques. Both evaluations show a statistically significant quality enhancement of the noisy in-ear microphone speech with ρ<0.0001 after denoising and ρ<0.01 after bandwidth extension.

Variations in voice level and fundamental frequency with changing background noise level and talker-to-listener distance while wearing hearing protectors: A pilot study.

OBJECTIVE: Speech production in noise with varying talker-to-listener distance has been well studied for the open ear condition. However, occluding the ear canal can affect the auditory feedback and cause deviations from the models presented for the open-ear condition. Communication is a main concern for people wearing hearing protection devices (HPD). Although practical, radio communication is cumbersome, as it does not distinguish designated receivers. A smarter radio communication protocol must be developed to alleviate this problem. Thus, it is necessary to model speech production in noise while wearing HPDs. Such a model opens the door to radio communication systems that distinguish receivers and offer more efficient communication between persons wearing HPDs. DESIGN: This paper presents the results of a pilot study aimed to investigate the effects of occluding the ear on changes in voice level and fundamental frequency in noise and with varying talker-to-listener distance. STUDY SAMPLE: Twelve participants with a mean age of 28 participated in this study. RESULTS: Compared to existing data, results show a trend similar to the open ear condition with the exception of the occluded quiet condition. CONCLUSIONS: This implies that a model can be developed to better understand speech production for the occluded ear.

Objective Quality and Intelligibility Prediction for Users of Assistive Listening Devices.

This article presents an overview of twelve existing objective speech quality and intelligibility prediction tools. Two classes of algorithms are presented, namely intrusive and non-intrusive, with the former requiring the use of a reference signal, while the latter does not. Investigated metrics include both those developed for normal hearing listeners, as well as those tailored particularly for hearing impaired (HI) listeners who are users of assistive listening devices (i.e., hearing aids, HAs, and cochlear implants, CIs). Representative examples of those optimized for HI listeners include the speech-to-reverberation modulation energy ratio, tailored to hearing aids (SRMR-HA) and to cochlear implants (SRMR-CI); the modulation spectrum area (ModA); the hearing aid speech quality (HASQI) and perception indices (HASPI); and the PErception MOdel - hearing impairment quality (PEMO-Q-HI). The objective metrics are tested on three subjectively-rated speech datasets covering reverberation-alone, noise-alone, and reverberation-plus-noise degradation conditions, as well as degradations resultant from nonlinear frequency compression and different speech enhancement strategies. The advantages and limitations of each measure are highlighted and recommendations are given for suggested uses of the different tools under specific environmental and processing conditions.

Approaches to understanding the impact of technologies for aging in place: a mini-review.

BACKGROUND: There are many approaches to evaluating aging-in-place technologies. While there are standard measures for outcomes such as health and caregiver burden, which lend themselves to statistical analysis, researchers have a harder time identifying why a particular information and communication technology (ICT) intervention worked (or not). OBJECTIVE: The purpose of this paper is to review a variety of methods that can help answer these deeper questions of when people will utilize an ICT for aging in place, how they use it, and most importantly why. This review is sensitive to the special context of aging in place, which necessitates an evaluation that can explore the nuances of the experiences of older adults and their caregivers with the technology in order to fully understand the potential impact of ICTs to support aging in place. METHODS: The authors searched both health (PubMed) and technology (ACM Digital Library) venues, reviewing 115 relevant papers that had an emphasis on understanding the use of aging-in-place technologies. This mini-review highlights a number of popular methods used in both the health and technology fields, including qualitative methods (e.g. interviews, focus groups, contextual observations, diaries, and cultural probes) and quantitative methods (e.g. surveys, the experience sampling method, and technology logs). RESULTS: This review highlights that a single evaluation method often is not adequate for understanding why people adopt ICTs for aging in place. The review ends with two examples of multifaceted evaluations attempting to get at these deeper issues. CONCLUSION: There is no proscriptive formula for evaluating the intricate nuances of technology acceptance and use in the aging-in-place context. Researchers should carefully examine a wide range of evaluation techniques to select those that will provide the richest insights for their particular project.

A model that predicts the binaural advantage to speech intelligibility from the mixed target and interferer signals.

A model is presented that predicts the binaural advantage to speech intelligibility by analyzing the right and left recordings at the two ears containing mixed target and interferer signals. This auditory-inspired model implements an equalization-cancellation stage to predict the binaural unmasking (BU) component, in conjunction with a modulation-frequency estimation block to estimate the "better ear" effect (BE) component of the binaural advantage. The model's performance was compared to experimental data obtained under anechoic and reverberant conditions using a single speech-shaped noise interferer paradigm. The internal BU and BE components were compared to those of the speech intelligibility model recently proposed by Lavandier et al. [J. Acoust. Soc. Am. 131, 218-231 (2012)], which requires separate inputs for target and interferer. The data indicate that the proposed model provides comparably good predictions from a mixed-signals input under both anechoic and reverberant conditions.

WavRx: a Disease-Agnostic, Generalizable, and Privacy-Preserving Speech Health Diagnostic Model.

Speech is known to carry health-related attributes, which has emerged as a novel venue for remote and long-term health monitoring. However, existing models are usually tailored for a specific type of disease, and have been shown to lack generalizability across datasets. Furthermore, concerns have been raised recently towards the leakage of speaker identity from health embeddings. To mitigate these limitations, we propose WavRx, a speech health diagnostics model that captures the respiration and articulation related dynamics from a universal speech representation. Our in-domain and cross-domain experiments on six pathological speech datasets demonstrate WavRx as a new state-of-the-art health diagnostic model. Furthermore, we show that the amount of speaker identity entailed in the WavRx health embeddings is significantly reduced without extra guidance during training. An in-depth analysis of the model was performed, thus providing physiological interpretation of its improved generalizability and privacy-preserving ability.

MSPB: a longitudinal multi-sensor dataset with phenotypic trait measurements from honey bees.

We present a one-year-long multi-sensor dataset collected from honey bee colonies (Apis mellifera) with rich phenotypic measurements. Data were collected non-stop from April 2020 to April 2021 from 53 hives located at two apiaries in Québec, Canada. The sensor data included audio features, temperature, and relative humidity. The phenotypic measurements contained beehive population, number of brood cells (eggs, larva and pupa), Varroa destructor infestation levels, defensive and hygienic behaviors, honey yield, and winter mortality. Our study is amongst the first to combine a wide variety of phenotypic trait measurements annotated by apicultural science experts with multi-sensor data, which facilitate a broader scope of analysis. We first summarize the data collection procedure, sensor data pre-processing steps, and data composition. We then provide an overview of the phenotypic data distribution as well as a visualization of the sensor data patterns. Lastly, we showcase several hive monitoring applications based on sensor data analysis and machine learning, such as winter mortality prediction, hive population estimation, and the presence of an active and laying queen.

On a reference-free speech quality estimator for hearing aids.

A reference-free speech quality measure is proposed and assessed for hearing aid applications. The proposed speech quality metric is validated with subjective ratings obtained from hearing impaired listeners under a number of noisy and reverberant conditions. In addition, a comparison is drawn between the proposed measure and a state-of-the-art electroacoustic measure that relies on a clean reference signal. The results showed that the reference-free measure had a lower correlation with the subjective ratings of hearing aid speech quality in comparison to the correlations achieved by the measure utilizing a reference signal. Nevertheless, advantages of the reference-free approach are discussed.

Objective speech intelligibility measurement for cochlear implant users in complex listening environments.

Objective intelligibility measurement allows for reliable, low-cost, and repeatable assessment of innovative speech processing technologies, thus dispensing costly and time-consuming subjective tests. To date, existing objective measures have focused on normal hearing model, and limited use has been found for restorative hearing instruments such as cochlear implants (CIs). In this paper, we have evaluated the performance of five existing objective measures, as well as proposed two refinements to one particular measure to better emulate CI hearing, under complex listening conditions involving noise-only, reverberation-only, and noise-plus-reverberation. Performance is assessed against subjectively rated data. Experimental results show that the proposed CI-inspired objective measures outperformed all existing measures; gains by as much as 22% could be achieved in rank correlation.

Enhancing motor imagery detection efficacy using multisensory virtual reality priming.

Brain-computer interfaces (BCI) have been developed to allow users to communicate with the external world by translating brain activity into control signals. Motor imagery (MI) has been a popular paradigm in BCI control where the user imagines movements of e.g., their left and right limbs and classifiers are then trained to detect such intent directly from electroencephalography (EEG) signals. For some users, however, it is difficult to elicit patterns in the EEG signal that can be detected with existing features and classifiers. As such, new user control strategies and training paradigms have been highly sought-after to help improve motor imagery performance. Virtual reality (VR) has emerged as one potential tool where improvements in user engagement and level of immersion have shown to improve BCI accuracy. Motor priming in VR, in turn, has shown to further enhance BCI accuracy. In this pilot study, we take the first steps to explore if multisensory VR motor priming, where haptic and olfactory stimuli are present, can improve motor imagery detection efficacy in terms of both improved accuracy and faster detection. Experiments with 10 participants equipped with a biosensor-embedded VR headset, an off-the-shelf scent diffusion device, and a haptic glove with force feedback showed that significant improvements in motor imagery detection could be achieved. Increased activity in the six common spatial pattern filters used were also observed and peak accuracy could be achieved with analysis windows that were 2 s shorter. Combined, the results suggest that multisensory motor priming prior to motor imagery could improve detection efficacy.

Using CNN Saliency Maps and EEG Modulation Spectra for Improved and More Interpretable Machine Learning-Based Alzheimer's Disease Diagnosis.

Biomarkers based on resting-state electroencephalography (EEG) signals have emerged as a promising tool in the study of Alzheimer's disease (AD). Recently, a state-of-the-art biomarker was found based on visual inspection of power modulation spectrograms where three "patches" or regions from the modulation spectrogram were proposed and used for AD diagnostics. Here, we propose the use of deep neural networks, in particular convolutional neural networks (CNNs) combined with saliency maps, trained on power modulation spectrogram inputs to find optimal patches in a data-driven manner. Experiments are conducted on EEG data collected from fifty-four participants, including 20 healthy controls, 19 patients with mild AD, and 15 moderate-to-severe AD patients. Five classification tasks are explored, including the three-class problem, early-stage detection (control vs. mild-AD), and severity level detection (mild vs. moderate-to-severe). Experimental results show the proposed biomarkers outperform the state-of-the-art benchmark across all five tasks, as well as finding complementary modulation spectrogram regions not previously seen via visual inspection. Lastly, experiments are conducted on the proposed biomarkers to test their sensitivity to age, as this is a known confound in AD characterization. Across all five tasks, none of the proposed biomarkers showed a significant relationship with age, thus further highlighting their usefulness for automated AD diagnostics.

Quantifying time perception during virtual reality gameplay using a multimodal biosensor-instrumented headset: a feasibility study.

We have all experienced the sense of time slowing down when we are bored or speeding up when we are focused, engaged, or excited about a task. In virtual reality (VR), perception of time can be a key aspect related to flow, immersion, engagement, and ultimately, to overall quality of experience. While several studies have explored changes in time perception using questionnaires, limited studies have attempted to characterize them objectively. In this paper, we propose the use of a multimodal biosensor-embedded VR headset capable of measuring electroencephalography (EEG), electrooculography (EOG), electrocardiography (ECG), and head movement data while the user is immersed in a virtual environment. Eight gamers were recruited to play a commercial action game comprised of puzzle-solving tasks and first-person shooting and combat. After gameplay, ratings were given across multiple dimensions, including (1) the perception of time flowing differently than usual and (2) the gamers losing sense of time. Several features were extracted from the biosignals, ranked based on a two-step feature selection procedure, and then mapped to a predicted time perception rating using a Gaussian process regressor. Top features were found to come from the four signal modalities and the two regressors, one for each time perception scale, were shown to achieve results significantly better than chance. An in-depth analysis of the top features is presented with the hope that the insights can be used to inform the design of more engaging and immersive VR experiences.

Neuromorphic Computing via Fission-based Broadband Frequency Generation.

The performance limitations of traditional computer architectures have led to the rise of brain-inspired hardware, with optical solutions gaining popularity due to the energy efficiency, high speed, and scalability of linear operations. However, the use of optics to emulate the synaptic activity of neurons has remained a challenge since the integration of nonlinear nodes is power-hungry and, thus, hard to scale. Neuromorphic wave computing offers a new paradigm for energy-efficient information processing, building upon transient and passively nonlinear interactions between optical modes in a waveguide. Here, an implementation of this concept is presented using broadband frequency conversion by coherent higher-order soliton fission in a single-mode fiber. It is shown that phase encoding on femtosecond pulses at the input, alongside frequency selection and weighting at the system output, makes transient spectro-temporal system states interpretable and allows for the energy-efficient emulation of various digital neural networks. The experiments in a compact, fully fiber-integrated setup substantiate an anticipated enhancement in computational performance with increasing system nonlinearity. The findings suggest that broadband frequency generation, accessible on-chip and in-fiber with off-the-shelf components, may challenge the traditional approach to node-based brain-inspired hardware design, ultimately leading to energy-efficient, scalable, and dependable computing with minimal optical hardware requirements.