Contactless Blood Oxygen Saturation Estimation from Facial Videos Using Deep Learning.

Blood oxygen saturation (SpO2) is an essential physiological parameter for evaluating a person's health. While conventional SpO2 measurement devices like pulse oximeters require skin contact, advanced computer vision technology can enable remote SpO2 monitoring through a regular camera without skin contact. In this paper, we propose novel deep learning models to measure SpO2 remotely from facial videos and evaluate them using a public benchmark database, VIPL-HR. We utilize a spatial-temporal representation to encode SpO2 information recorded by conventional RGB cameras and directly pass it into selected convolutional neural networks to predict SpO2. The best deep learning model achieves 1.274% in mean absolute error and 1.71% in root mean squared error, which exceed the international standard of 4% for an approved pulse oximeter. Our results significantly outperform the conventional analytical Ratio of Ratios model for contactless SpO2 measurement. Results of sensitivity analyses of the influence of spatial-temporal representation color spaces, subject scenarios, acquisition devices, and SpO2 ranges on the model performance are reported with explainability analyses to provide more insights for this emerging research field.

Identification of the human cerebral cortical hemodynamic response to passive whole-body movements using near-infrared spectroscopy.

The human vestibular system is crucial for motion perception, balance control, and various higher cognitive functions. Exploring how the cerebral cortex responds to vestibular signals is not only valuable for a better understanding of how the vestibular system participates in cognitive and motor functions but also clinically significant in diagnosing central vestibular disorders. Near-infrared spectroscopy (NIRS) provides a portable and non-invasive brain imaging technology to monitor cortical hemodynamics under physical motion. Objective: This study aimed to investigate the cerebral cortical response to naturalistic vestibular stimulation induced by real physical motion and to validate the vestibular cerebral cortex previously identified using alternative vestibular stimulation. Approach: Functional NIRS data were collected from 12 right-handed subjects when they were sitting in a motion platform that generated three types of whole-body passive translational motion (circular, lateral, and fore-and-aft). Main results: The study found that different cortical regions were activated by the three types of motion. The cortical response was more widespread under circular motion in two dimensions compared to lateral and fore-and-aft motions in one dimensions. Overall, the identified regions were consistent with the cortical areas found to be activated in previous brain imaging studies. Significance: The results provide new evidence of brain selectivity to different types of motion and validate previous findings on the vestibular cerebral cortex.

Vection induced by a pair of patches of synchronized visual motion stimuli covering total field of views as small as 10 square-degrees.

Vection (illusion of self-motion) is known to be induced by watching large field-of-view (FOV) moving scenes. In our study, we investigated vection induced by small FOV stimuli. Three experiments were conducted in 45 sessions to analyze vection provoked by moving scenes covering total FOVs as small as 10 square-degrees. Results indicated that 88% of the participants reported vection while watching two small patches of moving dots (1° horizontal by 5° vertical, each) placed on the left and right sides of the observers. This is less than a quarter of the total visual area of two Apple Watches viewed at a distance of 40 cm. Occlusion of the visual field between the two display patches significantly increased the levels of rated vection. Similarly, increasing the speed of the moving dots of the two display patches from about 5 to 25 °/sec increased the levels of rated vection significantly. The location of the two patches in the horizontal visual field did not affect the vection perception significantly. When the two straight stripes of dots were moving in opposite directions, participants perceived circular vection. The observers connected the two stimuli in their minds and perceived them as parts of a single occluded background. The findings of this study are relevant to the design of mobile devices (e.g., smartphones) and wearable technology (e.g., smart watches) with small display areas.

Robust Heart Rate Variability Measurement from Facial Videos.

Remote Photoplethysmography (rPPG) is a contactless method that enables the detection of various physiological signals from facial videos. rPPG utilizes a digital camera to detect subtle changes in skin color to measure vital signs such as heart rate variability (HRV), an important biomarker related to the autonomous nervous system. This paper presents a novel contactless HRV extraction algorithm, WaveHRV, based on the Wavelet Scattering Transform technique, followed by adaptive bandpass filtering and inter-beat-interval (IBI) analysis. Furthermore, a novel method is introduced to preprocess noisy contact-based PPG signals. WaveHRV is bench-marked against existing algorithms and public datasets. Our results show that WaveHRV is promising and achieves the lowest mean absolute error (MAE) of 10.5 ms and 6.15 ms for RMSSD and SDNN on the UBFCrPPG dataset.

Deep Learning Methods for Remote Heart Rate Measurement: A Review and Future Research Agenda.

Heart rate (HR) is one of the essential vital signs used to indicate the physiological health of the human body. While traditional HR monitors usually require contact with skin, remote photoplethysmography (rPPG) enables contactless HR monitoring by capturing subtle light changes of skin through a video camera. Given the vast potential of this technology in the future of digital healthcare, remote monitoring of physiological signals has gained significant traction in the research community. In recent years, the success of deep learning (DL) methods for image and video analysis has inspired researchers to apply such techniques to various parts of the remote physiological signal extraction pipeline. In this paper, we discuss several recent advances of DL-based methods specifically for remote HR measurement, categorizing them based on model architecture and application. We further detail relevant real-world applications of remote physiological monitoring and summarize various common resources used to accelerate related research progress. Lastly, we analyze the implications of research findings and discuss research gaps to guide future explorations.

Active head rolls enhance sonar-based auditory localization performance.

Animals utilize a variety of active sensing mechanisms to perceive the world around them. Echolocating bats are an excellent model for the study of active auditory localization. The big brown bat (Eptesicus fuscus), for instance, employs active head roll movements during sonar prey tracking. The function of head rolls in sound source localization is not well understood. Here, we propose an echolocation model with multi-axis head rotation to investigate the effect of active head roll movements on sound localization performance. The model autonomously learns to align the bat's head direction towards the target. We show that a model with active head roll movements better localizes targets than a model without head rolls. Furthermore, we demonstrate that active head rolls also reduce the time required for localization in elevation. Finally, our model offers key insights to sound localization cues used by echolocating bats employing active head movements during echolocation.

Motion sickness-susceptible participants exposed to coherent rotating dot patterns show excessive N2 amplitudes and impaired theta-band phase synchronization.

Visually induced motion sickness (VIMS) can occur via prolonged exposure to visual stimulation that generates the illusion of self-motion (vection). Not everyone is susceptible to VIMS and the neural mechanism underlying susceptibility is unclear. This study explored the differences of electroencephalographic (EEG) signatures between VIMS-susceptible and VIMS-resistant groups. Thirty-two-channel EEG data were recorded from 12 VIMS-susceptible and 15 VIMS-resistant university students while they were watching two patterns of moving dots: (1) a coherent rotation pattern (vection-inducing and potentially VIMS-provoking pattern), and (2) a random movement pattern (non-VIMS-provoking control). The VIMS-susceptible group exhibited a significantly larger increase in the parietal N2 response when exposed to the coherent rotating pattern than when exposed to control patterns. In members of the VIMS-resistant group, before vection onset, global connectivity from all other EEG electrodes to the right-temporal-parietal and to the right-central areas increased, whereas after vection onset the global connectivity to the right-frontal area reduced. Such changes were not observed in the susceptible group. Further, the increases in N2 amplitude and the identified phase synchronization index were significantly correlated with individual motion sickness susceptibility. Results suggest that VIMS susceptibility is associated with systematic impairment of dynamic cortical coordination as captured by the phase synchronization of cortical activities. Analyses of dynamic EEG signatures could be a means to unlock the neural mechanism of VIMS.

Adversarial relationship between combined medial olivocochlear (MOC) and middle-ear-muscle (MEM) reflexes and alarm-in-noise detection thresholds under negative signal-to-noise ratios (SNRs).

The role of auditory efferent feedback from the medial olivocochlear system (MOCS) and the middle-ear-muscle (MEM) reflex in tonal detection tasks for humans in the presence of noise is not clearly understood. Past studies have yielded inconsistent results on the relationship between efferent feedback and tonal detection thresholds. This study attempts to address this inconsistency. Fifteen human subjects with normal hearing participated in an experiment where they were asked to identify an alarm signal in the presence of 80 dBA background (pink) noise. Masked detection thresholds were estimated using the method of two-interval forced choice (2IFC). Contralateral suppression of transient-evoked otoacoustic emissions (TEOAEs) was measured to estimate the strength of auditory efferent feedback. Subsequent correlation analysis revealed that the contralateral suppression of TEOAEs was significantly negatively correlated (r = -0.526, n = 15, p = 0.0438) with alarm-in-noise (AIN) detection thresholds under negative signal-to-noise conditions. The result implies that the stronger the auditory efferent feedback, the worse the detection thresholds and thus the poorer the tonal detection performance in the presence of loud noise.

Allocating less attention to central vision during vection is correlated with less motion sickness.

Visually induced motion sickness (VIMS) is a common discomfort response associated with vection-provoking stimuli. It has been suggested that susceptibility to VIMS depends on the ability to regulate visual performance during vection. To test this, 29 participants, with VIMS susceptibility assessed by Motion Sickness Susceptibility Questionnaire, were recruited to undergo three series of sustained attention to response tests (SARTs) while watching dot pattern stimuli known to provoke roll-vection. In general, SARTs performance was impaired in the central visual field (CVF), but improved in peripheral visual field (PVF), suggesting the reallocation of attention during vection. Moreover, VIMS susceptibility was negatively correlated with the effect sizes, suggesting that participants who were less susceptible to VIMS showed better performance in attention re-allocation. Finally, when trained to re-allocation attention from the CVF to the PVF, participants experienced more stable vection. Findings provide a better understanding of VIMS and shed light on possible preventive measures. Practitioner Summary: Allocating less visual attention to central visual field during visual motion stimulation is associated with stronger vection and higher resistance to motion sickness. Virtual reality application designers may utilise the location of visual tasks to strengthen and stabilise vection, while reducing the potential of visually induced motion sickness.

Prevalent osteoporotic vertebral fractures more likely involve the upper endplate than the lower endplate and even more so in males.

BACKGROUND: While the importance of identifying osteoporotic vertebral endplate fracture (EPF) is being recognized; the pathophysiological understanding of EPF till now remain insufficient. In this population-based cross-sectional radiograph study, we aim to investigate the anatomic location characteristics of osteoporotic EPF. METHODS: This study analyzed the anatomical location of osteoporotic EPFs in elderly Chinese population (age ≥65 years). The T4-L4 radiographs of 1,954 elderly Chinese men (mean: 72.3 years) and 1,953 elderly Chinese women (mean: 72.5 years) were evaluated to identify EPF, and vertebral bodies were graded according to Genant's vertebral deformity criteria. RESULTS: Of the 101,582 endplates analyzed, there were 505 EPFs (males: 27.7%; females: 72.3%). Excluding those with both upper endplate and lower endplate involvements, the ratio of upper EPF to lower EPF was 9.63 for males and 4.3 for females (P<0.05). Thoracolumbar junction, particularly L1 (26.4% for males and 24.1% for females) and followed by T12 (20.7% for males and 19.7% for females), had highest prevalence of EPF. With an endplate divided into 5 segments of equal length in the anteroposterior direction and grade 0.5 & 1, grade 2 vertebral deformities analyzed, fractures occurred mostly at the middle segment (70.1% for upper endplates in males and 78.6% for upper endplates in females), followed by second anterior segment (20.9% for upper endplates in males and 14.4% for upper endplates in females). The most anterior and most posterior segments were not primarily involved in EPF. CONCLUSIONS: Osteoporotic EPFs more likely involve the upper endplate rather than lower endplate, with a trend for this effect to be greater in men than in women. These characteristics may help radiographic differential diagnosis for osteoporotic EPF.

Correlations between individual susceptibility to visually induced motion sickness and decaying time constant of after-nystagmus.

This study examines the correlations between optokinetic after-nystagmus (OKAN) parameters and individual susceptibility to visually induced motion sickness (VIMS). Twenty-seven participants were exposed to vertical black-and-white stripes drifting along the yaw axis at 60° per second for 30 min to collect individual VIMS data (Phase 1). Two weeks after the exposure, OKANs were measured (Phase 2). 19 out of 27 participants (i.e., 70%) exhibited consistent OKAN patterns. Significant correlations between the time constants of OKAN and levels of VIMS experienced by the same viewers were found. Four months later, these 27 participants were invited back for a second OKAN measurement (Phase 3). Twenty-one participants came back. Their two OKAN measurements were significantly correlated (r = 0.69, p = 0.001). Rated levels of VIMS in phase 1 significantly correlated with the time constant of OKAN in both Phase 2 (r = 0.51, p = 0.044) and Phase 3 (r = 0.74, p = 0.006). The implications of the correlation results are discussed.

Visually induced motion sickness when viewing visual oscillations of different frequencies along the fore-and-aft axis: keeping velocity versus amplitude constant.

Exposure to visual oscillations (VOs) can lead to visually induced motion sickness (VIMS). The level of VIMS among viewers has been shown to vary when the frequency of the VOs is changed either by manipulating their amplitude or velocity. The present study investigates whether the level of VIMS would change if we keep the root mean square (rms) velocity or amplitude of VOs constant while manipulating the VO frequency. A total of 25 individuals were exposed to random-dot and checkerboard VOs along the fore-and-aft axis in two experiments. Changing the amplitude (or frequency) of VOs while keeping the rms velocity constant did not affect the level of VIMS; however, increasing the rms velocity (or frequency) of VOs while keeping the amplitude constant made VIMS significantly worse. Practitioner Summary: Exposure to VOs of the same frequency can cause different levels of nausea depending on the combination of oscillation amplitude and velocity. Results suggest an opportunity for game designers to reduce symptoms of game sickness by using the correct combinations of velocity and amplitude of the visual motions.

Effects of spectral manipulation on nonindividualized head-related transfer functions (HRTFs).

BACKGROUND: Directional sounds simulated using nonindividualized head-related transfer functions (HRTFs) often result in front-back confusion. OBJECTIVE: This study was designed to examine how manipulating these nonindividualized HRTF spectra can reduce front-back confusion in headphone-simulated directional sounds. METHOD: HRTFs of six ear-level directions were studied (angles of 0 degrees, 45 degrees, 135 degrees, 180 degrees, 225 degrees, and 315 degrees). The HRTF gains in each of six frequency bands (200 to 690 Hz, 690 to 2400 Hz, 2400 to 6500 Hz, 6500 to 10000 Hz, 10000 to 14000 Hz, and 14000 to 22000 Hz) were amplified or attenuated by 0, 12, or 18 dB. Each manipulated HRTF generated a directional sound stimulus. For this study, 32 participants were invited to localize the randomly ordered stimuli. RESULTS: The results indicate that a 12- or 18-dB manipulation of five of the six frequency bands produced significantly better directional accuracy, with significantly less front-back confusion. A reduction of up to 70% in localization error was obtained, along with 66% less front-back confusion. Significant interactions were found between the manipulation level and frequency. CONCLUSION: A 12-dB spectral manipulation of selected HRTF frequency bands produces better directional accuracy. APPLICATION: The results of this research could be applied to the development of tunable nonindividualized HRTFs for audio products.

Detection of time-varying harmonic amplitude alterations due to spectral interpolations between musical instrument tones.

Gradated spectral interpolations between musical instrument tone pairs were used to investigate discrimination as a function of time-averaged spectral difference. All possible nonidentical pairs taken from a collection of eight musical instrument sounds consisting of bassoon, clarinet, flute, horn, oboe, saxophone, trumpet, and violin were tested. For each pair, several tones were generated with different balances between the primary and secondary instruments, where the balance was fixed across the duration of each tone. Among primary instruments it was found that changes to horn and bassoon [corrected] were most easily discriminable, while changes to saxophone and trumpet timbres were least discriminable. Among secondary instruments, the clarinet had the strongest effect on discrimination, whereas the bassoon had the least effect. For primary instruments, strong negative correlations were found between discrimination and their spectral incoherences, suggesting that the presence of dynamic spectral variations tends to increase the difficulty of detecting time-varying alterations such as spectral interpolation.

Isolating the effects of vection and optokinetic nystagmus on optokinetic rotation-induced motion sickness.

OBJECTIVE: This study investigates isolated effects of vection and optokinetic nystagmus (OKN) on visually induced motion sickness (VIMS) provoked by rotating optokinetic drum patterns. BACKGROUND: VIMS was the subject of recent standardization activities, but the effects of OKN have not been studied in the absence ofvection. METHOD: Experiment 1 suppressed OKN by eye fixation and examined VIMS severity (both ordinal and ratio scale) and time spent in saturated vection at four pattern rotating velocities of 0, 2, 14, and 34 degrees per second (dps). Experiment 2 suppressed vection by adding a peripheral visual field rotating in the opposite direction to the rotating patterns. VIMS severity and OKN slow-phase velocity were studied at four rotating velocities of 0, 30, 60, and 90 dps. RESULTS: Results from Experiment 1 indicated that VIMS severity increased as the pattern velocity increased from 0 dps to 34 dps. Results from Experiment 2 indicated that as the velocity of the rotating pattern increased, the slow-phase velocity of OKN and the severity of VIMS increased and peaked in the 60-dps condition. In both experiments, ratio-scaled nausea data significantly correlated with ordinal-scaled nausea ratings. CONCLUSION: VIMS can still occur in the absence of either vection or OKN. Interestingly, the profile of the summed results of the two experiments matches nicely with the profile reported by Hu et al. in which neither OKN nor vection were controlled. APPLICATION: Potential applications include modeling and reduction of VIMS in computer gaming environments.

Sensation of presence and cybersickness in applications of virtual reality for advanced rehabilitation.

Around three years ago, in the special issue on augmented and virtual reality in rehabilitation, the topics of simulator sickness was briefly discussed in relation to vestibular rehabilitation. Simulator sickness with virtual reality applications have also been referred to as visually induced motion sickness or cybersickness. Recently, study on cybersickness has been reported in entertainment, training, game, and medical environment in several journals. Virtual stimuli can enlarge sensation of presence, but they sometimes also evoke unpleasant sensation. In order to safely apply augmented and virtual reality for long-term rehabilitation treatment, sensation of presence and cybersickness should be appropriately controlled. This issue presents the results of five studies conducted to evaluate visually-induced effects and speculate influences of virtual rehabilitation. In particular, the influence of visual and vestibular stimuli on cardiovascular responses are reported in terms of academic contribution.