Automatic Separation of Respiratory Flow from Motion in Thermal Videos for Infant Apnea Detection.

Both Respiratory Flow (RF) and Respiratory Motion (RM) are visible in thermal recordings of infants. Monitoring these two signals usually requires landmark detection for the selection of a region of interest. Other approaches combine respiratory signals coming from both RF and RM, obtaining a Mixed Respiratory (MR) signal. The detection and classification of apneas, particularly common in preterm infants with low birth weight, would benefit from monitoring both RF and RM, or MR, signals. Therefore, we propose in this work an automatic RF pixel detector not based on facial/body landmarks. The method is based on the property of RF pixels in thermal videos, which are in areas with a smooth circular gradient. We defined 5 features combined with the use of a bank of Gabor filters that together allow selection of the RF pixels. The algorithm was tested on thermal recordings of 9 infants amounting to a total of 132 min acquired in a neonatal ward. On average the percentage of correctly identified RF pixels was 84%. Obstructive Apneas (OAs) were simulated as a proof of concept to prove the advantage in monitoring the RF signal compared to the MR signal. The sensitivity in the simulated OA detection improved for the RF signal reaching 73% against the 23% of the MR signal. Overall, the method yielded promising results, although the positioning and number of cameras used could be further optimized for optimal RF visibility.

Towards Continuous Camera-Based Respiration Monitoring in Infants.

Aiming at continuous unobtrusive respiration monitoring, motion robustness is paramount. However, some types of motion can completely hide the respiration information and the detection of these events is required to avoid incorrect rate estimations. Therefore, this work proposes a motion detector optimized to specifically detect severe motion of infants combined with a respiration rate detection strategy based on automatic pixels selection, which proved to be robust to motion of the infants involving head and limbs. A dataset including both thermal and RGB (Red Green Blue) videos was used amounting to a total of 43 h acquired on 17 infants. The method was successfully applied to both RGB and thermal videos and compared to the chest impedance signal. The Mean Absolute Error (MAE) in segments where some motion is present was 1.16 and 1.97 breaths/min higher than the MAE in the ideal moments where the infants were still for testing and validation set, respectively. Overall, the average MAE on the testing and validation set are 3.31 breaths/min and 5.36 breaths/min, using 64.00% and 69.65% of the included video segments (segments containing events such as interventions were excluded based on a manual annotation), respectively. Moreover, we highlight challenges that need to be overcome for continuous camera-based respiration monitoring. The method can be applied to different camera modalities, does not require skin visibility, and is robust to some motion of the infants.

Camera-Based Vital Signs Monitoring During Sleep - A Proof of Concept Study.

Polysomnography (PSG) is the current gold standard for the diagnosis of sleep disorders. However, this multi-parametric sleep monitoring tool also has some drawbacks, e.g. it limits the patient's mobility during the night and it requires the patient to come to a specialized sleep clinic or hospital to attach the sensors. Unobtrusive techniques for the detection of sleep disorders such as sleep apnea are therefore gaining increasing interest. Remote photoplethysmography using video is a technique which enables contactless detection of hemodynamic information. Promising results in near-infrared have been reported for the monitoring of sleep-relevant physiological parameters pulse rate, respiration and blood oxygen saturation. In this study we validate a contactless monitoring system on eight patients with a high likelihood of relevant obstructive sleep apnea, which are enrolled for a sleep study at a specialized sleep center. The dataset includes 46.5 hours of video recordings, full polysomnography and metadata. The camera can detect pulse and respiratory rate within 2 beats/breaths per minute accuracy 92% and 91% of the time, respectively. Estimated blood oxygen values are within 4 percentage points of the finger-oximeter 89% of the time. These results demonstrate the potential of a camera as a convenient diagnostic tool for sleep apnea, and sleep disorders in general.

Multi-camera infrared thermography for infant respiration monitoring.

Respiration is monitored in neonatal wards using chest impedance (CI), which is obtrusive and can cause skin damage to the infants. Therefore, unobtrusive solutions based on infrared thermography are being investigated. This work proposes an algorithm to merge multiple thermal camera views and automatically detect the pixels containing respiration motion or flow using three features. The method was tested on 152 minutes of recordings acquired on seven infants. We performed a comparison with the CI respiration rate yielding a mean absolute error equal to 2.07 breaths/min. Merging the three features resulted in reducing the dependency on the window size typical of spectrum-based features.

Analysis of CNN-based remote-PPG to understand limitations and sensitivities.

Deep learning based on convolutional neural network (CNN) has shown promising results in various vision-based applications, recently also in camera-based vital signs monitoring. The CNN-based photoplethysmography (PPG) extraction has, so far, been focused on performance rather than understanding. In this paper, we try to answer four questions with experiments aiming at improving our understanding of this methodology as it gains popularity. We conclude that the network exploits the blood absorption variation to extract the physiological signals, and that the choice and parameters (phase, spectral content, etc.) of the reference-signal may be more critical than anticipated. The availability of multiple convolutional kernels is necessary for CNN to arrive at a flexible channel combination through the spatial operation, but may not provide the same motion-robustness as a multi-site measurement using knowledge-based PPG extraction. We also find that the PPG-related prior knowledge may still be helpful for the CNN-based PPG extraction, and recommend further investigation of hybrid CNN-based methods that include prior knowledge in their design.

Discriminative Signatures for Remote-PPG.

Near-infrared (NIR) remote photoplethysmography (PPG) promises attractive applications in darkness, as it involves unobtrusive, invisible light. However, since the PPG strength (AC/DC) is much lower in the NIR spectrum than in the RGB spectrum, robust vital signs monitoring is more challenging. In this paper, we propose a new PPG-extraction method, DIScriminative signature based extraction (DIS), to significantly improve the pulse-rate measurement in NIR. Our core idea is to use both the color signals containing blood absorption variations and additional disturbance signals as input for PPG extraction. By defining a discriminative signature, we use one-step least-squares regression (joint optimization) to retrieve the pulsatile component from color signals and suppress disturbance signals simultaneously. A large-scale lab experiment, recorded in NIR with heavy body motions, shows the significant improvement of DIS over the state-of-the-art method, whereas its principle is simple and generally applicable.

Single Element Remote-PPG.

Camera-based remote photoplethysmography technology (remote-PPG) has shown great potential for contactless pulse-rate monitoring. However, remote-PPG systems typically analyze face images, which may restrict applications in view of privacy-preserving regulations such as the recently announced General Data Protection Regulation in the European Union. In this paper, we investigate the case of using single-element sensing as an input for remote-PPG extraction, which prohibits facial analysis and thus evades privacy issues. It also improves the efficiency of data storage and transmission. In contrast to known remote-PPG solutions using skin-selection techniques, the input signals in a single-element setup will contain a non-negligible degree of signal components associated with non-skin areas. Current remote-PPG extraction methods based on physiological and optical properties of skin reflections are therefore no longer valid. A new remote-PPG method, named Soft Signature based extraction (SoftSig), is proposed to deal with this situation by softening the dependence of pulse extraction on prior knowledge. A large scale experiment validates the concept of single-element remote-PPG monitoring and shows the improvement of SoftSig over general purpose solutions.

A Model for Waveform Dissimilarities in Dual-Depth Reflectance-PPG.

The pressure wave is attenuated as it travels through the vascular bed of tissue. Consequently, reflectance photoplethysmography (PPG) waveforms probed using dual-penetrating wavelengths, such as green (G) and red R; the deepest) are dissimilar. To unravel the dual-depth aspect of PPG, we modeled the wavelength-dependency of the shape of reflection-PPG signals in G (520-580 nm) and R (625-720nm). Skin compression perturbs the relative contributions of the dermal and subdermal blood volume variations sources (BVVs) to PPG and was used to verify our model. We acquired reflectance-PPG in G and R on the finger of nine subjects (ages, 26-32 yrs). Two parameters were used for describing dual-depth dissimilarities: the phase shift, $\phi $, between the first harmonics of the subdermal and dermal BVVs, and the observed phase shift (PS) between PPG signals in G and R. The average $\phi $ was 37.6, CI 95% [22.0, 53.2] degrees. At uncompressed skin, this corresponds to an average PS of 12.5, [7.8, 17.2] degrees. Our results suggest that phase parameters may enable microvascular characterization and diagnosis.

The Importance of Posture and Skin-Site Selection on Remote Measurements of Neck Pulsations: An Ultrasonographic Study.

Laser Doppler vibrometry (LDV) and camerabased vibrocardiography imaging (cVCGI) systems can sense cardiac-related displacements of the skin. This allows that carotid artery (CA) or jugular vein (JV) wall movements are acquired, non-obtrusively, at the neck and used for assessing cardiovascular health. However, skin-neck measurements are invalid if the CA and JV pulsations overlap. The concern is plausible since these vessels are anatomically close to one another until the carotid sinus. In this paper, we build on ultrasonographic (US) insights to verify whether trunk posture and skin-site variability within the neck influence cVCGI outcomes. Using ultrasound (US), we recorded the wall movements of the CA and JV in 4 subjects (ages, 28-41 yrs) in the supine, recumbent and seated positions at sites in the vicinity of the common CA. Skin-displacement waveforms were subsequently recorded by cVCGI and compared with US recordings. Our results show that CA displacements are dominant at the upper neck in the seated-to-recumbent positions whereas JV pulsations are best probed in recumbent-to-supine positions at the lower neck. These insights help to recognize the possible value of cVCGI in early-stage diagnosis or ambulatory monitoring.

Full video pulse extraction.

This paper introduces a new method to automate heart-rate detection using remote photoplethysmography (rPPG). The method replaces the commonly used region of interest (RoI) detection and tracking, and does not require initialization. Instead, it combines a number of candidate pulse-signals computed in the parallel, each biased towards differently colored objects in the scene. The method is based on the observation that the temporally averaged colors of video objects (skin and background) are usually quite stable over time in typical application-driven scenarios, such as the monitoring of a subject sleeping in bed, or an infant in an incubator. The resulting system, called full video pulse extraction (FVP), allows the direct use of raw video streams for pulse extraction. Our benchmark set of diverse videos shows that FVP enables long-term sleep monitoring in visible light and in infrared, and works for adults and neonates. Although we only demonstrate the concept for heart-rate monitoring, we foresee the adaptation to a range of vital signs, thus benefiting the larger video health monitoring field.

New insights into the origin of remote PPG signals in visible light and infrared.

Remote photoplethysmography (PPG) is an optical measurement technique with established applications in vital signs monitoring. Recently, the consensual understanding of blood volume variations (BVVs) as the origin of PPG signals was challenged, raising validity concerns about the remote SpO2 methodology. Recognizing the imperative for new opto-physiological evidence, this investigation supports the volumetric hypothesis with living skin experiments and Monte Carlo simulations of remote PPG-amplitude in visible light (VIS) and infrared (IR). Multilayered models of the skin were developed to simulate the separate contributions from skin layers containing pulsatile arterioles to the PPG signal in the 450-1000 nm range. The simulated spectra were qualitatively compared with observations of the resting and compressed finger pad, and complemented with videocapillaroscopy. Our results indicate that remote PPG systems indeed probe arterial blood. Green wavelengths probe dermal arterioles while red-IR wavelengths also reach subcutaneous BVVs. Owing to stable penetration depths, the red-IR diagnostic window promotes the invariance of SpO2 measurements to skin non-homogeneities.

Camera-based pulse-oximetry - validated risks and opportunities from theoretical analysis.

Camera-based pulse-oximetry has recently shown to be feasible, even when the signal is corrupted by noise and motion artifacts. Earlier work showed that using three instead of the common two wavelengths improves robustness of the measurement, however without a thorough investigation on the optimal wavelength selection. We therefore performed a search to identify these wavelengths to further improve the robustness of the measurement. Besides motion, it is empirically known that there are several other factors that influence the measurement leading to falsely-low or falsely-high SpO2 readings. These factors include the presence of dyshemoglobins or other species. In this paper, we use a theoretical skin-model to study how these factors influence the measurement, and how a proper wavelength selection can reduce the impact on the measurement. Additionally, we show that adding a third wavelength does not only improve robustness, but can also be exploited to create a reliability index for the measurement. Finally, we show that the presence of dyshemoglobins in arterial blood can not only be detected but also quantified. We illustrate this by comparing the estimated COHb levels of a small group of smokers and non-smokers, which typically have different CO-levels.

Attacks on Heartbeat-Based Security Using Remote Photoplethysmography.

The time interval between consecutive heartbeats (interpulse interval, IPI) has previously been suggested for securing mobile-health solutions. This time interval is known to contain a degree of randomness, permitting the generation of a time- and person-specific identifier. It is commonly assumed that only devices trusted by a person can make physical contact with him/her, and that this physical contact allows each device to generate a similar identifier based on its own cardiac recordings. Under these conditions, the identifiers generated by different trusted devices can facilitate secure authentication. Recently, a wide range of techniques have been proposed for measuring heartbeats remotely, a prominent example of which is remote photoplethysmography (rPPG). These techniques may pose a significant threat to heartbeat-based security, as an adversary may pretend to be a trusted device by generating a similar identifier without physical contact, thus bypassing one of the core security conditions. In this paper, we assess the feasibility of such remote attacks using state-of-the-art rPPG methods. Our evaluation shows that rPPG has similar accuracy as contact PPG and, thus, forms a substantial threat to heartbeat-based-security systems that permit trusted devices to obtain their identifiers from contact PPG recordings. Conversely, rPPG cannot obtain an accurate representation of an identifier generated from electrical cardiac signals, making the latter invulnerable to state-of-the-art remote attacks.

Amplitude-selective filtering for remote-PPG.

Biometric signatures of remote photoplethysmography (rPPG), including the pulse-induced characteristic color absorptions and pulse frequency range, have been used to design robust algorithms for extracting the pulse-signal from a video. In this paper, we look into a new biometric signature, i.e., the relative pulsatile amplitude, and use it to design a very effective yet computationally low-cost filtering method for rPPG, namely "amplitude-selective filtering" (ASF). Based on the observation that the human relative pulsatile amplitude varies in a specific lower range as a function of RGB channels, our basic idea is using the spectral amplitude of, e.g., the R-channel, to select the RGB frequency components inside the assumed pulsatile amplitude-range for pulse extraction. Similar to band-pass filtering (BPF), the proposed ASF can be applied to a broad range of rPPG algorithms to pre-process the RGB-signals before extracting the pulse. The benchmark in challenging fitness use-cases shows that applying ASF (ASF+BPF) as a pre-processing step brings significant and consistent improvements to all multi-channel pulse extraction methods. It improves different (multi-wavelength) rPPG algorithms to the extent where quality differences between the individual approaches almost disappear. The novelty of the proposed method is its simplicity and effectiveness in providing a solution for the extremely challenging application of rPPG to a fitness setting. The proposed method is easy to understand, simple to implement, and low-cost in running. It is the first time that the physiological property of pulsatile amplitude is used as a biometric signature for generic signal filtering.

Robust heart rate from fitness videos.

Remote photoplethysmography (rPPG) enables contactless heart-rate monitoring using a regular video camera. OBJECTIVE: This paper aims to improve the rPPG technology targeting continuous heart-rate measurement during fitness exercises. The fundamental limitation of the existing (multi-wavelength) rPPG methods is that they can suppress at most n - 1 independent distortions by linearly combining n wavelength color channels. Their performance are highly restricted when more than n - 1 independent distortions appear in a measurement, as typically occurs in fitness applications with vigorous body motions. APPROACH: To mitigate this limitation, we propose an effective yet very simple method that algorithmically extends the number of possibly suppressed distortions without using more wavelengths. Our core idea is to increase the degrees-of-freedom of noise reduction by decomposing the n wavelength camera-signals into multiple orthogonal frequency bands and extracting the pulse-signal per band-basis. This processing, namely Sub-band rPPG (SB), can suppress different distortion-frequencies using independent combinations of color channels. MAIN RESULTS: A challenging fitness benchmark dataset is created, including 25 videos recorded from 7 healthy adult subjects (ages from 25 to 40 yrs; six male and one female) running on a treadmill in an indoor environment. Various practical challenges are simulated in the recordings, such as different skin-tones, light sources, illumination intensities, and exercising modes. The basic form of SB is benchmarked against a state-of-the-art method (POS) on the fitness dataset. Using non-biased parameter settings, the average signal-to-noise-ratio (SNR) for POS varies in [-4.18, -2.07] dB, for SB varies in [-1.08, 4.77] dB. The ANOVA test shows that the improvement of SB over POS is statistically significant for almost all settings (p-value <0.05). SIGNIFICANCE: The results suggest that the proposed SB method considerably increases the robustness of heart-rate measurement in challenging fitness applications, and outperforms the state-of-the-art method.

Living-Skin Classification via Remote-PPG.

Detecting living-skin tissue in a video on the basis of induced color changes due to blood pulsation is emerging for automatic region of interest localization in remote photoplethysmography (rPPG). However, the state-of-the-art method performing unsupervised living-skin detection in a video is rather time consuming, which is mainly due to the high complexity of its unsupervised online learning for pulse/noise separation. In this paper, we address this issue by proposing a fast living-skin classification method. Our basic idea is to transform the time-variant rPPG-signals into signal shape descriptors called "multiresolution iterative spectrum," where pulse and noise have different patterns enabling accurate binary classification. The proposed technique is a proof-of-concept that has only been validated in lab conditions but not in real clinical conditions. The benchmark, including synthetic and realistic (nonclinical) experiments, shows that it achieves a high detection accuracy better than the state-of-the-art method, and a high detection speed at hundreds of frames per second in MATLAB, enabling real-time living-skin detection.

Algorithmic Principles of Remote PPG.

This paper introduces a mathematical model that incorporates the pertinent optical and physiological properties of skin reflections with the objective to increase our understanding of the algorithmic principles behind remote photoplethysmography (rPPG). The model is used to explain the different choices that were made in existing rPPG methods for pulse extraction. The understanding that comes from the model can be used to design robust or application-specific rPPG solutions. We illustrate this by designing an alternative rPPG method, where a projection plane orthogonal to the skin tone is used for pulse extraction. A large benchmark on the various discussed rPPG methods shows that their relative merits can indeed be understood from the proposed model.

Robust respiration detection from remote photoplethysmography.

Continuous monitoring of respiration is essential for early detection of critical illness. Current methods require sensors attached to the body and/or are not robust to subject motion. Alternative camera-based solutions have been presented using motion vectors and remote photoplethysmography. In this work, we present a non-contact camera-based method to detect respiration, which can operate in both visible and dark lighting conditions by detecting the respiratory-induced colour differences of the skin. We make use of the close similarity between skin colour variations caused by the beating of the heart and those caused by respiration, leading to a much improved signal quality compared to single-channel approaches. Essentially, we propose to find the linear combination of colour channels which suppresses the distortions best in a frequency band including pulse rate, and subsequently we use this same linear combination to extract the respiratory signal in a lower frequency band. Evaluation results obtained from recordings on healthy subjects which perform challenging scenarios, including motion, show that respiration can be accurately detected over the entire range of respiratory frequencies, with a correlation coefficient of 0.96 in visible light and 0.98 in infrared, compared to 0.86 with the best-performing non-contact benchmark algorithm. Furthermore, evaluation on a set of videos recorded in a Neonatal Intensive Care Unit (NICU) shows that this technique looks promising as a future alternative to current contact-sensors showing a correlation coefficient of 0.87.

New principle for measuring arterial blood oxygenation, enabling motion-robust remote monitoring.

Finger-oximeters are ubiquitously used for patient monitoring in hospitals worldwide. Recently, remote measurement of arterial blood oxygenation (SpO2) with a camera has been demonstrated. Both contact and remote measurements, however, require the subject to remain static for accurate SpO2 values. This is due to the use of the common ratio-of-ratios measurement principle that measures the relative pulsatility at different wavelengths. Since the amplitudes are small, they are easily corrupted by motion-induced variations. We introduce a new principle that allows accurate remote measurements even during significant subject motion. We demonstrate the main advantage of the principle, i.e. that the optimal signature remains the same even when the SNR of the PPG signal drops significantly due to motion or limited measurement area. The evaluation uses recordings with breath-holding events, which induce hypoxemia in healthy moving subjects. The events lead to clinically relevant SpO2 levels in the range 80-100%. The new principle is shown to greatly outperform current remote ratio-of-ratios based methods. The mean-absolute SpO2-error (MAE) is about 2 percentage-points during head movements, where the benchmark method shows a MAE of 24 percentage-points. Consequently, we claim ours to be the first method to reliably measure SpO2 remotely during significant subject motion.

Skin inhomogeneity as a source of error in remote PPG-imaging.

Remote photoplethysmography (rPPG) imaging is an optical technique to remotely assess the local cutaneous microcirculation. Despite its potential for enabling health-related applications, the current understanding of the outcome images/maps remains incomplete. In this paper, we present a model and supporting experiments confirming the contribution of skin inhomogeneity to the morphology of PPG waveforms. Since rPPG imagers rely on the complex inner-product operator and may combine multiple wavelengths, the derived phase measurements reflect morphological heterogeneity of PPG signals to a larger extent than propagation-related phase differences. The influence of light penetration depth on PPG was observed and modeled on the green and red wavelengths at the hand region. We further show how our work contributes to understanding reproducibility issues in recent papers on pulse wave velocity (PWV) estimation.