Sensor Fusion for the Robust Detection of Facial Regions of Neonates Using Neural Networks.

The monitoring of vital signs and increasing patient comfort are cornerstones of modern neonatal intensive care. Commonly used monitoring methods are based on skin contact which can cause irritations and discomfort in preterm neonates. Therefore, non-contact approaches are the subject of current research aiming to resolve this dichotomy. Robust neonatal face detection is essential for the reliable detection of heart rate, respiratory rate and body temperature. While solutions for adult face detection are established, the unique neonatal proportions require a tailored approach. Additionally, sufficient open-source data of neonates on the NICU is lacking. We set out to train neural networks with the thermal-RGB-fusion data of neonates. We propose a novel indirect fusion approach including the sensor fusion of a thermal and RGB camera based on a 3D time-of-flight (ToF) camera. Unlike other approaches, this method is tailored for close distances encountered in neonatal incubators. Two neural networks were used with the fusion data and compared to RGB and thermal networks. For the class "head" we reached average precision values of 0.9958 (RetinaNet) and 0.9455 (YOLOv3) for the fusion data. Compared with the literature, similar precision was achieved, but we are the first to train a neural network with fusion data of neonates. The advantage of this approach is in calculating the detection area directly from the fusion image for the RGB and thermal modality. This increases data efficiency by 66%. Our results will facilitate the future development of non-contact monitoring to further improve the standard of care for preterm neonates.

Automated Non-Contact Respiratory Rate Monitoring of Neonates Based on Synchronous Evaluation of a 3D Time-of-Flight Camera and a Microwave Interferometric Radar Sensor.

This paper introduces an automatic non-contact monitoring method based on the synchronous evaluation of a 3D time-of-flight (ToF) camera and a microwave interferometric radar sensor for measuring the respiratory rate of neonates. The current monitoring on the Neonatal Intensive Care Unit (NICU) has several issues which can cause pressure marks, skin irritations and eczema. To minimize these risks, a non-contact system made up of a 3D time-of-flight camera and a microwave interferometric radar sensor is presented. The 3D time-of-flight camera delivers 3D point clouds which can be used to calculate the change in distance of the moving chest and from it the respiratory rate. The disadvantage of the ToF camera is that the heartbeat cannot be determined. The microwave interferometric radar sensor determines the change in displacement caused by the respiration and is even capable of measuring the small superimposed movements due to the heartbeat. The radar sensor is very sensitive towards movement artifacts due to, e.g., the baby moving its arms. To allow a robust vital parameter detection the data of both sensors was evaluated synchronously. In this publication, we focus on the first step: determining the respiratory rate. After all processing steps, the respiratory rate determined by the radar sensor was compared to the value received from the 3D time-of-flight camera. The method was validated against our gold standard: a self-developed neonatal simulation system which can simulate different breathing patterns. In this paper, we show that we are the first to determine the respiratory rate by evaluating the data of an interferometric microwave radar sensor and a ToF camera synchronously. Our system delivers very precise breaths per minute (BPM) values within the norm range of 20-60 BPM with a maximum difference of 3 BPM (for the ToF camera itself at 30 BPM in normal mode). Especially in lower respiratory rate regions, i.e., 5 and 10 BPM, the synchronous evaluation is required to compensate the drawbacks of the ToF camera. In the norm range, the ToF camera performs slightly better than the radar sensor.

A Time-of-Flight and Radar Dataset of a neonatal Thorax Simulator with synchronized Reference Sensor Signals for respiratory Rate Detection.

In this paper we present an open-source Time-of-Flight and radar dataset of a neonatal thorax simulator for the development of respiratory rate detection algorithms. As it is very difficult to gain recordings of (preterm) neonates and there is hardly any open-source data available, we built our own neonatal thorax simulator which simulates the movement of the thorax due to respiration. We recorded Time-of-Flight (ToF) and radar data at different respiratory rates in a range of 5 to 80 breaths per minute (BPM) and with varying upstroke heights. As gold standard a laser micrometer was used. The open-source data can be used to test new algorithms for non-contact respiratory rate detection.

Automatic non-contact monitoring of the respiratory rate of neonates using a structured light camera.

This paper introduces an automatic non-contact monitoring method for measuring the respiratory rate of neonates using a structured light camera. The current monitoring bears several issues causing pressure marks, skin irritations and eczema. A structured light camera provides distance data. Our non-contact approach detects the thorax area automatically using a plane segmentation and calculates the respiratory rate from the movement of the thorax. Our method was tested and validated using the baby simulator SimBaby by Laerdal. We used different breathing rates corresponding to preterm neonates, mature neonates and babies aged up to nine months as well as two different breathing modes with differing breathing strokes. Furthermore, measurements were taken of two positions: the baby lying on its back and on its stomach.