Multi-modal body part segmentation of infants using deep learning.

BACKGROUND: Monitoring the body temperature of premature infants is vital, as it allows optimal temperature control and may provide early warning signs for severe diseases such as sepsis. Thermography may be a non-contact and wireless alternative to state-of-the-art, cable-based methods. For monitoring use in clinical practice, automatic segmentation of the different body regions is necessary due to the movement of the infant. METHODS: This work presents and evaluates algorithms for automatic segmentation of infant body parts using deep learning methods. Based on a U-Net architecture, three neural networks were developed and compared. While the first two only used one imaging modality (visible light or thermography), the third applied a feature fusion of both. For training and evaluation, a dataset containing 600 visible light and 600 thermography images from 20 recordings of infants was created and manually labeled. In addition, we used transfer learning on publicly available datasets of adults in combination with data augmentation to improve the segmentation results. RESULTS: Individual optimization of the three deep learning models revealed that transfer learning and data augmentation improved segmentation regardless of the imaging modality. The fusion model achieved the best results during the final evaluation with a mean Intersection-over-Union (mIoU) of 0.85, closely followed by the RGB model. Only the thermography model achieved a lower accuracy (mIoU of 0.75). The results of the individual classes showed that all body parts were well-segmented, only the accuracy on the torso is inferior since the models struggle when only small areas of the skin are visible. CONCLUSION: The presented multi-modal neural networks represent a new approach to the problem of infant body segmentation with limited available data. Robust results were obtained by applying feature fusion, cross-modality transfer learning and classical augmentation strategies.

Conditional Generative Adversarial Networks for Data Augmentation of a Neonatal Image Dataset.

In today's neonatal intensive care units, monitoring vital signs such as heart rate and respiration is fundamental for neonatal care. However, the attached sensors and electrodes restrict movement and can cause medical-adhesive-related skin injuries due to the immature skin of preterm infants, which may lead to serious complications. Thus, unobtrusive camera-based monitoring techniques in combination with image processing algorithms based on deep learning have the potential to allow cable-free vital signs measurements. Since the accuracy of deep-learning-based methods depends on the amount of training data, proper validation of the algorithms is difficult due to the limited image data of neonates. In order to enlarge such datasets, this study investigates the application of a conditional generative adversarial network for data augmentation by using edge detection frames from neonates to create RGB images. Different edge detection algorithms were used to validate the input images' effect on the adversarial network's generator. The state-of-the-art network architecture Pix2PixHD was adapted, and several hyperparameters were optimized. The quality of the generated RGB images was evaluated using a Mechanical Turk-like multistage survey conducted by 30 volunteers and the FID score. In a fake-only stage, 23% of the images were categorized as real. A direct comparison of generated and real (manually augmented) images revealed that 28% of the fake data were evaluated as more realistic. An FID score of 103.82 was achieved. Therefore, the conducted study shows promising results for the training and application of conditional generative adversarial networks to augment highly limited neonatal image datasets.

A Setup for Camera-Based Detection of Simulated Pathological States Using a Neonatal Phantom.

Premature infants are among the most vulnerable patients in a hospital. Due to numerous complications associated with immaturity, a continuous monitoring of vital signs with a high sensitivity and accuracy is required. Today, wired sensors are attached to the patient's skin. However, adhesive electrodes can be potentially harmful as they can damage the very thin immature skin. Although unobtrusive monitoring systems using cameras show the potential to replace cable-based techniques, advanced image processing algorithms are data-driven and, therefore, need much data to be trained. Due to the low availability of public neonatal image data, a patient phantom could help to implement algorithms for the robust extraction of vital signs from video recordings. In this work, a camera-based system is presented and validated using a neonatal phantom, which enabled a simulation of common neonatal pathologies such as hypo-/hyperthermia and brady-/tachycardia. The implemented algorithm was able to continuously measure and analyze the heart rate via photoplethysmography imaging with a mean absolute error of 0.91 bpm, as well as the distribution of a neonate's skin temperature with a mean absolute error of less than 0.55 °C. For accurate measurements, a temperature gain offset correction on the registered image from two infrared thermography cameras was performed. A deep learning-based keypoint detector was applied for temperature mapping and guidance for the feature extraction. The presented setup successfully detected several levels of hypo- and hyperthermia, an increased central-peripheral temperature difference, tachycardia and bradycardia.

A Deep Learning-Based Camera Approach for Vital Sign Monitoring Using Thermography Images for ICU Patients.

Infrared thermography for camera-based skin temperature measurement is increasingly used in medical practice, e.g., to detect fevers and infections, such as recently in the COVID-19 pandemic. This contactless method is a promising technology to continuously monitor the vital signs of patients in clinical environments. In this study, we investigated both skin temperature trend measurement and the extraction of respiration-related chest movements to determine the respiratory rate using low-cost hardware in combination with advanced algorithms. In addition, the frequency of medical examinations or visits to the patients was extracted. We implemented a deep learning-based algorithm for real-time vital sign extraction from thermography images. A clinical trial was conducted to record data from patients on an intensive care unit. The YOLOv4-Tiny object detector was applied to extract image regions containing vital signs (head and chest). The infrared frames were manually labeled for evaluation. Validation was performed on a hold-out test dataset of 6 patients and revealed good detector performance (0.75 intersection over union, 0.94 mean average precision). An optical flow algorithm was used to extract the respiratory rate from the chest region. The results show a mean absolute error of 2.69 bpm. We observed a computational performance of 47 fps on an NVIDIA Jetson Xavier NX module for YOLOv4-Tiny, which proves real-time capability on an embedded GPU system. In conclusion, the proposed method can perform real-time vital sign extraction on a low-cost system-on-module and may thus be a useful method for future contactless vital sign measurements.

Early Prediction of Neonatal Sepsis From Synthetic Clinical Data Using Machine Learning.

Neonatal sepsis is one of the most serious complications in neonatal intensive care units. Due to the often immature immune system, sepsis-related comorbidities are the major contributors to increased neonatal mortality. The rapid progression of the disease makes early treatment critical for patient survival. However, early diagnosis of sepsis remains difficult due to its non-specific symptoms. In recent years, Machine Learning-based techniques have been used in various medical applications to predict diseases using clinical data. In this work, we optimized and evaluated four prediction models with different architectural concepts. Two public datasets containing clinical data from adults and neonates were used for training. The adult data were collected to pre-train the models. Since neonatal data with sepsis diagnosis are very limited, we propose an augmentation method to generate synthetic clinical data. For the final evaluation, the real data of neonatal patients were defined as a test set. An AUROC of 0.91 and an AUPRC of 0.38 were obtained. These results are promising for early prediction of neonatal sepsis using artificial data for augmentation.Clinical relevance- This work demonstrates the potential of Machine Learning-based prediction models for the detection of sepsis to improve the early diagnosis of life-threatening conditions in neonatal intensive care units.

Shape analysis of training data for neural networks in Electrical Impedance Tomography.

Electrical Impedance Tomography (EIT) is a cost-effective and fast way to visualize dielectric properties of the human body, through the injection of alternating currents and measurement of the resulting potential on the bodies surface. However, this comes at the cost of low resolution as EIT is a non-linear ill-posed inverse problem. Recently Deep Learning methods have gained the interest in this field, as they provide a way to mimic non-linear functions. Most of the approaches focus on the structure of the Artificial Neural Networks (ANNs), while only glancing over the used training data. However, the structure of the training data is of great importance, as it needs to be simulated. In this work, we analyze the effect of basic shapes to be included as targets in the training data set. We compared inclusions of ellipsoids, cubes and octahedra as training data for ANNs in terms of reconstruction quality. For that, we used the well-established GREIT figures of merit on laboratory tank measurements. We found that ellipsoids resulted in the best reconstruction quality of EIT images. This shows that the choice of simulation data has an impact on the ANN reconstruction quality.Clinical relevance- This work helps to improve time independent EIT reconstruction, which in turn allows for extraction of time independent features of e.g., the lung.

A Physical Phantom for the Simulation of Neonatal Thermoregulation.

Preterm infants are at an increased health risk due to their low maturity. To monitor their health, vital signs are measured using contact-based methods. The adhesive sensors used to detect body temperature can damage the sensitive skin of neonates. Thus, a subject of current research is non-invasive measurement methods based on infrared thermography. In this context, thermal phantoms can be used to develop contactless temperature measurement systems and, furthermore, investigate the thermal behavior of preterm infants. In this work, an improved thermal phantom is introduced to simulate the thermoregulation of a premature infant. The shape and size are adapted to the body of a premature infant in the 29th week of pregnancy. The phantom consists of a 3D-printed frame to which carbon fiber heating elements and Pt1000 temperature sensors are attached. The frame is enclosed by a thermally conductive skin layer made of a silicone boron nitride mixture. Ball joints allow the body parts to tilt and rotate, enabling the phantom to model different body postures. Using PI controllers, the thermal phantom can achieve desired temperatures in 13 different areas of the body while maintaining a homogeneous temperature distribution on the skin surface. In addition, pathological temperature scenarios such as a central-peripheral temperature difference or a change in body temperature can be simulated with a maximum deviation of ± 0.4 °C.

On the cardiorespiratory coordination assessed by the photoplethysmography imaging technique.

Cardiorespiratory coordination (CRC) probes the interaction between cardiac and respiratory oscillators in which cardiac and respiratory activity are synchronized, with individual heartbeats occurring at approximately the same temporal positions during several breathing cycles. An increase of CRC has previously been related to pathological stressful states. We studied CRC employing coordigrams computed from non-contact photoplethysmography imaging (PPGI) and respiratory data using the optical flow method. In a blocked study design, we applied the cold pressure test (CPT), water at ambient temperature (AWT), and intermittent resting conditions. In controls (no intervention), CRC remained on initial low levels throughout measurements. In the experimental group (AWT and CPT intervention), CRC decreased during AWT and CPT. Following both interventions, CRC increased significantly, with a rebound effect following AWT. In controls, HR increased steadily over time. CPT evoked a significant HR increase which correlated with subjective stress/pain ratings. The CRC increase following AWT correlated significantly with subjective pain (r = .79) and stress (r = .63) ratings. Furthermore, we observed a significant correlation (r = - .80) between mean RMSSD and mean duration of CRC, which further supports an association between autonomic state and CRC level. CRC analysis obtained from cutaneous tissue perfusion data therefore appears to be a sensitive and useful method for the study of CRC and ANS activity. Future studies need to investigate the physiological principles and clinical significance of these findings.

On the spatial phase distribution of cutaneous low-frequency perfusion oscillations.

Distributed cutaneous tissue blood volume oscillations contain information on autonomic nervous system (ANS) regulation of cardiorespiratory activity as well as dominating thermoregulation. ANS associated with low-frequency oscillations can be quantified in terms of frequencies, amplitudes, and phase shifts. The relative order between these faculties may be disturbed by conditions colloquially termed 'stress'. Photoplethysmography imaging, an optical non-invasive diagnostic technique provides information on cutaneous tissue perfusion in the temporal and spatial domains. Using the cold pressure test (CPT) in thirteen healthy volunteers as a well-studied experimental intervention, we present a method for evaluating phase shifts in low- and intermediate frequency bands in forehead cutaneous perfusion mapping. Phase shift changes were analysed in low- and intermediate frequency ranges from 0.05 Hz to 0.18 Hz. We observed that time waveforms increasingly desynchronised in various areas of the scanned area throughout measurements. An increase of IM band phase desynchronization observed throughout measurements was comparable in experimental and control group, suggesting a time effect possibly due to overshooting the optimal relaxation duration. CPT triggered an increase in the number of points phase-shifted to the reference that was specific to the low frequency range for phase-shift thresholds defined as π/4, 3π/8, and π/2 rad, respectively. Phase shifts in forehead blood oscillations may infer changes of vascular tone due to activity of various neural systems. We present an innovative method for the phase shift analysis of cutaneous tissue perfusion that appears promising to assess ANS change processes related to physical or psychological stress. More comprehensive studies are needed to further investigate the reliability and physiological significance of findings.

Camera fusion for real-time temperature monitoring of neonates using deep learning.

The continuous monitoring of vital signs is a crucial aspect of medical care in neonatal intensive care units. Since cable-based sensors pose a potential risk for the immature skin of preterm infants, unobtrusive monitoring techniques using camera systems are increasingly investigated. The combination of deep learning-based algorithms and camera modalities such as RGB and infrared thermography can improve the development of cable-free methods for the extraction of vital parameters. In this study, a real-time approach for local extraction of temperatures on the body surface of neonates using a multi-modal clinical dataset was implemented. Therefore, a trained deep learning-based keypoint detector was used for body landmark prediction in RGB. Image registration was conducted to transfer the RGB points to the corresponding thermographic recordings. These landmarks were used to extract the body surface temperature in various regions to determine the central-peripheral temperature difference. A validation of the keypoint detector showed a mean average precision of 0.82. The registration resulted in mean absolute errors of 16.4 px (8.2 mm) for x and 22.4 px (11.2 mm) for y. The evaluation of the temperature extraction revealed a mean absolute error of 0.55 [Formula: see text]C. A final performance of 31 fps was observed on the NVIDIA Jetson Xavier NX module, which proves real-time capability on an embedded GPU system. As a result, the approach can perform real-time temperature extraction on a low-cost GPU module.

A Neonatal Phantom for Vital Signs Simulation.

Neonatal intensive care units provide vital medical support for premature infants. The key aspect in neonatal care is the continuous monitoring of vital signs measured using adhesive skin sensors. Since sensors can cause irritation of the skin and lead to infections, research focuses on contact-free, camera-based methods such as infrared thermography and photoplethysmography imaging. The development of image processing algorithms requires large datasets, but recording the necessary data from studies brings tremendous effort and costs. Therefore, realistic patient phantoms would be feasible to create a comprehensive dataset and validate image-based algorithms. This work describes the realization of a neonatal phantom which can simulate physiological vital parameters such as pulse rate and thermoregulation. It mimics the outer appearance of premature infants using a 3D printed base structure coated with several layers of modified, skin-colored silicone. A distribution of red and infrared LEDs in the scaffold enables the simulation of a PPG signal by mimicking pulsative light intensity changes on the skin. Additionally, the body temperature of the phantom is individually adjustable in several regions using heating elements. In the validation process for PPG simulation, the feasibility of setting different pulse frequencies and the variation of oxygen saturation levels was obtained. Furthermore, heating tests showed region-dependent temperature variations between 0.19 °C and 0.81 °C around the setpoint. In conclusion, the proposed neonatal phantom can be used to simulate a variety of vital parameters of preterm infants and, therefore, enables the implementation of image processing algorithms for the analysis of the medical state.

In Silico and in Vitro Conductivity Models of the Left Heart Ventricle.

Ventricular Assist Devices (VADs) are used to treat patients with cardiogenic shock. As the heart is unable to supply the organs with sufficient oxygenated blood and nutrients, a VAD maintains the circulation to keep the patient alive. The observation of the patient's hemodynamics is crucial for an individual treatment; therefore, sensors to measure quantifiable hemodynmaic parameters are desirable. In addition to pressure measurement, the volume of the left ventricle and the progress of muscle recovery seem to be promising parameters. Ongoing research aims to estimate ventricular volume and changes in electrical properties of cardiac muscle tissue by applying bioimpedance measurement. In the case where ventricular insufficiency is treated by a catheter-based VAD, this very catheter could be used to conduct bioimpedance measurement inside the assisted heart. However, the simultaneous measurement of bioimpedance and VAD support has not yet been realized, although this would allow the determination of various loading conditions of the ventricle. For this purpose, it is necessary to develop models to validate and quantify bioimpedance measurement during VAD support. In this study, we present an in silico and an in vitro conductivity model of a left ventricle to study the application of bioimpedance measurement in the context of VAD therapy. The in vitro model is developed from casting two anatomical silicone phantoms: One phantom of pure silicone, and one phantom enriched with carbon, to obtain a conductive behavior close to the properties of heart muscle tissue. Additionally, a measurement device to record the impedance inside the ventricle is presented. Equivalent to the in vitro model, the in silico model was designed. This finite element model offers changes in material properties for myocardium and the blood cavity. The measurements in the in vitro models show a strong correlation with the results of the simulation of the in silico model. The measurements and the simulation demonstrate a decrease in impedance, when conductive muscle properties are applied and higher impedances correspond to smaller ventricle cross sections. The in silico and in vitro models are used to further investigate the application of bioimpedance measurement inside the left heart ventricle during VAD support. We are confident that the models presented will allow for future evaluation of hemodynamic monitoring during VAD therapy at an early stage of research and development.

Photoplethysmography imaging:camera performance evaluation by means of an optoelectronic skin perfusion phantom.

OBJECTIVE: Photoplethysmography imaging (PPGI) is a promising contactless camera-based method of non-invasive cardiovascular diagnostics. To achieve the best results, it is important to choose the most suitable camera for a specific application. The settings of the camera influence the quality of the detected signal. APPROACH: The standard (European Machine Vision Association 2016 EMVA Standard 1288-Standard for Characterization of Image Sensors and Cameras pp 1-39 (available at: https://www.emva.org/wp-content/uploads/EMVA1288-3.1a.pdf)) for evaluating the imaging performance of machine vision cameras (MVC) helps at the initial decision of the sensor, but the camera should always be tested in terms of usability for a specific application. So far, PPGI lacks a standardized measurement scenario for evaluating the performance of individual cameras. For this, we realized a controllable optoelectronic phantom with artificial silicone skin allowing reproducible tests of cameras to verify their suitability for PPGI. The entire system is housed in a light-tight box. We tested an MVC, a digital single-lens reflex camera (DSLR) camera and a webcam. Each camera varies in used technology and price. MAIN RESULTS: We simulated real PPGI measurement conditions simulating the ratio of pulse (AC) and non-pulse (DC) component of the photoplethysmographic signal and achieved AC/DC ratios of 0.5 % on average. An additional OLED panel ensures proper DC providing reproducible measurement conditions. We evaluated the signal morphological features, amplitude spectrum, signal-to-noise ratio (SNR) and spatially dependent changes of simulated subcutaneous perfusion. Here, the MVC proved to be the most suitable device. A DSLR is also suitable for PPGI, but a larger smoothing kernel is required to obtain a perfusion map. The webcam, as the weakest contender, proved to be very susceptible to any inhomogeneous illumination of the examined artificial skin surface. However, it is still able to detect cardiac rhythm. SIGNIFICANCE: The result of our work is an optoelectronic phantom for reproducible testing of PPGI camera performance in terms of signal quality and measurement equipment costs.

Fast body part segmentation and tracking of neonatal video data using deep learning.

Photoplethysmography imaging (PPGI) for non-contact monitoring of preterm infants in the neonatal intensive care unit (NICU) is a promising technology, as it could reduce medical adhesive-related skin injuries and associated complications. For practical implementations of PPGI, a region of interest has to be detected automatically in real time. As the neonates' body proportions differ significantly from adults, existing approaches may not be used in a straightforward way, and color-based skin detection requires RGB data, thus prohibiting the use of less-intrusive near-infrared (NIR) acquisition. In this paper, we present a deep learning-based method for segmentation of neonatal video data. We augmented an existing encoder-decoder semantic segmentation method with a modified version of the ResNet-50 encoder. This reduced the computational time by a factor of 7.5, so that 30 frames per second can be processed at 960 × 576 pixels. The method was developed and optimized on publicly available databases with segmentation data from adults. For evaluation, a comprehensive dataset consisting of RGB and NIR video recordings from 29 neonates with various skin tones recorded in two NICUs in Germany and India was used. From all recordings, 643 frames were manually segmented. After pre-training the model on the public adult data, parts of the neonatal data were used for additional learning and left-out neonates are used for cross-validated evaluation. On the RGB data, the head is segmented well (82% intersection over union, 88% accuracy), and performance is comparable with those achieved on large, public, non-neonatal datasets. On the other hand, performance on the NIR data was inferior. By employing data augmentation to generate additional virtual NIR data for training, results could be improved and the head could be segmented with 62% intersection over union and 65% accuracy. The method is in theory capable of performing segmentation in real time and thus it may provide a useful tool for future PPGI applications. Graphical Abstract This work presents the development of a customized, real-time capable Deep Learning architecture for segmenting of neonatal videos recorded in the intensive care unit. In addition to hand-annotated data, transfer learning is exploited to improve performance.

A Broader Look: Camera-Based Vital Sign Estimation across the Spectrum.

OBJECTIVES: Camera-based vital sign estimation allows the contactless assessment of important physiological parameters. Seminal contributions were made in the 1930s, 1980s, and 2000s, and the speed of development seems ever increasing. In this suivey, we aim to overview the most recent works in this area, describe their common features as well as shortcomings, and highlight interesting "outliers". METHODS: We performed a comprehensive literature research and quantitative analysis of papers published between 2016 and 2018. Quantitative information about the number of subjects, studies with healthy volunteers vs. pathological conditions, public datasets, laboratory vs. real-world works, types of camera, usage of machine learning, and spectral properties of data was extracted. Moreover, a qualitative analysis of illumination used and recent advantages in terms of algorithmic developments was also performed. RESULTS: Since 2016, 116 papers were published on camera-based vital sign estimation and 59% of papers presented results on 20 or fewer subjects. While the average number of participants increased from 15.7 in 2016 to 22.9 in 2018, the vast majority of papers (n=100) were on healthy subjects. Four public datasets were used in 10 publications. We found 27 papers whose application scenario could be considered a real-world use case, such as monitoring during exercise or driving. These include 16 papers that dealt with non-healthy subjects. The majority of papers (n=61) presented results based on visual, red-green-blue (RGB) information, followed by RGB combined with other parts of the electromagnetic spectrum (n=18), and thermography only (n=12), while other works (n=25) used other mono- or polychromatic non-RGB data. Surprisingly, a minority of publications (n=39) made use of consumer-grade equipment. Lighting conditions were primarily uncontrolled or ambient. While some works focused on specialized aspects such as the removal of vital sign information from video streams to protect privacy or the influence of video compression, most algorithmic developments were related to three areas: region of interest selection, tracking, or extraction of a one-dimensional signal. Seven papers used deep learning techniques, 17 papers used other machine learning approaches, and 92 made no explicit use of machine learning. CONCLUSION: Although some general trends and frequent shortcomings are obvious, the spectrum of publications related to camera-based vital sign estimation is broad. While many creative solutions and unique approaches exist, the lack of standardization hinders comparability of these techniques and of their performance. We believe that sharing algorithms and/ or datasets will alleviate this and would allow the application of newer techniques such as deep learning.