A cGAN-based network for depth estimation from bronchoscopic images.

PURPOSE: Depth estimation is the basis of 3D reconstruction of airway structure from 2D bronchoscopic scenes, which can be further used to develop a vision-based bronchoscopic navigation system. This work aims to improve the performance of depth estimation directly from bronchoscopic images by training a depth estimation network on both synthetic and real datasets. METHODS: We propose a cGAN-based network Bronchoscopic-Depth-GAN (BronchoDep-GAN) to estimate depth from bronchoscopic images by translating bronchoscopic images into depth maps. The network is trained in a supervised way learning from synthetic textured bronchoscopic image-depth pairs and virtual bronchoscopic image-depth pairs, and simultaneously, also in an unsupervised way learning from unpaired real bronchoscopic images and depth maps to adapt the model to real bronchoscopic scenes. RESULTS: Our method is tested on both synthetic data and real data. However, the tests on real data are only qualitative, as no ground truth is available. The results show that our network obtains better accuracy in all cases in estimating depth from bronchoscopic images compared to the well-known cGANs pix2pix. CONCLUSIONS: Including virtual and real bronchoscopic images in the training phase of the depth estimation networks can improve depth estimation's performance on both synthetic and real scenes. Further validation of this work is planned on 3D clinical phantoms. Based on the depth estimation results obtained in this work, the accuracy of locating bronchoscopes with corresponding pre-operative CTs will also be evaluated in comparison with the current clinical status.

3D-localization of single point-like gamma sources with a coded aperture camera.

Objective.3D-localization of gamma sources has the potential to improve the outcome of radio-guided surgery. The goal of this paper is to analyze the localization accuracy for point-like sources with a single coded aperture camera.Approach.We both simulated and measured a point-like241Am source at 17 positions distributed within the field of view of an experimental gamma camera. The setup includes a0.11mmthick Tungsten sheet with a MURA mask of rank 31 and pinholes of0.08mmin diameter and a detector based on the photon counting readout circuit Timepix3. Two methods, namely an iterative search including either a symmetric Gaussian fitting or an exponentially modified Gaussian fitting (EMG) and a center of mass method were compared to estimate the 3D source position.Main results.Considering the decreasing axial resolution with source-to-mask distance, the EMG improved the results by a factor of 4 compared to the Gaussian fitting based on the simulated data. Overall, we obtained a mean localization error of0.77mmon the simulated and2.64mmon the experimental data in the imaging range of20-100mm.Significance.This paper shows that despite the low axial resolution, point-like sources in the nearfield can be localized as well as with more sophisticated imaging devices such as stereo cameras. The influence of the source size and the photon count on the imaging and localization accuracy remains an important issue for further research.

Assessment of the axial resolution of a compact gamma camera with coded aperture collimator.

PURPOSE: Handheld gamma cameras with coded aperture collimators are under investigation for intraoperative imaging in nuclear medicine. Coded apertures are a promising collimation technique for applications such as lymph node localization due to their high sensitivity and the possibility of 3D imaging. We evaluated the axial resolution and computational performance of two reconstruction methods. METHODS: An experimental gamma camera was set up consisting of the pixelated semiconductor detector Timepix3 and MURA mask of rank 31 with round holes of 0.08 mm in diameter in a 0.11 mm thick Tungsten sheet. A set of measurements was taken where a point-like gamma source was placed centrally at 21 different positions within the range of 12-100 mm. For each source position, the detector image was reconstructed in 0.5 mm steps around the true source position, resulting in an image stack. The axial resolution was assessed by the full width at half maximum (FWHM) of the contrast-to-noise ratio (CNR) profile along the z-axis of the stack. Two reconstruction methods were compared: MURA Decoding and a 3D maximum likelihood expectation maximization algorithm (3D-MLEM). RESULTS: While taking 4400 times longer in computation, 3D-MLEM yielded a smaller axial FWHM and a higher CNR. The axial resolution degraded from 5.3 mm and 1.8 mm at 12 mm to 42.2 mm and 13.5 mm at 100 mm for MURA Decoding and 3D-MLEM respectively. CONCLUSION: Our results show that the coded aperture enables the depth estimation of single point-like sources in the near field. Here, 3D-MLEM offered a better axial resolution but was computationally much slower than MURA Decoding, whose reconstruction time is compatible with real-time imaging.

Texture synthesis for generating realistic-looking bronchoscopic videos.

PURPOSE: Synthetic realistic-looking bronchoscopic videos are needed to develop and evaluate depth estimation methods as part of investigating vision-based bronchoscopic navigation system. To generate these synthetic videos under the circumstance where access to real bronchoscopic images/image sequences is limited, we need to create various realistic-looking image textures of the airway inner surface with large size using a small number of real bronchoscopic image texture patches. METHODS: A generative adversarial networks-based method is applied to create realistic-looking textures of the airway inner surface by learning from a limited number of small texture patches from real bronchoscopic images. By applying a purely convolutional architecture without any fully connected layers, this method allows the production of textures with arbitrary size. RESULTS: Authentic image textures of airway inner surface are created. An example of the synthesized textures and two frames of the thereby generated bronchoscopic video are shown. The necessity and sufficiency of the generated textures as image features for further depth estimation methods are demonstrated. CONCLUSIONS: The method can generate textures of the airway inner surface that meet the requirements for the texture itself and for the thereby generated bronchoscopic videos, including "realistic-looking," "long-term temporal consistency," "sufficient image features for depth estimation," and "large size and variety of synthesized textures." Besides, it also shows advantages with respect to the easy accessibility to required data source. A further validation of this approach is planned by utilizing the realistic-looking bronchoscopic videos with textures generated by this method as training and test data for some depth estimation networks.

Review of Artificial Intelligence-Based Signal Processing in Dialysis: Challenges for Machine-Embedded and Complementary Applications.

Artificial intelligence technology is trending in nearly every medical area. It offers the possibility for improving analytics, therapy outcome, and user experience during therapy. In dialysis, the application of artificial intelligence as a therapy-individualization tool is led more by start-ups than consolidated players, and innovation in dialysis seems comparably stagnant. Factors such as technical requirements or regulatory processes are important and necessary but can slow down the implementation of artificial intelligence due to missing data infrastructure and undefined approval processes. Current research focuses mainly on analyzing health records or wearable technology to add to existing health data. It barely uses signal data from treatment devices to apply artificial intelligence models. This article, therefore, discusses requirements for signal processing through artificial intelligence in health care and compares these with the status quo in dialysis therapy. It offers solutions for given barriers to speed up innovation with sensor data, opening access to existing and untapped sources, and shows the unique advantage of signal processing in dialysis compared to other health care domains. This research shows that even though the combination of different data is vital for improving patients' therapy, adding signal-based treatment data from dialysis devices to the picture can benefit the understanding of treatment dynamics, improving and individualizing therapy.

Impact of Lens Opacity on Optical Coherence Tomography Angiography Metrics.

PURPOSE: To evaluate the impact of lens opacity on the reliability of optical coherence tomography angiography metrics and to find a vessel caliber threshold that is reproducible in cataract patients. METHODS: A prospective cohort study of 31 patients, examining one eye per patient, by applying 3 × 3 mm macular optical coherence tomography angiography before (18.94 ± 12.22 days) and 3 months (111 ± 23.45 days) after uncomplicated cataract surgery. We extracted superficial (SVC) and deep vascular plexuses (DVC) for further analysis and evaluated changes in image contrast, vessel metrics (perfusion density, flow deficit and vessel-diameter index) and foveal avascular area (FAZ). RESULTS: After surgery, the blood flow signal in smaller capillaries was enhanced as image contrast improved. Signal strength correlated to average lens density defined by objective measurement in Scheimpflug images (Pearson's r: -.40, p: .027) and to flow deficit (r= -.70, p < .001). Perfusion density correlated to the signal strength index (r = .70, p < .001). Vessel metrics and FAZ area, except for FAZ area in DVC, were significantly different after cataract surgery, but the mean change was approximately 3-6%. A stepwise approach in extracting vessels according to their pixel caliber showed a threshold of > 6 pixels caliber (∼20-30 µm) was comparable before and after lens removal. CONCLUSION: In patients with cataract, OCTA vessel metrics should be interpreted with caution. In addition to signal strength, contrast and pixel properties can serve as supplementary quality metrics to improve the interpretation of OCTA metrics. Vessels with ∼20-30 µm in caliber seem to be reproducible.

The Use of Artificial Intelligence for the Classification of Craniofacial Deformities.

Positional cranial deformities are a common finding in toddlers, yet differentiation from craniosynostosis can be challenging. The aim of this study was to train convolutional neural networks (CNNs) to classify craniofacial deformities based on 2D images generated using photogrammetry as a radiation-free imaging technique. A total of 487 patients with photogrammetry scans were included in this retrospective cohort study: children with craniosynostosis (n = 227), positional deformities (n = 206), and healthy children (n = 54). Three two-dimensional images were extracted from each photogrammetry scan. The datasets were divided into training, validation, and test sets. During the training, fine-tuned ResNet-152s were utilized. The performance was quantified using tenfold cross-validation. For the detection of craniosynostosis, sensitivity was at 0.94 with a specificity of 0.85. Regarding the differentiation of the five existing classes (trigonocephaly, scaphocephaly, positional plagiocephaly left, positional plagiocephaly right, and healthy), sensitivity ranged from 0.45 (positional plagiocephaly left) to 0.95 (scaphocephaly) and specificity ranged from 0.87 (positional plagiocephaly right) to 0.97 (scaphocephaly). We present a CNN-based approach to classify craniofacial deformities on two-dimensional images with promising results. A larger dataset would be required to identify rarer forms of craniosynostosis as well. The chosen 2D approach enables future applications for digital cameras or smartphones.

3D-2D Distance Maps Conversion Enhances Classification of Craniosynostosis.

OBJECTIVE: Diagnosis of craniosynostosis using photogrammetric 3D surface scans is a promising radiation-free alternative to traditional computed tomography. We propose a 3D surface scan to 2D distance map conversion enabling the usage of the first convolutional neural networks (CNNs)-based classification of craniosynostosis. Benefits of using 2D images include preserving patient anonymity, enabling data augmentation during training, and a strong under-sampling of the 3D surface with good classification performance. METHODS: The proposed distance maps sample 2D images from 3D surface scans using a coordinate transformation, ray casting, and distance extraction. We introduce a CNN-based classification pipeline and compare our classifier to alternative approaches on a dataset of 496 patients. We investigate into low-resolution sampling, data augmentation, and attribution mapping. RESULTS: Resnet18 outperformed alternative classifiers on our dataset with an F1-score of 0.964 and an accuracy of 98.4%. Data augmentation on 2D distance maps increased performance for all classifiers. Under-sampling allowed 256-fold computation reduction during ray casting while retaining an F1-score of 0.92. Attribution maps showed high amplitudes on the frontal head. CONCLUSION: We demonstrated a versatile mapping approach to extract a 2D distance map from the 3D head geometry increasing classification performance, enabling data augmentation during training on 2D distance maps, and the usage of CNNs. We found that low-resolution images were sufficient for a good classification performance. SIGNIFICANCE: Photogrammetric surface scans are a suitable craniosynostosis diagnosis tool for clinical practice. Domain transfer to computed tomography seems likely and can further contribute to reducing ionizing radiation exposure for infants.

Impact of data synthesis strategies for the classification of craniosynostosis.

Introduction: Photogrammetric surface scans provide a radiation-free option to assess and classify craniosynostosis. Due to the low prevalence of craniosynostosis and high patient restrictions, clinical data are rare. Synthetic data could support or even replace clinical data for the classification of craniosynostosis, but this has never been studied systematically. Methods: We tested the combinations of three different synthetic data sources: a statistical shape model (SSM), a generative adversarial network (GAN), and image-based principal component analysis for a convolutional neural network (CNN)-based classification of craniosynostosis. The CNN is trained only on synthetic data but is validated and tested on clinical data. Results: The combination of an SSM and a GAN achieved an accuracy of 0.960 and an F1 score of 0.928 on the unseen test set. The difference to training on clinical data was smaller than 0.01. Including a second image modality improved classification performance for all data sources. Conclusions: Without a single clinical training sample, a CNN was able to classify head deformities with similar accuracy as if it was trained on clinical data. Using multiple data sources was key for a good classification based on synthetic data alone. Synthetic data might play an important future role in the assessment of craniosynostosis.

Generation of Synthetic Data for the Comparison of Different 3D-3D Registration Approaches in Laparoscopic Surgery.

In laparoscopic surgery image-guided navigation systems could support the surgeon by providing subsurface information such as the positions of tumors and vessels. For this purpose, one option is to perform a reliable registration of preoperative 3D data and a surface patch from laparo-scopic video data. A robust and automatic 3D-3D registration pipeline for the application during laparoscopic surgery has not yet been found due to application-specific challenges. To gain a better insight, we propose a framework enabling a qualitative and quantitative comparison of different registration approaches. The introduced framework is able to evaluate 3D feature descriptors and registration algorithms by generating and modifying synthetic data from clinical examples. Different confounding factors are considered and thus the reality can be reflected in any simplified or more complex way. Two exemplary experiments with a liver model, using the RANSAC algorithm, showed an increasing registration error for a decreasing size of the surface patch size and after introducing modifications. Moreover, the registration accuracy was dependent on the position and structure of the surface patch. The framework helps to quantitatively assess and optimize the registration pipeline, and hereby suggests future software improvements even with only few clinical examples. Clinical relevance- The introduced framework permits a quantitative and comprehensive comparison of different registration approaches which forms the basis for a supportive navigation tool in laparoscopic surgery.

A Radiation-Free Classification Pipeline for Craniosynostosis Using Statistical Shape Modeling.

BACKGROUND: Craniosynostosis is a condition caused by the premature fusion of skull sutures, leading to irregular growth patterns of the head. Three-dimensional photogrammetry is a radiation-free alternative to the diagnosis using computed tomography. While statistical shape models have been proposed to quantify head shape, no shape-model-based classification approach has been presented yet. METHODS: We present a classification pipeline that enables an automated diagnosis of three types of craniosynostosis. The pipeline is based on a statistical shape model built from photogrammetric surface scans. We made the model and pathology-specific submodels publicly available, making it the first publicly available craniosynostosis-related head model, as well as the first focusing on infants younger than 1.5 years. To the best of our knowledge, we performed the largest classification study for craniosynostosis to date. RESULTS: Our classification approach yields an accuracy of 97.8 %, comparable to other state-of-the-art methods using both computed tomography scans and stereophotogrammetry. Regarding the statistical shape model, we demonstrate that our model performs similar to other statistical shape models of the human head. CONCLUSION: We present a state-of-the-art shape-model-based classification approach for a radiation-free diagnosis of craniosynostosis. Our publicly available shape model enables the assessment of craniosynostosis on realistic and synthetic data.

Simulation-Based Estimation of the Number of Cameras Required for 3D Reconstruction in a Narrow-Baseline Multi-Camera Setup.

Graphical visualization systems are a common clinical tool for displaying digital images and three-dimensional volumetric data. These systems provide a broad spectrum of information to support physicians in their clinical routine. For example, the field of radiology enjoys unrestricted options for interaction with the data, since information is pre-recorded and available entirely in digital form. However, some fields, such as microsurgery, do not benefit from this yet. Microscopes, endoscopes, and laparoscopes show the surgical site as it is. To allow free data manipulation and information fusion, 3D digitization of surgical sites is required. We aimed to find the number of cameras needed to add this functionality to surgical microscopes. For this, we performed in silico simulations of the 3D reconstruction of representative models of microsurgical sites with different numbers of cameras in narrow-baseline setups. Our results show that eight independent camera views are preferable, while at least four are necessary for a digital surgical site. In most cases, eight cameras allow the reconstruction of over 99% of the visible part. With four cameras, still over 95% can be achieved. This answers one of the key questions for the development of a prototype microscope. In future, such a system can provide functionality which is unattainable today.

Transit Time Measurement in Indicator Dilution Curves: Overcoming the Missing Ground Truth and Quantifying the Error.

The vascular function of a vessel can be qualitatively and intraoperatively checked by recording the blood dynamics inside the vessel via fluorescence angiography (FA). Although FA is the state of the art in proving the existence of blood flow during interventions such as bypass surgery, it still lacks a quantitative blood flow measurement that could decrease the recurrence rate and postsurgical mortality. Previous approaches show that the measured flow has a significant deviation compared to the gold standard reference (ultrasonic flow meter). In order to systematically address the possible sources of error, we investigated the error in transit time measurement of an indicator. Obtaining in vivo indicator dilution curves with a known ground truth is complex and often not possible. Further, the error in transit time measurement should be quantified and reduced. To tackle both issues, we first computed many diverse indicator dilution curves using an in silico simulation of the indicator's flow. Second, we post-processed these curves to mimic measured signals. Finally, we fitted mathematical models (parabola, gamma variate, local density random walk, and mono-exponential model) to re-continualize the obtained discrete indicator dilution curves and calculate the time delay of two analytical functions. This re-continualization showed an increase in the temporal accuracy up to a sub-sample accuracy. Thereby, the Local Density Random Walk (LDRW) model performed best using the cross-correlation of the first derivative of both indicator curves with a cutting of the data at 40% of the peak intensity. The error in frames depends on the noise level and is for a signal-to-noise ratio (SNR) of 20 dB and a sampling rate of f s = 60 Hz at f s - 1 · 0 . 25 ( ± 0 . 18 ) , so this error is smaller than the distance between two consecutive samples. The accurate determination of the transit time and the quantification of the error allow the calculation of the error propagation onto the flow measurement. Both can assist surgeons as an intraoperative quality check and thereby reduce the recurrence rate and post-surgical mortality.

3D-Guided Face Manipulation of 2D Images for the Prediction of Post-Operative Outcome After Cranio-Maxillofacial Surgery.

Cranio-maxillofacial surgery often alters the aesthetics of the face which can be a heavy burden for patients to decide whether or not to undergo surgery. Today, physicians can predict the post-operative face using surgery planning tools to support the patient's decision-making. While these planning tools allow a simulation of the post-operative face, the facial texture must usually be captured by another 3D texture scan and subsequently mapped on the simulated face. This approach often results in face predictions that do not appear realistic or lively looking and are therefore ill-suited to guide the patient's decision-making. Instead, we propose a method using a generative adversarial network to modify a facial image according to a 3D soft-tissue estimation of the post-operative face. To circumvent the lack of available data pairs between pre- and post-operative measurements we propose a semi-supervised training strategy using cycle losses that only requires paired open-source data of images and 3D surfaces of the face's shape. After training on "in-the-wild" images we show that our model can realistically manipulate local regions of a face in a 2D image based on a modified 3D shape. We then test our model on four clinical examples where we predict the post-operative face according to a 3D soft-tissue prediction of surgery outcome, which was simulated by a surgery planning tool. As a result, we aim to demonstrate the potential of our approach to predict realistic post-operative images of faces without the need of paired clinical data, physical models, or 3D texture scans.

In Silico Modelling of Blood Vessel Segmentations for Estimation of Discretization Error in Spatial Measurement and its Impact on Quantitative Fluorescence Angiography.

Today the vascular function after interventions as Bypass surgeries are checked qualitatively by observing the blood dynamics inside the vessel via Indocyanine Green (ICG) Fluorescence Angiography. This state-of-the-art should be upgraded and has to be improved and converted towards a quantitatively measured blood flow. Previous approaches show that the blood flow measured from fluorescence angiography cannot be easily calibrated to a gold standard reference. In order to systematically address the possible source of error we investigate as a first step the discretization error in a camera-based measurement of the vessel's geometry. In order to generate an error-free ground truth, a vessel model has been developed based on mathematical functions. This database is then used to determine the error in discretizing the centerline of the structure and estimate its effects on the accuracy of the flow calculation. As result the model is implemented according to the conditions which are set up to ensure transferability on camera-based segmentations of vessels. In this paper the relative discretization error for estimating the centerline length of segmented vessels could be calculated in the range of 6.3%. This would reveal significant error propagated to the estimation of the blood flow value derived by camera-based angiography.