Photoacoustic quantification of tissue oxygenation using conditional invertible neural networks.

Intelligent systems in interventional healthcare depend on the reliable perception of the environment. In this context, photoacoustic tomography (PAT) has emerged as a non-invasive, functional imaging modality with great clinical potential. Current research focuses on converting the high-dimensional, not human-interpretable spectral data into the underlying functional information, specifically the blood oxygenation. One of the largely unexplored issues stalling clinical advances is the fact that the quantification problem is ambiguous, i.e. that radically different tissue parameter configurations could lead to almost identical photoacoustic spectra. In the present work, we tackle this problem with conditional Invertible Neural Networks (cINNs). Going beyond traditional point estimates, our network is used to compute an approximation of the conditional posterior density of tissue parameters given the measurement. To this end, an automatic mode detection algorithm extracts the plausible solution from the sample-based posterior. According to a comprehensive validation study based on both synthetic and real images, our approach is well-suited for exploring ambiguity in quantitative PAT.

Correction: Comparison of Four Active SARS-CoV-2 Surveillance Strategies in Representative Population Sample Points: Two-Factor Factorial Randomized Controlled Trial.

[This corrects the article DOI: 10.2196/44204.].

Comparison of Four Active SARS-CoV-2 Surveillance Strategies in Representative Population Sample Points: Two-Factor Factorial Randomized Controlled Trial.

BACKGROUND: The COVID-19 pandemic is characterized by rapid increases in infection burden owing to the emergence of new variants with higher transmissibility and immune escape. To date, monitoring the COVID-19 pandemic has mainly relied on passive surveillance, yielding biased epidemiological measures owing to the disproportionate number of undetected asymptomatic cases. Active surveillance could provide accurate estimates of the true prevalence to forecast the evolution of the pandemic, enabling evidence-based decision-making. OBJECTIVE: This study compared 4 different approaches of active SARS-CoV-2 surveillance focusing on feasibility and epidemiological outcomes. METHODS: A 2-factor factorial randomized controlled trial was conducted in 2020 in a German district with 700,000 inhabitants. The epidemiological outcome comprised SARS-CoV-2 prevalence and its precision. The 4 study arms combined 2 factors: individuals versus households and direct testing versus testing conditioned on symptom prescreening. Individuals aged ≥7 years were eligible. Altogether, 27,908 addresses from 51 municipalities were randomly allocated to the arms and 15 consecutive recruitment weekdays. Data collection and logistics were highly digitized, and a website in 5 languages enabled low-barrier registration and tracking of results. Gargle sample collection kits were sent by post. Participants collected a gargle sample at home and mailed it to the laboratory. Samples were analyzed with reverse transcription loop-mediated isothermal amplification (RT-LAMP); positive and weak results were confirmed with real-time reverse transcription-polymerase chain reaction (RT-PCR). RESULTS: Recruitment was conducted between November 18 and December 11, 2020. The response rates in the 4 arms varied between 34.31% (2340/6821) and 41.17% (2043/4962). The prescreening classified 16.61% (1207/7266) of the patients as COVID-19 symptomatic. Altogether, 4232 persons without prescreening and 7623 participating in the prescreening provided 5351 gargle samples, of which 5319 (99.4%) could be analyzed. This yielded 17 confirmed SARS-CoV-2 infections and a combined prevalence of 0.36% (95% CI 0.14%-0.59%) in the arms without prescreening and 0.05% (95% CI 0.00%-0.108%) in the arms with prescreening (initial contacts only). Specifically, we found a prevalence of 0.31% (95% CI 0.06%-0.58%) for individuals and 0.35% (95% CI 0.09%-0.61%) for households, and lower estimates with prescreening (0.07%, 95% CI 0.0%-0.15% for individuals and 0.02%, 95% CI 0.0%-0.06% for households). Asymptomatic infections occurred in 27% (3/11) of the positive cases with symptom data. The 2 arms without prescreening performed the best regarding effectiveness and accuracy. CONCLUSIONS: This study showed that postal mailing of gargle sample kits and returning home-based self-collected liquid gargle samples followed by high-sensitivity RT-LAMP analysis is a feasible way to conduct active SARS-CoV-2 population surveillance without burdening routine diagnostic testing. Efforts to improve participation rates and integration into the public health system may increase the potential to monitor the course of the pandemic. TRIAL REGISTRATION: Deutsches Register Klinischer Studien (DRKS) DRKS00023271; https://tinyurl.com/3xenz68a. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): RR2-10.1186/s13063-021-05619-5.

Beyond rankings: Learning (more) from algorithm validation.

Challenges have become the state-of-the-art approach to benchmark image analysis algorithms in a comparative manner. While the validation on identical data sets was a great step forward, results analysis is often restricted to pure ranking tables, leaving relevant questions unanswered. Specifically, little effort has been put into the systematic investigation on what characterizes images in which state-of-the-art algorithms fail. To address this gap in the literature, we (1) present a statistical framework for learning from challenges and (2) instantiate it for the specific task of instrument instance segmentation in laparoscopic videos. Our framework relies on the semantic meta data annotation of images, which serves as foundation for a General Linear Mixed Models (GLMM) analysis. Based on 51,542 meta data annotations performed on 2,728 images, we applied our approach to the results of the Robust Medical Instrument Segmentation Challenge (ROBUST-MIS) challenge 2019 and revealed underexposure, motion and occlusion of instruments as well as the presence of smoke or other objects in the background as major sources of algorithm failure. Our subsequent method development, tailored to the specific remaining issues, yielded a deep learning model with state-of-the-art overall performance and specific strengths in the processing of images in which previous methods tended to fail. Due to the objectivity and generic applicability of our approach, it could become a valuable tool for validation in the field of medical image analysis and beyond.

Spectral imaging enables contrast agent-free real-time ischemia monitoring in laparoscopic surgery.

Laparoscopic surgery has evolved as a key technique for cancer diagnosis and therapy. While characterization of the tissue perfusion is crucial in various procedures, such as partial nephrectomy, doing so by means of visual inspection remains highly challenging. We developed a laparoscopic real-time multispectral imaging system featuring a compact and lightweight multispectral camera and the possibility to complement the conventional surgical view of the patient with functional information at a video rate of 25 Hz. To enable contrast agent-free ischemia monitoring during laparoscopic partial nephrectomy, we phrase the problem of ischemia detection as an out-of-distribution detection problem that does not rely on data from any other patient and uses an ensemble of invertible neural networks at its core. An in-human trial demonstrates the feasibility of our approach and highlights the potential of spectral imaging combined with advanced deep learning-based analysis tools for fast, efficient, reliable, and safe functional laparoscopic imaging.

Robust deep learning-based semantic organ segmentation in hyperspectral images.

Semantic image segmentation is an important prerequisite for context-awareness and autonomous robotics in surgery. The state of the art has focused on conventional RGB video data acquired during minimally invasive surgery, but full-scene semantic segmentation based on spectral imaging data and obtained during open surgery has received almost no attention to date. To address this gap in the literature, we are investigating the following research questions based on hyperspectral imaging (HSI) data of pigs acquired in an open surgery setting: (1) What is an adequate representation of HSI data for neural network-based fully automated organ segmentation, especially with respect to the spatial granularity of the data (pixels vs. superpixels vs. patches vs. full images)? (2) Is there a benefit of using HSI data compared to other modalities, namely RGB data and processed HSI data (e.g. tissue parameters like oxygenation), when performing semantic organ segmentation? According to a comprehensive validation study based on 506 HSI images from 20 pigs, annotated with a total of 19 classes, deep learning-based segmentation performance increases - consistently across modalities - with the spatial context of the input data. Unprocessed HSI data offers an advantage over RGB data or processed data from the camera provider, with the advantage increasing with decreasing size of the input to the neural network. Maximum performance (HSI applied to whole images) yielded a mean DSC of 0.90 ((standard deviation (SD)) 0.04), which is in the range of the inter-rater variability (DSC of 0.89 ((standard deviation (SD)) 0.07)). We conclude that HSI could become a powerful image modality for fully-automatic surgical scene understanding with many advantages over traditional imaging, including the ability to recover additional functional tissue information. Our code and pre-trained models are available at https://github.com/IMSY-DKFZ/htc.

Robust hand tracking for surgical telestration.

PURPOSE: As human failure has been shown to be one primary cause for post-operative death, surgical training is of the utmost socioeconomic importance. In this context, the concept of surgical telestration has been introduced to enable experienced surgeons to efficiently and effectively mentor trainees in an intuitive way. While previous approaches to telestration have concentrated on overlaying drawings on surgical videos, we explore the augmented reality (AR) visualization of surgical hands to imitate the direct interaction with the situs. METHODS: We present a real-time hand tracking pipeline specifically designed for the application of surgical telestration. It comprises three modules, dedicated to (1) the coarse localization of the expert's hand and the subsequent (2) segmentation of the hand for AR visualization in the field of view of the trainee and (3) regression of keypoints making up the hand's skeleton. The semantic representation is obtained to offer the ability for structured reporting of the motions performed as part of the teaching. RESULTS: According to a comprehensive validation based on a large data set comprising more than 14,000 annotated images with varying application-relevant conditions, our algorithm enables real-time hand tracking and is sufficiently accurate for the task of surgical telestration. In a retrospective validation study, a mean detection accuracy of 98%, a mean keypoint regression accuracy of 10.0 px and a mean Dice Similarity Coefficient of 0.95 were achieved. In a prospective validation study, it showed uncompromised performance when the sensor, operator or gesture varied. CONCLUSION: Due to its high accuracy and fast inference time, our neural network-based approach to hand tracking is well suited for an AR approach to surgical telestration. Future work should be directed to evaluating the clinical value of the approach.

Heidelberg colorectal data set for surgical data science in the sensor operating room.

Image-based tracking of medical instruments is an integral part of surgical data science applications. Previous research has addressed the tasks of detecting, segmenting and tracking medical instruments based on laparoscopic video data. However, the proposed methods still tend to fail when applied to challenging images and do not generalize well to data they have not been trained on. This paper introduces the Heidelberg Colorectal (HeiCo) data set - the first publicly available data set enabling comprehensive benchmarking of medical instrument detection and segmentation algorithms with a specific emphasis on method robustness and generalization capabilities. Our data set comprises 30 laparoscopic videos and corresponding sensor data from medical devices in the operating room for three different types of laparoscopic surgery. Annotations include surgical phase labels for all video frames as well as information on instrument presence and corresponding instance-wise segmentation masks for surgical instruments (if any) in more than 10,000 individual frames. The data has successfully been used to organize international competitions within the Endoscopic Vision Challenges 2017 and 2019.

Learned spectral decoloring enables photoacoustic oximetry.

The ability of photoacoustic imaging to measure functional tissue properties, such as blood oxygenation sO[Formula: see text], enables a wide variety of possible applications. sO[Formula: see text] can be computed from the ratio of oxyhemoglobin HbO[Formula: see text] and deoxyhemoglobin Hb, which can be distuinguished by multispectral photoacoustic imaging due to their distinct wavelength-dependent absorption. However, current methods for estimating sO[Formula: see text] yield inaccurate results in realistic settings, due to the unknown and wavelength-dependent influence of the light fluence on the signal. In this work, we propose learned spectral decoloring to enable blood oxygenation measurements to be inferred from multispectral photoacoustic imaging. The method computes sO[Formula: see text] pixel-wise, directly from initial pressure spectra [Formula: see text], which represent initial pressure values at a fixed spatial location [Formula: see text] over all recorded wavelengths [Formula: see text]. The method is compared to linear unmixing approaches, as well as pO[Formula: see text] and blood gas analysis reference measurements. Experimental results suggest that the proposed method is able to obtain sO[Formula: see text] estimates from multispectral photoacoustic measurements in silico, in vitro, and in vivo.

Light source calibration for multispectral imaging in surgery.

PURPOSE: Live intra-operative functional imaging has multiple potential clinical applications, such as localization of ischemia, assessment of organ transplantation success and perfusion monitoring. Recent research has shown that live monitoring of functional tissue properties, such as tissue oxygenation and blood volume fraction, is possible using multispectral imaging in laparoscopic surgery. While the illuminant spectrum is typically kept constant in laparoscopic surgery and can thus be estimated from preoperative calibration images, a key challenge in open surgery originates from the dynamic changes of lighting conditions. METHODS: The present paper addresses this challenge with a novel approach to light source calibration based on specular highlight analysis. It involves the acquisition of low-exposure time images serving as a basis for recovering the illuminant spectrum from pixels that contain a dominant specular reflectance component. RESULTS: Comprehensive in silico and in vivo experiments with a range of different light sources demonstrate that our approach enables an accurate and robust recovery of the illuminant spectrum in the field of view of the camera, which results in reduced errors with respect to the estimation of functional tissue properties. Our approach further outperforms state-of-the-art methods proposed in the field of computer vision. CONCLUSION: Our results suggest that low-exposure multispectral images are well suited for light source calibration via specular highlight analysis. This work thus provides an important first step toward live functional imaging in open surgery.

Uncertainty-aware performance assessment of optical imaging modalities with invertible neural networks.

PURPOSE: Optical imaging is evolving as a key technique for advanced sensing in the operating room. Recent research has shown that machine learning algorithms can be used to address the inverse problem of converting pixel-wise multispectral reflectance measurements to underlying tissue parameters, such as oxygenation. Assessment of the specific hardware used in conjunction with such algorithms, however, has not properly addressed the possibility that the problem may be ill-posed. METHODS: We present a novel approach to the assessment of optical imaging modalities, which is sensitive to the different types of uncertainties that may occur when inferring tissue parameters. Based on the concept of invertible neural networks, our framework goes beyond point estimates and maps each multispectral measurement to a full posterior probability distribution which is capable of representing ambiguity in the solution via multiple modes. Performance metrics for a hardware setup can then be computed from the characteristics of the posteriors. RESULTS: Application of the assessment framework to the specific use case of camera selection for physiological parameter estimation yields the following insights: (1) estimation of tissue oxygenation from multispectral images is a well-posed problem, while (2) blood volume fraction may not be recovered without ambiguity. (3) In general, ambiguity may be reduced by increasing the number of spectral bands in the camera. CONCLUSION: Our method could help to optimize optical camera design in an application-specific manner.