Computer-assisted 3D bowel length measurement for quantitative laparoscopy.

BACKGROUND: This study aimed at developing and evaluating a tool for computer-assisted 3D bowel length measurement (BMS) to improve objective measurement in minimally invasive surgery. Standardization and quality of surgery as well as its documentation are currently limited by lack of objective intraoperative measurements. To solve this problem, we developed BMS as a clinical application of Quantitative Laparoscopy (QL). METHODS: BMS processes images from a conventional 3D laparoscope. Computer vision algorithms are used to measure the distance between laparoscopic instruments along a 3D reconstruction of the bowel surface. Preclinical evaluation was performed in phantom, ex vivo porcine, and in vivo porcine models. A bowel length of 70 cm was measured with BMS and compared to a manually obtained ground truth. Afterwards 70 cm of bowel (ground truth) was measured and compared to BMS. RESULTS: Ground truth was 66.1 ± 2.7 cm (relative error + 5.8%) in phantom, 65.8 ± 2.5 cm (relative error + 6.4%) in ex vivo, and 67.5 ± 6.6 cm (relative error + 3.7%) in in vivo porcine evaluation when 70 cm was measured with BMS. Using 70 cm of bowel, BMS measured 75.0 ± 2.9 cm (relative error + 7.2%) in phantom and 74.4 ± 2.8 cm (relative error + 6.3%) in ex vivo porcine evaluation. After thorough preclinical evaluation, BMS was successfully used in a patient undergoing laparoscopic Roux-en-Y gastric bypass for morbid obesity. CONCLUSIONS: QL using BMS was shown to be feasible and was successfully translated from studies on phantom, ex vivo, and in vivo porcine bowel to a clinical feasibility study.

E-DQN-Based Path Planning Method for Drones in Airsim Simulator under Unknown Environment.

To improve the rapidity of path planning for drones in unknown environments, a new bio-inspired path planning method using E-DQN (event-based deep Q-network), referring to introducing event stream to reinforcement learning network, is proposed. Firstly, event data are collected through an airsim simulator for environmental perception, and an auto-encoder is presented to extract data features and generate event weights. Then, event weights are input into DQN (deep Q-network) to choose the action of the next step. Finally, simulation and verification experiments are conducted in a virtual obstacle environment built with an unreal engine and airsim. The experiment results show that the proposed algorithm is adaptable for drones to find the goal in unknown environments and can improve the rapidity of path planning compared with that of commonly used methods.

Dense GPU-enhanced surface reconstruction from stereo endoscopic images for intraoperative registration.

PURPOSE: In laparoscopic surgery, soft tissue deformations substantially change the surgical site, thus impeding the use of preoperative planning during intraoperative navigation. Extracting depth information from endoscopic images and building a surface model of the surgical field-of-view is one way to represent this constantly deforming environment. The information can then be used for intraoperative registration. Stereo reconstruction is a typical problem within computer vision. However, most of the available methods do not fulfill the specific requirements in a minimally invasive setting such as the need of real-time performance, the problem of view-dependent specular reflections and large curved areas with partly homogeneous or periodic textures and occlusions. METHODS: In this paper, the authors present an approach toward intraoperative surface reconstruction based on stereo endoscopic images. The authors describe our answer to this problem through correspondence analysis, disparity correction and refinement, 3D reconstruction, point cloud smoothing and meshing. Real-time performance is achieved by implementing the algorithms on the gpu. The authors also present a new hybrid cpu-gpu algorithm that unifies the advantages of the cpu and the gpu version. RESULTS: In a comprehensive evaluation using in vivo data, in silico data from the literature and virtual data from a newly developed simulation environment, the cpu, the gpu, and the hybrid cpu-gpu versions of the surface reconstruction are compared to a cpu and a gpu algorithm from the literature. The recommended approach toward intraoperative surface reconstruction can be conducted in real-time depending on the image resolution (20 fps for the gpu and 14fps for the hybrid cpu-gpu version on resolution of 640 × 480). It is robust to homogeneous regions without texture, large image changes, noise or errors from camera calibration, and it reconstructs the surface down to sub millimeter accuracy. In all the experiments within the simulation environment, the mean distance to ground truth data is between 0.05 and 0.6 mm for the hybrid cpu-gpu version. The hybrid cpu-gpu algorithm shows a much more superior performance than its cpu and gpu counterpart (mean distance reduction 26% and 45%, respectively, for the experiments in the simulation environment). CONCLUSIONS: The recommended approach for surface reconstruction is fast, robust, and accurate. It can represent changes in the intraoperative environment and can be used to adapt a preoperative model within the surgical site by registration of these two models.

Velocity navigator for motion compensated thermometry.

OBJECT: Proton resonance frequency shift thermometry is sensitive to breathing motion that leads to incorrect phase differences. In this work, a novel velocity-sensitive navigator technique for triggering MR thermometry image acquisition is presented. MATERIALS AND METHODS: A segmented echo planar imaging pulse sequence was modified for velocity-triggered temperature mapping. Trigger events were generated when the estimated velocity value was less than 0.2 cm/s during the slowdown phase in parallel to the velocity-encoding direction. To remove remaining high-frequency spikes from pulsation in real time, a Kalman filter was applied to the velocity navigator data. A phantom experiment with heating and an initial volunteer experiment without heating were performed to show the applicability of this technique. Additionally, a breath-hold experiment was conducted for comparison. RESULTS: A temperature rise of ΔT = +37.3°C was seen in the phantom experiment, and a root mean square error (RMSE) outside the heated region of 2.3°C could be obtained for periodic motion. In the volunteer experiment, a RMSE of 2.7°C/2.9°C (triggered vs. breath hold) was measured. CONCLUSION: A novel velocity navigator with Kalman filter postprocessing in real time significantly improves the temperature accuracy over non-triggered acquisitions and suggests being comparable to a breath-held acquisition. The proposed technique might be clinically applied for monitoring of thermal ablations in abdominal organs.

Web-based interactive volume rendering.

In this paper we present a web-based remote visualization system. The system makes use of video stream based techniques to reduce the network bandwidth requirements and is capable of performing interactive volume rendering on computed tomography data in real-time. The technique allows embedding interactive volume rendering into a website. The concrete contribution of this paper is twofold. First, we outline a Microsoft Silverlight based implementation of the prototype and describe the applied video encoding techniques. Furthermore we present experimental results that allow evaluating the system in terms of latency and image quality. In particular, we show that the additional delay of stream based remote visualization is very small if compared to picture based techniques.

Benchmarking Highly Parallel Hardware for Spiking Neural Networks in Robotics.

Animal brains still outperform even the most performant machines with significantly lower speed. Nonetheless, impressive progress has been made in robotics in the areas of vision, motion- and path planning in the last decades. Brain-inspired Spiking Neural Networks (SNN) and the parallel hardware necessary to exploit their full potential have promising features for robotic application. Besides the most obvious platform for deploying SNN, brain-inspired neuromorphic hardware, Graphical Processing Units (GPU) are well capable of parallel computing as well. Libraries for generating CUDA-optimized code, like GeNN and affordable embedded systems make them an attractive alternative due to their low price and availability. While a few performance tests exist, there has been a lack of benchmarks targeting robotic applications. We compare the performance of a neural Wavefront algorithm as a representative of use cases in robotics on different hardware suitable for running SNN simulations. The SNN used for this benchmark is modeled in the simulator-independent declarative language PyNN, which allows using the same model for different simulator backends. Our emphasis is the comparison between Nest, running on serial CPU, SpiNNaker, as a representative of neuromorphic hardware, and an implementation in GeNN. Beyond that, we also investigate the differences of GeNN deployed to different hardware. A comparison between the different simulators and hardware is performed with regard to total simulation time, average energy consumption per run, and the length of the resulting path. We hope that the insights gained about performance details of parallel hardware solutions contribute to developing more efficient SNN implementations for robotics.

Neuromorphic Stereo Vision: A Survey of Bio-Inspired Sensors and Algorithms.

Any visual sensor, whether artificial or biological, maps the 3D-world on a 2D-representation. The missing dimension is depth and most species use stereo vision to recover it. Stereo vision implies multiple perspectives and matching, hence it obtains depth from a pair of images. Algorithms for stereo vision are also used prosperously in robotics. Although, biological systems seem to compute disparities effortless, artificial methods suffer from high energy demands and latency. The crucial part is the correspondence problem; finding the matching points of two images. The development of event-based cameras, inspired by the retina, enables the exploitation of an additional physical constraint-time. Due to their asynchronous course of operation, considering the precise occurrence of spikes, Spiking Neural Networks take advantage of this constraint. In this work, we investigate sensors and algorithms for event-based stereo vision leading to more biologically plausible robots. Hereby, we focus mainly on binocular stereo vision.

Generating Pointing Motions for a Humanoid Robot by Combining Motor Primitives.

The human motor system is robust, adaptive and very flexible. The underlying principles of human motion provide inspiration for robotics. Pointing at different targets is a common robotics task, where insights about human motion can be applied. Traditionally in robotics, when a motion is generated it has to be validated so that the robot configurations involved are appropriate. The human brain, in contrast, uses the motor cortex to generate new motions reusing and combining existing knowledge before executing the motion. We propose a method to generate and control pointing motions for a robot using a biological inspired architecture implemented with spiking neural networks. We outline a simplified model of the human motor cortex that generates motions using motor primitives. The network learns a base motor primitive for pointing at a target in the center, and four correction primitives to point at targets up, down, left and right from the base primitive, respectively. The primitives are combined to reach different targets. We evaluate the performance of the network with a humanoid robot pointing at different targets marked on a plane. The network was able to combine one, two or three motor primitives at the same time to control the robot in real-time to reach a specific target. We work on extending this work from pointing to a given target to performing a grasping or tool manipulation task. This has many applications for engineering and industry involving real robots.

Embodied Synaptic Plasticity With Online Reinforcement Learning.

The endeavor to understand the brain involves multiple collaborating research fields. Classically, synaptic plasticity rules derived by theoretical neuroscientists are evaluated in isolation on pattern classification tasks. This contrasts with the biological brain which purpose is to control a body in closed-loop. This paper contributes to bringing the fields of computational neuroscience and robotics closer together by integrating open-source software components from these two fields. The resulting framework allows to evaluate the validity of biologically-plausibe plasticity models in closed-loop robotics environments. We demonstrate this framework to evaluate Synaptic Plasticity with Online REinforcement learning (SPORE), a reward-learning rule based on synaptic sampling, on two visuomotor tasks: reaching and lane following. We show that SPORE is capable of learning to perform policies within the course of simulated hours for both tasks. Provisional parameter explorations indicate that the learning rate and the temperature driving the stochastic processes that govern synaptic learning dynamics need to be regulated for performance improvements to be retained. We conclude by discussing the recent deep reinforcement learning techniques which would be beneficial to increase the functionality of SPORE on visuomotor tasks.

Concepts for visualization of multidirectional phase-contrast MRI of the heart and large thoracic vessels.

RATIONALE AND OBJECTIVES: Multidirectional phase-contrast magnetic resonance imaging allows the acquisition of time-resolved velocity fields (vectors) of cardiac and vascular blood flow. Its unique ability to provide vectorial flow information promises to give new insights into hemodynamic physiology. However, up to now appropriate and standardized procedures and software tools are missing to take advantage of all the information contained in the data. The objective of this work is to present a new versatile software tool and to demonstrate its practical value for the examination of multidirectional blood flow. MATERIALS AND METHODS: An exemplary selection of data sets from healthy volunteers, patients with cardiovascular pathologies, and healthy domestic pigs has been acquired using a phase-contrast magnetic resonance imaging sequence based on FLASH (fast low angle shot) that encodes velocity as field of three-dimensional vectors. For data processing, we have developed a software tool that integrates the whole workflow, including noise filtering, interactive visualization, and flow quantification. RESULTS: Using the software tool visualization of complex flow data is easily generated within 5 minutes; interactive exploration of the data is possible in real-time. Exemplary physiologic and pathologic flow patterns were visualized in an intuitive manner. The visual results suggest valuable diagnostic information; its significance, however, must be further evaluated together with the development of more specific data processing. CONCLUSIONS: Multidirectional phase-contrast magnetic resonance imaging is a valuable tool for assessment of cardiac and vascular hemodynamics. With the development of tools that offer standardized and thus comparable visualizations it may be integrated into the clinical routine in the near future.

[Biomechanical modeling of the cervical spine on the basis of tomographic data]. / Biomechanische Modellierung der Halswirbelsäule auf Basis tomografischer Bilddaten.

Navigation systems have been successfully established in spine surgery over the past years. Although there is rather good intraoperative support, preoperative planning still lacks a technologically mature software basis. In particular, the computation of the effect of intervention would be of great importance to the surgeon, which would allow him to choose the right strategy. For the patient, this would result in reduced trauma and optimized postoperative mobility. For a correct simulation of the behavior of the cervical spine, a patient individual finite element model is needed which would precisely reflect the geometrical and biomechanical properties of the spine. Therefore, a modeling process starting from tomographic data and ending in an applicable finite element model must be elaborated. A possible solution will be presented in this paper.

Real-time smoke and bleeding simulation in virtual surgery.

We present a particle-based smoke simulation and a particle-based fluid simulation in an interactive environment with rigid and deformable objects. Many smoke and fluid simulations offer high physical and visual accuracy, but the underlaying models are to complex to run in real-time while performing soft-tissue simulation, collision detection, and haptic device support at the same time. Our algorithms are based on simple models that allow the surgery simulation to run in real-time.

Scaling up liquid state machines to predict over address events from dynamic vision sensors.

Short-term visual prediction is important both in biology and robotics. It allows us to anticipate upcoming states of the environment and therefore plan more efficiently. In theoretical neuroscience, liquid state machines have been proposed as a biologically inspired method to perform asynchronous prediction without a model. However, they have so far only been demonstrated in simulation or small scale pre-processed camera images. In this paper, we use a liquid state machine to predict over the whole [Formula: see text] event stream provided by a real dynamic vision sensor (DVS, or silicon retina). Thanks to the event-based nature of the DVS, the liquid is constantly fed with data when an object is in motion, fully embracing the asynchronicity of spiking neural networks. We propose a smooth continuous representation of the event stream for the short-term visual prediction task. Moreover, compared to previous works (2002 Neural Comput. 2525 282-93 and Burgsteiner H et al 2007 Appl. Intell. 26 99-109), we scale the input dimensionality that the liquid operates on by two order of magnitudes. We also expose the current limits of our method by running experiments in a challenging environment where multiple objects are in motion. This paper is a step towards integrating biologically inspired algorithms derived in theoretical neuroscience to real world robotic setups. We believe that liquid state machines could complement current prediction algorithms used in robotics, especially when dealing with asynchronous sensors.

Projective biomechanical depth matching for soft tissue registration in laparoscopic surgery.

PURPOSE: A key component of computer- assisted surgery systems is the accurate and robust registration of preoperative planning data with intraoperative sensor data. In laparoscopic surgery, this image-based registration remains challenging due to soft tissue deformations. This paper presents a novel approach for biomechanical soft tissue registration of preoperative CT data with stereo endoscopic image data. METHODS: The proposed method consists of two registrations steps. First, we use a 3D surface mosaic from partial surfaces reconstructed from stereo endoscopic images to initially align the biomechanical model with the intraoperative position and shape of the organ. After this initialization, the biomechanical model is projected onto newly captured surfaces, resulting in displacement boundary conditions, which in turn are used to update the biomechanical model. RESULTS: The method is evaluated in silico, using a human liver model, and in vivo, using porcine data. The quantitative in silico data shows a stable behaviour of the biomechanical model and root-mean-square deviation of volume vertices of under 3 mm with adjusted biomechanical parameters. CONCLUSION: This work contributes a fully automatic featureless non-rigid registration approach. The results of the in silico and in vivo experiments suggest that our method is able to handle dynamic deformations during surgery. Additional experiments, especially regarding human tissue behaviour, are an important next step towards clinical applications.

Image-based laparoscopic bowel measurement.

PURPOSE: Minimally invasive interventions offer benefits for patients, while also entailing drawbacks for surgeons, such as the loss of depth perception. Thus estimating distances, which is of particular importance in gastric bypasses, becomes difficult. In this paper, we propose an approach based on stereo endoscopy that segments organs on-the-fly and measures along their surface during a minimally invasive interventions. Here, the application of determining the length of bowel segments during a laparoscopic bariatric gastric bypass is the main focus, but the proposed method can easily be used for other types of measurements, e.g., the size of a hernia. METHODS: As input, image pairs from a calibrated stereo endoscope are used. Our proposed method is then divided into three steps: First, we located structures of interest, such as organs and instruments, via random forest segmentation. Two modes of instrument detection are used. The first mode is based on an automatic segmentation, and the second mode uses input from the user. These regions are then reconstructed, and the distance along the surface of the reconstruction is measured. For measurement, we propose three methods. The first one is based on the direct distance of the instruments, while the second one finds the shortest path along a surface. The third method smooths the shortest path with a spline interpolation. Our measuring system is currently one shot, meaning a signal to begin a measurement is required. RESULTS: To evaluate our approach, data sets from multiple users were recorded during minimally invasive bowel measurements performed on phantoms and pigs. On the phantom data sets, the overall average error was [Formula: see text] on shorter pieces of bowel ([Formula: see text]5 cm) and [Formula: see text] on longer pieces ([Formula: see text]10 cm). On the porcine data sets, the average error was [Formula: see text]. CONCLUSION: We present and evaluate a novel approach that makes measuring on-the-fly during minimally invasive surgery possible. Furthermore, we compare different methods for determining the length of bowel segments. The only requirement for our approach is a calibrated stereo endoscope, thereby keeping the impact on the surgical workflow to a minimum.

Bridging the gap between formal and experience-based knowledge for context-aware laparoscopy.

PURPOSE: Computer assistance is increasingly common in surgery. However, the amount of information is bound to overload processing abilities of surgeons. We propose methods to recognize the current phase of a surgery for context-aware information filtering. The purpose is to select the most suitable subset of information for surgical situations which require special assistance. METHODS: We combine formal knowledge, represented by an ontology, and experience-based knowledge, represented by training samples, to recognize phases. For this purpose, we have developed two different methods. Firstly, we use formal knowledge about possible phase transitions to create a composition of random forests. Secondly, we propose a method based on cultural optimization to infer formal rules from experience to recognize phases. RESULTS: The proposed methods are compared with a purely formal knowledge-based approach using rules and a purely experience-based one using regular random forests. The comparative evaluation on laparoscopic pancreas resections and adrenalectomies employs a consistent set of quality criteria on clean and noisy input. The rule-based approaches proved best with noisefree data. The random forest-based ones were more robust in the presence of noise. CONCLUSION: Formal and experience-based knowledge can be successfully combined for robust phase recognition.

Toward cognitive pipelines of medical assistance algorithms.

PURPOSE: Assistance algorithms for medical tasks have great potential to support physicians with their daily work. However, medicine is also one of the most demanding domains for computer-based support systems, since medical assistance tasks are complex and the practical experience of the physician is crucial. Recent developments in the area of cognitive computing appear to be well suited to tackle medicine as an application domain. METHODS: We propose a system based on the idea of cognitive computing and consisting of auto-configurable medical assistance algorithms and their self-adapting combination. The system enables automatic execution of new algorithms, given they are made available as Medical Cognitive Apps and are registered in a central semantic repository. Learning components can be added to the system to optimize the results in the cases when numerous Medical Cognitive Apps are available for the same task. Our prototypical implementation is applied to the areas of surgical phase recognition based on sensor data and image progressing for tumor progression mappings. RESULTS: Our results suggest that such assistance algorithms can be automatically configured in execution pipelines, candidate results can be automatically scored and combined, and the system can learn from experience. Furthermore, our evaluation shows that the Medical Cognitive Apps are providing the correct results as they did for local execution and run in a reasonable amount of time. CONCLUSION: The proposed solution is applicable to a variety of medical use cases and effectively supports the automated and self-adaptive configuration of cognitive pipelines based on medical interpretation algorithms.

An automatic robust meshing algorithm for soft tissue modeling.

Soft tissue simulation with the finite element method is based on discretizations of anatomical objects. The structure of organs is mostly heterogeneous. To compute the deformation of composite materials correctly borders between adjacent tissues must be preserved. In this paper a new meshing algorithm is presented that fulfill this requirement. Additionally it works automatically in a straight forward manner to produce high quality 3d meshes from input triangle surfaces. The algorithm shows robust behavior and allows even to generate mesh from the most complex geometries. The processing time depends directly on the size of the input mesh and not on topology. This makes this algorithm calculable hence useful for time dependent cases, for example to adapt intraoperatively to modified operation status.

Intraoperative augmented reality: the surgeons view.

Augmented Reality (AR) is a promising tool for intraoperative visualization. Two different AR systems, one projector based, one based on see-through glasses were used on patients. The task was the transfer of preoperative planning into the intraoperative reality, or the visualization of space occupying lesions, respectively. The intraoperative application of both systems is discussed from the surgeons point of view.

Intraoperative on-the-fly organ-mosaicking for laparoscopic surgery.

The goal of computer-assisted surgery is to provide the surgeon with guidance during an intervention, e.g., using augmented reality. To display preoperative data, soft tissue deformations that occur during surgery have to be taken into consideration. Laparoscopic sensors, such as stereo endoscopes, can be used to create a three-dimensional reconstruction of stereo frames for registration. Due to the small field of view and the homogeneous structure of tissue, reconstructing just one frame, in general, will not provide enough detail to register preoperative data, since every frame only contains a part of an organ surface. A correct assignment to the preoperative model is possible only if the patch geometry can be unambiguously matched to a part of the preoperative surface. We propose and evaluate a system that combines multiple smaller reconstructions from different viewpoints to segment and reconstruct a large model of an organ. Using graphics processing unit-based methods, we achieved four frames per second. We evaluated the system with in silico, phantom, ex vivo, and in vivo (porcine) data, using different methods for estimating the camera pose (optical tracking, iterative closest point, and a combination). The results indicate that the proposed method is promising for on-the-fly organ reconstruction and registration.