E-DQN-Based Path Planning Method for Drones in Airsim Simulator under Unknown Environment.

To improve the rapidity of path planning for drones in unknown environments, a new bio-inspired path planning method using E-DQN (event-based deep Q-network), referring to introducing event stream to reinforcement learning network, is proposed. Firstly, event data are collected through an airsim simulator for environmental perception, and an auto-encoder is presented to extract data features and generate event weights. Then, event weights are input into DQN (deep Q-network) to choose the action of the next step. Finally, simulation and verification experiments are conducted in a virtual obstacle environment built with an unreal engine and airsim. The experiment results show that the proposed algorithm is adaptable for drones to find the goal in unknown environments and can improve the rapidity of path planning compared with that of commonly used methods.

Benchmarking Highly Parallel Hardware for Spiking Neural Networks in Robotics.

Animal brains still outperform even the most performant machines with significantly lower speed. Nonetheless, impressive progress has been made in robotics in the areas of vision, motion- and path planning in the last decades. Brain-inspired Spiking Neural Networks (SNN) and the parallel hardware necessary to exploit their full potential have promising features for robotic application. Besides the most obvious platform for deploying SNN, brain-inspired neuromorphic hardware, Graphical Processing Units (GPU) are well capable of parallel computing as well. Libraries for generating CUDA-optimized code, like GeNN and affordable embedded systems make them an attractive alternative due to their low price and availability. While a few performance tests exist, there has been a lack of benchmarks targeting robotic applications. We compare the performance of a neural Wavefront algorithm as a representative of use cases in robotics on different hardware suitable for running SNN simulations. The SNN used for this benchmark is modeled in the simulator-independent declarative language PyNN, which allows using the same model for different simulator backends. Our emphasis is the comparison between Nest, running on serial CPU, SpiNNaker, as a representative of neuromorphic hardware, and an implementation in GeNN. Beyond that, we also investigate the differences of GeNN deployed to different hardware. A comparison between the different simulators and hardware is performed with regard to total simulation time, average energy consumption per run, and the length of the resulting path. We hope that the insights gained about performance details of parallel hardware solutions contribute to developing more efficient SNN implementations for robotics.

Embodied Synaptic Plasticity With Online Reinforcement Learning.

The endeavor to understand the brain involves multiple collaborating research fields. Classically, synaptic plasticity rules derived by theoretical neuroscientists are evaluated in isolation on pattern classification tasks. This contrasts with the biological brain which purpose is to control a body in closed-loop. This paper contributes to bringing the fields of computational neuroscience and robotics closer together by integrating open-source software components from these two fields. The resulting framework allows to evaluate the validity of biologically-plausibe plasticity models in closed-loop robotics environments. We demonstrate this framework to evaluate Synaptic Plasticity with Online REinforcement learning (SPORE), a reward-learning rule based on synaptic sampling, on two visuomotor tasks: reaching and lane following. We show that SPORE is capable of learning to perform policies within the course of simulated hours for both tasks. Provisional parameter explorations indicate that the learning rate and the temperature driving the stochastic processes that govern synaptic learning dynamics need to be regulated for performance improvements to be retained. We conclude by discussing the recent deep reinforcement learning techniques which would be beneficial to increase the functionality of SPORE on visuomotor tasks.

Generating Pointing Motions for a Humanoid Robot by Combining Motor Primitives.

The human motor system is robust, adaptive and very flexible. The underlying principles of human motion provide inspiration for robotics. Pointing at different targets is a common robotics task, where insights about human motion can be applied. Traditionally in robotics, when a motion is generated it has to be validated so that the robot configurations involved are appropriate. The human brain, in contrast, uses the motor cortex to generate new motions reusing and combining existing knowledge before executing the motion. We propose a method to generate and control pointing motions for a robot using a biological inspired architecture implemented with spiking neural networks. We outline a simplified model of the human motor cortex that generates motions using motor primitives. The network learns a base motor primitive for pointing at a target in the center, and four correction primitives to point at targets up, down, left and right from the base primitive, respectively. The primitives are combined to reach different targets. We evaluate the performance of the network with a humanoid robot pointing at different targets marked on a plane. The network was able to combine one, two or three motor primitives at the same time to control the robot in real-time to reach a specific target. We work on extending this work from pointing to a given target to performing a grasping or tool manipulation task. This has many applications for engineering and industry involving real robots.

Neuromorphic Stereo Vision: A Survey of Bio-Inspired Sensors and Algorithms.

Any visual sensor, whether artificial or biological, maps the 3D-world on a 2D-representation. The missing dimension is depth and most species use stereo vision to recover it. Stereo vision implies multiple perspectives and matching, hence it obtains depth from a pair of images. Algorithms for stereo vision are also used prosperously in robotics. Although, biological systems seem to compute disparities effortless, artificial methods suffer from high energy demands and latency. The crucial part is the correspondence problem; finding the matching points of two images. The development of event-based cameras, inspired by the retina, enables the exploitation of an additional physical constraint-time. Due to their asynchronous course of operation, considering the precise occurrence of spikes, Spiking Neural Networks take advantage of this constraint. In this work, we investigate sensors and algorithms for event-based stereo vision leading to more biologically plausible robots. Hereby, we focus mainly on binocular stereo vision.

Computer-assisted 3D bowel length measurement for quantitative laparoscopy.

BACKGROUND: This study aimed at developing and evaluating a tool for computer-assisted 3D bowel length measurement (BMS) to improve objective measurement in minimally invasive surgery. Standardization and quality of surgery as well as its documentation are currently limited by lack of objective intraoperative measurements. To solve this problem, we developed BMS as a clinical application of Quantitative Laparoscopy (QL). METHODS: BMS processes images from a conventional 3D laparoscope. Computer vision algorithms are used to measure the distance between laparoscopic instruments along a 3D reconstruction of the bowel surface. Preclinical evaluation was performed in phantom, ex vivo porcine, and in vivo porcine models. A bowel length of 70 cm was measured with BMS and compared to a manually obtained ground truth. Afterwards 70 cm of bowel (ground truth) was measured and compared to BMS. RESULTS: Ground truth was 66.1 ± 2.7 cm (relative error + 5.8%) in phantom, 65.8 ± 2.5 cm (relative error + 6.4%) in ex vivo, and 67.5 ± 6.6 cm (relative error + 3.7%) in in vivo porcine evaluation when 70 cm was measured with BMS. Using 70 cm of bowel, BMS measured 75.0 ± 2.9 cm (relative error + 7.2%) in phantom and 74.4 ± 2.8 cm (relative error + 6.3%) in ex vivo porcine evaluation. After thorough preclinical evaluation, BMS was successfully used in a patient undergoing laparoscopic Roux-en-Y gastric bypass for morbid obesity. CONCLUSIONS: QL using BMS was shown to be feasible and was successfully translated from studies on phantom, ex vivo, and in vivo porcine bowel to a clinical feasibility study.

Scaling up liquid state machines to predict over address events from dynamic vision sensors.

Short-term visual prediction is important both in biology and robotics. It allows us to anticipate upcoming states of the environment and therefore plan more efficiently. In theoretical neuroscience, liquid state machines have been proposed as a biologically inspired method to perform asynchronous prediction without a model. However, they have so far only been demonstrated in simulation or small scale pre-processed camera images. In this paper, we use a liquid state machine to predict over the whole [Formula: see text] event stream provided by a real dynamic vision sensor (DVS, or silicon retina). Thanks to the event-based nature of the DVS, the liquid is constantly fed with data when an object is in motion, fully embracing the asynchronicity of spiking neural networks. We propose a smooth continuous representation of the event stream for the short-term visual prediction task. Moreover, compared to previous works (2002 Neural Comput. 2525 282-93 and Burgsteiner H et al 2007 Appl. Intell. 26 99-109), we scale the input dimensionality that the liquid operates on by two order of magnitudes. We also expose the current limits of our method by running experiments in a challenging environment where multiple objects are in motion. This paper is a step towards integrating biologically inspired algorithms derived in theoretical neuroscience to real world robotic setups. We believe that liquid state machines could complement current prediction algorithms used in robotics, especially when dealing with asynchronous sensors.

Projective biomechanical depth matching for soft tissue registration in laparoscopic surgery.

PURPOSE: A key component of computer- assisted surgery systems is the accurate and robust registration of preoperative planning data with intraoperative sensor data. In laparoscopic surgery, this image-based registration remains challenging due to soft tissue deformations. This paper presents a novel approach for biomechanical soft tissue registration of preoperative CT data with stereo endoscopic image data. METHODS: The proposed method consists of two registrations steps. First, we use a 3D surface mosaic from partial surfaces reconstructed from stereo endoscopic images to initially align the biomechanical model with the intraoperative position and shape of the organ. After this initialization, the biomechanical model is projected onto newly captured surfaces, resulting in displacement boundary conditions, which in turn are used to update the biomechanical model. RESULTS: The method is evaluated in silico, using a human liver model, and in vivo, using porcine data. The quantitative in silico data shows a stable behaviour of the biomechanical model and root-mean-square deviation of volume vertices of under 3 mm with adjusted biomechanical parameters. CONCLUSION: This work contributes a fully automatic featureless non-rigid registration approach. The results of the in silico and in vivo experiments suggest that our method is able to handle dynamic deformations during surgery. Additional experiments, especially regarding human tissue behaviour, are an important next step towards clinical applications.

Bridging the gap between formal and experience-based knowledge for context-aware laparoscopy.

PURPOSE: Computer assistance is increasingly common in surgery. However, the amount of information is bound to overload processing abilities of surgeons. We propose methods to recognize the current phase of a surgery for context-aware information filtering. The purpose is to select the most suitable subset of information for surgical situations which require special assistance. METHODS: We combine formal knowledge, represented by an ontology, and experience-based knowledge, represented by training samples, to recognize phases. For this purpose, we have developed two different methods. Firstly, we use formal knowledge about possible phase transitions to create a composition of random forests. Secondly, we propose a method based on cultural optimization to infer formal rules from experience to recognize phases. RESULTS: The proposed methods are compared with a purely formal knowledge-based approach using rules and a purely experience-based one using regular random forests. The comparative evaluation on laparoscopic pancreas resections and adrenalectomies employs a consistent set of quality criteria on clean and noisy input. The rule-based approaches proved best with noisefree data. The random forest-based ones were more robust in the presence of noise. CONCLUSION: Formal and experience-based knowledge can be successfully combined for robust phase recognition.

Toward cognitive pipelines of medical assistance algorithms.

PURPOSE: Assistance algorithms for medical tasks have great potential to support physicians with their daily work. However, medicine is also one of the most demanding domains for computer-based support systems, since medical assistance tasks are complex and the practical experience of the physician is crucial. Recent developments in the area of cognitive computing appear to be well suited to tackle medicine as an application domain. METHODS: We propose a system based on the idea of cognitive computing and consisting of auto-configurable medical assistance algorithms and their self-adapting combination. The system enables automatic execution of new algorithms, given they are made available as Medical Cognitive Apps and are registered in a central semantic repository. Learning components can be added to the system to optimize the results in the cases when numerous Medical Cognitive Apps are available for the same task. Our prototypical implementation is applied to the areas of surgical phase recognition based on sensor data and image progressing for tumor progression mappings. RESULTS: Our results suggest that such assistance algorithms can be automatically configured in execution pipelines, candidate results can be automatically scored and combined, and the system can learn from experience. Furthermore, our evaluation shows that the Medical Cognitive Apps are providing the correct results as they did for local execution and run in a reasonable amount of time. CONCLUSION: The proposed solution is applicable to a variety of medical use cases and effectively supports the automated and self-adaptive configuration of cognitive pipelines based on medical interpretation algorithms.

Image-based laparoscopic bowel measurement.

PURPOSE: Minimally invasive interventions offer benefits for patients, while also entailing drawbacks for surgeons, such as the loss of depth perception. Thus estimating distances, which is of particular importance in gastric bypasses, becomes difficult. In this paper, we propose an approach based on stereo endoscopy that segments organs on-the-fly and measures along their surface during a minimally invasive interventions. Here, the application of determining the length of bowel segments during a laparoscopic bariatric gastric bypass is the main focus, but the proposed method can easily be used for other types of measurements, e.g., the size of a hernia. METHODS: As input, image pairs from a calibrated stereo endoscope are used. Our proposed method is then divided into three steps: First, we located structures of interest, such as organs and instruments, via random forest segmentation. Two modes of instrument detection are used. The first mode is based on an automatic segmentation, and the second mode uses input from the user. These regions are then reconstructed, and the distance along the surface of the reconstruction is measured. For measurement, we propose three methods. The first one is based on the direct distance of the instruments, while the second one finds the shortest path along a surface. The third method smooths the shortest path with a spline interpolation. Our measuring system is currently one shot, meaning a signal to begin a measurement is required. RESULTS: To evaluate our approach, data sets from multiple users were recorded during minimally invasive bowel measurements performed on phantoms and pigs. On the phantom data sets, the overall average error was [Formula: see text] on shorter pieces of bowel ([Formula: see text]5 cm) and [Formula: see text] on longer pieces ([Formula: see text]10 cm). On the porcine data sets, the average error was [Formula: see text]. CONCLUSION: We present and evaluate a novel approach that makes measuring on-the-fly during minimally invasive surgery possible. Furthermore, we compare different methods for determining the length of bowel segments. The only requirement for our approach is a calibrated stereo endoscope, thereby keeping the impact on the surgical workflow to a minimum.

Intraoperative on-the-fly organ-mosaicking for laparoscopic surgery.

The goal of computer-assisted surgery is to provide the surgeon with guidance during an intervention, e.g., using augmented reality. To display preoperative data, soft tissue deformations that occur during surgery have to be taken into consideration. Laparoscopic sensors, such as stereo endoscopes, can be used to create a three-dimensional reconstruction of stereo frames for registration. Due to the small field of view and the homogeneous structure of tissue, reconstructing just one frame, in general, will not provide enough detail to register preoperative data, since every frame only contains a part of an organ surface. A correct assignment to the preoperative model is possible only if the patch geometry can be unambiguously matched to a part of the preoperative surface. We propose and evaluate a system that combines multiple smaller reconstructions from different viewpoints to segment and reconstruct a large model of an organ. Using graphics processing unit-based methods, we achieved four frames per second. We evaluated the system with in silico, phantom, ex vivo, and in vivo (porcine) data, using different methods for estimating the camera pose (optical tracking, iterative closest point, and a combination). The results indicate that the proposed method is promising for on-the-fly organ reconstruction and registration.

LapOntoSPM: an ontology for laparoscopic surgeries and its application to surgical phase recognition.

PURPOSE: The rise of intraoperative information threatens to outpace our abilities to process it. Context-aware systems, filtering information to automatically adapt to the current needs of the surgeon, are necessary to fully profit from computerized surgery. To attain context awareness, representation of medical knowledge is crucial. However, most existing systems do not represent knowledge in a reusable way, hindering also reuse of data. Our purpose is therefore to make our computational models of medical knowledge sharable, extensible and interoperational with established knowledge representations in the form of the LapOntoSPM ontology. To show its usefulness, we apply it to situation interpretation, i.e., the recognition of surgical phases based on surgical activities. METHODS: Considering best practices in ontology engineering and building on our ontology for laparoscopy, we formalized the workflow of laparoscopic adrenalectomies, cholecystectomies and pancreatic resections in the framework of OntoSPM, a new standard for surgical process models. Furthermore, we provide a rule-based situation interpretation algorithm based on SQWRL to recognize surgical phases using the ontology. RESULTS: The system was evaluated on ground-truth data from 19 manually annotated surgeries. The aim was to show that the phase recognition capabilities are equal to a specialized solution. The recognition rates of the new system were equal to the specialized one. However, the time needed to interpret a situation rose from 0.5 to 1.8 s on average which is still viable for practical application. CONCLUSION: We successfully integrated medical knowledge for laparoscopic surgeries into OntoSPM, facilitating knowledge and data sharing. This is especially important for reproducibility of results and unbiased comparison of recognition algorithms. The associated recognition algorithm was adapted to the new representation without any loss of classification power. The work is an important step to standardized knowledge and data representation in the field on context awareness and thus toward unified benchmark data sets.

Noninvasive 4D pressure difference mapping derived from 4D flow MRI in patients with repaired aortic coarctation: comparison with young healthy volunteers.

To assess spatial and temporal pressure characteristics in patients with repaired aortic coarctation compared to young healthy volunteers using time-resolved velocity-encoded three-dimensional phase-contrast magnetic resonance imaging (4D flow MRI) and derived 4D pressure difference maps. After in vitro validation against invasive catheterization as gold standard, 4D flow MRI of the thoracic aorta was performed at 1.5T in 13 consecutive patients after aortic coarctation repair without recoarctation and 13 healthy volunteers. Using in-house developed processing software, 4D pressure difference maps were computed based on the Navier-Stokes equation. Pressure difference amplitudes, maximum slope of pressure amplitudes and spatial pressure range at mid systole were retrospectively measured by three readers, and twice by one reader to assess inter- and intraobserver agreement. In vitro, pressure differences derived from 4D flow MRI showed excellent agreement to invasive catheter measurements. In vivo, pressure difference amplitudes, maximum slope of pressure difference amplitudes and spatial pressure range at mid systole were significantly increased in patients compared to volunteers in the aortic arch, the proximal descending and the distal descending thoracic aorta (p < 0.05). Greatest differences occurred in the proximal descending aorta with values of the three parameters for patients versus volunteers being 19.7 ± 7.5 versus 10.0 ± 2.0 (p < 0.001), 10.9 ± 10.4 versus 1.9 ± 0.4 (p = 0.002), and 8.7 ± 6.3 versus 1.6 ± 0.9 (p < 0.001). Inter- and intraobserver agreements were excellent (p < 0.001). Noninvasive 4D pressure difference mapping derived from 4D flow MRI enables detection of altered intraluminal aortic pressures and showed significant spatial and temporal changes in patients with repaired aortic coarctation.

Physics-based shape matching for intraoperative image guidance.

PURPOSE: Soft-tissue deformations can severely degrade the validity of preoperative planning data during computer assisted interventions. Intraoperative imaging such as stereo endoscopic, time-of-flight or, laser range scanner data can be used to compensate these movements. In this context, the intraoperative surface has to be matched to the preoperative model. The shape matching is especially challenging in the intraoperative setting due to noisy sensor data, only partially visible surfaces, ambiguous shape descriptors, and real-time requirements. METHODS: A novel physics-based shape matching (PBSM) approach to register intraoperatively acquired surface meshes to preoperative planning data is proposed. The key idea of the method is to describe the nonrigid registration process as an electrostatic-elastic problem, where an elastic body (preoperative model) that is electrically charged slides into an oppositely charged rigid shape (intraoperative surface). It is shown that the corresponding energy functional can be efficiently solved using the finite element (FE) method. It is also demonstrated how PBSM can be combined with rigid registration schemes for robust nonrigid registration of arbitrarily aligned surfaces. Furthermore, it is shown how the approach can be combined with landmark based methods and outline its application to image guidance in laparoscopic interventions. RESULTS: A profound analysis of the PBSM scheme based on in silico and phantom data is presented. Simulation studies on several liver models show that the approach is robust to the initial rigid registration and to parameter variations. The studies also reveal that the method achieves submillimeter registration accuracy (mean error between 0.32 and 0.46 mm). An unoptimized, single core implementation of the approach achieves near real-time performance (2 TPS, 7-19 s total registration time). It outperforms established methods in terms of speed and accuracy. Furthermore, it is shown that the method is able to accurately match partial surfaces. Finally, a phantom experiment demonstrates how the method can be combined with stereo endoscopic imaging to provide nonrigid registration during laparoscopic interventions. CONCLUSIONS: The PBSM approach for surface matching is fast, robust, and accurate. As the technique is based on a preoperative volumetric FE model, it naturally recovers the position of volumetric structures (e.g., tumors and vessels). It cannot only be used to recover soft-tissue deformations from intraoperative surface models but can also be combined with landmark data from volumetric imaging. In addition to applications in laparoscopic surgery, the method might prove useful in other areas that require soft-tissue registration from sparse intraoperative sensor data (e.g., radiation therapy).

Noninvasive pressure difference mapping derived from 4D flow MRI in patients with unrepaired and repaired aortic coarctation.

PURPOSE: To develop a method for computing and visualizing pressure differences derived from time-resolved velocity-encoded three-dimensional phase-contrast magnetic resonance imaging (4D flow MRI) and to compare pressure difference maps of patients with unrepaired and repaired aortic coarctation to young healthy volunteers. METHODS: 4D flow MRI data of four patients with aortic coarctation either before or after repair (mean age 17 years, age range 3-28, one female, three males) and four young healthy volunteers without history of cardiovascular disease (mean age 24 years, age range 20-27, one female, three males) was acquired using a 1.5-T clinical MR scanner. Image analysis was performed with in-house developed image processing software. Relative pressures were computed based on the Navier-Stokes equation. RESULTS: A standardized method for intuitive visualization of pressure difference maps was developed and successfully applied to all included patients and volunteers. Young healthy volunteers exhibited smooth and regular distribution of relative pressures in the thoracic aorta at mid systole with very similar distribution in all analyzed volunteers. Patients demonstrated disturbed pressures compared to volunteers. Changes included a pressure drop at the aortic isthmus in all patients, increased relative pressures in the aortic arch in patients with residual narrowing after repair, and increased relative pressures in the descending aorta in a patient after patch aortoplasty. CONCLUSIONS: Pressure difference maps derived from 4D flow MRI can depict alterations of spatial pressure distribution in patients with repaired and unrepaired aortic coarctation. The technique might allow identifying pathophysiological conditions underlying complications after aortic coarctation repair.

The medical simulation markup language - simplifying the biomechanical modeling workflow.

Modeling and simulation of the human body by means of continuum mechanics has become an important tool in diagnostics, computer-assisted interventions and training. This modeling approach seeks to construct patient-specific biomechanical models from tomographic data. Usually many different tools such as segmentation and meshing algorithms are involved in this workflow. In this paper we present a generalized and flexible description for biomechanical models. The unique feature of the new modeling language is that it not only describes the final biomechanical simulation, but also the workflow how the biomechanical model is constructed from tomographic data. In this way, the MSML can act as a middleware between all tools used in the modeling pipeline. The MSML thus greatly facilitates the prototyping of medical simulation workflows for clinical and research purposes. In this paper, we not only detail the XML-based modeling scheme, but also present a concrete implementation. Different examples highlight the flexibility, robustness and ease-of-use of the approach.

Towards open-source, low-cost haptics for surgery simulation.

In minimally invasive surgery (MIS), virtual reality (VR) training systems have become a promising education tool. However, the adoption of these systems in research and clinical settings is still limited by the high costs of dedicated haptics hardware for MIS. In this paper, we present ongoing research towards an open-source, low-cost haptic interface for MIS simulation. We demonstrate the basic mechanical design of the device, the sensor setup as well as its software integration.

Estimation of aortic pressure waveforms from 4D phase-contrast MRI.

Several approaches for the non-invasive MRI-based measurement of the aortic pressure waveform over the heart cycle have been proposed in the last years. These methods are normally based on time-resolved, two-dimensional phase-contrast sequences with uni-directionally encoded velocities (2D PC-MRI). In contrast, three-dimensional acquisitions with tridirectional velocity encoding (4D PC-MRI) have been shown to be a suitable data source for detailed investigations of blood flow and spatial blood pressure maps. In order to avoid additional MR acquisitions, it would be advantageous if the aortic pressure waveform could also be computed from this particular form of MRI. Therefore, we propose an approach for the computation of the aortic pressure waveform which can be completely performed using 4D PC-MRI. After the application of a segmentation algorithm, the approach automatically computes the aortic pressure waveform without any manual steps. We show that our method agrees well with catheter measurements in an experimental phantom setup and produces physiologically realistic results in three healthy volunteers.