First-in-human real-time AI-assisted instrument deocclusion during augmented reality robotic surgery.

The integration of Augmented Reality (AR) into daily surgical practice is withheld by the correct registration of pre-operative data. This includes intelligent 3D model superposition whilst simultaneously handling real and virtual occlusions caused by the AR overlay. Occlusions can negatively impact surgical safety and as such deteriorate rather than improve surgical care. Robotic surgery is particularly suited to tackle these integration challenges in a stepwise approach as the robotic console allows for different inputs to be displayed in parallel to the surgeon. Nevertheless, real-time de-occlusion requires extensive computational resources which further complicates clinical integration. This work tackles the problem of instrument occlusion and presents, to the authors' best knowledge, the first-in-human on edge deployment of a real-time binary segmentation pipeline during three robot-assisted surgeries: partial nephrectomy, migrated endovascular stent removal, and liver metastasectomy. To this end, a state-of-the-art real-time segmentation and 3D model pipeline was implemented and presented to the surgeon during live surgery. The pipeline allows real-time binary segmentation of 37 non-organic surgical items, which are never occluded during AR. The application features real-time manual 3D model manipulation for correct soft tissue alignment. The proposed pipeline can contribute towards surgical safety, ergonomics, and acceptance of AR in minimally invasive surgery.

Histostitcher™: An informatics software platform for reconstructing whole-mount prostate histology using the extensible imaging platform framework.

CONTEXT: Co-registration of ex-vivo histologic images with pre-operative imaging (e.g., magnetic resonance imaging [MRI]) can be used to align and map disease extent, and to identify quantitative imaging signatures. However, ex-vivo histology images are frequently sectioned into quarters prior to imaging. AIMS: This work presents Histostitcher™, a software system designed to create a pseudo whole mount histology section (WMHS) from a stitching of four individual histology quadrant images. MATERIALS AND METHODS: Histostitcher™ uses user-identified fiducials on the boundary of two quadrants to stitch such quadrants. An original prototype of Histostitcher™ was designed using the Matlab programming languages. However, clinical use was limited due to slow performance, computer memory constraints and an inefficient workflow. The latest version was created using the extensible imaging platform (XIP™) architecture in the C++ programming language. A fast, graphics processor unit renderer was designed to intelligently cache the visible parts of the histology quadrants and the workflow was significantly improved to allow modifying existing fiducials, fast transformations of the quadrants and saving/loading sessions. RESULTS: The new stitching platform yielded significantly more efficient workflow and reconstruction than the previous prototype. It was tested on a traditional desktop computer, a Windows 8 Surface Pro table device and a 27 inch multi-touch display, with little performance difference between the different devices. CONCLUSIONS: Histostitcher™ is a fast, efficient framework for reconstructing pseudo WMHS from individually imaged quadrants. The highly modular XIP™ framework was used to develop an intuitive interface and future work will entail mapping the disease extent from the pseudo WMHS onto pre-operative MRI.

Disocclusion-based 2D-3D registration for aortic interventions.

Occlusions introduced by medical instruments affect the accuracy and robustness of existing intensity-based medical image registration algorithms. In this paper, we present disocclusion-based 2D-3D registration handling occlusion and dissimilarity during registration. Therefore, we introduce two disocclusion techniques, Spline Interpolation and Stent-editing, and two robust similarity measures, Huber and Tukey Gradient Correlation. Our techniques are validated on synthetic and real interventional data and compared with well-known approaches. Results prove that an integration of disocclusion into the registration procedure yield higher accuracy and robustness. It is also shown that the robust measures have different effects depending on the type of occluding structure.

Interactive 3D visualization of a single-view X-ray image.

In this paper, we present an interactive X-Ray perceptual visualization technique (IXPV) to improve 3D perception in standard single-view X-Ray images. Based on a priori knowledge from CT data, we re-introduce lost depth information into the original single-view X-Ray image without jeopardizing information of the original X-Ray. We propose a novel approach that is suitable for correct fusion of intraoperative X-Ray and ultrasound, co-visualization of X-Ray and surgical tools, and for improving the 3D perception of standard radiographs. Phantom and animal cadaver datasets were used during experimentation to demonstrate the impact of our technique. Results from a questionnaire completed by 11 clinicians and computer scientists demonstrate the added value of introduced depth cues directly in an X-Ray image. In conclusion, we propose IXPV as a futuristic alternative to the standard radiographic image found in today's clinical setting.

Regurgitation quantification using 3D PISA in volume echocardiography.

We present the first system for measurement of proximal isovelocity surface area (PISA) on a 3D ultrasound acquisition using modified ultrasound hardware, volumetric image segmentation and a simple efficient workflow. Accurate measurement of the PISA in 3D flow through a valve is an emerging method for quantitatively assessing cardiac valve regurgitation and function. Current state of the art protocols for assessing regurgitant flow require laborious and time consuming user interaction with the data, where a precise execution is crucial for an accurate diagnosis. We propose a new improved 3D PISA workflow that is initialized interactively with two points, followed by fully automatic segmentation of the valve annulus and isovelocity surface area computation. Our system is first validated against several in vitro phantoms to verify the calculations of surface area, orifice area and regurgitant flow. Finally, we use our system to compare orifice area calculations obtained from in vivo patient imaging measurements to an independent measurement and then use our system to successfully classify patients into mild-moderate regurgitation and moderate-severe regurgitation categories.

Linear intensity-based image registration by Markov random fields and discrete optimization.

We propose a framework for intensity-based registration of images by linear transformations, based on a discrete Markov random field (MRF) formulation. Here, the challenge arises from the fact that optimizing the energy associated with this problem requires a high-order MRF model. Currently, methods for optimizing such high-order models are less general, easy to use, and efficient, than methods for the popular second-order models. Therefore, we propose an approximation to the original energy by an MRF with tractable second-order terms. The approximation at a certain point p in the parameter space is the normalized sum of evaluations of the original energy at projections of p to two-dimensional subspaces. We demonstrate the quality of the proposed approximation by computing the correlation with the original energy, and show that registration can be performed by discrete optimization of the approximated energy in an iteration loop. A search space refinement strategy is employed over iterations to achieve sub-pixel accuracy, while keeping the number of labels small for efficiency. The proposed framework can encode any similarity measure is robust to the settings of the internal parameters, and allows an intuitive control of the parameter ranges. We demonstrate the applicability of the framework by intensity-based registration, and 2D-3D registration of medical images. The evaluation is performed by random studies and real registration tasks. The tests indicate increased robustness and precision compared to corresponding standard optimization of the original energy, and demonstrate robustness to noise. Finally, the proposed framework allows the transfer of advances in MRF optimization to linear registration problems.

Visualization and GPU-accelerated simulation of medical ultrasound from CT images.

We present a fast GPU-based method for simulation of ultrasound images from volumetric CT scans and their visualization. The method uses a ray-based model of the ultrasound to generate view-dependent ultrasonic effects such as occlusions, large-scale reflections and attenuation combined with speckle patterns derived from pre-processing the CT image using a wave-based model of ultrasound propagation in soft tissue. The main applications of the method are ultrasound training and registration of ultrasound and CT images.

Multi-modal registration based ultrasound mosaicing.

Recent US systems allow the real-time acquisition of volume data, either by freehand 3D techniques or novel transducer hardware. However, the acquisition of large volumes is limited by the field of view of the US transducer and anatomical scanning windows into the patient. Mosaicing of several 3D US scans has been proposed to generate large US volumes. Still, US imaging specific characteristics and artifacts make it challenging to create high quality mosaics. For many clinical cases, especially interventions, additional high quality CT data is available. In this paper we present a novel multi-variate, multi-modal 3D US registration and mosaicing approach, which reduces the effects of ultrasound imaging artifacts on mosaic quality, by incorporating information from co-registered CT.

Dynamic model-driven quantitative and visual evaluation of the aortic valve from 4D CT.

Aortic valve disease is an important cardio-vascular disorder, which affects 2.5% of the global population and often requires elaborate clinical management. Experts agree that visual and quantitative evaluation of the valve, crucial throughout the clinical workflow, is currently limited to 2D imaging which can potentially yield inaccurate measurements. In this paper, we propose a novel approach for morphological and functional quantification of the aortic valve based on a 4D model estimated from computed tomography data. A physiological model of the aortic valve, capable to express large shape variations, is generated using parametric splines together with anatomically-driven topological and geometrical constraints. Recent advances in discriminative learning and incremental searching methods allow rapid estimation of the model parameters from 4D Cardiac CT specifically for each patient. The proposed approach enables precise valve evaluation with model-based dynamic measurements and advanced visualization. Extensive experiments and initial clinical validation demonstrate the efficiency and accuracy of the proposed approach. To the best of our knowledge this is the first time such a patient specific 4D aortic valve model is proposed.

Simulation and fully automatic multimodal registration of medical ultrasound.

The fusion of 3D freehand ultrasound with CT and CTA has benefits for a variety of clinical applications, however a lot of manual work is usually required for correct registration. We developed new methods that allow one to simulate medical ultrasound from CT in real-time, reproducing the majority of ultrasonic imaging effects. The second novelty is a robust similarity measure that assesses the correlation of a combination of multiple signals extracted from CT with ultrasound, without knowing the influence of each signal. This serves as the foundation of a fully automatic registration, which aligns a freehand ultrasound sweep with the corresponding 3D modality using a rigid or an affine transformation model, without any manual interaction. We also present the used initialization, global and local parameter optimization schemes, and validation on abdominal CTA and ultrasound imaging of 10 patients.