Robust augmented reality registration method for localization of solid organs' tumors using CT-derived virtual biomechanical model and fluorescent fiducials.

BACKGROUND: Augmented reality (AR) is the fusion of computer-generated and real-time images. AR can be used in surgery as a navigation tool, by creating a patient-specific virtual model through 3D software manipulation of DICOM imaging (e.g., CT scan). The virtual model can be superimposed to real-time images enabling transparency visualization of internal anatomy and accurate localization of tumors. However, the 3D model is rigid and does not take into account inner structures' deformations. We present a concept of automated AR registration, while the organs undergo deformation during surgical manipulation, based on finite element modeling (FEM) coupled with optical imaging of fluorescent surface fiducials. METHODS: Two 10 × 1 mm wires (pseudo-tumors) and six 10 × 0.9 mm fluorescent fiducials were placed in ex vivo porcine kidneys (n = 10). Biomechanical FEM-based models were generated from CT scan. Kidneys were deformed and the shape changes were identified by tracking the fiducials, using a near-infrared optical system. The changes were registered automatically with the virtual model, which was deformed accordingly. Accuracy of prediction of pseudo-tumors' location was evaluated with a CT scan in the deformed status (ground truth). In vivo: fluorescent fiducials were inserted under ultrasound guidance in the kidney of one pig, followed by a CT scan. The FEM-based virtual model was superimposed on laparoscopic images by automatic registration of the fiducials. RESULTS: Biomechanical models were successfully generated and accurately superimposed on optical images. The mean measured distance between the estimated tumor by biomechanical propagation and the scanned tumor (ground truth) was 0.84 ± 0.42 mm. All fiducials were successfully placed in in vivo kidney and well visualized in near-infrared mode enabling accurate automatic registration of the virtual model on the laparoscopic images. CONCLUSIONS: Our preliminary experiments showed the potential of a biomechanical model with fluorescent fiducials to propagate the deformation of solid organs' surface to their inner structures including tumors with good accuracy and automatized robust tracking.

Validation method of a middle ear mechanical model to develop a surgical simulator.

Ossicular surgery requires a high dexterity for the manipulation of the fragile and small middle ear components. Currently, the only efficient technique for training residents in otological surgery is through the use of temporal bone specimens, where any existing surgical simulator does not provide useful feedback. The objective of this study was to develop a finite-element model of the human ossicular chain dedicated to surgical simulation and to propose a method to evaluate its behavior. A model was developed based on human middle ear micromagnetic resonance imaging. The mechanical parameters were determined according to published data. To assess its performance, the middle ear transfer function was analyzed. The robustness of our model and the influence of different middle ear components were also evaluated at low frequency by static force pressure simulations. The mechanical behavior of our model in nominal and pathological conditions was in good agreement with published human temporal bone measurements. We showed that the cochlea influences the transfer function only at high frequency and could be omitted from a surgical simulator. In addition, surgeons were able to manipulate the validated middle ear model with a real-time haptic feedback. The computational efficiency of our approach allowed real-time interactions, making it suitable for use in a training simulator.

A zero-shot reinforcement learning strategy for autonomous guidewire navigation.

PURPOSE: The treatment of cardiovascular diseases requires complex and challenging navigation of a guidewire and catheter. This often leads to lengthy interventions during which the patient and clinician are exposed to X-ray radiation. Deep reinforcement learning approaches have shown promise in learning this task and may be the key to automating catheter navigation during robotized interventions. Yet, existing training methods show limited capabilities at generalizing to unseen vascular anatomies, requiring to be retrained each time the geometry changes. METHODS: In this paper, we propose a zero-shot learning strategy for three-dimensional autonomous endovascular navigation. Using a very small training set of branching patterns, our reinforcement learning algorithm is able to learn a control that can then be applied to unseen vascular anatomies without retraining. RESULTS: We demonstrate our method on 4 different vascular systems, with an average success rate of 95% at reaching random targets on these anatomies. Our strategy is also computationally efficient, allowing the training of our controller to be performed in only 2 h. CONCLUSION: Our training method proved its ability to navigate unseen geometries with different characteristics, thanks to a nearly shape-invariant observation space.

Deformation-based augmented reality for hepatic surgery.

In this paper we introduce a method for augmenting the laparoscopic view during hepatic tumor resection. Using augmented reality techniques, vessels, tumors and cutting planes computed from pre-operative data can be overlaid onto the laparoscopic video. Compared to current techniques, which are limited to a rigid registration of the pre-operative liver anatomy with the intra-operative image, we propose a real-time, physics-based, non-rigid registration. The main strength of our approach is that the deformable model can also be used to regularize the data extracted from the computer vision algorithms. We show preliminary results on a video sequence which clearly highlights the interest of using physics-based model for elastic registration.

Hysteroscopic placement of tubal sterilization implants: virtual reality simulator training.

STUDY OBJECTIVE: To assess face and construct validity of a new virtual reality (VR) training simulator for hysteroscopic placement of tubal sterilization implants. DESIGN: Nonrandomized, controlled trial comparing responses and performance of novices and experts on the simulator. DESIGN CLASSIFICATION: Canadian task force II-1. SETTING: Forty-six gynecologists were personally invited or recruited at the 33rd Conference of the French National College of Gynecologists and Obstetricians (CNGOF) from December 9 to 12, 2009, grouped as 20 experts and 26 novices. They all performed the defined sequence of virtual procedures on the simulator (case 1 for familiarization, case 4 for study assessment) and finally completed the study questionnaire. MEASUREMENTS AND MAIN RESULTS: Responses to realism, educational potential, and general opinion were excellent, proving face validity. Significant differences between novices and experts were assessed for 7 of the 15 metrics analyzed, proving construct validity. CONCLUSIONS: We established face and construct validity for EssureSim, an educational VR simulator for hysteroscopic tubal sterilization implant placement. The next steps are to investigate convergent and predictive validity to affirm the real capacity of transferring the skills learned on the training simulator to the patient in the operating room.

Augmented Reality during Open Liver Surgery Using a Markerless Non-rigid Registration System.

INTRODUCTION: Intraoperative navigation during liver resection remains difficult and requires high radiologic skills because liver anatomy is complex and patient-specific. Augmented reality (AR) during open liver surgery could be helpful to guide hepatectomies and optimize resection margins but faces many challenges when large parenchymal deformations take place. We aimed to experiment a new vision-based AR to assess its clinical feasibility and anatomical accuracy. PATIENTS AND METHODS: Based on preoperative CT scan 3-D segmentations, we applied a non-rigid registration method, integrating a physics-based elastic model of the liver, computed in real time using an efficient finite element method. To fit the actual deformations, the model was driven by data provided by a single RGB-D camera. Five livers were considered in this experiment. In vivo AR was performed during hepatectomy (n = 4), with manual handling of the livers resulting in large realistic deformations. Ex vivo experiment (n = 1) consisted in repeated CT scans of explanted whole organ carrying internal metallic landmarks, in fixed deformations, and allowed us to analyze our estimated deformations and quantify spatial errors. RESULTS: In vivo AR tests were successfully achieved in all patients with a fast and agile setup installation (< 10 min) and real-time overlay of the virtual anatomy onto the surgical field displayed on an external screen. In addition, an ex vivo quantification demonstrated a 7.9 mm root mean square error for the registration of internal landmarks. CONCLUSION: These first experiments of a markerless AR provided promising results, requiring very little equipment and setup time, yet providing real-time AR with satisfactory 3D accuracy. These results must be confirmed in a larger prospective study to definitively assess the impact of such minimally invasive technology on pathological margins and oncological outcomes.

Estimation of boundary conditions for patient-specific liver simulation during augmented surgery.

PURPOSE: Augmented reality can improve the outcome of hepatic surgeries, assuming an accurate liver model is available to estimate the position of internal structures. While researchers have proposed patient-specific liver simulations, very few have addressed the question of boundary conditions. Resulting mainly from ligaments attached to the liver, they are not visible in preoperative images, yet play a key role in the computation of the deformation. METHOD: We propose to estimate both the location and stiffness of ligaments by using a combination of a statistical atlas, numerical simulation, and Bayesian inference. Ligaments are modeled as polynomial springs connected to a liver finite element model. They are initialized using an anatomical atlas and stiffness properties taken from the literature. These characteristics are then corrected using a reduced-order unscented Kalman filter based on observations taken from the laparoscopic image stream. RESULTS: Our approach is evaluated using synthetic data and phantom data. By relying on a simplified representation of the ligaments to speed up computation times, it is not estimating the true characteristics of ligaments. However, results show that our estimation of the boundary conditions still improves the accuracy of the simulation by 75% when compared to typical methods involving Dirichlet boundary conditions. CONCLUSION: By estimating patient-specific boundary conditions, using tracked liver motion from RGB-D data, our approach significantly improves the accuracy of the liver model. The method inherently handles noisy observations, a substantial feature in the context of augmented reality.

Constrained stochastic state estimation of deformable 1D objects: Application to single-view 3D reconstruction of catheters with radio-opaque markers.

Minimally invasive fluoroscopy-based procedures are the gold standard for diagnosis and treatment of various pathologies of the cardiovascular system. This kind of procedures imply for the clinicians to infer the 3D shape of the device from 2D images, which is known to be an ill-posed problem. In this paper we present a method to reconstruct the 3D shape of the interventional device, with the aim of improving the navigation. The method combines a physics-based simulation with non-linear Bayesian filter. Whereas the physics-based model provides a prediction of the shape of the device navigating within the blood vessels (taking into account non-linear interactions between the catheter and the surrounding anatomy), an Unscented Kalman Filter is used to correct the navigation model using 2D image features as external observations. The proposed framework has been evaluated on both synthetic and real data, under different model parameterizations, filter parameters tuning and external observations data-sets. Comparing the reconstructed 3D shape with a known ground truth, for the synthetic data-set, we obtained average values for 3D Hausdorff Distance of 0.81±0.53mm, for the 3D mean distance at the segment of 0.37±0.17mm and an average 3D tip error of 0.24±0.13mm. For the real data-set,we obtained an average 3D Hausdorff distance of 1.74±0.77mm, a average 3D mean distance at the distal segment of 0.91±0.14mm, an average 3D error on the tip of 0.53±0.09mm. These results show the ability of our method to retrieve the 3D shape of the device, under a variety of filter parameterizations and challenging conditions: uncertainties on model parameterization, ambiguous views and non-linear complex phenomena such as stick and slip motions.

Simulation of hyperelastic materials in real-time using deep learning.

The finite element method (FEM) is among the most commonly used numerical methods for solving engineering problems. Due to its computational cost, various ideas have been introduced to reduce computation times, such as domain decomposition, parallel computing, adaptive meshing, and model order reduction. In this paper we present U-Mesh: A data-driven method based on a U-Net architecture that approximates the non-linear relation between a contact force and the displacement field computed by a FEM algorithm. We show that deep learning, one of the latest machine learning methods based on artificial neural networks, can enhance computational mechanics through its ability to encode highly non-linear models in a compact form. Our method is applied to three benchmark examples: a cantilever beam, an L-shape and a liver model subject to moving punctual loads. A comparison between our method and proper orthogonal decomposition (POD) is done through the paper. The results show that U-Mesh can perform very fast simulations on various geometries and topologies, mesh resolutions and number of input forces with very small errors.

Elastic Registration Based on Compliance Analysis and Biomechanical Graph Matching.

An automatic elastic registration method suited for vascularized organs is proposed. The vasculature in both the preoperative and intra-operative images is represented as a graph. A typical application of this method is the fusion of pre-operative information onto the organ during surgery, to compensate for the limited details provided by the intra-operative imaging modality (e.g. cone beam CT) and to cope with changes in the shape of the organ. Due to image modalities differences and organ deformation, each graph has a different topology and shape. The adaptive compliance graph matching (ACGM) method presented does not require any manual initialization, handles intra-operative nonrigid deformations of up to 65 mm and computes a complete displacement field over the organ from only the matched vasculature. ACGM is better than the previous biomechanical graph matching method (Garcia Guevara et al. IJCARS, 2018) (BGM) because it uses an efficient biomechanical vascularized liver model to compute the organ's transformation and the vessels bifurcations compliance. This allows to efficiently find the best graph matches with a novel compliance-based adaptive search. These contributions are evaluated on 10 realistic synthetic and 2 porcine automatically segmented datasets. ACGM obtains better target registration error (TRE) than BGM, with an average TRE in the real datasets of 4.2 mm compared to 6.5 mm, respectively. It also is up to one order of magnitude faster, less dependent on the parameters used and more robust to noise.

Deformation Aware Augmented Reality for Craniotomy using 3D/2D Non-rigid Registration of Cortical Vessels.

Intra-operative brain shift is a well-known phenomenon that describes non-rigid deformation of brain tissues due to gravity and loss of cerebrospinal fluid among other phenomena. This has a negative influence on surgical outcome that is often based on pre-operative planning where the brain shift is not considered. We present a novel brain-shift aware Augmented Reality method to align pre-operative 3D data onto the deformed brain surface viewed through a surgical microscope. We formulate our non-rigid registration as a Shape-from-Template problem. A pre-operative 3D wire-like deformable model is registered onto a single 2D image of the cortical vessels, which is automatically segmented. This 3D/2D registration drives the underlying brain structures, such as tumors, and compensates for the brain shift in sub-cortical regions. We evaluated our approach on simulated and real data composed of 6 patients. It achieved good quantitative and qualitative results making it suitable for neurosurgical guidance.

Alignment of Cortical Vessels viewed through the Surgical Microscope with Preoperative Imaging to Compensate for Brain Shift.

Brain shift is a non-rigid deformation of brain tissue that is affected by loss of cerebrospinal fluid, tissue manipulation and gravity among other phenomena. This deformation can negatively influence the outcome of a surgical procedure since surgical planning based on pre-operative image becomes less valid. We present a novel method to compensate for brain shift that maps preoperative image data to the deformed brain during intra-operative neurosurgical procedures and thus increases the likelihood of achieving a gross total resection while decreasing the risk to healthy tissue surrounding the tumor. Through a 3D/2D non-rigid registration process, a 3D articulated model derived from pre-operative imaging is aligned onto 2D images of the vessels viewed through the surgical miscroscopic intra-operatively. The articulated 3D vessels constrain a volumetric biomechanical model of the brain to propagate cortical vessel deformation to the parenchyma and in turn to the tumor. The 3D/2D non-rigid registration is performed using an energy minimization approach that satisfies both projective and physical constraints. Our method is evaluated on real and synthetic data of human brain showing both quantitative and qualitative results and exhibiting its particular suitability for real-time surgical guidance.

Definitions of Computer-Assisted Surgery and Intervention, Image-Guided Surgery and Intervention, Hybrid Operating Room, and Guidance Systems: Strasbourg International Consensus Study.

OBJECTIVE: To develop consensus definitions of image-guided surgery, computer-assisted surgery, hybrid operating room, and surgical navigation systems. SUMMARY BACKGROUND DATA: The use of minimally invasive procedures has increased tremendously over the past 2 decades, but terminology related to image-guided minimally invasive procedures has not been standardized, which is a barrier to clear communication. METHODS: Experts in image-guided techniques and specialized engineers were invited to engage in a systematic process to develop consensus definitions of the key terms listed above. The process was designed following review of common consensus-development methodologies and included participation in 4 online surveys and a post-surveys face-to-face panel meeting held in Strasbourg, France. RESULTS: The experts settled on the terms computer-assisted surgery and intervention, image-guided surgery and intervention, hybrid operating room, and guidance systems and agreed-upon definitions of these terms, with rates of consensus of more than 80% for each term. The methodology used proved to be a compelling strategy to overcome the current difficulties related to data growth rates and technological convergence in this field. CONCLUSIONS: Our multidisciplinary collaborative approach resulted in consensus definitions that may improve communication, knowledge transfer, collaboration, and research in the rapidly changing field of image-guided minimally invasive techniques.

Fiber-based fracture model for simulating soft tissue tearing.

In this paper, we propose a novel approach for simulating soft tissue tearing, using a model that takes into account the existence of fibers within the tissue. These fibers influence the deformation by introducing anisotropy, and impact the direction of propagation for the fracture during tearing. We describe our approach for simulating, in real-time, the deformation and fracture of anisotropic membranes, and we illustrate our method with the simulation of capsulorhexis, one of the critical steps of cataract surgery.

CT scan merging to enhance navigation in interventional radiology simulation.

We present a method to merge two distinct CT scans acquired from different patients such that the second scan can supplement the first when it is missing necessary supporting anatomy. The aim is to provide vascular intervention simulations with full body anatomy. Often, patient CT scans are confined to a localised region so that the patient is not exposed to more radiation than necessary and to increase scanner throughput. Unfortunately, this localised scanning region may be limiting for some applications where surrounding anatomy may be required and where approximate supporting anatomy is acceptable. The resulting merged scan can enhance body navigation simulations with X-ray rendering by providing a complete anatomical reference which may be useful in training and rehearsal. An example of the use of our CT scan merging technique in the field of interventional radiology is described.

Force classification during robotic interventions through simulation-trained neural networks.

PURPOSE: Intravitreal injection is among the most frequent treatment strategies for chronic ophthalmic diseases. The last decade has seen a serious increase in the number of intravitreal injections, and with it, adverse effects and drawbacks. To tackle these problems, medical assistive devices for robotized injections have been suggested and are projected to enhance delivery mechanisms for a new generation of pharmacological solutions. In this paper, we present a method aimed at improving the safety characteristics of upcoming robotic systems. Our vision-based method uses a combination of 2D OCT data, numerical simulation and machine learning to classify the range of the force applied by an injection needle on the sclera. METHODS: We design a neural network to classify force ranges from optical coherence tomography (OCT) images of the sclera directly. To avoid the need for large real data sets, the network is trained on images of simulated deformed sclera. This simulation is based on a finite element method, and the model is parameterized using a Bayesian filter applied to observations of the deformation in OCT images. RESULTS: We validate our approach on real OCT data collected on five ex vivo porcine eyes using a robotically guided needle. The thorough parameterization of the simulations leads to a very good agreement between the virtually generated samples used to train the network and the real OCT acquisitions. Results show that the applied force range on real data can be predicted with 93% accuracy. CONCLUSIONS: Through a simulation-trained neural network, our approach estimates the force range applied by a robotically guided needle on the sclera based solely on a single OCT slice of the deformed sclera. Being real-time, this solution can be integrated in the control loop of the system, permitting the prompt withdrawal of the needle for safety reasons.

An open electromagnetic tracking framework applied to targeted liver tumour ablation.

PURPOSE: Electromagnetic tracking is a core platform technology in the navigation and visualisation of image-guided procedures. The technology provides high tracking accuracy in non-line-of-sight environments, allowing instrument navigation in locations where optical tracking is not feasible. EMT can be beneficial in applications such as percutaneous radiofrequency ablation for the treatment of hepatic lesions where the needle tip may be obscured due to difficult liver environments (e.g subcutaneous fat or ablation artefacts). Advances in the field of EMT include novel methods of improving tracking system accuracy, precision and error compensation capabilities, though such system-level improvements cannot be readily incorporated in current therapy applications due to the 'blackbox' nature of commercial tracking solving algorithms. METHODS: This paper defines a software framework to allow novel EMT designs, and improvements become part of the global design process for image-guided interventions. An exemplary framework is implemented in the Python programming language and demonstrated with the open-source Anser EMT system. The framework is applied in the preclinical setting though targeted liver ablation therapy on an animal model. RESULTS: The developed framework was tested with the Anser EMT electromagnetic tracking platform. Liver tumour targeting was performed using the tracking framework with the CustusX navigation platform using commercially available electromagnetically tracked needles. Ablation of two tumours was performed with a commercially available ablation system. Necropsy of the tumours indicated ablations within 5 mm of the tumours. CONCLUSIONS: An open-source framework for electromagnetic tracking was presented and effectively demonstrated in the preclinical setting. We believe that this framework provides a structure for future advancement in EMT system in and customised instrument design.

Real-Time Error Control for Surgical Simulation.

OBJECTIVE: To present the first a posteriori error-driven adaptive finite element approach for real-time simulation, and to demonstrate the method on a needle insertion problem. METHODS: We use corotational elasticity and a frictional needle/tissue interaction model. The problem is solved using finite elements within SOFA.1 For simulating soft tissue deformation, the refinement strategy relies upon a hexahedron-based finite element method, combined with a posteriori error estimation driven local -refinement. RESULTS: We control the local and global error level in the mechanical fields (e.g., displacement or stresses) during the simulation. We show the convergence of the algorithm on academic examples, and demonstrate its practical usability on a percutaneous procedure involving needle insertion in a liver. For the latter case, we compare the force-displacement curves obtained from the proposed adaptive algorithm with that obtained from a uniform refinement approach. CONCLUSIONS: Error control guarantees that a tolerable error level is not exceeded during the simulations. Local mesh refinement accelerates simulations. SIGNIFICANCE: Our work provides a first step to discriminate between discretization error and modeling error by providing a robust quantification of discretization error during simulations.

Biomechanics-based graph matching for augmented CT-CBCT.

PURPOSE: Augmenting intraoperative cone beam computed tomography (CBCT) images with preoperative computed tomography data in the context of image-guided liver therapy is proposed. The expected benefit is an improved visualization of tumor(s), vascular system and other internal structures of interest. METHOD: An automatic elastic registration based on matching of vascular trees extracted from both the preoperative and intraoperative images is presented. Although methods dedicated to nonrigid graph matching exist, they are not efficient when large intraoperative deformations of tissues occur, as is the case during the liver surgery. The contribution is an extension of the graph matching algorithm using Gaussian process regression (GPR) (Serradell et al. in IEEE Trans Pattern Anal Mach Intell 37(3):625-638, 2015): First, an improved GPR matching is introduced by imposing additional constraints during the matching when the number of hypothesis is large; like the original algorithm, this extended version does not require a manual initialization of matching. Second, a fast biomechanical model is employed to make the method capable of handling large deformations. RESULTS: The proposed automatic intraoperative augmentation is evaluated on both synthetic and real data. It is demonstrated that the algorithm is capable of handling large deformations, thus being more robust and reliable than previous approaches. Moreover, the time required to perform the elastic registration is compatible with the intraoperative navigation scenario. CONCLUSION: A biomechanics-based graph matching method, which can handle large deformations and augment intraoperative CBCT, is presented and evaluated.