WatchPose: A View-Aware Approach for Camera Pose Data Collection in Industrial Environments.

Collecting correlated scene images and camera poses is an essential step towards learning absolute camera pose regression models. While the acquisition of such data in living environments is relatively easy by following regular roads and paths, it is still a challenging task in constricted industrial environments. This is because industrial objects have varied sizes and inspections are usually carried out with non-constant motions. As a result, regression models are more sensitive to scene images with respect to viewpoints and distances. Motivated by this, we present a simple but efficient camera pose data collection method, WatchPose, to improve the generalization and robustness of camera pose regression models. Specifically, WatchPose tracks nested markers and visualizes viewpoints in an Augmented Reality- (AR) based manner to properly guide users to collect training data from broader camera-object distances and more diverse views around the objects. Experiments show that WatchPose can effectively improve the accuracy of existing camera pose regression models compared to the traditional data acquisition method. We also introduce a new dataset, Industrial10, to encourage the community to adapt camera pose regression methods for more complex environments.

Multimodal acquisition of articulatory data: Geometrical and temporal registration.

Acquisition of dynamic articulatory data is of major importance for studying speech production. It turns out that one technique alone often is not enough to get a correct coverage of the whole vocal tract at a sufficient sampling rate. Ultrasound (US) imaging has been proposed as a good acquisition technique for the tongue surface because it offers a good temporal sampling, does not alter speech production, is cheap, and is widely available. However, it cannot be used alone and this paper describes a multimodal acquisition system which uses electromagnetography sensors to locate the US probe. The paper particularly focuses on the calibration of the US modality which is the key point of the system. This approach enables US data to be merged with other data. The use of the system is illustrated via an experiment consisting of measuring the minimal tongue to palate distance in order to evaluate and design Magnetic Resonance Imaging protocols well suited for the acquisition of three-dimensional images of the vocal tract. Compared to manual registration of acquisition modalities which is often used in acquisition of articulatory data, the approach presented relies on automatic techniques well founded from geometrical and mathematical points of view.

Deformation-based augmented reality for hepatic surgery.

In this paper we introduce a method for augmenting the laparoscopic view during hepatic tumor resection. Using augmented reality techniques, vessels, tumors and cutting planes computed from pre-operative data can be overlaid onto the laparoscopic video. Compared to current techniques, which are limited to a rigid registration of the pre-operative liver anatomy with the intra-operative image, we propose a real-time, physics-based, non-rigid registration. The main strength of our approach is that the deformable model can also be used to regularize the data extracted from the computer vision algorithms. We show preliminary results on a video sequence which clearly highlights the interest of using physics-based model for elastic registration.

Capturing contact in mitral valve dynamic closure with fluid-structure interaction simulation.

PURPOSE: Realistic fluid-structure interaction (FSI) simulation of the mitral valve opens the way toward planning for surgical repair. In the literature, blood leakage is identified by measuring the flow rate, but detailed information about closure efficiency is missing. We present in this paper an FSI model that improves the detection of blood leakage by building a map of contact. METHODS: Our model is based on the immersed boundary method that captures a map of contact and perfect closure of the mitral valve, without the presence of orifice holes, which often appear with existing methods. We also identified important factors influencing convergence issues. RESULTS: The method is demonstrated in three typical clinical situations: mitral valve with leakage, bulging, and healthy. In addition to the classical ways of evaluating MV closure, such as stress distribution and flow rate, the contact map provides easy detection of leakage with identification of the sources of leakage and a quality assessment of the closure. CONCLUSIONS: Our method significantly improves the quality of the simulation and allows the identification of regurgitation as well as a spatial evaluation of the quality of valve closure. Comparably fast simulation, ability to simulate large deformation, and capturing detailed contact are the main aspects of the study.

Automatic extraction of the mitral valve chordae geometry for biomechanical simulation.

PURPOSE: Mitral valve computational models are widely studied in the literature. They can be used for preoperative planning or anatomical understanding. Manual extraction of the valve geometry on medical images is tedious and requires special training, while automatic segmentation is still an open problem. METHODS: We propose here a fully automatic pipeline to extract the valve chordae architecture compatible with a computational model. First, an initial segmentation is obtained by sub-mesh topology analysis and RANSAC-like model-fitting procedure. Then, the chordal structure is optimized with respect to objective functions based on mechanical, anatomical, and image-based considerations. RESULTS: The approach has been validated on 5 micro-CT scans with a graph-based metric and has shown an [Formula: see text] accuracy rate. The method has also been tested within a structural simulation of the mitral valve closed state. CONCLUSION: Our results show that the chordae architecture resulting from our algorithm can give results similar to experienced users while providing an equivalent biomechanical simulation.

Elastic Registration Based on Compliance Analysis and Biomechanical Graph Matching.

An automatic elastic registration method suited for vascularized organs is proposed. The vasculature in both the preoperative and intra-operative images is represented as a graph. A typical application of this method is the fusion of pre-operative information onto the organ during surgery, to compensate for the limited details provided by the intra-operative imaging modality (e.g. cone beam CT) and to cope with changes in the shape of the organ. Due to image modalities differences and organ deformation, each graph has a different topology and shape. The adaptive compliance graph matching (ACGM) method presented does not require any manual initialization, handles intra-operative nonrigid deformations of up to 65 mm and computes a complete displacement field over the organ from only the matched vasculature. ACGM is better than the previous biomechanical graph matching method (Garcia Guevara et al. IJCARS, 2018) (BGM) because it uses an efficient biomechanical vascularized liver model to compute the organ's transformation and the vessels bifurcations compliance. This allows to efficiently find the best graph matches with a novel compliance-based adaptive search. These contributions are evaluated on 10 realistic synthetic and 2 porcine automatically segmented datasets. ACGM obtains better target registration error (TRE) than BGM, with an average TRE in the real datasets of 4.2 mm compared to 6.5 mm, respectively. It also is up to one order of magnitude faster, less dependent on the parameters used and more robust to noise.

A methodology for validating a 3D imaging modality for brain AVM delineation: application to 3DRA.

A general methodology is described to validate a 3D imaging modality with respect to 2D digital subtracted angiography (DSA) for brain AVMs (BAVM) delineation. It relies on the assessment of the statistical compatibility of the radiosurgical target delineated in 3D with its delineations in 2D. This methodology is demonstrated through a preliminary evaluation of 3D rotational angiography (3DRA). Generally speaking, BAVM delineation cannot be performed on 3DRA alone. However, in our study, 3DRA showed similar performances to DSA for rather easy cases, and even better for three patients. Conversely, three problematic cases are identified and discussed.

Biomechanics-based graph matching for augmented CT-CBCT.

PURPOSE: Augmenting intraoperative cone beam computed tomography (CBCT) images with preoperative computed tomography data in the context of image-guided liver therapy is proposed. The expected benefit is an improved visualization of tumor(s), vascular system and other internal structures of interest. METHOD: An automatic elastic registration based on matching of vascular trees extracted from both the preoperative and intraoperative images is presented. Although methods dedicated to nonrigid graph matching exist, they are not efficient when large intraoperative deformations of tissues occur, as is the case during the liver surgery. The contribution is an extension of the graph matching algorithm using Gaussian process regression (GPR) (Serradell et al. in IEEE Trans Pattern Anal Mach Intell 37(3):625-638, 2015): First, an improved GPR matching is introduced by imposing additional constraints during the matching when the number of hypothesis is large; like the original algorithm, this extended version does not require a manual initialization of matching. Second, a fast biomechanical model is employed to make the method capable of handling large deformations. RESULTS: The proposed automatic intraoperative augmentation is evaluated on both synthetic and real data. It is demonstrated that the algorithm is capable of handling large deformations, thus being more robust and reliable than previous approaches. Moreover, the time required to perform the elastic registration is compatible with the intraoperative navigation scenario. CONCLUSION: A biomechanics-based graph matching method, which can handle large deformations and augment intraoperative CBCT, is presented and evaluated.

Monocular 3D Reconstruction and Augmentation of Elastic Surfaces with Self-Occlusion Handling.

This paper focuses on the 3D shape recovery and augmented reality on elastic objects with self-occlusions handling, using only single view images. Shape recovery from a monocular video sequence is an underconstrained problem and many approaches have been proposed to enforce constraints and resolve the ambiguities. State-of-the art solutions enforce smoothness or geometric constraints, consider specific deformation properties such as inextensibility or resort to shading constraints. However, few of them can handle properly large elastic deformations. We propose in this paper a real-time method that uses a mechanical model and able to handle highly elastic objects. The problem is formulated as an energy minimization problem accounting for a non-linear elastic model constrained by external image points acquired from a monocular camera. This method prevents us from formulating restrictive assumptions and specific constraint terms in the minimization. In addition, we propose to handle self-occluded regions thanks to the ability of mechanical models to provide appropriate predictions of the shape. Our method is compared to existing techniques with experiments conducted on computer-generated and real data that show the effectiveness of recovering and augmenting 3D elastic objects. Additionally, experiments in the context of minimally invasive liver surgery are also provided and results on deformations with the presence of self-occlusions are exposed.

Impact of Soft Tissue Heterogeneity on Augmented Reality for Liver Surgery.

This paper presents a method for real-time augmented reality of internal liver structures during minimally invasive hepatic surgery. Vessels and tumors computed from pre-operative CT scans can be overlaid onto the laparoscopic view for surgery guidance. Compared to current methods, our method is able to locate the in-depth positions of the tumors based on partial three-dimensional liver tissue motion using a real-time biomechanical model. This model permits to properly handle the motion of internal structures even in the case of anisotropic or heterogeneous tissues, as it is the case for the liver and many anatomical structures. Experimentations conducted on phantom liver permits to measure the accuracy of the augmentation while real-time augmentation on in vivo human liver during real surgery shows the benefits of such an approach for minimally invasive surgery.

Statistical modeling and recognition of surgical workflow.

In this paper, we contribute to the development of context-aware operating rooms by introducing a novel approach to modeling and monitoring the workflow of surgical interventions. We first propose a new representation of interventions in terms of multidimensional time-series formed by synchronized signals acquired over time. We then introduce methods based on Dynamic Time Warping and Hidden Markov Models to analyze and process this data. This results in workflow models combining low-level signals with high-level information such as predefined phases, which can be used to detect actions and trigger an event. Two methods are presented to train these models, using either fully or partially labeled training surgeries. Results are given based on tool usage recordings from sixteen laparoscopic cholecystectomies performed by several surgeons.