Visadapt: Catadioptric adaptive camera for scenes of variable density of visual information.

This paper presents the design method of a multi-resolution camera, named Visadapt. It is made of a conventional compact camera with a sensor and a lens pointed to a new deformable mirror so that the mirror in a flat state is parallel to the image plane. The main novelty of the latter mirror, to our knowledge, is the ability to control automatically strokes of several millimeters. This allows Visadapt to capture scenes with a spatially variable density of visual information. A grid of linear actuators, set underneath the mirror surface, deforms the mirror to reach the desired shape computed to capture several areas of different resolutions. Mechanical simulations are allowed to iterate on Visadapt's design, to reduce the geometrical distortions in the images. Evaluations made on an actual prototype of Visadapt show that, by adapting the mirror shape, this camera can magnify a scene object up to 20%, even off-centered in the field-of-view, while still perceiving the rest of the scene.

SofaGym: An Open Platform for Reinforcement Learning Based on Soft Robot Simulations.

OpenAI Gym is one of the standard interfaces used to train Reinforcement Learning (RL) Algorithms. The Simulation Open Framework Architecture (SOFA) is a physics-based engine that is used for soft robotics simulation and control based on real-time models of deformation. The aim of this article is to present SofaGym, an open-source software to create OpenAI Gym interfaces, called environments, out of soft robot digital twins. The link between soft robotics and RL offers new challenges for both fields: representation of the soft robot in an RL context, complex interactions with the environment, use of specific mechanical tools to control soft robots, transfer of policies learned in simulation to the real world, etc. The article presents the large possible uses of SofaGym to tackle these challenges by using RL and planning algorithms. This publication contains neither new algorithms nor new models but proposes a new platform, open to the community, that offers non existing possibilities of coupling RL to physics-based simulation of soft robots. We present 11 environments, representing a wide variety of soft robots and applications; we highlight the challenges showcased by each environment. We propose methods of solving the task using traditional control, RL, and planning and point out research perspectives using the platform.

Monocular 3D Reconstruction and Augmentation of Elastic Surfaces with Self-Occlusion Handling.

This paper focuses on the 3D shape recovery and augmented reality on elastic objects with self-occlusions handling, using only single view images. Shape recovery from a monocular video sequence is an underconstrained problem and many approaches have been proposed to enforce constraints and resolve the ambiguities. State-of-the art solutions enforce smoothness or geometric constraints, consider specific deformation properties such as inextensibility or resort to shading constraints. However, few of them can handle properly large elastic deformations. We propose in this paper a real-time method that uses a mechanical model and able to handle highly elastic objects. The problem is formulated as an energy minimization problem accounting for a non-linear elastic model constrained by external image points acquired from a monocular camera. This method prevents us from formulating restrictive assumptions and specific constraint terms in the minimization. In addition, we propose to handle self-occluded regions thanks to the ability of mechanical models to provide appropriate predictions of the shape. Our method is compared to existing techniques with experiments conducted on computer-generated and real data that show the effectiveness of recovering and augmenting 3D elastic objects. Additionally, experiments in the context of minimally invasive liver surgery are also provided and results on deformations with the presence of self-occlusions are exposed.

Impact of Soft Tissue Heterogeneity on Augmented Reality for Liver Surgery.

This paper presents a method for real-time augmented reality of internal liver structures during minimally invasive hepatic surgery. Vessels and tumors computed from pre-operative CT scans can be overlaid onto the laparoscopic view for surgery guidance. Compared to current methods, our method is able to locate the in-depth positions of the tumors based on partial three-dimensional liver tissue motion using a real-time biomechanical model. This model permits to properly handle the motion of internal structures even in the case of anisotropic or heterogeneous tissues, as it is the case for the liver and many anatomical structures. Experimentations conducted on phantom liver permits to measure the accuracy of the augmentation while real-time augmentation on in vivo human liver during real surgery shows the benefits of such an approach for minimally invasive surgery.

Registration by interactive inverse simulation: application for adaptive radiotherapy.

PURPOSE: This paper introduces a new methodology for semi-automatic registration of anatomical structure deformations. The contribution is to use an interactive inverse simulation of physics-based deformable model, computed in real time. METHODS: The method relies on nonlinear finite element method (FEM) within a constraint-based framework. Given a set of few registered points provided by the user, a real-time optimization adapts the boundary conditions and(/or) some parameters of the FEM in order to obtain the adequate geometrical deformations. To dramatically fasten the process, the method relies on a projection of the model in the space of the optimization variables. In this reduced space, a quadratic programming problem is formulated and solved very quickly. The method is validated with numerical examples for retrieving some unknown parameters such as the Young's modulus and some pressures on the boundaries of the model. RESULTS: The approach is employed in the context of radiotherapy of the neck where weight loss during the 7 weeks of the therapy modifies the volume of the anatomical structures and induces large deformations. Indeed, sensitive structures such as the parotid glands may cross the target volume due to these deformations which leads to adverse effects for the patient. We thus apply the approach for the registration of the parotid glands during the radiotherapy of the head and neck cancer. CONCLUSIONS: The results show how the method could be used in a clinical routine and be employed in the planning in order to limit the radiations of these glands.

Introducing interactive inverse FEM simulation and its application for adaptive radiotherapy.

We introduce a new methodology for semi-automatic deformable registration of anatomical structures, using interactive inverse simulations. The method relies on non-linear real-time Finite Element Method (FEM) within a constraint-based framework. Given a set of few registered points provided by the user, a real-time optimization adapts the boundary conditions and(/or) some parameters of the FEM in order to obtain the adequate geometrical deformations. To dramatically fasten the process, the method relies on a projection of the model in the space of the optimization variables. In this reduced space, a quadratic programming problem is formulated and solved very quickly. The method is validated with numerical examples for retrieving Young's modulus and some pressures on the boundaries. Then, we apply the approach for the registration of the parotid glands during the radiotherapy of the head and neck cancer. Radiotherapy treatment induces weight loss that modifies the shape and the positions of these structures and they eventually intersect the target volume. We show how we could adapt the planning to limit the radiation of these glands.

Deformation-based augmented reality for hepatic surgery.

In this paper we introduce a method for augmenting the laparoscopic view during hepatic tumor resection. Using augmented reality techniques, vessels, tumors and cutting planes computed from pre-operative data can be overlaid onto the laparoscopic video. Compared to current techniques, which are limited to a rigid registration of the pre-operative liver anatomy with the intra-operative image, we propose a real-time, physics-based, non-rigid registration. The main strength of our approach is that the deformable model can also be used to regularize the data extracted from the computer vision algorithms. We show preliminary results on a video sequence which clearly highlights the interest of using physics-based model for elastic registration.

Towards a better understanding of pelvic system disorders using numerical simulation.

Genital prolapse is a pathologic hyper-mobility of the organs that forms the pelvic system. Although this is common condition, the pathophysiology of this disorder is not well known. In order to improve the understanding of its origins, we recreate--virtually--this biomechanical pathology using numerical simulation. The approach builds on a finite element model with parameters measured on several fresh cadavers. The meshes are created from a MRI of a healthy woman and the simulation includes the mechanical interactions between organs (contacts, ligaments, adhesion...). The model is validated through comparison of functional mobilities of the pelvic system observed on a dynamic MRI. We then propose to modify, step by step, the model and its parameters to produce a pathologic situation and have a better understanding of the process. It is not a formal proof but the numerical experiments reinforce the clinical hypothesis on the multifactorial origins of the pathology.

Biomechanical simulation of electrode migration for deep brain stimulation.

Deep Brain Stimulation is a modern surgical technique for treating patients who suffer from affective or motion disorders such as Parkinson's disease. The efficiency of the procedure relies heavily on the accuracy of the placement of a micro-electrode which sends electrical pulses to a specific part of the brain that controls motion and affective symptoms. However, targeting this small anatomical structure is rendered difficult due to a series of brain shifts that take place during and after the procedure. This paper introduces a biomechanical simulation of the intra and postoperative stages of the procedure in order to determine lead deformation and electrode migration due to brain shift. To achieve this goal, we propose a global approach, which accounts for brain deformation but also for the numerous interactions that take place during the procedure (contacts between the brain and the inner part of the skull and falx cerebri, effect of the cerebro-spinal fluid, and biomechanical interactions between the brain and the electrodes and cannula used during the procedure). Preliminary results show a good correlation between our simulations and various results reported in the literature.

Towards interactive planning of coil embolization in brain aneurysms.

Many vascular pathologies can now be treated in a minimally invasive way thanks to interventional radiology. Instead of open surgery, it allows to reach the lesion of the arteries with therapeutic devices through a catheter. As a particular case, intracranial aneurysms are treated by filling the localized widening of the artery with a set of coils to prevent a rupture due to the weakened arterial wall. Considering the location of the lesion, close to the brain, and its very small size, the procedure requires a combination of careful planning and excellent technical skills. An interactive and reliable simulation, adapted to the patient anatomy, would be an interesting tool for helping the interventional neuroradiologist plan and rehearse a coil embolization procedure. This paper describes an original method to perform interactive simulations of coil embolization and proposes a clinical metric to quantitatively measure how the first coil fills the aneurysm. The simulation relies on an accurate reconstruction of the aneurysm anatomy and a real-time model of the coil for which sliding and friction contacts are taken into account. Simulation results are compared to real embolization procedure and exhibit good adequacy.

Interactive simulation of embolization coils: modeling and experimental validation.

Coil embolization offers a new approach to treat aneurysms. This medical procedure is namely less invasive than an open-surgery as it relies on the deployment of very thin platinum-based wires within the aneurysm through the arteries. When performed intracranially, this procedure must be particularly accurate and therefore carefully planned and performed by experienced radiologists. A simulator of the coil deployment represents an interesting and helpful tool for the physician by providing information on the coil behavior. In this paper, an original modeling is proposed to obtain interactive and accurate simulations of coil deployment. The model takes into account geometric nonlinearities and uses a shape memory formulation to describe its complex geometry. An experimental validation is performed in a contact-free environment to identify the mechanical properties of the coil and to quantitatively compare the simulation with real data. Computational performances are also measured to insure an interactive simulation.

Interactive contacts resolution using smooth surface representation.

Accurately describing interactions between medical devices and anatomical structures, or between anatomical structures themselves, is an essential step towards the adoption of computer-based medical simulation as an alternative to traditional training methods. However, while substantial work has been done in the area of real-time soft tissue modeling, little has been done to study the problem of contacts occurring during tissue manipulation. In this paper we introduce a new method for correctly handling complex contacts between various combination of rigid and deformable objects. Our approach verifies Signorini's law by combining Lagrange multipliers and the status method to solve unilateral constraints. Our method handles both concave and convex surfaces by using a displacement subdivision strategy, and the proposed algorithm allows interactive computation times even in very constrained situations. We demonstrate the efficiency of our approach in the context of interventional radiology, with the navigation of catheters and guidewires in tortuous vessels and with the deployment of coils to treat aneurysms.