Keyhole-aware laparoscopic augmented reality.

Augmented Reality (AR) from preoperative data is a promising approach to improve intraoperative tumour localisation in Laparoscopic Liver Resection (LLR). Existing systems register the preoperative tumour model with the laparoscopic images and render it by direct camera projection, as if the organ were transparent. However, a simple geometric reasoning shows that this may induce serious surgeon misguidance. This is because the tools enter in a different keyhole than the laparoscope. As AR is particularly important for deep tumours, this problem potentially hinders the whole interest of AR guidance. A remedy to this issue is to project the tumour from its internal position to the liver surface towards the tool keyhole, and only then to the camera. This raises the problem of estimating the tool keyhole position in laparoscope coordinates. We propose a keyhole-aware pipeline which resolves the problem by using the observed tool to probe the keyhole position and by showing a keyhole-aware visualisation of the tumour. We assess the benefits of our pipeline quantitatively on a geometric in silico model and on a liver phantom model, as well as qualitatively on three patient data.

Augmented Reality Guided Laparoscopic Liver Resection: A Phantom Study With Intraparenchymal Tumors.

INTRODUCTION: Augmented reality (AR) in laparoscopic liver resection (LLR) can improve intrahepatic navigation by creating a virtual liver transparency. Our team has recently developed Hepataug, an AR software that projects the invisible intrahepatic tumors onto the laparoscopic images and allows the surgeon to localize them precisely. However, the accuracy of registration according to the location and size of the tumors, as well as the influence of the projection axis, have never been measured. The aim of this work was to measure the three-dimensional (3D) tumor prediction error of Hepataug. METHODS: Eight 3D virtual livers were created from the computed tomography scan of a healthy human liver. Reference markers with known coordinates were virtually placed on the anterior surface. The virtual livers were then deformed and 3D printed, forming 3D liver phantoms. After placing each 3D phantom inside a pelvitrainer, registration allowed Hepataug to project virtual tumors along two axes: the laparoscope axis and the operator port axis. The surgeons had to point the center of eight virtual tumors per liver with a pointing tool whose coordinates were precisely calculated. RESULTS: We obtained 128 pointing experiments. The average pointing error was 29.4 ± 17.1 mm and 9.2 ± 5.1 mm for the laparoscope and operator port axes respectively (P = 0.001). The pointing errors tended to increase with tumor depth (correlation coefficients greater than 0.5 with P < 0.001). There was no significant dependency of the pointing error on the tumor size for both projection axes. CONCLUSIONS: Tumor visualization by projection toward the operating port improves the accuracy of AR guidance and partially solves the problem of the two-dimensional visual interface of monocular laparoscopy. Despite a lower precision of AR for tumors located in the posterior part of the liver, it could allow the surgeons to access these lesions without completely mobilizing the liver, hence decreasing the surgical trauma.

Using multiple images and contours for deformable 3D-2D registration of a preoperative CT in laparoscopic liver surgery.

PURPOSE: Laparoscopic liver resection is a challenging procedure because of the difficulty to localise inner structures such as tumours and vessels. Augmented reality overcomes this problem by overlaying preoperative 3D models on the laparoscopic views. It requires deformable registration of the preoperative 3D models to the laparoscopic views, which is a challenging task due to the liver flexibility and partial visibility. METHODS: We propose several multi-view registration methods exploiting information from multiple views simultaneously in order to improve registration accuracy. They are designed to work on two scenarios: on rigidly related views and on non-rigidly related views. These methods exploit the liver's anatomical landmarks and texture information available in all the views to constrain registration. RESULTS: We evaluated the registration accuracy of our methods quantitatively on synthetic and phantom data, and qualitatively on patient data. We measured 3D target registration errors in mm for the whole liver for the quantitative case, and 2D reprojection errors in pixels for the qualitative case. CONCLUSION: The proposed rigidly related multi-view methods improve registration accuracy compared to the baseline single-view method. They comply with the 1 cm oncologic resection margin advised for hepatocellular carcinoma interventions, depending on the available registration constraints. The non-rigidly related multi-view method does not provide a noticeable improvement. This means that using multiple views with the rigidity assumption achieves the best overall registration error.

Augmented reality in laparoscopic liver resection evaluated on an ex-vivo animal model with pseudo-tumours.

BACKGROUND: The aim of this study was to assess the performance of our augmented reality (AR) software (Hepataug) during laparoscopic resection of liver tumours and compare it to standard ultrasonography (US). MATERIALS AND METHODS: Ninety pseudo-tumours ranging from 10 to 20 mm were created in sheep cadaveric livers by injection of alginate. CT-scans were then performed and 3D models reconstructed using a medical image segmentation software (MITK). The livers were placed in a pelvi-trainer on an inclined plane, approximately perpendicular to the laparoscope. The aim was to obtain free resection margins, as close as possible to 1 cm. Laparoscopic resection was performed using US alone (n = 30, US group), AR alone (n = 30, AR group) and both US and AR (n = 30, ARUS group). R0 resection, maximal margins, minimal margins and mean margins were assessed after histopathologic examination, adjusted to the tumour depth and to a liver zone-wise difficulty level. RESULTS: The minimal margins were not different between the three groups (8.8, 8.0 and 6.9 mm in the US, AR and ARUS groups, respectively). The maximal margins were larger in the US group compared to the AR and ARUS groups after adjustment on depth and zone difficulty (21 vs. 18 mm, p = 0.001 and 21 vs. 19.5 mm, p = 0.037, respectively). The mean margins, which reflect the variability of the measurements, were larger in the US group than in the ARUS group after adjustment on depth and zone difficulty (15.2 vs. 12.8 mm, p < 0.001). When considering only the most difficult zone (difficulty 3), there were more R1/R2 resections in the US group than in the AR + ARUS group (50% vs. 21%, p = 0.019). CONCLUSION: Laparoscopic liver resection using AR seems to provide more accurate resection margins with less variability than the gold standard US navigation, particularly in difficult to access liver zones with deep tumours.

A case series study of augmented reality in laparoscopic liver resection with a deformable preoperative model.

BACKGROUND: Previous work in augmented reality (AR) guidance in monocular laparoscopic hepatectomy requires the surgeon to manually overlay a rigid preoperative model onto a laparoscopy image. This may be fairly inaccurate because of significant liver deformation. We have proposed a technique which overlays a deformable preoperative model semi-automatically onto a laparoscopic image using a new software called Hepataug. The aim of this study is to show the feasibility of Hepataug to perform AR with a deformable model in laparoscopic hepatectomy. METHODS: We ran Hepataug during the procedures, as well as the usual means of laparoscopic ultrasonography (LUS) and visual inspection of the preoperative CT or MRI. The primary objective was to assess the feasibility of Hepataug, in terms of minimal disruption of the surgical workflow. The secondary objective was to assess the potential benefit of Hepataug, by subjective comparison with LUS. RESULTS: From July 2017 to March 2019, 17 consecutive patients were included in this study. AR was feasible in all procedures, with good correlation with LUS. However, for 2 patients, LUS did not reveal the location of the tumors. Hepataug gave a prediction of the tumor locations, which was confirmed and refined by careful inspection of the preoperative CT or MRI. CONCLUSION: Hepataug showed a minimal disruption of the surgical workflow and can thus be feasibly used in real hepatectomy procedures. Thanks to its new mechanism of semi-automatic deformable alignment, Hepataug also showed a good agreement with LUS and visual CT or MRI inspection in subsurface tumor localization. Importantly, Hepataug yields reproducible results. It is easy to use and could be deployed in any existing operating room. Nevertheless, comparative prospective studies are needed to study its efficacy.

Combining Visual Cues with Interactions for 3D-2D Registration in Liver Laparoscopy.

Augmented Reality (AR) in monocular liver laparoscopy requires one to register a preoperative 3D liver model to a laparoscopy image. This is a difficult problem because the preoperative shape may significantly differ from the unknown intraoperative shape and the liver is only partially visible in the laparoscopy image. Previous approaches are either manual, using a rigid model, or automatic, using visual cues and a biomechanical model. We propose a new approach called the hybrid approach combining the best of both worlds. The visual cues allow us to capture the machine perception while user interaction allows us to take advantage of the surgeon's prior knowledge and spatial understanding of the patient anatomy. The registration accuracy and repeatability were evaluated on phantom, animal ex vivo and patient data respectively. The proposed registration outperforms the state of the art methods both in terms of accuracy and repeatability. An average registration error below the 1 cm oncologic margin advised in the literature for tumour resection in laparoscopy hepatectomy was obtained.