A decade of progress: bringing mixed reality image-guided surgery systems in the operating room.

Advancements in mixed reality (MR) have led to innovative approaches in image-guided surgery (IGS). In this paper, we provide a comprehensive analysis of the current state of MR in image-guided procedures across various surgical domains. Using the Data Visualization View (DVV) Taxonomy, we analyze the progress made since a 2013 literature review paper on MR IGS systems. In addition to examining the current surgical domains using MR systems, we explore trends in types of MR hardware used, type of data visualized, visualizations of virtual elements, and interaction methods in use. Our analysis also covers the metrics used to evaluate these systems in the operating room (OR), both qualitative and quantitative assessments, and clinical studies that have demonstrated the potential of MR technologies to enhance surgical workflows and outcomes. We also address current challenges and future directions that would further establish the use of MR in IGS.

A Tool-free Neuronavigation Method based on Single-view Hand Tracking.

This work presents a novel tool-free neuronavigation method that can be used with a single RGB commodity camera. Compared with freehand craniotomy placement methods, the proposed system is more intuitive and less error prone. The proposed method also has several advantages over standard neuronavigation platforms. First, it has a much lower cost, since it doesn't require the use of an optical tracking camera or electromagnetic field generator, which are typically the most expensive parts of a neuronavigation system, making it much more accessible. Second, it requires minimal setup, meaning that it can be performed at the bedside and in circumstances where using a standard neuronavigation system is impractical. Our system relies on machine-learning-based hand pose estimation that acts as a proxy for optical tool tracking, enabling a 3D-3D pre-operative to intra-operative registration. Qualitative assessment from clinical users showed that the concept is clinically relevant. Quantitative assessment showed that on average a target registration error (TRE) of 1.3cm can be achieved. Furthermore, the system is framework-agnostic, meaning that future improvements to hand-tracking frameworks would directly translate to a higher accuracy.

NousNav: A low-cost neuronavigation system for deployment in lower-resource settings.

PURPOSE: NousNav is a complete low-cost neuronavigation system that aims to democratize access to higher-quality healthcare in lower-resource settings. NousNav's goal is to provide a model for local actors to be able to reproduce, build and operate a fully functional neuronavigation system at an affordable cost. METHODS: NousNav is entirely open source and relies on low-cost off-the-shelf components, which makes it easy to reproduce and deploy in any region. NousNav's software is also specifically devised with the low-resource setting in mind. RESULTS: It offers means for intuitive intraoperative control. The designed interface is also clean and simple. This allows for easy intraoperative use by either the practicing clinician or a nurse. It thus alleviates the need for a dedicated technician for operation. CONCLUSION: A prototype implementation of the design was built. Hardware and algorithms were designed for robustness, ruggedness, modularity, to be standalone and data-agnostic. The built prototype demonstrates feasibility of the objectives.

MARIN: an open-source mobile augmented reality interactive neuronavigation system.

PURPOSE: Neuronavigation systems making use of augmented reality (AR) have been the focus of much research in the last couple of decades. In recent years, there has been considerable interest in using mobile devices for AR in the operating room (OR). We propose a complete system that performs real-time AR video augmentation on a mobile device in the context of image-guided neurosurgery. METHODS: MARIN (mobile augmented reality interactive neuronavigation system) improves upon the state of the art in terms of performance, allowing real-time augmentation, and interactivity by allowing users to interact with the displayed data. The system was tested in a user study with 17 subjects for qualitative and quantitative evaluation in the context of target localization and brought into the OR for preliminary feasibility tests, where qualitative feedback from surgeons was obtained. RESULTS: The results of the user study showed that MARIN performs significantly better in terms of both time ([Formula: see text]) and accuracy ([Formula: see text]) for the task of target localization in comparison with a traditional image-guided neurosurgery (IGNS) navigation system. Further, MARIN AR visualization was found to be more intuitive and allowed users to estimate target depth more easily. CONCLUSION: MARIN improves upon previously proposed mobile AR neuronavigation systems with its real-time performance, higher accuracy, full integration in the normal workflow and greater interactivity and customizability of the displayed information. The improvement in efficiency and usability over previous systems will facilitate bringing AR into the OR.

Gesture-based registration correction using a mobile augmented reality image-guided neurosurgery system.

In image-guided neurosurgery, a registration between the patient and their pre-operative images and the tracking of surgical tools enables GPS-like guidance to the surgeon. However, factors such as brainshift, image distortion, and registration error cause the patient-to-image alignment accuracy to degrade throughout the surgical procedure no longer providing accurate guidance. The authors present a gesture-based method for manual registration correction to extend the usage of augmented reality (AR) neuronavigation systems. The authors' method, which makes use of the touchscreen capabilities of a tablet on which the AR navigation view is presented, enables surgeons to compensate for the effects of brainshift, misregistration, or tracking errors. They tested their system in a laboratory user study with ten subjects and found that they were able to achieve a median registration RMS error of 3.51 mm on landmarks around the craniotomy of interest. This is comparable to the level of accuracy attainable with previously proposed methods and currently available commercial systems while being simpler and quicker to use. The method could enable surgeons to quickly and easily compensate for most of the observed shift. Further advantages of their method include its ease of use, its small impact on the surgical workflow and its small-time requirement.

Quantifying attention shifts in augmented reality image-guided neurosurgery.

Image-guided surgery (IGS) has allowed for more minimally invasive procedures, leading to better patient outcomes, reduced risk of infection, less pain, shorter hospital stays and faster recoveries. One drawback that has emerged with IGS is that the surgeon must shift their attention from the patient to the monitor for guidance. Yet both cognitive and motor tasks are negatively affected with attention shifts. Augmented reality (AR), which merges the realworld surgical scene with preoperative virtual patient images and plans, has been proposed as a solution to this drawback. In this work, we studied the impact of two different types of AR IGS set-ups (mobile AR and desktop AR) and traditional navigation on attention shifts for the specific task of craniotomy planning. We found a significant difference in terms of the time taken to perform the task and attention shifts between traditional navigation, but no significant difference between the different AR set-ups. With mobile AR, however, users felt that the system was easier to use and that their performance was better. These results suggest that regardless of where the AR visualisation is shown to the surgeon, AR may reduce attention shifts, leading to more streamlined and focused procedures.

Dynamic interaction between fetal adversity and a genetic score reflecting dopamine function on developmental outcomes at 36 months.

BACKGROUND: Fetal adversity, evidenced by poor fetal growth for instance, is associated with increased risk for several diseases later in life. Classical cut-offs to characterize small (SGA) and large for gestational age (LGA) newborns are used to define long term vulnerability. We aimed at exploring the possible dynamism of different birth weight cut-offs in defining vulnerability in developmental outcomes (through the Bayley Scales of Infant and Toddler Development), using the example of a gene vs. fetal adversity interaction considering gene choices based on functional relevance to the studied outcome. METHODS: 36-month-old children from an established prospective birth cohort (Maternal Adversity, Vulnerability, and Neurodevelopment) were classified according to birth weight ratio (BWR) (SGA ≤0.85, LGA >1.15, exploring a wide range of other cut-offs) and genotyped for polymorphisms associated with dopamine signaling (TaqIA-A1 allele, DRD2-141C Ins/Ins, DRD4 7-repeat, DAT1-10- repeat, Met/Met-COMT), composing a score based on the described function, in which hypofunctional variants received lower scores. RESULTS: There were 251 children (123 girls and 128 boys). Using the classic cut-offs (0.85 and 1.15), there were no statistically significant interactions between the neonatal groups and the dopamine genetic score. However, when changing the cut-offs, it is possible to see ranges of BWR that could be associated with vulnerability to poorer development according to the variation in the dopamine function. CONCLUSION: The classic birth weight cut-offs to define SGA and LGA newborns should be seen with caution, as depending on the outcome in question, the protocols for long-term follow up could be either too inclusive-therefore most costly, or unable to screen true vulnerabilities-and therefore ineffective to establish early interventions and primary prevention.