Registration Sanity Check for AR-guided Surgical Interventions: Experience From Head and Face Surgery.

Achieving and maintaining proper image registration accuracy is an open challenge of image-guided surgery. This work explores and assesses the efficacy of a registration sanity check method for augmented reality-guided navigation (AR-RSC), based on the visual inspection of virtual 3D models of landmarks. We analyze the AR-RSC sensitivity and specificity by recruiting 36 subjects to assess the registration accuracy of a set of 114 AR images generated from camera images acquired during an AR-guided orthognathic intervention. Translational or rotational errors of known magnitude up to ±1.5 mm/±15.5°, were artificially added to the image set in order to simulate different registration errors. This study analyses the performance of AR-RSC when varying (1) the virtual models selected for misalignment evaluation (e. g., the model of brackets, incisor teeth, and gingival margins in our experiment), (2) the type (translation/rotation) of registration error, and (3) the level of user experience in using AR technologies. Results show that: 1) the sensitivity and specificity of the AR-RSC depends on the virtual models (globally, a median true positive rate of up to 79.2% was reached with brackets, and a median true negative rate of up to 64.3% with incisor teeth), 2) there are error components that are more difficult to identify visually, 3) the level of user experience does not affect the method. In conclusion, the proposed AR-RSC, tested also in the operating room, could represent an efficient method to monitor and optimize the registration accuracy during the intervention, but special attention should be paid to the selection of the AR data chosen for the visual inspection of the registration accuracy.

Design, Fabrication, and Preliminary Validation of Patient-Specific Spine Section Phantoms for Use in Training Spine Surgeons Outside the Operating Room/Theatre.

Pedicle screw fixation (PSF) demands rigorous training to mitigate the risk of severe neurovascular complications arising from screw misplacement. This paper introduces a patient-specific phantom designed for PSF training, extending a portion of the learning process beyond the confines of the surgical room. Six phantoms of the thoracolumbar region were fabricated from radiological datasets, combining 3D printing and casting techniques. The phantoms were employed in three training sessions by a fifth-year resident who performed full training on all six phantoms; he/she placed a total of 57 pedicle screws. Analysis of the learning curve, focusing on time per screw and positioning accuracy, revealed attainment of an asymptotic performance level (around 3 min per screw) after 40 screws. The phantom's efficacy was evaluated by three experts and six residents, each inserting a minimum of four screws. Initial assessments confirmed face, content, and construct validity, affirming the patient-specific phantoms as a valuable training resource. These proposed phantoms exhibit great promise as an essential tool in surgical training as they exhibited a demonstrable learning effect on the PSF technique. This study lays the foundation for further exploration and underscores the potential impact of these patient-specific phantoms on the future of spinal surgical education.

Preliminary Results of Preoperative Planning Using 3D Printing and Augmented Reality in Cryotherapy Treatment of Giant Cell Tumor of Bone-CRIO2AR Project.

Giant Cell Tumor of Bone is a benign tumor with high local aggressive expansion, which, in rare cases, spreads metastasis. Surgical treatment, which often consists of wide curettage to reduce recurrence risk, can lower the quality of life for those affected. Along with aggressive surgery, adjuvant intraoperative techniques have been implemented such as PMMA and cryotherapy. One of the most widely used cryotherapy techniques involves the use of probes to generate ice balls, which have been scientifically shown to have various impacts on the tumor. Although this has been acknowledged, no one has yet tested a way to accurately plan the positioning of cryotherapy probes before surgery, according to the research conducted by the authors. CRIO2AR is a randomized clinical prospective ongoing study by which it will be experimented via preoperative planning of ice probes placement using AR and 3D printing technologies. By studying a single clinical case with these technologies, the surgeon gains better awareness of patient's anatomy and tumor localization. Preliminary results are shown in the article. The first results are confirming that these technologies are applicable in clinical practice. Secondly, preoperative planning is proving to be reliable, easily replicable, and useful for the surgeon.

Analysis of the Robotic-Based In Situ Bioprinting Workflow for the Regeneration of Damaged Tissues through a Case Study.

This study aims to critically analyse the workflow of the in situ bioprinting procedure, presenting a simulated neurosurgical case study, based on a real traumatic event, for collecting quantitative data in support of this innovative approach. After a traumatic event involving the head, bone fragments may have to be removed and a replacement implant placed through a highly demanding surgical procedure in terms of surgeon dexterity. A promising alternative to the current surgical technique is the use of a robotic arm to deposit the biomaterials directly onto the damaged site of the patient following a planned curved surface, which can be designed pre-operatively. Here we achieved an accurate planning-patient registration through pre-operative fiducial markers positioned around the surgical area, reconstructed starting from computed tomography images. Exploiting the availability of multiple degrees of freedom for the regeneration of complex and also overhanging parts typical of anatomical defects, in this work the robotic platform IMAGObot was used to regenerate a cranial defect on a patient-specific phantom. The in situ bioprinting process was then successfully performed showing the great potential of this innovative technology in the field of cranial surgery. In particular, the accuracy of the deposition process was quantified, as well as the duration of the whole procedure was compared to a standard surgical practice. Further investigations include a biological characterisation over time of the printed construct as well as an in vitro and in vivo analysis of the proposed approach, to better analyse the biomaterial performances in terms of osteo-integration with the native tissue.

Head-Mounted Projector for Manual Precision Tasks: Performance Assessment.

The growing interest in augmented reality applications has led to an in-depth look at the performance of head-mounted displays and their testing in numerous domains. Other devices for augmenting the real world with virtual information are presented less frequently and usually focus on the description of the device rather than on its performance analysis. This is the case of projected augmented reality, which, compared to head-worn AR displays, offers the advantages of being simultaneously accessible by multiple users whilst preserving user awareness of the environment and feeling of immersion. This work provides a general evaluation of a custom-made head-mounted projector for the aid of precision manual tasks through an experimental protocol designed for investigating spatial and temporal registration and their combination. The results of the tests show that the accuracy (0.6±0.1 mm of spatial registration error) and motion-to-photon latency (113±12 ms) make the proposed solution suitable for guiding precision tasks.

Magic Leap 1 versus Microsoft HoloLens 2 for the Visualization of 3D Content Obtained from Radiological Images.

The adoption of extended reality solutions is growing rapidly in the healthcare world. Augmented reality (AR) and virtual reality (VR) interfaces can bring advantages in various medical-health sectors; it is thus not surprising that the medical MR market is among the fastest-growing ones. The present study reports on a comparison between two of the most popular MR head-mounted displays, Magic Leap 1 and Microsoft HoloLens 2, for the visualization of 3D medical imaging data. We evaluate the functionalities and performance of both devices through a user-study in which surgeons and residents assessed the visualization of 3D computer-generated anatomical models. The digital content is obtained through a dedicated medical imaging suite (Verima imaging suite) developed by the Italian start-up company (Witapp s.r.l.). According to our performance analysis in terms of frame rate, there are no significant differences between the two devices. The surgical staff expressed a clear preference for Magic Leap 1, particularly for the better visualization quality and the ease of interaction with the 3D virtual content. Nonetheless, even though the results of the questionnaire were slightly more positive for Magic Leap 1, the spatial understanding of the 3D anatomical model in terms of depth relations and spatial arrangement was positively evaluated for both devices.

Bioengineering, augmented reality, and robotic surgery in vascular surgery: A literature review.

Biomedical engineering integrates a variety of applied sciences with life sciences to improve human health and reduce the invasiveness of surgical procedures. Technological advances, achieved through biomedical engineering, have contributed to significant improvements in the field of vascular and endovascular surgery. This paper aims to review the most cutting-edge technologies of the last decade involving the use of augmented reality devices and robotic systems in vascular surgery, highlighting benefits and limitations. Accordingly, two distinct literature surveys were conducted through the PubMed database: the first review provides a comprehensive assessment of augmented reality technologies, including the different techniques available for the visualization of virtual content (11 papers revised); the second review collects studies with bioengineering content that highlight the research trend in robotic vascular surgery, excluding works focused only on the clinical use of commercially available robotic systems (15 papers revised). Technological flow is constant and further advances in imaging techniques and hardware components will inevitably bring new tools for a clinical translation of innovative therapeutic strategies in vascular surgery.

Brain Tumor and Augmented Reality: New Technologies for the Future.

In recent years, huge progress has been made in the management of brain tumors, due to the availability of imaging devices, which provide fundamental anatomical and pathological information not only for diagnostic purposes [...].

Augmented Reality-Assisted Craniotomy for Parasagittal and Convexity En Plaque Meningiomas and Custom-Made Cranio-Plasty: A Preliminary Laboratory Report.

BACKGROUND: This report discusses the utility of a wearable augmented reality platform in neurosurgery for parasagittal and convexity en plaque meningiomas with bone flap removal and custom-made cranioplasty. METHODS: A real patient with en plaque cranial vault meningioma with diffuse and extensive dural involvement, extracranial extension into the calvarium, and homogeneous contrast enhancement on gadolinium-enhanced T1-weighted MRI, was selected for this case study. A patient-specific manikin was designed starting with the segmentation of the patient's preoperative MRI images to simulate a craniotomy procedure. Surgical planning was performed according to the segmented anatomy, and customized bone flaps were designed accordingly. During the surgical simulation stage, the VOSTARS head-mounted display was used to accurately display the planned craniotomy trajectory over the manikin skull. The precision of the craniotomy was assessed based on the evaluation of previously prepared custom-made bone flaps. RESULTS: A bone flap with a radius 0.5 mm smaller than the radius of an ideal craniotomy fitted perfectly over the performed craniotomy, demonstrating an error of less than ±1 mm in the task execution. The results of this laboratory-based experiment suggest that the proposed augmented reality platform helps in simulating convexity en plaque meningioma resection and custom-made cranioplasty, as carefully planned in the preoperative phase. CONCLUSIONS: Augmented reality head-mounted displays have the potential to be a useful adjunct in tumor surgical resection, cranial vault lesion craniotomy and also skull base surgery, but more study with large series is needed.

In Situ Visualization for 3D Ultrasound-Guided Interventions with Augmented Reality Headset.

Augmented Reality (AR) headsets have become the most ergonomic and efficient visualization devices to support complex manual tasks performed under direct vision. Their ability to provide hands-free interaction with the augmented scene makes them perfect for manual procedures such as surgery. This study demonstrates the reliability of an AR head-mounted display (HMD), conceived for surgical guidance, in navigating in-depth high-precision manual tasks guided by a 3D ultrasound imaging system. The integration between the AR visualization system and the ultrasound imaging system provides the surgeon with real-time intra-operative information on unexposed soft tissues that are spatially registered with the surrounding anatomic structures. The efficacy of the AR guiding system was quantitatively assessed with an in vitro study simulating a biopsy intervention aimed at determining the level of accuracy achievable. In the experiments, 10 subjects were asked to perform the biopsy on four spherical lesions of decreasing sizes (10, 7, 5, and 3 mm). The experimental results showed that 80% of the subjects were able to successfully perform the biopsy on the 5 mm lesion, with a 2.5 mm system accuracy. The results confirmed that the proposed integrated system can be used for navigation during in-depth high-precision manual tasks.

Hybrid Simulation and Planning Platform for Cryosurgery with Microsoft HoloLens.

Cryosurgery is a technique of growing popularity involving tissue ablation under controlled freezing. Technological advancement of devices along with surgical technique improvements have turned cryosurgery from an experimental to an established option for treating several diseases. However, cryosurgery is still limited by inaccurate planning based primarily on 2D visualization of the patient's preoperative images. Several works have been aimed at modelling cryoablation through heat transfer simulations; however, most software applications do not meet some key requirements for clinical routine use, such as high computational speed and user-friendliness. This work aims to develop an intuitive platform for anatomical understanding and pre-operative planning by integrating the information content of radiological images and cryoprobe specifications either in a 3D virtual environment (desktop application) or in a hybrid simulator, which exploits the potential of the 3D printing and augmented reality functionalities of Microsoft HoloLens. The proposed platform was preliminarily validated for the retrospective planning/simulation of two surgical cases. Results suggest that the platform is easy and quick to learn and could be used in clinical practice to improve anatomical understanding, to make surgical planning easier than the traditional method, and to strengthen the memorization of surgical planning.

Evaluation of a Wearable AR Platform for Guiding Complex Craniotomies in Neurosurgery.

Today, neuronavigation is widely used in daily clinical routine to perform safe and efficient surgery. Augmented reality (AR) interfaces can provide anatomical models and preoperative planning contextually blended with the real surgical scenario, overcoming the limitations of traditional neuronavigators. This study aims to demonstrate the reliability of a new-concept AR headset in navigating complex craniotomies. Moreover, we aim to prove the efficacy of a patient-specific template-based methodology for fast, non-invasive, and fully automatic planning-to-patient registration. The AR platform navigation performance was assessed with an in-vitro study whose goal was twofold: to measure the real-to-virtual 3D target visualization error (TVE), and assess the navigation accuracy through a user study involving 10 subjects in tracing a complex craniotomy. The feasibility of the template-based registration was preliminarily tested on a volunteer. The TVE mean and standard deviation were 1.3 and 0.6 mm. The results of the user study, over 30 traced craniotomies, showed that 97% of the trajectory length was traced within an error margin of 1.5 mm, and 92% within a margin of 1 mm. The in-vivo test confirmed the feasibility and reliability of the patient-specific template for registration. The proposed AR headset allows ergonomic and intuitive fruition of preoperative planning, and it can represent a valid option to support neurosurgical tasks.

ValveTech: A Novel Robotic Approach for Minimally Invasive Aortic Valve Replacement.

OBJECTIVE: Aortic valve disease is the most common heart disease in the elderly calling for replacement with an artificial valve. The presented surgical robot aims to provide a highly controllable instrument for efficient delivery of an artificial valve by the help of integrated endoscopic vision. METHODS: A robot (called ValveTech), intended for minimally invasive surgery (MIS) and consisting of a flexible cable driven manipulator, a passive arm, and a control unit has been designed and prototyped. The flexible manipulator has several features (e.g., stabilizing flaps, tiny cameras, dexterous introducer and custom cartridge) to help the proper valve placement. It provides 5 degrees of freedom for reaching the operative site via mini-thoracotomy; it adjusts the valve and expands it at the optimal position. The robot was evaluated by ten cardiac surgeons following a real surgical scenario in artificial chest simulator with an aortic mockup. Moreover, after each delivery, the expanded valve was evaluated objectively in comparison with the ideal position. RESULTS: The robot performances were evaluated positively by surgeons. The trials resulted in faster delivery and an average misalignment distance of 3.8 mm along the aorta axis; 16.3 degrees rotational angle around aorta axis and 8.8 degrees misalignment of the valve commissure plane to the ideal plane were measured. CONCLUSION: The trials successfully proved the proposed system for valve delivery under endoscopic vision. SIGNIFICANCE: The ValveTech robot can be an alternative solution for minimally invasive aortic valve surgery and improve the quality of the operation both for surgeons and patients.

The Wearable VOSTARS System for Augmented Reality-Guided Surgery: Preclinical Phantom Evaluation for High-Precision Maxillofacial Tasks.

BACKGROUND: In the context of guided surgery, augmented reality (AR) represents a groundbreaking improvement. The Video and Optical See-Through Augmented Reality Surgical System (VOSTARS) is a new AR wearable head-mounted display (HMD), recently developed as an advanced navigation tool for maxillofacial and plastic surgery and other non-endoscopic surgeries. In this study, we report results of phantom tests with VOSTARS aimed to evaluate its feasibility and accuracy in performing maxillofacial surgical tasks. METHODS: An early prototype of VOSTARS was used. Le Fort 1 osteotomy was selected as the experimental task to be performed under VOSTARS guidance. A dedicated set-up was prepared, including the design of a maxillofacial phantom, an ad hoc tracker anchored to the occlusal splint, and cutting templates for accuracy assessment. Both qualitative and quantitative assessments were carried out. RESULTS: VOSTARS, used in combination with the designed maxilla tracker, showed excellent tracking robustness under operating room lighting. Accuracy tests showed that 100% of Le Fort 1 trajectories were traced with an accuracy of ±1.0 mm, and on average, 88% of the trajectory's length was within ±0.5 mm accuracy. CONCLUSIONS: Our preliminary results suggest that the VOSTARS system can be a feasible and accurate solution for guiding maxillofacial surgical tasks, paving the way to its validation in clinical trials and for a wide spectrum of maxillofacial applications.

Interactive serious game for shoulder rehabilitation based on real-time hand tracking.

BACKGROUND: Virtual reality is becoming popular in the rehabilitation field thanks to the several advantages it can offer to patients and physicians. Indeed, serious games can: motivate and engage the patient; offer different levels of challenge and difficulty based on the patient baseline, and integrate objective measures of the patient's performance during each rehabilitation session. OBJECTIVE: We designed and implemented a serious game for shoulder rehabilitation based on real-time hand tracking. The aim was to maintain the medical benefits of traditional rehabilitation, while reducing human resources and costs and facilitating active patient participation. METHODS: Our software application provides the user with a shoulder horizontal adduction exercise. This exercise takes place in a 2D interactive game environment, controlled by hand movements on a desk pad. The hardware includes a standard desktop computer and screen, and the Leap Motion Controller: a hand tracking system. Changing the desk pad material allows the physiotherapist to vary the friction between the user hand and the supporting surface. RESULTS: Fourteen healthy volunteers and six rehabilitation experts tested our serious game. The results showed that the application is attractive, ergonomic and clinically useful. CONCLUSION: Despite promising results, clinical validation is necessary to demonstrate the efficacy of the serious game.

Wearable Augmented Reality Platform for Aiding Complex 3D Trajectory Tracing.

Augmented reality (AR) Head-Mounted Displays (HMDs) are emerging as the most efficient output medium to support manual tasks performed under direct vision. Despite that, technological and human-factor limitations still hinder their routine use for aiding high-precision manual tasks in the peripersonal space. To overcome such limitations, in this work, we show the results of a user study aimed to validate qualitatively and quantitatively a recently developed AR platform specifically conceived for guiding complex 3D trajectory tracing tasks. The AR platform comprises a new-concept AR video see-through (VST) HMD and a dedicated software framework for the effective deployment of the AR application. In the experiments, the subjects were asked to perform 3D trajectory tracing tasks on 3D-printed replica of planar structures or more elaborated bony anatomies. The accuracy of the trajectories traced by the subjects was evaluated by using templates designed ad hoc to match the surface of the phantoms. The quantitative results suggest that the AR platform could be used to guide high-precision tasks: on average more than 94% of the traced trajectories stayed within an error margin lower than 1 mm. The results confirm that the proposed AR platform will boost the profitable adoption of AR HMDs to guide high precision manual tasks in the peripersonal space.

Perceptual Limits of Optical See-Through Visors for Augmented Reality Guidance of Manual Tasks.

OBJECTIVE: The focal length of available optical see-through (OST) head-mounted displays (HMDs) is at least 2 m; therefore, during manual tasks, the user eye cannot keep in focus both the virtual and real content at the same time. Another perceptual limitation is related to the vergence-accommodation conflict, the latter being present in binocular vision only. This paper investigates the effect of incorrect focus cues on the user performance, visual comfort, and workload during the execution of augmented reality (AR)-guided manual task with one of the most advanced OST HMD, the Microsoft HoloLens. METHODS: An experimental study was designed to investigate the performance of 20 subjects in a connect-the-dots task, with and without the use of AR. The following tests were planned: AR-guided monocular and binocular, and naked-eye monocular and binocular. Each trial was analyzed to evaluate the accuracy in connecting dots. NASA Task Load Index and Likert questionnaires were used to assess the workload and the visual comfort. RESULTS: No statistically significant differences were found in the workload, and in the perceived comfort between the AR-guided binocular and monocular test. User performances were significantly better during the naked eye tests. No statistically significant differences in performances were found in the monocular and binocular tests. The maximum error in AR tests was 5.9 mm. CONCLUSION: Even if there is a growing interest in using commercial OST HMD, for guiding high-precision manual tasks, attention should be paid to the limitations of the available technology not designed for the peripersonal space.