VolumetricOR: A New Approach to Simulate Surgical Interventions in Virtual Reality for Training and Education.

BACKGROUND: Surgical training is primarily carried out through observation during assistance or on-site classes, by watching videos as well as by different formats of simulation. The simulation of physical presence in the operating theatre in virtual reality might complement these necessary experiences. A prerequisite is a new education concept for virtual classes that communicates the unique workflows and decision-making paths of surgical health professions (i.e. surgeons, anesthesiologists and surgical assistants) in an authentic and immersive way. For this project, media scientists, designers and surgeons worked together to develop the foundations for new ways of conveying knowledge using virtual reality in surgery. MATERIALS AND METHOD: A technical workflow to record and present volumetric videos of surgical interventions in a photorealistic virtual operating room was developed. Situated in the virtual reality demonstrator called VolumetricOR, users can experience and navigate through surgical workflows as if they are physically present. The concept is compared with traditional video-based formats of digital simulation in surgical training. RESULTS: VolumetricOR let trainees experience surgical action and workflows (a) three-dimensionally, (b) from any perspective and (c) in real scale. This improves the linking of theoretical expertise and practical application of knowledge and shifts the learning experience from observation to participation. DISCUSSION: Volumetric training environments allow trainees to acquire procedural knowledge before going to the operating room and could improve the efficiency and quality of the learning and training process for professional staff by communicating techniques and workflows when the possibilities of training on-site are limited.

Real world usability analysis of two augmented reality headsets in visceral surgery.

Recent developments in the field of augmented reality (AR) have enabled new use cases in surgery. Initial set-up of an appropriate infrastructure for maintaining an AR surgical workflow requires investment in appropriate hardware. We compared the usability of the Microsoft HoloLens and Meta 2 head mounted displays (HMDs). Fifteen medicine students tested each device and were questioned with a variant of the System Usability Scale (SUS). Two surgeons independently tested the devices in an intraoperative setting. In our adapted SUS, ergonomics, ease of use, and visual clarity of the display did not differ significantly between HMD groups. The field of view (FOV) was smaller in the Microsoft HoloLens than the Meta 2 and significantly more study subjects (80% vs. 13.3%; P < 0.001) felt limited through the FOV. Intraoperatively, decreased mobility due to the necessity of an AC adapter and additional computing device for the Meta 2 proved to be limiting. Object stability was rated superior in the Microsoft HoloLens than the Meta 2 by our surgeons and lead to increased use. In summary, after examination of the Meta 2 and the Microsoft HoloLens, we found key advantages in the Microsoft HoloLens which provided palpable benefits in a surgical setting.

Mixed Reality in Visceral Surgery: Development of a Suitable Workflow and Evaluation of Intraoperative Use-cases.

OBJECTIVE: The paper evaluates the application of a mixed reality (MR) headmounted display (HMD) for the visualization of anatomical structures in complex visceral-surgical interventions. A workflow was developed and technical feasibility was evaluated. SUMMARY OF BACKGROUND DATA: Medical images are still not seamlessly integrated into surgical interventions and, thus, remain separated from the surgical procedure.Surgeons need to cognitively relate 2-dimensional sectional images to the 3-dimensional (3D) during the actual intervention. MR applications simulate 3D images and reduce the offset between working space and visualization allowing for improved spatial-visual approximation of patient and image. METHODS: The surgeon's field of vision was superimposed with a 3D-model of the patient's relevant liver structures displayed on a MR-HMD. This set-up was evaluated during open hepatic surgery. RESULTS: A suitable workflow for segmenting image masks and texture mapping of tumors, hepatic artery, portal vein, and the hepatic veins was developed. The 3D model was positioned above the surgical site. Anatomical reassurance was possible simply by looking up. Positioning in the room was stable without drift and minimal jittering. Users reported satisfactory comfort wearing the device without significant impairment of movement. CONCLUSION: MR technology has a high potential to improve the surgeon's action and perception in open visceral surgery by displaying 3D anatomical models close to the surgical site. Superimposing anatomical structures directly onto the organs within the surgical site remains challenging, as the abdominal organs undergo major deformations due to manipulation, respiratory motion, and the interaction with the surgical instruments during the intervention. A further application scenario would be intraoperative ultrasound examination displaying the image directly next to the transducer. Displays and sensor-technologies as well as biomechanical modeling and object-recognition algorithms will facilitate the application of MR-HMD in surgery in the near future.

Surgical planning in virtual reality: a systematic review.

Purpose: Virtual reality (VR) technology has emerged as a promising tool for physicians, offering the ability to assess anatomical data in 3D with visuospatial interaction qualities. The last decade has witnessed a remarkable increase in the number of studies focusing on the application of VR to assess patient-specific image data. This systematic review aims to provide an up-to-date overview of the latest research on VR in the field of surgical planning. Approach: A comprehensive literature search was conducted based on the preferred reporting items for systematic reviews and meta-analyses covering the period from April 1, 2021 to May 10, 2023. It includes research articles reporting on preoperative surgical planning using patient-specific medical images in virtual reality using head-mounted displays. The review summarizes the current state of research in this field, identifying key findings, technologies, study designs, methods, and potential directions for future research. Results: The selected studies show a positive impact on surgical decision-making and anatomy understanding compared to other visualization modalities. A substantial number of studies are reporting anecdotal evidence and case-specific outcomes. Notably, surgical planning using VR led to more frequent changes in surgical plans compared to planning with other visualization methods when surgeons reassessed their initial plans. VR demonstrated benefits in reducing planning time and improving spatial localization of pathologies. Conclusions: Results show that the application of VR for surgical planning is still in an experimental stage but is gradually advancing toward clinical use. The diverse study designs, methodologies, and varying reporting hinder a comprehensive analysis. Some findings lack statistical evidence and rely on subjective assumptions. To strengthen evaluation, future research should focus on refining study designs, improving technical reporting, defining visual and technical proficiency requirements, and enhancing VR software usability and design. Addressing these areas could pave the way for an effective implementation of VR in clinical settings.