An analysis on the effect of body tissues and surgical tools on workflow recognition in first person surgical videos.

PURPOSE: Analysis of operative fields is expected to aid in estimating procedural workflow and evaluating surgeons' procedural skills by considering the temporal transitions during the progression of the surgery. This study aims to propose an automatic recognition system for the procedural workflow by employing machine learning techniques to identify and distinguish elements in the operative field, including body tissues such as fat, muscle, and dermis, along with surgical tools. METHODS: We conducted annotations on approximately 908 first-person-view images of breast surgery to facilitate segmentation. The annotated images were used to train a pixel-level classifier based on Mask R-CNN. To assess the impact on procedural workflow recognition, we annotated an additional 43,007 images. The network, structured on the Transformer architecture, was then trained with surgical images incorporating masks for body tissues and surgical tools. RESULTS: The instance segmentation of each body tissue in the segmentation phase provided insights into the trend of area transitions for each tissue. Simultaneously, the spatial features of the surgical tools were effectively captured. In regard to the accuracy of procedural workflow recognition, accounting for body tissues led to an average improvement of 3 % over the baseline. Furthermore, the inclusion of surgical tools yielded an additional increase in accuracy by 4 % compared to the baseline. CONCLUSION: In this study, we revealed the contribution of the temporal transition of the body tissues and surgical tools spatial features to recognize procedural workflow in first-person-view surgical videos. Body tissues, especially in open surgery, can be a crucial element. This study suggests that further improvements can be achieved by accurately identifying surgical tools specific to each procedural workflow step.

Enhanced Precision in Genioplasty: A Novel Intraoperative Spatial Repositioning Using Computer-Aided Design and Manufacturing Technology and a Holographic Mixed Reality Application.

Genioplasty is performed for the orthognathic surgical correction of dentofacial deformities. This article reports a safe and accurate method for genioplasty combining a novel three-dimensional (3D) device with mixed reality (MR)-assisted surgery using a registration marker and a head-mounted display. Four types of devices were designed based on the virtual operation: a surgical splint with a connector; an osteotomy device; a repositioning device; and a registration marker. Microsoft HoloLens 2 and Holoeyes MD were used to project holograms created using computed tomography (CT) data onto the surgical field to improve the accuracy of the computer-aided designed and manufactured (CAD/CAM) surgical guides. After making an incision on the oral vestibule, the splint was fitted on the teeth and the osteotomy device was mounted at the junction site, placed directly on the exposed mandible bone surface. Temporary screws were fixed into the screw hole. An ultrasonic cutting instrument was used for the osteotomy. After separating the bone, a repositioning device was connected to the splint junction and bone segment, and repositioning was performed. At the time of repositioning, the registration marker was connected to the splint junction, and mandible repositioning was confirmed three-dimensionally through HoloLens 2 into the position specified in the virtual surgery. The rate of overlay error between the preoperative virtual operation and one-month postoperative CT data within 2 mm was 100%. CAD/CAM combined with MR enabled accurate genioplasty.

Transanal lateral lymph node dissection with intraoperative hologram support in low rectal cancer.

BACKGROUND: In Japan, the standard treatment for stage II/III advanced low rectal cancer is total mesorectal excision plus lateral lymph node dissection (LLND). There are also recent reports on the use of transanal LLND. However, the transanal anatomy is difficult to understand, and additional support tools are required to improve the surgical safety. The present study examined the utility of holograms with mixed reality as an intraoperative support tool for assessing the complex pelvic anatomy. METHODS: Polygon (stereolithography) files of patients' pelvic organs were created and exported from the SYNAPSE VINCENT imaging system and uploaded into the Holoeyes MD virtual reality software. Three-dimensional images were automatically converted into patient-specific holograms. Each hologram was then installed into a head mount display (HoloLens2), and the surgeons and assistants wore the HoloLens2 when they performed transanal LLND. Twelve digestive surgeons with prior practice in hologram manipulation evaluated the utility of the intraoperative hologram support by means of a questionnaire. RESULTS: Intraoperative hologram support improved the surgical understanding of the lateral lymph node region anatomy. In the questionnaire, 75% of the surgeons answered that the hologram accurately reflected the anatomy, and 92% of the surgeons answered that the anatomy was better understood by simulating the hologram intraoperatively than preoperatively. Moreover, 92% of the surgeons agreed that intraoperative holograms were a useful support tool for improving the surgical safety. CONCLUSIONS: Intraoperative hologram support improved the surgical understanding of the pelvic anatomy for transanal LLND. Intraoperative holograms may represent a next-generation surgical tool for transanal LLND.

Virtual Reality Simulation for Minimally Invasive Coronary Artery Bypass Grafting With Aortic No-Touch Total Arterial Grafting Technique.

OBJECTIVE: Virtual reality can be applied preoperatively by surgeons to gain precise insights into a patient's anatomy for planning minimally invasive coronary artery bypass grafting (CABG) with in situ arterial grafts. This study aimed to examine virtual reality simulation for minimally invasive CABG with in situ arterial grafts. METHODS: Preoperative stereolithographic files in 35 in situ arterial grafts were converted using 320-slice computed tomography and workstation. The accurate length and direction of each graft were confirmed through virtual reality glasses. The simulation of graft designs was performed by using an immersive virtual reality platform. RESULTS: The mean harvested lengths of in situ left internal thoracic artery (n = 17), right internal thoracic artery (n = 12), and gastroepiploic artery (n = 6) grafts predicted by virtual reality simulation were 21.4 ± 3.4 cm, 21.2 ± 3.6 cm, and 22.8 ± 4.8 cm. The required lengths of these grafts predicted by virtual reality simulation were 15.8 ± 2.3 cm, 16.4 ± 2.1 cm, and 14.5 ± 4.4 cm. Minimally invasive CABG using virtual reality simulation was completed in 17 patients, of whom 16 patients underwent aortic no-touch total arterial CABG. The surgical strategy was adjusted in 11.8% of the cases due to the 3-dimensional virtual reality-based anatomy evaluation. The early mortality and morbidity were 0%, and the patency of the graft was 100%. The median time to return to full physical activity was 7.1 days. CONCLUSIONS: This study demonstrated the successful development and clinical application of the first dedicated virtual reality platform for planning aortic no-touch total arterial minimally invasive CABG. Virtual reality simulation can allow the accurate preoperative understanding of anatomy and appropriate planning of the graft design with acceptable postoperative outcomes.

Intraoperative holographic image-guided surgery in a transanal approach for rectal cancer.

PURPOSE: Urethral injury is one of the most important complications in transanal total mesorectal excision (TaTME) in male patients with rectal cancer. The purpose of this study was to investigate holographic image-guided surgery in TaTME. METHODS: Polygon (stereolithography) files were created and exported from SYNAPSE VINCENT, and then uploaded into the Holoeyes MD system (Holoeyes Inc., Tokyo, Japan). After uploading the data, the three-dimensional image was automatically converted into a case-specific hologram. The hologram was then installed into the head mount display, HoloLens (Microsoft Corporation, Redmond, WA). The surgeons and assistants wore the HoloLens when they performed TaTME. RESULTS: In a Wi-Fi-enabled operating room, each surgeon, wearing a HoloLens, shared the same hologram and succeeded in adjusting the hologram by making simple hand gestures from their respective angles. The hologram contributed to better comprehension of the positional relationships between the urethra and the surrounding pelvic organs during surgery. All surgeons were able to properly determine the dissection line. CONCLUSIONS: This first experience suggests that intraoperative holograms contributed to reducing the risk of urethral injury and understanding transanal anatomy. Intraoperative holograms have the potential to become a new next-generation surgical support tool for use in spatial awareness and the sharing of information between surgeons.

Embodiment of supernumerary robotic limbs in virtual reality.

The supernumerary robotic limb system expands the motor function of human users by adding extra artificially designed limbs. It is important for us to embody the system as if it is a part of one's own body and to maintain cognitive transparency in which the cognitive load is suppressed. Embodiment studies have been conducted with an expansion of bodily functions through a "substitution" and "extension". However, there have been few studies on the "addition" of supernumerary body parts. In this study, we developed a supernumerary robotic limb system that operates in a virtual environment, and then evaluated whether the extra limb can be regarded as a part of one's own body using a questionnaire and whether the perception of peripersonal space changes with a visuotactile crossmodal congruency task. We found that the participants can embody the extra-limbs after using the supernumerary robotic limb system. We also found a positive correlation between the perceptual change in the crossmodal congruency task and the subjective feeling that the number of one's arms had increased (supernumerary limb sensation). These results suggest that the addition of an extra body part may cause the participants to feel that they had acquired a new body part that differs from their original body part through a functional expansion.

Digital Transformation Will Change Medical Education and Rehabilitation in Spine Surgery.

The concept of minimally invasive spine therapy (MIST) has been proposed as a treatment strategy to reduce the need for overall patient care, including not only minimally invasive spine surgery (MISS) but also conservative treatment and rehabilitation. To maximize the effectiveness of patient care in spine surgery, the educational needs of medical students, residents, and patient rehabilitation can be enhanced by digital transformation (DX), including virtual reality (VR), augmented reality (AR), mixed reality (MR), and extended reality (XR), three-dimensional (3D) medical images and holograms; wearable sensors, high-performance video cameras, fifth-generation wireless system (5G) and wireless fidelity (Wi-Fi), artificial intelligence, and head-mounted displays (HMDs). Furthermore, to comply with the guidelines for social distancing due to the unexpected COVID-19 pandemic, the use of DX to maintain healthcare and education is becoming more innovative than ever before. In medical education, with the evolution of science and technology, it has become mandatory to provide a highly interactive educational environment and experience using DX technology for residents and medical students, known as digital natives. This study describes an approach to pre- and intraoperative medical education and postoperative rehabilitation using DX in the field of spine surgery that was implemented during the COVID-19 pandemic and will be utilized thereafter.

Essential updates 2020/2021: Current topics of simulation and navigation in hepatectomy.

With the development of three-dimensional (3D) simulation software, preoperative simulation technology is almost completely established. The remaining issue is how to recognize anatomy three-dimensionally. Extended reality is a newly developed technology with several merits for surgical application: no requirement for a sterilized display monitor, better spatial awareness, and the ability to share 3D images among all surgeons. Various technology or devices for intraoperative navigation have also been developed to support the safety and certainty of liver surgery. Consensus recommendations regarding indocyanine green fluorescence were determined in 2021. Extended reality has also been applied to intraoperative navigation, and artificial intelligence (AI) is one of the topics of real-time navigation. AI might overcome the problem of liver deformity with automatic registration. Including the issues described above, this article focuses on recent advances in simulation and navigation in liver surgery from 2020 to 2021.

XR (Extended Reality: Virtual Reality, Augmented Reality, Mixed Reality) Technology in Spine Medicine: Status Quo and Quo Vadis.

In recent years, with the rapid advancement and consumerization of virtual reality, augmented reality, mixed reality, and extended reality (XR) technology, the use of XR technology in spine medicine has also become increasingly popular. The rising use of XR technology in spine medicine has also been accelerated by the recent wave of digital transformation (i.e., case-specific three-dimensional medical images and holograms, wearable sensors, video cameras, fifth generation, artificial intelligence, and head-mounted displays), and further accelerated by the COVID-19 pandemic and the increase in minimally invasive spine surgery. The COVID-19 pandemic has a negative impact on society, but positive impacts can also be expected, including the continued spread and adoption of telemedicine services (i.e., tele-education, tele-surgery, tele-rehabilitation) that promote digital transformation. The purpose of this narrative review is to describe the accelerators of XR (VR, AR, MR) technology in spine medicine and then to provide a comprehensive review of the use of XR technology in spine medicine, including surgery, consultation, education, and rehabilitation, as well as to identify its limitations and future perspectives (status quo and quo vadis).

Intraoperative support with three-dimensional holographic cholangiography in hepatobiliary surgery.

PURPOSE: This study was performed to investigate the potential of intraoperative three-dimensional (3D) holographic cholangiography, which provides a computer graphics model of the biliary tract, with mixed reality techniques. METHODS: Two patients with intraductal papillary neoplasm of the bile duct were enrolled in the study. Intraoperative 3D cholangiography was performed in a hybrid operating room. Three-dimensional polygon data using the acquired cholangiography data were installed into a head mount display (HoloLens; Microsoft Corporation, Redmond, WA, USA). RESULTS: Upon completion of intraoperative 3D cholangiography, a hologram was immediately and successfully made in the operating room using the acquired cholangiography data, and several surgeons wearing the HoloLens succeeded in sharing the same hologram. Compared with usual two-dimensional cholangiography, this 3D holographic cholangiography technique contributed to more accurate reappearance of the bile ducts, especially the B1 origination site, and moving the hologram from the respective operators' angles by means of easy gesture-handling without any monitors. CONCLUSION: Intraoperative 3D holographic cholangiography might be a new next-generation operation-support tool in terms of immediacy, accurate anatomical reappearance, and ease of handling.

Intraoperative holography navigation using a mixed-reality wearable computer during laparoscopic cholecystectomy.

BACKGROUND: Mixed-reality technology, a new digital holographic image technology, is used to present 3-dimensional (3D) images in the surgical space using a wearable mixed-reality device. This study aimed to assess the safety and efficacy of laparoscopic cholecystectomy using a holography-guided navigation system as an intraoperative support image.In this prospective observational study, 27 patients with cholelithiasis or mild cholecystitis underwent laparoscopic cholecystectomy between April 2020 and November 2020. Nine patients underwent laparoscopic cholecystectomy with 3D models generated by a wearable mixed-reality device (laparoscopic cholecystectomy with 3D models) and 18 underwent laparoscopic cholecystectomy with conventional two-dimensional images (laparoscopic cholecystectomy with 2D images) as surgical support images. Surgical outcomes such as operative time, blood loss, and perioperative complication rate were measured, and a four-item questionnaire was used for subjective assessment. All surgeries were performed by a mid-career and an experienced surgeon. RESULTS: Median operative times of laparoscopic cholecystectomy with 3-dimensional models and 2-dimensional images were 74.0 and 58.0 minutes, respectively. No intraoperative blood loss or perioperative complications occurred. Although the midcareer surgeon indicated that laparoscopic cholecystectomy with 3-dimensional models was "normal" or "easy" compared with 2-dimensional images in all cases, the experienced surgeon rated 3-dimensional models as more difficult in 3 (33%) of 9 cases. CONCLUSION: This study provides evidence that laparoscopic cholecystectomy with 3-dimensional models is feasible. However, the efficacy of laparoscopic cholecystectomy with 3-dimensional models may depend on the surgeon's experience, as indicated by the different ratings provided by the surgeons.

Patient-specific virtual and mixed reality for immersive, experiential anatomy education and for surgical planning in temporal bone surgery.

OBJECTIVE: The recent development of extended reality technology has attracted interest in medicine. We explored the use of patient-specific virtual reality (VR) and mixed reality (MR) temporal bone models in anatomical teaching, pre-operative surgical planning and intra-operative surgical referencing. METHODS: VR and MR temporal bone models were created and visualized on head-mounted display (HMD) and MR headset respectively, by a novel webservice that allows users to convert computed tomography images to VR and MR images without specific knowledge of programming. Eleven otorhinolaryngology trainees and specialists were asked to manipulate the healthy VR temporal bone model and to assess its validity by filling out a questionnaire. Additionally, VR and MR pathological models of petrous apex cholesteatoma were utilized for surgical planning pre-operatively and for referring to the anatomy during the surgery. RESULTS: Most participants were favorable about the VR model and considered HMD as superior to a flat computer screen. 91% of the participants agreed or somewhat agreed that VR through HMD is cost effective. In addition, the VR pathological model was used for planning and sharing the surgical approach during a pre-operative surgical conference. The MR headset was worn intra-operatively to clarify the relationship between the pathological lesion and vital anatomical structures. CONCLUSION: Regardless of the participants' training level in otorhinolaryngology or VR experience, all participants agreed that the VR temporal bone model is useful for anatomical education. Furthermore, the creation of patient-specific VR and MR models using the webservice and their pre- and intra-operative usages indicated the potential of innovative adjunctive surgical instrument.

Mixed reality and three dimensional printed models for resection of maxillary tumor: a case report.

In the field of oral and maxillofacial surgery, many institutions have recently begun using three-dimensional printers to create three-dimensional models and mixed reality in a variety of diseases. Here, we report the actual situation model which we made using three-dimensional printer from virtual operation data and the resection that was performed while grasping a maxillary benign tumor and neighboring three-dimensional structure by designing an application for Microsoft® HoloLens, and using Mixed Reality surgery support during the procedure.

Augmented, virtual and mixed reality in spinal surgery: A real-world experience.

This review aims to identify the role of augmented, virtual or mixed reality (AR, VR or MR) technologies in setting of spinal surgery. The authors address the challenges surrounding the implementation of this technology in the operating room. A technical standpoint addresses the efficacy of these imaging modalities based on the current literature in the field. Ultimately, these technologies must be cost-effective to ensure widespread adoption. This may be achieved through reduced surgical times and decreased incidence of post-operative complications and revisions while maintaining equivalent safety profile to alternative surgical approaches. While current studies focus mainly on the successful placement of pedicle screws via AR-guided instrumentation, a wider scope of procedures may be assisted using AR, VR or MR technology once efficacy and safety have been validated. These emerging technologies offer a significant advantage in the guidance of complex procedures that require high precision and accuracy using minimally invasive interventions.

Holographic image-guided thoracoscopic surgery: possibility of usefulness for esophageal cancer patients with abnormal artery.

BACKGROUND: With the recent improvement of medical image analysis technology, three-dimensional (3D) holograms technology is beginning to be used as intraoperative image support. CASE PRESENTATION: We used a wearable holographic computer during thoracoscopic esophagectomy in a 70-year-old man with esophageal cancer. During lymph node dissection around the right recurrent laryngeal nerve, abnormal blood vessels were observed beside the right subclavian artery (RSA). As a result of confirming the anatomical variation of the right vertebral artery (RVA) using the 3D holograms, it was possible to understand that the RVA branched from a low position on the RSA. CONCLUSIONS: Holographic image-guided thoracoscopic esophagectomy using wearable holographic computer provided better spatial recognition of vascular variation and safe lymph node dissection.

Intraoperative 3D Hologram Support With Mixed Reality Techniques in Liver Surgery.

OBJECTIVE: The aim of this study was to investigate the potential of an intraoperative 3D hologram, which was a computer graphics model liver, with mixed reality techniques in liver surgery. SUMMARY BACKGROUND DATA: The merits for the application of a hologram for surgical support are: 1) no sterilized display monitor; 2) better spatial awareness; and 3) 3D images shared by all the surgeons. METHODS: 3D polygon data using preoperative computed tomography data was installed into head mount displays, HoloLens (Microsoft Corporation, Redmond, WA). RESULTS: In a Wi-Fi-enabled operative room, several surgeons wearing HoloLens succeeded in sharing the same hologram and moving that hologram from respective operators' angles by means of easy gesture-handling without any monitors. The intraoperative hologram contributed to better imagination of tumor locations, and for determining the parenchymal dissection line in the hepatectomy for the patients with more than 20 multiple colo-rectal liver metastases. In another case, the hologram enabled a safe Gliisonean pedicle approach for hepato-cellular carcinoma with a hilar anatomical anomaly. Surgeons could easily compare the real patient's anatomy and that of the hologram just before the hepatic hilar procedure. CONCLUSIONS: This initial experience suggested that an intraoperative hologram with mixed reality techniques contributed to "last-minute simulation," not for "navigation." The intraoperative hologram might be a new next-generation operation-supportive tool in terms of spatial awareness, sharing, and simplicity.