Transanal lateral lymph node dissection with intraoperative hologram support in low rectal cancer.

BACKGROUND: In Japan, the standard treatment for stage II/III advanced low rectal cancer is total mesorectal excision plus lateral lymph node dissection (LLND). There are also recent reports on the use of transanal LLND. However, the transanal anatomy is difficult to understand, and additional support tools are required to improve the surgical safety. The present study examined the utility of holograms with mixed reality as an intraoperative support tool for assessing the complex pelvic anatomy. METHODS: Polygon (stereolithography) files of patients' pelvic organs were created and exported from the SYNAPSE VINCENT imaging system and uploaded into the Holoeyes MD virtual reality software. Three-dimensional images were automatically converted into patient-specific holograms. Each hologram was then installed into a head mount display (HoloLens2), and the surgeons and assistants wore the HoloLens2 when they performed transanal LLND. Twelve digestive surgeons with prior practice in hologram manipulation evaluated the utility of the intraoperative hologram support by means of a questionnaire. RESULTS: Intraoperative hologram support improved the surgical understanding of the lateral lymph node region anatomy. In the questionnaire, 75% of the surgeons answered that the hologram accurately reflected the anatomy, and 92% of the surgeons answered that the anatomy was better understood by simulating the hologram intraoperatively than preoperatively. Moreover, 92% of the surgeons agreed that intraoperative holograms were a useful support tool for improving the surgical safety. CONCLUSIONS: Intraoperative hologram support improved the surgical understanding of the pelvic anatomy for transanal LLND. Intraoperative holograms may represent a next-generation surgical tool for transanal LLND.

Intraoperative support with three-dimensional holographic cholangiography in hepatobiliary surgery.

PURPOSE: This study was performed to investigate the potential of intraoperative three-dimensional (3D) holographic cholangiography, which provides a computer graphics model of the biliary tract, with mixed reality techniques. METHODS: Two patients with intraductal papillary neoplasm of the bile duct were enrolled in the study. Intraoperative 3D cholangiography was performed in a hybrid operating room. Three-dimensional polygon data using the acquired cholangiography data were installed into a head mount display (HoloLens; Microsoft Corporation, Redmond, WA, USA). RESULTS: Upon completion of intraoperative 3D cholangiography, a hologram was immediately and successfully made in the operating room using the acquired cholangiography data, and several surgeons wearing the HoloLens succeeded in sharing the same hologram. Compared with usual two-dimensional cholangiography, this 3D holographic cholangiography technique contributed to more accurate reappearance of the bile ducts, especially the B1 origination site, and moving the hologram from the respective operators' angles by means of easy gesture-handling without any monitors. CONCLUSION: Intraoperative 3D holographic cholangiography might be a new next-generation operation-support tool in terms of immediacy, accurate anatomical reappearance, and ease of handling.

Intraoperative holographic image-guided surgery in a transanal approach for rectal cancer.

PURPOSE: Urethral injury is one of the most important complications in transanal total mesorectal excision (TaTME) in male patients with rectal cancer. The purpose of this study was to investigate holographic image-guided surgery in TaTME. METHODS: Polygon (stereolithography) files were created and exported from SYNAPSE VINCENT, and then uploaded into the Holoeyes MD system (Holoeyes Inc., Tokyo, Japan). After uploading the data, the three-dimensional image was automatically converted into a case-specific hologram. The hologram was then installed into the head mount display, HoloLens (Microsoft Corporation, Redmond, WA). The surgeons and assistants wore the HoloLens when they performed TaTME. RESULTS: In a Wi-Fi-enabled operating room, each surgeon, wearing a HoloLens, shared the same hologram and succeeded in adjusting the hologram by making simple hand gestures from their respective angles. The hologram contributed to better comprehension of the positional relationships between the urethra and the surrounding pelvic organs during surgery. All surgeons were able to properly determine the dissection line. CONCLUSIONS: This first experience suggests that intraoperative holograms contributed to reducing the risk of urethral injury and understanding transanal anatomy. Intraoperative holograms have the potential to become a new next-generation surgical support tool for use in spatial awareness and the sharing of information between surgeons.

Digital Transformation Will Change Medical Education and Rehabilitation in Spine Surgery.

The concept of minimally invasive spine therapy (MIST) has been proposed as a treatment strategy to reduce the need for overall patient care, including not only minimally invasive spine surgery (MISS) but also conservative treatment and rehabilitation. To maximize the effectiveness of patient care in spine surgery, the educational needs of medical students, residents, and patient rehabilitation can be enhanced by digital transformation (DX), including virtual reality (VR), augmented reality (AR), mixed reality (MR), and extended reality (XR), three-dimensional (3D) medical images and holograms; wearable sensors, high-performance video cameras, fifth-generation wireless system (5G) and wireless fidelity (Wi-Fi), artificial intelligence, and head-mounted displays (HMDs). Furthermore, to comply with the guidelines for social distancing due to the unexpected COVID-19 pandemic, the use of DX to maintain healthcare and education is becoming more innovative than ever before. In medical education, with the evolution of science and technology, it has become mandatory to provide a highly interactive educational environment and experience using DX technology for residents and medical students, known as digital natives. This study describes an approach to pre- and intraoperative medical education and postoperative rehabilitation using DX in the field of spine surgery that was implemented during the COVID-19 pandemic and will be utilized thereafter.

Intraoperative 3D Hologram Support With Mixed Reality Techniques in Liver Surgery.

OBJECTIVE: The aim of this study was to investigate the potential of an intraoperative 3D hologram, which was a computer graphics model liver, with mixed reality techniques in liver surgery. SUMMARY BACKGROUND DATA: The merits for the application of a hologram for surgical support are: 1) no sterilized display monitor; 2) better spatial awareness; and 3) 3D images shared by all the surgeons. METHODS: 3D polygon data using preoperative computed tomography data was installed into head mount displays, HoloLens (Microsoft Corporation, Redmond, WA). RESULTS: In a Wi-Fi-enabled operative room, several surgeons wearing HoloLens succeeded in sharing the same hologram and moving that hologram from respective operators' angles by means of easy gesture-handling without any monitors. The intraoperative hologram contributed to better imagination of tumor locations, and for determining the parenchymal dissection line in the hepatectomy for the patients with more than 20 multiple colo-rectal liver metastases. In another case, the hologram enabled a safe Gliisonean pedicle approach for hepato-cellular carcinoma with a hilar anatomical anomaly. Surgeons could easily compare the real patient's anatomy and that of the hologram just before the hepatic hilar procedure. CONCLUSIONS: This initial experience suggested that an intraoperative hologram with mixed reality techniques contributed to "last-minute simulation," not for "navigation." The intraoperative hologram might be a new next-generation operation-supportive tool in terms of spatial awareness, sharing, and simplicity.

Holographic image-guided thoracoscopic surgery: possibility of usefulness for esophageal cancer patients with abnormal artery.

BACKGROUND: With the recent improvement of medical image analysis technology, three-dimensional (3D) holograms technology is beginning to be used as intraoperative image support. CASE PRESENTATION: We used a wearable holographic computer during thoracoscopic esophagectomy in a 70-year-old man with esophageal cancer. During lymph node dissection around the right recurrent laryngeal nerve, abnormal blood vessels were observed beside the right subclavian artery (RSA). As a result of confirming the anatomical variation of the right vertebral artery (RVA) using the 3D holograms, it was possible to understand that the RVA branched from a low position on the RSA. CONCLUSIONS: Holographic image-guided thoracoscopic esophagectomy using wearable holographic computer provided better spatial recognition of vascular variation and safe lymph node dissection.

[IMMERSIVE SURGICAL NAVIGATION USING SPATIAL INTERACTIVE VIRTUAL REALITY AND HOLOGRAPHIC AUGMENTED REALITY].

This paper describes the potential role and limitations of current three-dimensional (3D) virtual reality (VR), augmented reality (AR), and holography for image-guided surgery. We developed a new surgical spatial navigation system using VR, AR, and virtual holography. An interactive stereo display is used to view the interactions between the surgeon and stereo images of the patient's anatomy depicted on the display by tracking the surgeon's head and hand/arm positions. Sensing the surgeon's head position creates motion parallax information, an immersive depth cue that can be added to the binocular parallax already present in the display. The beneficial applications of VR/AR devices (head-mounted devices, 3D tablets, and motion sensors) are also discussed. They allow the user to manipulate the spatial attributes of VR, which can enhance spatial reasoning and AR.

An analysis on the effect of body tissues and surgical tools on workflow recognition in first person surgical videos.

PURPOSE: Analysis of operative fields is expected to aid in estimating procedural workflow and evaluating surgeons' procedural skills by considering the temporal transitions during the progression of the surgery. This study aims to propose an automatic recognition system for the procedural workflow by employing machine learning techniques to identify and distinguish elements in the operative field, including body tissues such as fat, muscle, and dermis, along with surgical tools. METHODS: We conducted annotations on approximately 908 first-person-view images of breast surgery to facilitate segmentation. The annotated images were used to train a pixel-level classifier based on Mask R-CNN. To assess the impact on procedural workflow recognition, we annotated an additional 43,007 images. The network, structured on the Transformer architecture, was then trained with surgical images incorporating masks for body tissues and surgical tools. RESULTS: The instance segmentation of each body tissue in the segmentation phase provided insights into the trend of area transitions for each tissue. Simultaneously, the spatial features of the surgical tools were effectively captured. In regard to the accuracy of procedural workflow recognition, accounting for body tissues led to an average improvement of 3 % over the baseline. Furthermore, the inclusion of surgical tools yielded an additional increase in accuracy by 4 % compared to the baseline. CONCLUSION: In this study, we revealed the contribution of the temporal transition of the body tissues and surgical tools spatial features to recognize procedural workflow in first-person-view surgical videos. Body tissues, especially in open surgery, can be a crucial element. This study suggests that further improvements can be achieved by accurately identifying surgical tools specific to each procedural workflow step.

MUC2 expression and prevalence of high-grade dysplasia and invasive carcinoma in mixed-type intraductal papillary mucinous neoplasm of the pancreas.

BACKGROUND/OBJECTIVES: Morphological types and mucin protein expressions classify intraductal papillary mucinous neoplasms (IPMNs). Main duct (MD)-IPMN mostly consists of intestinal type (I-type), which expresses MUC2. Branch duct (BD)-IPMN mostly consists of gastric type (G-type), which does not express MUC2. However, the definition of mixed-type IPMN has yet to be clarified and it contains various histological types. The aim of this study was to investigate the relationship between MUC2 expression and the presence of high-grade dysplasia (HGD) and invasive carcinoma, especially in mixed-type IPMN. METHODS: This retrospective study included 101 consecutive patients with surgically resected IPMNs between April 2001 and October 2012. All patients were morphologically classified into four distinct types (I-type, G-type, PB-type: pancreatobilliary, O-type: oncocytic) and immunohistochemical reactivity of various anti-mucin antibodies were investigated. RESULTS: According to the classification of the 2012 international guidelines, the numbers (and histomorphological types: I/G/PB/O) of MD, mixed-type, and BD-IPMNs were 16 (12/4/0/0), 45 (16/28/1/0), and 40 (0/38/1/1). Prevalence of MUC2 expression in MD, mixed-type, and BD-IPMNs were 75% (12/16), 36% (16/45), and 0% (0/40). In mixed-type IPMN, the prevalence of HGD and/or invasive carcinoma in MUC2-positive IPMN was significantly higher than that of MUC2-negative IPMN (HGD + invasive carcinoma: 88% vs. 38%, p = 0.0017; invasive carcinoma: 50% vs. 21%, p = 0.042). Multivariate analysis showed that MUC2 expression is an independent predictive factor of HGD and invasive carcinoma in mixed IPMN (odds ratio 14.6, 95% CI 2.5-87.4, p = 0.003). CONCLUSIONS: In mixed-type IPMN, MUC2 expression clearly identified HGD and invasive carcinoma and may provide most appropriate surgical indication.

[Clinical Application of Metaverse and Extended Reality in Neurosurgery: Spatial Image Analysis, Remote Surgical Assistance, and Re-live Training].

Metaverse and XR (extended reality) are rapidly gaining traction in the medical field, being used to assist surgeries. Leveraging high-speed communication network and advanced XR devices, medical images and surgical procedures are displayed in a three-dimensional space within the metaverse and are shared among physicians. Furthermore, there are initiatives in which novice doctors can relive the techniques of experienced doctors over time, with hopes that it will improve the overall quality and operational efficiency of the medical profession.

Analyzing the Effect of Diverse Gaze and Head Direction on Facial Expression Recognition With Photo-Reflective Sensors Embedded in a Head-Mounted Display.

As one of the facial expression recognition techniques for Head-Mounted Display (HMD) users, embedded photo-reflective sensors have been used. In this paper, we investigate how gaze and face directions affect facial expression recognition using the embedded photo-reflective sensors. First, we collected a dataset of five facial expressions (Neutral, Happy, Angry, Sad, Surprised) while looking in diverse directions by moving 1) the eyes and 2) the head. Using the dataset, we analyzed the effect of gaze and face directions by constructing facial expression classifiers in five ways and evaluating the classification accuracy of each classifier. The results revealed that the single classifier that learned the data for all gaze points achieved the highest classification performance. Then, we investigated which facial part was affected by the gaze and face direction. The results showed that the gaze directions affected the upper facial parts, while the face directions affected the lower facial parts. In addition, by removing the bias of facial expression reproducibility, we investigated the pure effect of gaze and face directions in three conditions. The results showed that, in terms of gaze direction, building classifiers for each direction significantly improved the classification accuracy. However, in terms of face directions, there were slight differences between the classifier conditions. Our experimental results implied that multiple classifiers corresponding to multiple gaze and face directions improved facial expression recognition accuracy, but collecting the data of the vertical movement of gaze and face is a practical solution to improving facial expression recognition accuracy.

Sensory Attenuation with a Virtual Robotic Arm Controlled Using Facial Movements.

When humans generate stimuli voluntarily, they perceive the stimuli more weakly than those produced by others, which is called sensory attenuation (SA). SA has been investigated in various body parts, but it is unclear whether an extended body induces SA. This study investigated the SA of audio stimuli generated by an extended body. SA was assessed using a sound comparison task in a virtual environment. We prepared robotic arms as extended bodies, and the robotic arms were controlled by facial movements. To evaluate the SA of robotic arms, we conducted two experiments. Experiment 1 investigated the SA of the robotic arms under four conditions. The results showed that robotic arms manipulated by voluntary actions attenuated audio stimuli. Experiment 2 investigated the SA of the robotic arm and innate body under five conditions. The results indicated that the innate body and robotic arm induced SA, while there were differences in the sense of agency between the innate body and robotic arm. Analysis of the results indicated three findings regarding the SA of the extended body. First, controlling the robotic arm with voluntary actions in a virtual environment attenuates the audio stimuli. Second, there were differences in the sense of agency related to SA between extended and innate bodies. Third, the SA of the robotic arm was correlated with the sense of body ownership.

Temporal visuomotor synchrony induces embodiment towards an avatar with biomechanically impossible arm movements.

Visuomotor synchrony in time and space induces a sense of embodiment towards virtual bodies experienced in first-person view using Virtual Reality (VR). Here, we investigated whether temporal visuomotor synchrony affects avatar embodiment even when the movements of the virtual arms are spatially altered from those of the user in a non-human-like manner. In a within-subjects design VR experiment, participants performed a reaching task controlling an avatar whose lower arms bent in inversed and biomechanically impossible directions from the elbow joints. They performed the reaching task using this "unnatural avatar" as well as a "natural avatar," whose arm movements and positions spatially matched the user. The reaching tasks were performed with and without a one second delay between the real and virtual movements. While the senses of body ownership and agency towards the unnatural avatar were significantly lower compared to those towards the natural avatar, temporal visuomotor synchrony did significantly increase the sense of embodiment towards the unnatural avatar as well as the natural avatar. These results suggest that temporal visuomotor synchrony is crucial for inducing embodiment even when the spatial match between the real and virtual limbs is disrupted with movements outside the pre-existing cognitive representations of the human body.

Enhanced Precision in Genioplasty: A Novel Intraoperative Spatial Repositioning Using Computer-Aided Design and Manufacturing Technology and a Holographic Mixed Reality Application.

Genioplasty is performed for the orthognathic surgical correction of dentofacial deformities. This article reports a safe and accurate method for genioplasty combining a novel three-dimensional (3D) device with mixed reality (MR)-assisted surgery using a registration marker and a head-mounted display. Four types of devices were designed based on the virtual operation: a surgical splint with a connector; an osteotomy device; a repositioning device; and a registration marker. Microsoft HoloLens 2 and Holoeyes MD were used to project holograms created using computed tomography (CT) data onto the surgical field to improve the accuracy of the computer-aided designed and manufactured (CAD/CAM) surgical guides. After making an incision on the oral vestibule, the splint was fitted on the teeth and the osteotomy device was mounted at the junction site, placed directly on the exposed mandible bone surface. Temporary screws were fixed into the screw hole. An ultrasonic cutting instrument was used for the osteotomy. After separating the bone, a repositioning device was connected to the splint junction and bone segment, and repositioning was performed. At the time of repositioning, the registration marker was connected to the splint junction, and mandible repositioning was confirmed three-dimensionally through HoloLens 2 into the position specified in the virtual surgery. The rate of overlay error between the preoperative virtual operation and one-month postoperative CT data within 2 mm was 100%. CAD/CAM combined with MR enabled accurate genioplasty.

Directed orientation of asymmetric composite dumbbells by electric field induced assembly.

Assembly and directed orientation of anisotropic particles with an external ac electric field in a range from 1 kHz to 2 MHz were studied for asymmetric composite dumbbells incorporating a silica, titania, or titania/silica (titania:silica = 75:25 vol %) sphere. The asymmetric composite dumbbells, which were composed of a polymethylmethacrylate (PMMA)-coated sphere (core-shell part) and a polystyrene (PSt) lobe, were synthesized with a soap-free emulsion polymerization to prepare PMMA-coated inorganic spheres and another soap-free emulsion polymerization to form a polystyrene (PSt) lobe from the PMMA-coated inorganic spheres. The composite dumbbells dispersed in water were directly observed with optical microscopy. The dumbbells incorporating a silica sphere oriented parallel to an electric field in the whole frequency range and they formed a pearl chain structure at a high frequency of 2 MHz. The titania-incorporated dumbbells formed chain structures, in which they contacted their core-shell parts and oriented perpendicularly to a low-frequency (kHz) field, whereas they oriented parallel to a high-frequency (MHz) field. Since the alignment of dumbbells in the chains depends not only on the interparticle forces but also on the torque that the induced dipoles in the dumbbells experience in the electric field, the orientation of dumbbells perpendicular to the electric field was the case dominated by the interparticle force, whereas the other orientation was the case dominated by the torque. The present experiments show that the incorporation of inorganic dumbbells is an effective way to control the assembled structure and orientation with an electric field.

Clinicopathological characteristics of abnormal micro-lesions at the oro-hypopharynx detected by a magnifying narrow band imaging system.

BACKGROUND: Narrow band imaging (NBI) with magnifying endoscopy (NBI-ME) allows the detection of abnormal micro-lesions smaller than 5 mm in diameter in the oro-hypopharynx that could not be visualized previously. The purpose of the present study was to clarify the clinicopathological characteristics of abnormal micro-lesions of the oro-hypopharynx detected by NBI-ME. METHODS: Of the 62 lesions detected by NBI-ME, 40 abnormal micro-lesions in 37 patients were removed by endoscopic treatment and were pathologically evaluated. We reviewed the medical records of patients with these lesions and investigated the relationship between NBI-ME findings and pathological findings. RESULTS: Pathological examination revealed the following: high-grade intraepithelial neoplasia (HGIN) in nine (23%) lesions, low-grade intraepithelial neoplasia (LGIN) in 22 (55%), pharyngitis in seven (18%) and papilloma in two (5%). Two NBI-ME findings, high microvascular density (MVD) and a brownish area (BA), were recognized more frequently as the grade of malignancy advanced. The likelihood ratio (confidential interval) for having HGIN in the patients with both MVD and BA was 13 (3.62-127). CONCLUSIONS: The pathological diagnosis of abnormal micro-lesions ranged from pharyngitis to HGIN. High MVD and BA may be important findings for grading the malignancy of abnormal micro-lesions.

Effects of virtual hands and feet on the onset time and duration of illusory body ownership.

In the illusory body ownership, humans feel as if a rubber hand or an avatar in a virtual environment is their own body through visual-tactile synchronization or visual-motor synchronization. Despite the onset time and duration of illusory body ownership has been investigated, it is not clear how the onset time and duration change when a part of the body is missing from the full-body. In this study, we investigated the completeness of the full-body for the illusion onset and duration by comparing the following conditions: complete avatar, avatar missing hands and feet, and avatar with hands and feet only. Our results suggest that avatar hands and feet only shorten the duration of the illusion, and missing body parts, such as only hands and feet or no hands and feet, reduce the sense of body ownership and of agency. However, the effects of avatar completeness on the onset time are unclear, and no conclusions can be made in either direction based on the current findings.

Intraoperative holography navigation using a mixed-reality wearable computer during laparoscopic cholecystectomy.

BACKGROUND: Mixed-reality technology, a new digital holographic image technology, is used to present 3-dimensional (3D) images in the surgical space using a wearable mixed-reality device. This study aimed to assess the safety and efficacy of laparoscopic cholecystectomy using a holography-guided navigation system as an intraoperative support image.In this prospective observational study, 27 patients with cholelithiasis or mild cholecystitis underwent laparoscopic cholecystectomy between April 2020 and November 2020. Nine patients underwent laparoscopic cholecystectomy with 3D models generated by a wearable mixed-reality device (laparoscopic cholecystectomy with 3D models) and 18 underwent laparoscopic cholecystectomy with conventional two-dimensional images (laparoscopic cholecystectomy with 2D images) as surgical support images. Surgical outcomes such as operative time, blood loss, and perioperative complication rate were measured, and a four-item questionnaire was used for subjective assessment. All surgeries were performed by a mid-career and an experienced surgeon. RESULTS: Median operative times of laparoscopic cholecystectomy with 3-dimensional models and 2-dimensional images were 74.0 and 58.0 minutes, respectively. No intraoperative blood loss or perioperative complications occurred. Although the midcareer surgeon indicated that laparoscopic cholecystectomy with 3-dimensional models was "normal" or "easy" compared with 2-dimensional images in all cases, the experienced surgeon rated 3-dimensional models as more difficult in 3 (33%) of 9 cases. CONCLUSION: This study provides evidence that laparoscopic cholecystectomy with 3-dimensional models is feasible. However, the efficacy of laparoscopic cholecystectomy with 3-dimensional models may depend on the surgeon's experience, as indicated by the different ratings provided by the surgeons.