A holographic telementoring system depicting surgical instrument movements for real-time guidance in open surgeries.

BACKGROUND AND OBJECTIVE: During open surgeries, telementoring serves as a valuable tool for transferring surgical knowledge from a specialist surgeon (mentor) to an operating surgeon (mentee). Depicting the intended movements of the surgical instruments over the operative field improves the understanding of the required tool-tissue interaction. The objective of this work is to develop a telementoring system tailored for open surgeries, enabling the mentor to remotely demonstrate the necessary motions of surgical instruments to the mentee. METHODS: A remote telementoring system for open surgery was implemented. The system generates visual cues in the form of virtual surgical instrument motion augmented onto the live view of the operative field. These cues can be rendered on both conventional screens in the operating room and as dynamic holograms on a head mounted display device worn by the mentee. The technical performance of the system was evaluated, where the operating room and remote location were geographically separated and connected via the Internet. Additionally, user studies were conducted to assess the effectiveness of the system as a mentoring tool. RESULTS: The system took 307 ± 12 ms to transmit an operative field view of 1920 × 1080 resolution, along with depth information spanning 36 cm, from the operating room to the remote location. Conversely, it took 145 ± 14 ms to receive the motion of virtual surgical instruments from the remote location back to the operating room. Furthermore, the user studies demonstrated: (a) mentor's capability to annotate the operative field with an accuracy of 3.92 ± 2.1 mm, (b) mentee's ability to comprehend and replicate the motion of surgical instruments in real-time with an average deviation of 12.8 ± 3 mm, (c) efficacy of the rendered dynamic holograms in conveying information intended for surgical instrument motion. CONCLUSIONS: The study demonstrates the feasibility of transmitting information over the Internet from the mentor to the mentee in the form of virtual surgical instruments' motion and projecting it as holograms onto the live view of the operative field. This holds potential to enhance real-time collaborative capabilities between the mentor and the mentee during an open surgery.

An Actuated Variable-View Rigid Scope System to Assist Visualization in Diagnostic Procedures.

OBJECTIVE: Variable-view rigid scopes offer advantages compared to traditional angled laparoscopes for examining a diagnostic site. However, altering the scope's view requires a high level of dexterity and understanding of spatial orientation. This requires an intuitive mechanism to allow an operator to easily understand the anatomical surroundings and smoothly adjust the scope's focus during diagnosis. To address this challenge, the objective of this work is to develop a mechanized arm that assists in visualization using variable-view rigid scopes during diagnostic procedures. METHODS: A system with a mechanized arm to maneuver a variable-view rigid scope (EndoCAMeleon - Karl Storz) was developed. A user study was conducted to assess the ability of the proposed mechanized arm for diagnosis in a preclinical navigation task and a simulated cystoscopy procedure. RESULTS: The mechanized arm performed significantly better than direct maneuvering of the rigid scope. In the preclinical navigation task, it reduced the percentage of time the scope's focus shifted outside a predefined track. Similarly, for simulated cystoscopy procedure, it reduced the duration and the perceived workload. CONCLUSION: The proposed mechanized arm enhances the operator's ability to accurately maneuver a variable-view rigid scope and reduces the effort in performing diagnostic procedures.Clinical and Translational Impact Statement: The preclinical research introduces a mechanized arm to intuitively maneuver a variable-view rigid scope during diagnostic procedures, while minimizing the mental and physical workload to the operator.

Technologies Used for Telementoring in Open Surgery: A Scoping Review.

Background: Telementoring technologies enable a remote mentor to guide a mentee in real-time during surgical procedures. This addresses challenges, such as lack of expertise and limited surgical training/education opportunities in remote locations. This review aims to provide a comprehensive account of these technologies tailored for open surgery. Methods: A comprehensive scoping review of the scientific literature was conducted using PubMed, ScienceDirect, ACM Digital Library, and IEEE Xplore databases. Broad and inclusive searches were done to identify articles reporting telementoring or teleguidance technologies in open surgery. Results: Screening of the search results yielded 43 articles describing surgical telementoring for open approach. The studies were categorized based on the type of open surgery (surgical specialty, surgical procedure, and stage of clinical trial), the telementoring technology used (information transferred between mentor and mentee, devices used for rendering the information), and assessment of the technology (experience level of mentor and mentee, study design, and assessment criteria). Majority of the telementoring technologies focused on trauma-related surgeries and mixed reality headsets were commonly used for rendering information (telestrations, surgical tools, or hand gestures) to the mentee. These technologies were primarily assessed on high-fidelity synthetic phantoms. Conclusions: Despite longer operative time, these telementoring technologies demonstrated clinical viability during open surgeries through improved performance and confidence of the mentee. In general, usage of immersive devices and annotations appears to be promising, although further clinical trials will be required to thoroughly assess its benefits.

A generic scope actuation system for flexible endoscopes.

BACKGROUND: A scope actuation system assists a surgeon in steering a scope for navigating an operative field during an interventional or diagnostic procedure. Each system is tailored for a specific surgical procedure. The development of a generic scope actuation system could assist various laparoscopic and endoscopic procedures. This has the potential to reduce the deployment and maintenance costs for a hospital, making it more accessible for clinical usage. METHODS: A modular actuation system (for maneuvering rigid laparoscopes) was adapted to enable incorporation of flexible endoscopes. The design simplifies the installation and disassembly processes. User studies were conducted to assess the ability of the system to focus onto a diagnostic area, and to navigate during a simulated esophagogastroduodenoscopy procedure. During the studies, the endoscope was maneuvered with (robotic mode) and without (manual mode) the actuation system to navigate the endoscope's focus on a predefined track. RESULTS: Results show that the robotic mode performed better than the manual mode on all the measured performance parameters including (a) the total duration to traverse a track, (b) the percentage of time spent outside a track while traversing, and (c) the number of times the scope focus shifts outside the track. Additionally, robotic mode also reduced the perceived workload based on the NASA-TLX scale. CONCLUSIONS: The proposed scope actuation system enhances the maneuverability of flexible endoscopes. It also lays the groundwork for future development of modular and generic scope assistant systems that can be used in both laparoscopic and endoscopic procedures.

Development and Evaluation of a Mixed-Reality Tele-ultrasound System.

OBJECTIVE: The objective of this feasibility study was to develop and assess a tele-ultrasound system that would enable an expert sonographer (situated at the remote site) to provide real-time guidance to an operator (situated at the imaging site) using a mixed-reality environment. METHODS: An architecture along with the operational workflow of the system is designed and a prototype is developed that enables guidance in form of audiovisual cues. The visual cues comprise holograms (of the ultrasound images and ultrasound probe) and is rendered to the operator using a head-mounted display device. The position and orientation of the ultrasound probe's hologram are remotely controlled by the expert sonographer and guide the placement of a physical ultrasound probe at the imaging site. The developed prototype was evaluated for its performance on a network. In addition, a user study (with 12 participants) was conducted to assess the operator's ability to align the probe under different guidance modes. RESULTS: The network performance revealed the view of the imaging site and ultrasound images were transferred to the remote site in 233 ± 42 and 158 ± 38 ms, respectively. The expert sonographer was able to transfer, to the imaging site, data related to position and orientation of the ultrasound probe's hologram in 78 ± 13 ms. The user study indicated that the audiovisual cues are sufficient for an operator to position and orient a physical probe for accurate depiction of the targeted tissue (p < 0.001). The probe's placement translational and rotational errors were 1.4 ± 0.6 mm and 5.4 ± 2.2º. CONCLUSION: The work illustrates the feasibility of using a mixed-reality environment for effective communication between an expert sonographer (ultrasound physician) and an operator. Further studies are required to determine its applicability in a clinical setting during tele-ultrasound.

User interfaces for actuated scope maneuvering in surgical systems: a scoping review.

BACKGROUND: A variety of human computer interfaces are used by robotic surgical systems to control and actuate camera scopes during minimally invasive surgery. The purpose of this review is to examine the different user interfaces used in both commercial systems and research prototypes. METHODS: A comprehensive scoping review of scientific literature was conducted using PubMed and IEEE Xplore databases to identify user interfaces used in commercial products and research prototypes of robotic surgical systems and robotic scope holders. Papers related to actuated scopes with human-computer interfaces were included. Several aspects of user interfaces for scope manipulation in commercial and research systems were reviewed. RESULTS: Scope assistance was classified into robotic surgical systems (for multiple port, single port, and natural orifice) and robotic scope holders (for rigid, articulated, and flexible endoscopes). Benefits and drawbacks of control by different user interfaces such as foot, hand, voice, head, eye, and tool tracking were outlined. In the review, it was observed that hand control, with its familiarity and intuitiveness, is the most used interface in commercially available systems. Control by foot, head tracking, and tool tracking are increasingly used to address limitations, such as interruptions to surgical workflow, caused by using a hand interface. CONCLUSION: Integrating a combination of different user interfaces for scope manipulation may provide maximum benefit for the surgeons. However, smooth transition between interfaces might pose a challenge while combining controls.

Scope actuation system for articulated laparoscopes.

BACKGROUND: An articulated laparoscope comprises a rigid shaft with an articulated distal end to change the viewing direction. The articulation provides improved navigation of the operating field in confined spaces. Furthermore, incorporation of an actuation system tends to enhance the control of an articulated laparoscope. METHODS: A preliminary prototype of a scope actuation system to maneuver an off-the-shelf articulated laparoscope (EndoCAMaleon by Karl Storz, Germany) was developed. A user study was conducted to evaluate this prototype for the surgical paradigm of video-assisted thoracic surgery. In the study, the subjects maneuvered an articulated scope under two modes of operation: (a) actuated mode where an operating surgeon maneuvers the scope using the developed prototype and (b) manual mode where a surgical assistant directly maneuvers the scope. The actuated mode was further assessed for multiple configurations based on the orientation of the articulated scope at the incision. RESULTS: The data show the actuated mode scored better than the manual mode on all the measured performance parameters including (a) total duration to visualize a marked region, (a) duration for which scope focus shifts outside a predefined visualization region, and (c) number of times for which scope focus shifts outside a predefined visualization region. Among the different configurations tested using the actuated mode, no significant difference was observed. CONCLUSIONS: The proposed articulated scope actuation system facilitates better navigation of an operative field as compared to a human assistant. Secondly, irrespective of the orientation in which an articulated scope's shaft is inserted through an incision, the proposed actuation system can navigate and visualize the operative field.

Evaluation of user-interfaces for controlling movements of virtual minimally invasive surgical instruments.

BACKGROUND: Recent tele-mentoring technologies for minimally invasive surgery (MIS) augments the operative field with movements of virtual surgical instruments as visual cues. The objective of this work is to assess different user-interfaces that effectively transfer mentor's hand gestures to the movements of virtual surgical instruments. METHODS: A user study was conducted to assess three different user-interface devices (Oculus-Rift, SpaceMouse, Touch Haptic device) under various scenarios. The devices were integrated with a MIS tele-mentoring framework for control of both manual and robotic virtual surgical instruments. RESULTS: The user study revealed that Oculus Rift is preferred during robotic scenarios, whereas the touch haptic device is more suitable during manual scenarios for tele-mentoring. CONCLUSION: A user-interface device in the form of a stylus controlled by fingers for pointing in 3D space is more suitable for manual MIS, whereas a user-interface that can be moved and oriented easily in 3D space by wrist motion is more suitable for robotic MIS.

Preliminary design and evaluation of a remote tele-mentoring system for minimally invasive surgery.

BACKGROUND: Tele-mentoring during surgery facilitates the transfer of surgical knowledge from a mentor (specialist surgeon) to a mentee (operating surgeon). The aim of this work is to develop a tele-mentoring system tailored for minimally invasive surgery (MIS) where the mentor can remotely demonstrate to the mentee the required motion of the surgical instruments. METHODS: A remote tele-mentoring system is implemented that generates visual cues in the form of virtual surgical instrument motion overlaid onto the live view of the operative field. The technical performance of the system is evaluated in a simulated environment, where the operating room and the central location of the mentor were physically located in different countries and connected over the internet. In addition, a user study was performed to assess the system as a mentoring tool. RESULTS: On average, it took 260 ms to send a view of the operative field of 1920 × 1080 resolution from the operating room to the central location of the mentor and an average of 132 ms to receive the motion of virtual surgical instruments from the central location to the operating room. The user study showed that it is feasible for the mentor to demonstrate and for the mentee to understand and replicate the motion of surgical instruments. CONCLUSION: The work demonstrates the feasibility of transferring information over the internet from a mentor to a mentee in the form of virtual surgical instruments. Their motion is overlaid onto the live view of the operative field enabling real-time interactions between both the surgeons.

Towards development of a tele-mentoring framework for minimally invasive surgeries.

BACKGROUND: Tele-mentoring facilitates the transfer of surgical knowledge. The objective of this work is to develop a tele-mentoring framework that enables a specialist surgeon to mentor an operating surgeon by transferring information in a form of surgical instruments' motion required during a minimally invasive surgery. METHOD: A tele-mentoring framework is developed to transfer video stream of the surgical field, poses of the scope and port placement from the operating room to a remote location. From the remote location, the motion of virtual surgical instruments augmented onto the surgical field is sent to the operating room. RESULTS: The proposed framework is suitable to be integrated with laparoscopic as well as robotic surgeries. It takes on average 1.56 s to send information from the operating room to the remote location and 0.089 s for vice versa over a local area network. CONCLUSIONS: The work demonstrates a tele-mentoring framework that enables a specialist surgeon to mentor an operating surgeon during a minimally invasive surgery.

Evaluation of how users interface with holographic augmented reality surgical scenes: Interactive planning MR-Guided prostate biopsies.

BACKGROUND: User interfaces play a vital role in the planning and execution of an interventional procedure. The objective of this study is to investigate the effect of using different user interfaces for planning transrectal robot-assisted MR-guided prostate biopsy (MRgPBx) in an augmented reality (AR) environment. METHOD: End-user studies were conducted by simulating an MRgPBx system with end- and side-firing modes. The information from the system to the operator was rendered on HoloLens as an output interface. Joystick, mouse/keyboard, and holographic menus were used as input interfaces to the system. RESULTS: The studies indicated that using a joystick improved the interactive capacity and enabled operator to plan MRgPBx in less time. It efficiently captures the operator's commands to manipulate the augmented environment representing the state of MRgPBx system. CONCLUSIONS: The study demonstrates an alternative to conventional input interfaces to interact and manipulate an AR environment within the context of MRgPBx planning.

End-user evaluation of software-generated intervention planning environment for transrectal magnetic resonance-guided prostate biopsies.

BACKGROUND: This study presents user evaluation studies to assess the effect of information rendered by an interventional planning software on the operator's ability to plan transrectal magnetic resonance (MR)-guided prostate biopsies using actuated robotic manipulators. METHODS: An intervention planning software was developed based on the clinical workflow followed for MR-guided transrectal prostate biopsies. The software was designed to interface with a generic virtual manipulator and simulate an intervention environment using 2D and 3D scenes. User studies were conducted with urologists using the developed software to plan virtual biopsies. RESULTS: User studies demonstrated that urologists with prior experience in using 3D software completed the planning less time. 3D scenes were required to control all degrees-of-freedom of the manipulator, while 2D scenes were sufficient for planar motion of the manipulator. CONCLUSIONS: The study provides insights on using 2D versus 3D environment from a urologist's perspective for different operational modes of MR-guided prostate biopsy systems.

A Platform Integrating Acquisition, Reconstruction, Visualization, and Manipulator Control Modules for MRI-Guided Interventions.

This work presents a platform that integrates a customized MRI data acquisition scheme with reconstruction and three-dimensional (3D) visualization modules along with a module for controlling an MRI-compatible robotic device to facilitate the performance of robot-assisted, MRI-guided interventional procedures. Using dynamically-acquired MRI data, the computational framework of the platform generates and updates a 3D model representing the area of the procedure (AoP). To image structures of interest in the AoP that do not reside inside the same or parallel slices, the MRI acquisition scheme was modified to collect a multi-slice set of intraoblique to each other slices; which are termed composing slices. Moreover, this approach interleaves the collection of the composing slices so the same k-space segments of all slices are collected during similar time instances. This time matching of the k-space segments results in spatial matching of the imaged objects in the individual composing slices. The composing slices were used to generate and update the 3D model of the AoP. The MRI acquisition scheme was evaluated with computer simulations and experimental studies. Computer simulations demonstrated that k-space segmentation and time-matched interleaved acquisition of these segments provide spatial matching of the structures imaged with composing slices. Experimental studies used the platform to image the maneuvering of an MRI-compatible manipulator that carried tubing filled with MRI contrast agent. In vivo experimental studies to image the abdomen and contrast enhanced heart on free-breathing subjects without cardiac triggering demonstrated spatial matching of imaged anatomies in the composing planes. The described interventional MRI framework could assist in performing real-time MRI-guided interventions.

Constrained Statistical Modelling of Knee Flexion From Multi-Pose Magnetic Resonance Imaging.

Reconstruction of the anterior cruciate ligament (ACL) through arthroscopy is one of the most common procedures in orthopaedics. It requires accurate alignment and drilling of the tibial and femoral tunnels through which the ligament graft is attached. Although commercial computer-assisted navigation systems exist to guide the placement of these tunnels, most of them are limited to a fixed pose without due consideration of dynamic factors involved in different knee flexion angles. This paper presents a new model for intraoperative guidance of arthroscopic ACL reconstruction with reduced error particularly in the ligament attachment area. The method uses 3D preoperative data at different flexion angles to build a subject-specific statistical model of knee pose. To circumvent the problem of limited training samples and ensure physically meaningful pose instantiation, homogeneous transformations between different poses and local-deformation finite element modelling are used to enlarge the training set. Subsequently, an anatomical geodesic flexion analysis is performed to extract the subject-specific flexion characteristics. The advantages of the method were also tested by detailed comparison to standard Principal Component Analysis (PCA), nonlinear PCA without training set enlargement, and other state-of-the-art articulated joint modelling methods. The method yielded sub-millimetre accuracy, demonstrating its potential clinical value.

Preliminary design of an actuated imaging probe for generation of additional visual cues in a robotic surgery.

BACKGROUND: The aim of this study was to enhance the visual feedback of surgeons, during robotic surgeries, by designing and developing an actuated 2D imaging probe, which is used in conjunction with the traditional stereoscopic camera of the da Vinci surgical system. The probe provides the surgeon with additional visual cues, overcoming visualization constraints encountered during certain scenarios of robot-assisted minimally invasive surgery. METHODS: The actuated imaging probe is implemented as a master-slave tele-manipulated system, and it is designed to be compatible with the da Vinci surgical system. The detachable probe design enables it to be mounted on any of the EndoWrist(®) instruments of the robot and is controlled by the surgeon using a custom-made pedal system. The image from the 2D probe is rendered along with the stereoscopic view on the surgeon's console. RESULTS: The experimental results demonstrate the effectiveness of the proposed actuated imaging probe when used as an additional visualization channel and in surgical scenarios presenting visual problems due to tissue occlusion. CONCLUSION: The study shows the potential benefits of an additional actuated imaging probe when used in conjunction with traditional surgical instruments to perform surgical tasks requiring visualization from multiple orientations and workspaces.

GPU-accelerated interactive visualization and planning of neurosurgical interventions.

Advances in computational methods and hardware platforms provide efficient processing of medical-imaging datasets for surgical planning. For neurosurgical interventions employing a straight access path, planning entails selecting a path from the scalp to the target area that's of minimal risk to the patient. A proposed GPU-accelerated method enables interactive quantitative estimation of the risk for a particular path. It exploits acceleration spatial data structures and efficient implementation of algorithms on GPUs. In evaluations of its computational efficiency and scalability, it achieved interactive rates even for high-resolution meshes. A user study and feedback from neurosurgeons identified this methods' potential benefits for preoperative planning and intraoperative replanning.

A framework for integrating real-time MRI with robot control: application to simulated transapical cardiac interventions.

The advent of intraoperative real-time image guidance has led to the emergence of new surgical interventional paradigms including image-guided robot assistance. Most often the use of an intraoperative imaging modality is limited to visual perception of the area of procedure. In this study, we propose a framework for performing robot-assisted interventions with real-time magnetic resonance imaging (rtMRI) guidance. The described computational core of this framework, processes on-the-fly rtMRI, integrates the processed information with robot control and renders it on the human-machine interfaces. This information is rendered on a visualization and force-feedback interface for enhanced perception of a dynamic area of procedure and for assisting the operator in the safe and accurate maneuvering of a robotic manipulator. The framework was experimentally tested by applying it to a simulated Transapical aortic valve implantation with a virtual robotic manipulator. rtMRI data were processed on-the-fly in a rolling-window scheme and together with a multithreaded and multihardware implementation, the core delivered appropriate speed of 20 Hz for visualization and 1000 Hz for force feedback. The experimental results demonstrate significant improvement in the simulated task by both decreasing the duration of the procedure by half and increasing safety in the presence of cardiac and breathing motion by reducing the duration or incidents the operator collides with the tissue.

MR-based real time path planning for cardiac operations with transapical access.

Minimally invasive surgeries (MIS) have been perpetually evolving due to their potential high impact on improving patient management and overall cost effectiveness. Currently, MIS are further strengthened by the incorporation of magnetic resonance imaging (MRI) for amended visualization and high precision. Motivated by the fact that real-time MRI is emerging as a feasible modality especially for guiding interventions and surgeries in the beating heart; in this paper we introduce a real-time path planning algorithm for intracardiac procedures. Our approach creates a volumetric safety zone inside a beating heart and updates it on-the-fly using real-time MRI during the deployment of a robotic device. In order to prove the concept and assess the feasibility of the introduced method, a realistic operational scenario of transapical aortic valve replacement in a beating heart is chosen as the virtual case study.