Tracking and mapping in medical computer vision: A review.

As computer vision algorithms increase in capability, their applications in clinical systems will become more pervasive. These applications include: diagnostics, such as colonoscopy and bronchoscopy; guiding biopsies, minimally invasive interventions, and surgery; automating instrument motion; and providing image guidance using pre-operative scans. Many of these applications depend on the specific visual nature of medical scenes and require designing algorithms to perform in this environment. In this review, we provide an update to the field of camera-based tracking and scene mapping in surgery and diagnostics in medical computer vision. We begin with describing our review process, which results in a final list of 515 papers that we cover. We then give a high-level summary of the state of the art and provide relevant background for those who need tracking and mapping for their clinical applications. After which, we review datasets provided in the field and the clinical needs that motivate their design. Then, we delve into the algorithmic side, and summarize recent developments. This summary should be especially useful for algorithm designers and to those looking to understand the capability of off-the-shelf methods. We maintain focus on algorithms for deformable environments while also reviewing the essential building blocks in rigid tracking and mapping since there is a large amount of crossover in methods. With the field summarized, we discuss the current state of the tracking and mapping methods along with needs for future algorithms, needs for quantification, and the viability of clinical applications. We then provide some research directions and questions. We conclude that new methods need to be designed or combined to support clinical applications in deformable environments, and more focus needs to be put into collecting datasets for training and evaluation.

Surgical Tattoos in Infrared: A Dataset for Quantifying Tissue Tracking and Mapping.

Quantifying performance of methods for tracking and mapping tissue in endoscopic environments is essential for enabling image guidance and automation of medical interventions and surgery. Datasets developed so far either use rigid environments, visible markers, or require annotators to label salient points in videos after collection. These are respectively: not general, visible to algorithms, or costly and error-prone. We introduce a novel labeling methodology along with a dataset that uses said methodology, Surgical Tattoos in Infrared (STIR). STIR has labels that are persistent but invisible to visible spectrum algorithms. This is done by labelling tissue points with IR-fluorescent dye, indocyanine green (ICG), and then collecting visible light video clips. STIR comprises hundreds of stereo video clips in both in vivo and ex vivo scenes with start and end points labelled in the IR spectrum. With over 3,000 labelled points, STIR will help to quantify and enable better analysis of tracking and mapping methods. After introducing STIR, we analyze multiple different frame-based tracking methods on STIR using both 3D and 2D endpoint error and accuracy metrics. STIR is available at https://dx.doi.org/10.21227/w8g4-g548.

Study on augmented reality for robotic surgery bedside assistants.

Robotic surgery bedside assistants play an important role in robotic procedures by performing intra-corporeal tasks while accommodating the physical presence of the robot. We hypothesized that an augmented reality headset enabling 3D intra-corporeal vision while facing the surgical field could decrease time and improve accuracy of robotic bedside tasks. Bedside assistants (one physician assistant, one medical student, three surgical trainees, and two attending surgeons) performed validated tasks within a mock abdominal cavity with a surgical robot docked. Tasks were performed with a bedside monitor providing 2D or 3D vision, or an optical see-through head-mounted augmented reality device with 2D or 3D vision. The effect of augmented reality device resolution on performance was also evaluated. For the simplest task of touching a straw, performance was generally high, regardless of mode of visualization. With more complex tasks, including stapling and pulling a ring along a path, 3D augmented reality decreased time and number of errors per task. 3D augmented reality allowed the physician assistant to perform at the level of an attending surgeon using 3D augmented reality (p = 0.08). All participants had improved times for the ring path task with better resolution (lower resolution 23 ± 11 s vs higher resolution 14 ± 4 s, p = 0.002). 3D augmented reality vision with high resolution decreased time and improved accuracy of more complex tasks, enabling a less experienced robotic surgical bedside assistant to function similar to attending surgeons. These data warrant further study with additional complex tasks and bedside assistants at various levels of training.

The Path to Surgical Robotics in Neurosurgery.

Robotic systems may help efficiently execute complicated tasks that require a high degree of accuracy, and this, in large part, explains why robotics have garnered widespread use in a variety of neurosurgical applications, including intracranial biopsies, spinal instrumentation, and placement of intracranial leads. The use of robotics in neurosurgery confers many benefits, and inherent limitations, to both surgeons and their patients. In this narrative review, we provide a historical overview of robotics and its implementation across various surgical specialties, and discuss the various robotic systems that have been developed specifically for neurosurgical applications. We also discuss the relative advantages of robotic systems compared to traditional surgical techniques, particularly as it pertains to integration of image guidance with the ability of the robotic arm to reliably execute pre-planned tasks. As more neurosurgeons adopt the use of robotics in their practice, we postulate that further technological advancements will become available that will help achieve improved technical capabilities, user experience, and overall patient clinical outcomes.

MRI-Compatible Pneumatic Robot for Transperineal Prostate Needle Placement.

Magnetic resonance imaging (MRI) can provide high-quality 3-D visualization of prostate and surrounding tissue, thus granting potential to be a superior medical imaging modality for guiding and monitoring prostatic interventions. However, the benefits cannot be readily harnessed for interventional procedures due to difficulties that surround the use of high-field (1.5T or greater) MRI. The inability to use conventional mechatronics and the confined physical space makes it extremely challenging to access the patient. We have designed a robotic assistant system that overcomes these difficulties and promises safe and reliable intraprostatic needle placement inside closed high-field MRI scanners. MRI compatibility of the robot has been evaluated under 3T MRI using standard prostate imaging sequences and average SNR loss is limited to 5%. Needle alignment accuracy of the robot under servo pneumatic control is better than 0.94 mm rms per axis. The complete system workflow has been evaluated in phantom studies with accurate visualization and targeting of five out of five 1 cm targets. The paper explains the robot mechanism and controller design, the system integration, and presents results of preliminary evaluation of the system.

Device connectivity for image-guided medical applications.

The integration of medical devices with software applications is crucial for image-guided medical applications. This work describes a general device interface that has been designed for high-frequency streaming of multi-modal events, thus providing maximum performance and flexibility for such applications. Several sample applications and performance tests are provided to demonstrate the usability of the concept.

3D simulation of needle-tissue interaction with application to prostate brachytherapy.

This paper presents a needle-tissue interaction model that is a 3D extension of prior work based on needle and tissue models discretized using the Finite Element Method. The use of flexible needles necessitates remeshing the tissue during insertion, since simple mesh-node snapping to the tip can be detrimental to the simulation. In this paper, node repositioning and node addition are the two methods of mesh modification examined for coarse meshes. Our focus is on numerical approaches for fast implementation of these techniques. Although the two approaches compared, namely the Woodbury formula (matrix inversion lemma) and the boundary condition switches, have the same computational complexity, the Woodbury formula is shown to perform faster due to its cache-efficient order of operations. Furthermore, node addition is applied in constant time for both approaches, whereas node repositioning requires longer and variable computational times. A method for rendering the needle forces during simulated insertions into a 3D prostate model has been implemented. Combined with a detailed anatomical segmentation, this will be useful in teaching the practice of prostate brachytherapy. Issues related to discretization of such coupled (e.g., needle-tissue) models are also discussed.

Magnetic resonance-guided prostate interventions.

We review our experience using an open 0.5-T magnetic resonance (MR) interventional unit to guide procedures in the prostate. This system allows access to the patient and real-time MR imaging simultaneously and has made it possible to perform prostate biopsy and brachytherapy under MR guidance. We review MR imaging of the prostate and its use in targeted therapy, and describe our use of image processing methods such as image registration to further facilitate precise targeting. We describe current developments with a robot assist system being developed to aid radioactive seed placement.

Needle steering and motion planning in soft tissues.

In this work, needle insertion into deformable tissue is formulated as a trajectory planning and control problem. A new concept of needle steering has been developed and a needle manipulation Jacobian defined using numerical needle insertion models that include needle deflection and soft tissue deformation. This concept is used in conjunction with a potential-field-based path planning technique to demonstrate needle tip placement and obstacle avoidance. Results from open loop insertion experiments are provided.

Interactive simulation of needle insertion models.

A novel interactive virtual needle insertion simulation is presented. The simulation models are based on measured planar tissue deformations and needle insertion forces. Since the force-displacement relationship is only of interest along the needle shaft, a condensation technique is shown to reduce the computational complexity of linear simulation models significantly. As the needle penetrates or is withdrawn from the tissue model, the boundary conditions that determine the tissue and needle motion change. Boundary condition and local material coordinate changes are facilitated by fast low-rank matrix updates. A large-strain elastic needle model is coupled to the tissue models to account for needle deflection and bending during simulated insertion. A haptic environment, based on these novel interactive simulation techniques, allows users to manipulate a three-degree-of-freedom virtual needle as it penetrates virtual tissue models, while experiencing steering torques and lateral needle forces through a planar haptic interface.

Robot-assisted laparoscopic ultrasonography for hepatic surgery.

INTRODUCTION: This study describes and evaluates a novel, robot-assisted laparoscopic ultrasonographic device for hepatic surgery. Laparoscopic liver surgery is being performed with increasing frequency. One major drawback of this approach is the limited capability of intraoperative ultrasonography (IOUS) using standard laparoscopic devices. Robotic surgery systems offer the opportunity to develop new tools to improve techniques in minimally invasive surgery. This study evaluates a new integrated ultrasonography (US) device with the da Vinci Surgical System for laparoscopic visualization, comparing it with conventional handheld laparoscopic IOUS for performing key tasks in hepatic surgery. METHODS: A prototype laparoscopic IOUS instrument was developed for the da Vinci Surgical System and compared with a conventional laparoscopic US device in simulation tasks: (1) In vivo porcine hepatic visualization and probe manipulation, (2) lesion detection accuracy, and (3) biopsy precision. Usability was queried by poststudy questionnaire. RESULTS: The robotic US proved better than conventional laparoscopic US in liver surface exploration (85% success vs 73%; P = .030) and tool manipulation (79% vs 57%; P = .028), whereas no difference was detected in lesion identification (63 vs 58; P = .41) and needle biopsy tasks (57 vs 48; P = .11). Subjects found the robotic US to facilitate better probe positioning (80%), decrease fatigue (90%), and be more useful overall (90%) on the post-task questionnaire. CONCLUSION: We found this robot-assisted IOUS system to be practical and useful in the performance of important tasks required for hepatic surgery, outperforming free-hand laparoscopic IOUS for certain tasks, and was more subjectively usable to the surgeon. Systems such as this may expand the use of robotic surgery for complex operative procedures requiring IOUS.

Integrated navigation and control software system for MRI-guided robotic prostate interventions.

A software system to provide intuitive navigation for MRI-guided robotic transperineal prostate therapy is presented. In the system, the robot control unit, the MRI scanner, and the open-source navigation software are connected together via Ethernet to exchange commands, coordinates, and images using an open network communication protocol, OpenIGTLink. The system has six states called "workphases" that provide the necessary synchronization of all components during each stage of the clinical workflow, and the user interface guides the operator linearly through these workphases. On top of this framework, the software provides the following features for needle guidance: interactive target planning; 3D image visualization with current needle position; treatment monitoring through real-time MR images of needle trajectories in the prostate. These features are supported by calibration of robot and image coordinates by fiducial-based registration. Performance tests show that the registration error of the system was 2.6mm within the prostate volume. Registered real-time 2D images were displayed 1.97 s after the image location is specified.

Compensation of geometric distortion effects on intraoperative magnetic resonance imaging for enhanced visualization in image-guided neurosurgery.

OBJECTIVE: Preoperative magnetic resonance imaging (MRI), functional MRI, diffusion tensor MRI, magnetic resonance spectroscopy, and positron-emission tomographic scans may be aligned to intraoperative MRI to enhance visualization and navigation during image-guided neurosurgery. However, several effects (both machine- and patient-induced distortions) lead to significant geometric distortion of intraoperative MRI. Therefore, a precise alignment of these image modalities requires correction of the geometric distortion. We propose and evaluate a novel method to compensate for the geometric distortion of intraoperative 0.5-T MRI in image-guided neurosurgery. METHODS: In this initial pilot study, 11 neurosurgical procedures were prospectively enrolled. The scheme used to correct the geometric distortion is based on a nonrigid registration algorithm introduced by our group. This registration scheme uses image features to establish correspondence between images. It estimates a smooth geometric distortion compensation field by regularizing the displacements estimated at the correspondences. A patient-specific linear elastic material model is used to achieve the regularization. The geometry of intraoperative images (0.5 T) is changed so that the images match the preoperative MRI scans (3 T). RESULTS: We compared the alignment between preoperative and intraoperative imaging using 1) only rigid registration without correction of the geometric distortion, and 2) rigid registration and compensation for the geometric distortion. We evaluated the success of the geometric distortion correction algorithm by measuring the Hausdorff distance between boundaries in the 3-T and 0.5-T MRIs after rigid registration alone and with the addition of geometric distortion correction of the 0.5-T MRI. Overall, the mean magnitude of the geometric distortion measured on the intraoperative images is 10.3 mm with a minimum of 2.91 mm and a maximum of 21.5 mm. The measured accuracy of the geometric distortion compensation algorithm is 1.93 mm. There is a statistically significant difference between the accuracy of the alignment of preoperative and intraoperative images, both with and without the correction of geometric distortion (P < 0.001). CONCLUSION: The major contributions of this study are 1) identification of geometric distortion of intraoperative images relative to preoperative images, 2) measurement of the geometric distortion, 3) application of nonrigid registration to compensate for geometric distortion during neurosurgery, 4) measurement of residual distortion after geometric distortion correction, and 5) phantom study to quantify geometric distortion.

Software strategy for robotic transperineal prostate therapy in closed-bore MRI.

A software strategy to provide intuitive navigation for MRI-guided robotic transperineal prostate therapy is presented. In the system, the robot control unit, the MRI scanner, and open-source navigation software are connected to one another via Ethernet to exchange commands, coordinates, and images. Six states of the system called "workphases" are defined based on the clinical scenario to synchronize behaviors of all components. The wizard-style user interface allows easy following of the clinical workflow. On top of this framework, the software provides features for intuitive needle guidance: interactive target planning; 3D image visualization with current needle position; treatment monitoring through real-time MRI. These features are supported by calibration of robot and image coordinates by the fiducial-based registration. The performance test shows that the registration error of the system was 2.6 mm in the prostate area, and it displayed real-time 2D image 1.7 s after the completion of image acquisition.

Transperineal prostate biopsy under magnetic resonance image guidance: a needle placement accuracy study.

PURPOSE: To quantify needle placement accuracy of magnetic resonance image (MRI)-guided core needle biopsy of the prostate. MATERIALS AND METHODS: A total of 10 biopsies were performed with 18-gauge (G) core biopsy needle via a percutaneous transperineal approach. Needle placement error was assessed by comparing the coordinates of preplanned targets with the needle tip measured from the intraprocedural coherent gradient echo images. The source of these errors was subsequently investigated by measuring displacement caused by needle deflection and needle susceptibility artifact shift in controlled phantom studies. Needle placement error due to misalignment of the needle template guide was also evaluated. RESULTS: The mean and standard deviation (SD) of errors in targeted biopsies was 6.5 +/- 3.5 mm. Phantom experiments showed significant placement error due to needle deflection with a needle with an asymmetrically beveled tip (3.2-8.7 mm depending on tissue type) but significantly smaller error with a symmetrical bevel (0.6-1.1 mm). Needle susceptibility artifacts observed a shift of 1.6 +/- 0.4 mm from the true needle axis. Misalignment of the needle template guide contributed an error of 1.5 +/- 0.3 mm. CONCLUSION: Needle placement error was clinically significant in MRI-guided biopsy for diagnosis of prostate cancer. Needle placement error due to needle deflection was the most significant cause of error, especially for needles with an asymmetrical bevel.

Robotic assistant for transperineal prostate interventions in 3T closed MRI.

Numerous studies have demonstrated the efficacy of image-guided needle-based therapy and biopsy in the management of prostate cancer. The accuracy of traditional prostate interventions performed using transrectal ultrasound (TRUS) is limited by image fidelity, needle template guides, needle deflection and tissue deformation. Magnetic Resonance Imaging (MRI) is an ideal modality for guiding and monitoring such interventions due to its excellent visualization of the prostate, its sub-structure and surrounding tissues. We have designed a comprehensive robotic assistant system that allows prostate biopsy and brachytherapy procedures to be performed entirely inside a 3T closed MRI scanner. We present a detailed design of the robotic manipulator and an evaluation of its usability and MR compatibility.

Challenges in image-guided therapy system design.

System development for image-guided therapy (IGT), or image-guided interventions (IGI), continues to be an area of active interest across academic and industry groups. This is an emerging field that is growing rapidly: major academic institutions and medical device manufacturers have produced IGT technologies that are in routine clinical use, dozens of high-impact publications are published in well regarded journals each year, and several small companies have successfully commercialized sophisticated IGT systems. In meetings between IGT investigators over the last two years, a consensus has emerged that several key areas must be addressed collaboratively by the community to reach the next level of impact and efficiency in IGT research and development to improve patient care. These meetings culminated in a two-day workshop that brought together several academic and industrial leaders in the field today. The goals of the workshop were to identify gaps in the engineering infrastructure available to IGT researchers, develop the role of research funding agencies and the recently established US-based National Center for Image Guided Therapy (NCIGT), and ultimately to facilitate the transfer of technology among research centers that are sponsored by the National Institutes of Health (NIH). Workshop discussions spanned many of the current challenges in the development and deployment of new IGT systems. Key challenges were identified in a number of areas, including: validation standards; workflows, use-cases, and application requirements; component reusability; and device interface standards. This report elaborates on these key points and proposes research challenges that are to be addressed by a joint effort between academic, industry, and NIH participants.

3D needle-tissue interaction simulation for prostate brachytherapy.

This paper presents a needle-tissue interaction model that is a 3D extension of a prior work based on the finite element method. The model is also adapted to accommodate arbitrary meshes so that the anatomy can effectively be meshed using third-party algorithms. Using this model a prostate brachytherapy simulator is designed to help medical residents acquire needle steering skills. This simulation uses a prostate mesh generated from clinical data segmented as contours on parallel slices. Node repositioning and addition, which are methods for achieving needle-tissue coupling, are discussed. In order to achieve realtime haptic rates, computational approaches to these methods are compared. Specifically, the benefit of using the Woodbury formula (matrix inversion lemma) is studied. Our simulation of needle insertion into a prostate is shown to run faster than 1 kHz.