3D Visualization and Augmented Reality for Orthopedics.

Augmented reality (AR) techniques play an important role in the field of minimally invasive surgery for orthopedics. AR can improve the hand-eye coordination by providing surgeons with the merged surgical scene, which enables surgeons to perform surgical operations more easily. To display the navigation information in the AR scene, medical image processing and three-dimensional (3D) visualization of the important anatomical structures are required. As a promising 3D display technique, integral videography (IV) can produce an autostereoscopic image with full parallax and continuous viewing points. Moreover, IV-based 3D AR navigation technique is proposed to present intuitive scene and has been applied in orthopedics, including oral surgery and spine surgery. The accurate patient-image registration, as well as the real-time target tracking for surgical tools and the patient, can be achieved. This paper overviews IV-based AR navigation and the applications in orthopedics, discusses the infrastructure required for successful implementation of IV-based approaches, and outlines the challenges that must be overcome for IV-based AR navigation to advance further development.

Vision-based markerless registration using stereo vision and an augmented reality surgical navigation system: a pilot study.

BACKGROUND: This study evaluated the use of an augmented reality navigation system that provides a markerless registration system using stereo vision in oral and maxillofacial surgery. METHOD: A feasibility study was performed on a subject, wherein a stereo camera was used for tracking and markerless registration. The computed tomography data obtained from the volunteer was used to create an integral videography image and a 3-dimensional rapid prototype model of the jaw. The overlay of the subject's anatomic site and its 3D-IV image were displayed in real space using a 3D-AR display. Extraction of characteristic points and teeth matching were done using parallax images from two stereo cameras for patient-image registration. RESULTS: Accurate registration of the volunteer's anatomy with IV stereoscopic images via image matching was done using the fully automated markerless system, which recognized the incisal edges of the teeth and captured information pertaining to their position with an average target registration error of < 1 mm. These 3D-CT images were then displayed in real space with high accuracy using AR. Even when the viewing position was changed, the 3D images could be observed as if they were floating in real space without using special glasses. CONCLUSION: Teeth were successfully used for registration via 3D image (contour) matching. This system, without using references or fiducial markers, displayed 3D-CT images in real space with high accuracy. The system provided real-time markerless registration and 3D image matching via stereo vision, which, combined with AR, could have significant clinical applications.

Analysis of carotid artery deformation in different head and neck positions for maxillofacial catheter navigation in advanced oral cancer treatment.

BACKGROUND: To improve the accuracy of catheter navigation, it is important to develop a method to predict shifts of carotid artery (CA) bifurcations caused by intraoperative deformation. An important factor affecting the accuracy of electromagnetic maxillofacial catheter navigation systems is CA deformations. We aimed to assess CA deformation in different head and neck positions. METHODS: Using two sets of computed tomography angiography (CTA) images of six patients, displacements of the skull (maxillofacial segments), C1-C4 cervical vertebrae, mandible (mandibular segment), and CA along with its branches were analyzed. Segmented rigid bones around CA were considered the main causes of CA deformation. After superimposition of maxillofacial segments, C1-C4 and mandible segments were superimposed separately for displacement measurements. Five bifurcation points (vA-vE) were assessed after extracting the CA centerline. A new standardized coordinate system, regardless of patient-specific scanning positions, was employed. It was created using the principal axes of inertia of the maxillofacial bone segments of patients. Position and orientation parameters were transferred to this coordinate system. CA deformation in different head and neck positions was assessed. RESULTS: Absolute shifts in the center of gravity in the bone models for different segments were C1, 1.02 ± 0.9; C2, 2.18 ± 1.81; C3, 4.25 ± 3.85; C4, 5.90 ± 5.14; and mandible, 1.75 ± 2.76 mm. Shifts of CA bifurcations were vA, 5.52 ± 4.12; vB, 4.02 ± 3.27; vC, 4.39 ± 2.42; vD, 4.48 ± 1.88; and vE, 2.47 ± 1.32. Displacements, position changes, and orientation changes of C1-C4 segments as well as the displacements of all CA bifurcation points were similar in individual patients. CONCLUSIONS: CA deformation was objectively proven as an important factor contributing to errors in maxillofacial navigation. Our study results suggest that small movements of the bones around CA can result in small CA deformations. Although patients' faces were not fixed properly during CT scanning, C1-C4 and vA-vE displacements were similar in individual patients. We proposed a novel method for accumulation of the displacement data, and this study indicated the importance of surrounding bone displacements in predicting CA bifurcation.

An accurate 3D augmented reality navigation system with enhanced autostereoscopic display for oral and maxillofacial surgery.

BACKGROUND: Oral and maxillofacial surgery demands high precision navigation to achieve optimal surgical outcomes. Augmented reality (AR) has been a promising solution for intuitive and accurate guidance, whereas existing systems have room for improvement. METHODS: We propose a high quality and accurate in situ AR navigation system. Enhanced image quality is achieved by deploying eye tracking in lenticular-based autostereoscopic display. Viewpoint tracking and optical tracking are integrated to assure accurate in situ images. RESULTS: The proposed system can provide in situ AR images in real-time, with a viewing angle of 59.5° × 49.1°, 3D image resolution of 0.35 × 0.21 mm, and image accuracy of 0.99 ± 0.70 mm. A maxillary drilling experiment obtains average position error of 1.12 ± 0.30 mm and localization time of 8.7 ± 3.9 s, significantly better than conventional 2D navigation system. CONCLUSIONS: The proposed system can display high-quality AR images and therefore reduce positioning error and operating time.

Augmented reality surgical navigation with accurate CBCT-patient registration for dental implant placement.

It is challenging to achieve high implant accuracy in dental implant placement, because high risk tissues need to be avoided. In this study, we present an augmented reality (AR) surgical navigation with an accurate cone beam computed tomography (CBCT)-patient registration method to provide clinically desired dental implant accuracy. A registration device is used for registration between preoperative data and patient outside the patient's mouth. After registration, the registration device is worn on the patient's teeth for tracking the patient. Naked-eye 3D images of the planning path and the mandibular nerve are superimposed onto the patient in situ to form an AR scene. Simultaneously, a 3D image of the drill is overlaid accurately on the real one to guide the implant procedure. Finally, implant accuracy is evaluated postoperatively. A model experiment was performed by an experienced dentist. Totally, ten parallel pins were inserted into five 3D-printed mandible models guided by our AR navigation method and through the dentist's experience, respectively. AR-guided dental implant placement showed better results than the dentist's experience (mean target error = 1.25 mm vs. 1.63 mm; mean angle error = 4.03° vs. 6.10°). Experimental results indicate that the proposed method is expected to be applied in the clinic. Graphical abstract ᅟ.

Evaluation of the 3D Augmented Reality-Guided Intraoperative Positioning of Dental Implants in Edentulous Mandibular Models.

PURPOSE: This research aimed to propose a three-dimensional (3D) augmented reality navigation method with point cloud-based image-patient registration that could merge virtual images in the real environment for dental implants using a 3D image overlay and to evaluate its feasibility. MATERIALS AND METHODS: A total of 12 rapid prototyping mandibular models were fabricated using a 3D printing method and were divided into two groups: 3D augmented reality-guided group and traditional two-dimensional (2D) image-guided group. A point cloud-based preoperative image-to-patient registration method was introduced to replace the traditional point-to-point registration. After the registration, dental implant surgery was performed in the two model groups using an augmented reality-guided navigation method and a traditional two-dimensional image-guided navigation method. The planned and actual postoperative implant positions were compared for measuring positional implantation errors. The surgery time was also recorded and compared between the two groups. RESULTS: In the model experiment, the root-mean-square deviation of registration was 0.54 mm, and the implant surgery results showed < 1.5-mm mean linear deviation and < 5.5-degree angular deviation. The augmented reality-guided implantation showed smaller horizontal, vertical, and angular errors in the apical areas of the central incisor and the canine region. The surgery time using the augmented reality-guided navigation method was significantly shorter than that using the two-dimensional (2D) image-guided navigation method (P < .05). Moreover, the volunteer experiment demonstrated that the preoperative 3D models in situ accurately overlaid onto the surgical site. CONCLUSION: The proposed point cloud-based registration method can achieve excellent registration accuracy. Dental implant placement guided by the proposed 3D augmented reality navigation method showed better accuracy and applicability, as well as higher efficiency, than the traditional 2D image navigation method.

Real-time computer-generated integral imaging and 3D image calibration for augmented reality surgical navigation.

Autostereoscopic 3D image overlay for augmented reality (AR) based surgical navigation has been studied and reported many times. For the purpose of surgical overlay, the 3D image is expected to have the same geometric shape as the original organ, and can be transformed to a specified location for image overlay. However, how to generate a 3D image with high geometric fidelity and quantitative evaluation of 3D image's geometric accuracy have not been addressed. This paper proposes a graphics processing unit (GPU) based computer-generated integral imaging pipeline for real-time autostereoscopic 3D display, and an automatic closed-loop 3D image calibration paradigm for displaying undistorted 3D images. Based on the proposed methods, a novel AR device for 3D image surgical overlay is presented, which mainly consists of a 3D display, an AR window, a stereo camera for 3D measurement, and a workstation for information processing. The evaluation on the 3D image rendering performance with 2560×1600 elemental image resolution shows the rendering speeds of 50-60 frames per second (fps) for surface models, and 5-8 fps for large medical volumes. The evaluation of the undistorted 3D image after the calibration yields sub-millimeter geometric accuracy. A phantom experiment simulating oral and maxillofacial surgery was also performed to evaluate the proposed AR overlay device in terms of the image registration accuracy, 3D image overlay accuracy, and the visual effects of the overlay. The experimental results show satisfactory image registration and image overlay accuracy, and confirm the system usability.

Augmented reality navigation with automatic marker-free image registration using 3-D image overlay for dental surgery.

Computer-assisted oral and maxillofacial surgery (OMS) has been rapidly evolving since the last decade. State-of-the-art surgical navigation in OMS still suffers from bulky tracking sensors, troublesome image registration procedures, patient movement, loss of depth perception in visual guidance, and low navigation accuracy. We present an augmented reality navigation system with automatic marker-free image registration using 3-D image overlay and stereo tracking for dental surgery. A customized stereo camera is designed to track both the patient and instrument. Image registration is performed by patient tracking and real-time 3-D contour matching, without requiring any fiducial and reference markers. Real-time autostereoscopic 3-D imaging is implemented with the help of a consumer-level graphics processing unit. The resulting 3-D image of the patient's anatomy is overlaid on the surgical site by a half-silvered mirror using image registration and IP-camera registration to guide the surgeon by exposing hidden critical structures. The 3-D image of the surgical instrument is also overlaid over the real one for an augmented display. The 3-D images present both stereo and motion parallax from which depth perception can be obtained. Experiments were performed to evaluate various aspects of the system; the overall image overlay error of the proposed system was 0.71 mm.

Real-time in situ three-dimensional integral videography and surgical navigation using augmented reality: a pilot study.

To evaluate the feasibility and accuracy of a three-dimensional augmented reality system incorporating integral videography for imaging oral and maxillofacial regions, based on preoperative computed tomography data. Three-dimensional surface models of the jawbones, based on the computed tomography data, were used to create the integral videography images of a subject's maxillofacial area. The three-dimensional augmented reality system (integral videography display, computed tomography, a position tracker and a computer) was used to generate a three-dimensional overlay that was projected on the surgical site via a half-silvered mirror. Thereafter, a feasibility study was performed on a volunteer. The accuracy of this system was verified on a solid model while simulating bone resection. Positional registration was attained by identifying and tracking the patient/surgical instrument's position. Thus, integral videography images of jawbones, teeth and the surgical tool were superimposed in the correct position. Stereoscopic images viewed from various angles were accurately displayed. Change in the viewing angle did not negatively affect the surgeon's ability to simultaneously observe the three-dimensional images and the patient, without special glasses. The difference in three-dimensional position of each measuring point on the solid model and augmented reality navigation was almost negligible (<1 mm); this indicates that the system was highly accurate. This augmented reality system was highly accurate and effective for surgical navigation and for overlaying a three-dimensional computed tomography image on a patient's surgical area, enabling the surgeon to understand the positional relationship between the preoperative image and the actual surgical site, with the naked eye.

Augmented reality system for oral surgery using 3D auto stereoscopic visualization.

We present an augmented reality system for oral and maxillofacial surgery in this paper. Instead of being displayed on a separated screen, three-dimensional (3D) virtual presentations of osseous structures and soft tissues are projected onto the patient's body, providing surgeons with exact knowledge of depth information of high risk tissues inside the bone. We employ a 3D integral imaging technique which produce motion parallax in both horizontal and vertical direction over a wide viewing area in this study. In addition, surgeons are able to check the progress of the operation in real-time through an intuitive 3D based interface which is content-rich, hardware accelerated. These features prevent surgeons from penetrating into high risk areas and thus help improve the quality of the operation. Operational tasks such as hole drilling, screw fixation were performed using our system and showed an overall positional error of less than 1 mm. Feasibility of our system was also verified with a human volunteer experiment.

Nonmagnetic rigid and flexible outer sheath with pneumatic interlocking mechanism for minimally invasive surgical approach.

We developed a nonmagnetic rigid and flexible outer sheath with pneumatic interlocking mechanism using flexible toothed links and a wire-driven bending distal end. The outer sheath can be switched between rigid and flexible modes easily depending on surgical scenes, and the angle of its distal end can be controlled by three nylon wires. All components of flexible parts are made of MRI-compatible nonmagnetic plastics. We manufactured the device with 300-mm long, 16-mm outer diameter, 7-mm inner diameter and 90-mm bending distal end. Holding power of the device in rigid mode was maximum 3.6 N, which was sufficient for surgical tasks in body cavity. In vivo experiment using a swine, our device performed smooth insertion of a flexible endoscope and a biopsy forceps into reverse side of the liver, intestines and spleen with a curved path. In conclusion, our device shows availability of secure approach of surgical instruments into deep cavity.