Vision-based markerless registration using stereo vision and an augmented reality surgical navigation system: a pilot study.

BACKGROUND: This study evaluated the use of an augmented reality navigation system that provides a markerless registration system using stereo vision in oral and maxillofacial surgery. METHOD: A feasibility study was performed on a subject, wherein a stereo camera was used for tracking and markerless registration. The computed tomography data obtained from the volunteer was used to create an integral videography image and a 3-dimensional rapid prototype model of the jaw. The overlay of the subject's anatomic site and its 3D-IV image were displayed in real space using a 3D-AR display. Extraction of characteristic points and teeth matching were done using parallax images from two stereo cameras for patient-image registration. RESULTS: Accurate registration of the volunteer's anatomy with IV stereoscopic images via image matching was done using the fully automated markerless system, which recognized the incisal edges of the teeth and captured information pertaining to their position with an average target registration error of < 1 mm. These 3D-CT images were then displayed in real space with high accuracy using AR. Even when the viewing position was changed, the 3D images could be observed as if they were floating in real space without using special glasses. CONCLUSION: Teeth were successfully used for registration via 3D image (contour) matching. This system, without using references or fiducial markers, displayed 3D-CT images in real space with high accuracy. The system provided real-time markerless registration and 3D image matching via stereo vision, which, combined with AR, could have significant clinical applications.

3-D endoscope using a single CCD camera and pneumatic vibration mechanism.

BACKGROUND: During endoscopic surgical procedures, surgeons must manipulate an endoscope inside the body cavity to observe a surgical area while estimating the distance between that area and the surgical instruments by reference to a monitor on which the movement and size of the surgical instruments are displayed in 2-D endoscopic images. Therefore, there is a risk of the endoscope or instruments physically damaging body tissues. To overcome this problem, we developed a Ø5-mm, 3-D endoscope using a single 1/10-inch CCD camera and pneumatic vibration mechanism. METHODS: The 3-D endoscope proposed in this paper consists of an outer and inner sleeve, a 1/10-inch CCD camera attached to its distal end, and a pneumatic vibration mechanism attached to its proximal end. This endoscope can acquire left and right endoscopic images for stereovision in synchrony with the periodical motion generated by the vibration mechanism. We measured the displacement at the proximal and distal end of the 3-D endoscope simultaneously, and evaluated the feasibility of its use in vivo. RESULTS: The displacement at the distal end of the endoscope to which the CCD camera is attached was approximately ±0.25 mm. The timing when the displacement of the CCD camera was at maximal amplitude coincided with the timing when the displacement of its proximal end was at maximal amplitude. In the in vivo experiment, this 3-D endoscope can provide clear 3-D images of the surgical area. CONCLUSIONS: The developed 3-D endoscope that uses a single CCD camera and pneumatic vibration mechanism can successfully visualize internal organs inside the body even though the CCD camera is moved by the vibration. Therefore, the risk of damage to fragile body tissues can be significantly decreased.

Magnetic resonance imaging-compatible tactile sensing device based on a piezoelectric array.

Minimally invasive surgery is a widely used medical technique, one of the drawbacks of which is the loss of direct sense of touch during the operation. Palpation is the use of fingertips to explore and make fast assessments of tissue morphology. Although technologies are developed to equip minimally invasive surgery tools with haptic feedback capabilities, the majority focus on tissue stiffness profiling and tool-tissue interaction force measurement. For greatly increased diagnostic capability, a magnetic resonance imaging-compatible tactile sensor design is proposed, which allows minimally invasive surgery to be performed under image guidance, combining the strong capability of magnetic resonance imaging soft tissue and intuitive palpation. The sensing unit is based on a piezoelectric sensor methodology, which conforms to the stringent mechanical and electrical design requirements imposed by the magnetic resonance environment The sensor mechanical design and the device integration to a 0.2 Tesla open magnetic resonance imaging scanner are described, together with the device's magnetic resonance compatibility testing. Its design limitations and potential future improvements are also discussed. A tactile sensing unit based on a piezoelectric sensor principle is proposed, which is designed for magnetic resonance imaging guided interventions.

Endoscope system with plasma flushing and coaxial round jet nozzle for off-pump cardiac surgery.

BACKGROUND: To develop a new endoscope for performing simple surgical tasks inside the blood-filled cardiac atrium/chamber, that is, "off-pump" cardiac surgeries. METHODS: We developed the endoscope system with plasma flushing and coaxial round jet nozzle. The "plasma flushing" system was invented to observe the interior of the blood-filled heart by displacing blood cells in front of the endoscope tip. However, some areas could not be observed with simple flushing of the liquid because the flushed liquid mixed with blood. Further, a large amount of liquid had to be flushed, which posed a risk of cardiac damage caused by excess volume. Therefore, to safely capture high-resolution images of the interior of the heart, an endoscope with a coaxial round jet nozzle through which plasma is flushed has been developed. And to reduce the volume of flushed liquid, the synchronization system of heartbeat and the endoscope system with plasma flushing has been developed. RESULTS: We conducted an in vivo experiment to determine whether we could observe intracardiac tissues in swine without the use of a heart-lung machine. As a result, we successfully observed intracardiac tissues without using a heart-lung machine. By using a coaxial nozzle, we could even observe the tricuspid valve. Moreover, we were able to save up to 30% of the flushed liquid by replacing the original system with a synchronization system. And we evaluated the performance of the endoscope with the coaxial round jet nozzle by conducting fluid analysis and an in vitro experiment. CONCLUSION: We successfully observed intracardiac tissues without using a heart-lung machine. By using a coaxial nozzle, we could even observe the tricuspid valve. And by replacing an original system to a synchronization system, we were able to save up to 30% of the flushed liquid. As a follow-up study, we plan to create a surgical flexible device for valve disease that can grasp, staple, and repair cardiac valves by endoscopic visualization.

A variational method for geometric regularization of vascular segmentation in medical images.

In this paper, a level-set-based geometric regularization method is proposed which has the ability to estimate the local orientation of the evolving front and utilize it as shape induced information for anisotropic propagation. We show that preserving anisotropic fronts can improve elongations of the extracted structures, while minimizing the risk of leakage. To that end, for an evolving front using its shape-offset level-set representation, a novel energy functional is defined. It is shown that constrained optimization of this functional results in an anisotropic expansion flow which is usefull for vessel segmentation. We have validated our method using synthetic data sets, 2-D retinal angiogram images and magnetic resonance angiography volumetric data sets. A comparison has been made with two state-of-the-art vessel segmentation methods. Quantitative results, as well as qualitative comparisons of segmentations, indicate that our regularization method is a promising tool to improve the efficiency of both techniques.

Improved viewing resolution of integral videography by use of rotated prism sheets.

We demonstrated that placing a pair of prism sheets in front of a display and rotating them overcomes the upper resolution limit of Integral Photograpy (IP) / Integral Videography (IV) imposed by the Nyquist sampling theorem. A pair of prism sheet with the same pitch placed in front of an IP or IV display parallel-shifts the light rays in the 3D space. Rotating the pair shifts the light rays, causing them to appear to rotate around their original positions. Changing the gap between the sheets changes the diameter of the apparent rotation. Changing the speed at which the sheets are rotated changes the speed of the image movement.. Experimental results showed that the quality of the IP and IV images is improved by using this technique. It is a simple and effective way to improve the viewing resolution of IP and IV images without reducing their 3D aspects, such as image depth. It also eliminates the need to move the lenslet array.

Intraoperative tumor segmentation and volume measurement in MRI-guided glioma surgery for tumor resection rate control.

RATIONALE AND OBJECTIVES: Gross-total surgery under intraoperative magnetic resonance imaging (MRI) is a promising method of glioma removal. The purpose of this article is intraoperative measurement of resected tumor volume in MRI-guided glioma surgery using semiautomatic image segmentation to unbiased resection rate control. MATERIALS AND METHODS: A newly developed software program based on a fuzzy connectedness (FC) segmentation algorithm was used to achieve fast and semiautomatic tumor segmentation and tumor volume measurement. The program was validated by retrospective study of eight glioma cases and then applied to seven glioma cases. All clinical cases underwent actual MRI-guided surgery using 0.3-T open magnets. RESULTS: The volume of the tumor before resection ranged from 10.1 to 206.7 mL. A comparison of the results of manual segmentation with those of the semiautomatic FC-based segmentation gave an average dice similarity coefficient of 0.80 and an average match of 76%. Volume measurement combined with a developed software program enabled quantitative monitoring of tumor removal, which was critical in the near-total resection of glioma in MRI-guided surgery. CONCLUSION: The FC-based tumor segmentation method can be used for intraoperative tumor segmentation and volume measurement in MRI-guided glioma surgery using 0.3-T open magnets. This method is useful for objective resection rate monitoring, which may ultimately minimize the amount of residual tumor in glioma surgery.

Feasibility study of imaging of intraductal papillary mucinous tumors of the pancreas based on integral photography.

BACKGROUND/AIMS: The principle of integral photography was first proposed by Lippman in 1908. Recently, the three-dimensional display system based on the principle of integral photography was developed. It projects three-dimensional objects using a lens array called a fly's eye lens and photographic film. Some groups have used it for surgical navigation in brain surgery. In this study, we tried to generate integral photographic images of intraductal papillary mucinous tumors of the pancreas, and evaluated the feasibility of this method. METHODOLOGY: Five patients with an intraductal papillary mucinous tumor of pancreas were studied. We used MRI data as three-dimensional data. MRI data were acquired with a 1.5-T clinical imager (Signal.5; GE Medical Systems, U.S.A). We used multi-slab single-shot fast spin-echo sequences. Section thickness was between 2 and 3mm in the coronal plane. From these data, integral photographic images were generated on a liquid crystal display by a three-dimensional rendering algorithm. RESULTS: Three-dimensional images using the principle of integral photography could be generated in all 5 cases. We could recognize these images as three-dimensional images by slightly moving the viewpoint to up, down, left, and right. From these images, we were able to grasp the three-dimensional relationship between the tumor, bile duct, and pancreatic duct. Using these three-dimensional images, a variety of minimally invasive surgical procedures were performed for intraductal papillary mucinous tumor of the pancreas, more safely and speedily than formerly. CONCLUSIONS: The images based on integral photography are suitable for medical imaging and are useful for surgical planning.

High-quality integral videography using a multiprojector.

Integral videography (IV) is an animated extension of integral photography. Despite IV's many advantages, the quality of its spatial images has thus far been poor; the pixel pitch of the display and the lens pitch are considered to be the main factors affecting the IV image format. Our solution for increasing pixel density is to use multiple projectors to create a high-resolution image and project the resultant image onto a small screen by using long-zoom-lens projection optics. We manufactured a lens array for the display device, and here we present experimental results on using two SXGA projectors. The pixel pitch and lens pitch of the new display are 85 mum and 1.016 mm, respectively. The multiprojector IV display device has a spatial resolution of approximately 1, 2, and 3 mm for image depths of 10, 35, and 60 mm, respectively, in front of and behind the lens array.

Motion tracking in MR-guided liver therapy by using navigator echoes and projection profile matching.

RATIONALE AND OBJECTIVES: Image registration in magnetic resonance (MR) image-guided liver therapy enhances surgical guidance by fusing preoperative multimodality images with intraoperative images, or by fusing intramodality images to correlate serial intraoperative images to monitor the effect of therapy. The objective of this paper is to describe the application of navigator echo and projection profile matching to fast two-dimensional image registration for MR-guided liver therapy. MATERIALS AND METHODS: We obtain navigator echoes along the read-out and phase-encoding directions by using modified gradient echo imaging. This registration is made possible by masking out the liver profile from the image and performing profile matching with cross-correlation or mutual information as similarity measures. The set of experiments include a phantom study with a 2.0-T experimental MR scanner, and a volunteer and a clinical study with a 0.5-T open-configuration MR scanner, and these evaluate the accuracy and effectiveness of this method for liver therapy. RESULTS: Both the phantom and volunteer study indicate that this method can perform registration in 34 ms with root-mean-square error of 1.6 mm when the given misalignment of a liver is 30 mm. The clinical studies demonstrate that the method can track liver motion of up to approximately 40 mm. Matching profiles with cross-correlation information perform better than with mutual information in terms of robustness and speed. CONCLUSION: The proposed image registration method has potential clinical impact on and advantages for MR-guided liver therapy.

Three-dimensional volume rendering of fetal MR images for the diagnosis of congenital cystic adenomatoid malformation.

RATIONALE AND OBJECTIVES: Great expertise is necessary to mentally compile a series of individual two-dimensional image sections into a three-dimensional (3D) composite view that can aid in differential diagnosis. The purpose of this study was to test 3D volume-rendering techniques for differentiating congenital cystic adenomatoid malformation from congenital diaphragmatic hernia. MATERIALS AND METHODS: The authors acquired T2-weighted magnetic resonance (MR) images of a 27-week fetus in the sagittal plane and then applied the 3D volume-rendering method to the MR image data sets to obtain a composite 3D image. RESULTS: It was unclear on the MR images whether the intestines were situated above or below the diaphragm. The composite image showed that the intestines were not herniated into the chest, and this facilitated a diagnosis of congenital cystic adenomatoid malformation rather than congenital diaphragmatic hernia. CONCLUSION: The 3D volume-rendering techniques aided in the assessment of fetal organ structure and could be applied also to preoperative simulation and planning of fetal surgery.

Study of Secondary Flow in Centrifugal Blood Pumps Using a Flow Visualization Method with a High-Speed Video Camera.

Four pump models with different vane configurations were evaluated with flow visualization techniques using a high-speed video camera. These models also were evaluated through in vivo hemolysis tests using bovine blood. The impeller having the greatest fluid velocity relative to the impeller, the largest velocity variance, and the most irregular local flow patterns in the flow passage caused the most hemolysis. Even if the pumps were operated at almost the same speed (rpm) at the same output, the impeller showing more irregular flow patterns had a statistically greater rate of hemolysis. This fact confirms that the existence of local irregular flow patterns in a centrifugal blood pump deteriorates its hemolytic performance. Thus, to optimize the design of the pump, it is very important to examine the secondary flow patterns in the centrifugal blood pump in detail using flow visualization with a high-speed video camera.

Surgical navigation by autostereoscopic image overlay of integral videography.

This paper describes an autostereoscopic image overlay technique that is integrated into a surgical navigation system to superimpose a real three-dimensional (3-D) image onto the patient via a half-silvered mirror. The images are created by employing a modified version of integral videography (IV), which is an animated extension of integral photography. IV records and reproduces 3-D images using a microconvex lens array and flat display; it can display geometrically accurate 3-D autostereoscopic images and reproduce motion parallax without the need for special devices. The use of semitransparent display devices makes it appear that the 3-D image is inside the patient's body. This is the first report of applying an autostereoscopic display with an image overlay system in surgical navigation. Experiments demonstrated that the fast IV rendering technique and patient-image registration method produce an average registration accuracy of 1.13 mm. Experiments using a target in phantom agar showed that the system can guide a needle toward a target with an average error of 2.6 mm. Improvement in the quality of the IV display will make this system practical and its use will increase surgical accuracy and reduce invasiveness.

[The overview of robot surgery].

Surgical operations have developed in the method which skillful surgeon's hands and eyes are used. However, to realize a new surgical therapy in the 21st century, it is necessary to use various advanced technologies; surgical robots, three dimensional medical images, computer graphics, computer simulation technology and others. Three dimensional medical image for surgical operation provides surgeons with advanced vision. Surgical robots provide surgeons with advanced hand, but it is not a machine to do the same action of a surgeon using scissors or a scalpel. The advanced vision and hands available to surgeons are creating new surgical fields which are minimally invasive surgery, non-invasive surgery, virtual reality micro-surgery, tele-surgery, fetus surgery, neuro-informatics surgery and others in the 21st century.

A beam-splitter-type 3-D endoscope for front view and front-diagonal view images.

PURPOSE: In endoscopic surgery, surgeons must manipulate an endoscope inside the body cavity to observe a large field-of-view while estimating the distance between surgical instruments and the affected area by reference to the size or motion of the surgical instruments in 2-D endoscopic images on a monitor. Therefore, there is a risk of the endoscope or surgical instruments physically damaging body tissues. To overcome this problem, we developed a Ø7- mm 3-D endoscope that can switch between providing front and front-diagonal view 3-D images by simply rotating its sleeves. METHODS: This 3-D endoscope consists of a conventional 3-D endoscope and an outer and inner sleeve with a beam splitter and polarization plates. The beam splitter was used for visualizing both the front and front-diagonal view and was set at 25° to the outer sleeve's distal end in order to eliminate a blind spot common to both views. Polarization plates were used to avoid overlap of the two views. We measured signal-to-noise ratio (SNR), sharpness, chromatic aberration (CA), and viewing angle of this 3-D endoscope and evaluated its feasibility in vivo. RESULTS: Compared to the conventional 3-D endoscope, SNR and sharpness of this 3-D endoscope decreased by 20 and 7 %, respectively. No significant difference was found in CA. The viewing angle for both the front and front-diagonal views was about 50°. In the in vivo experiment, this 3-D endoscope can provide clear 3-D images of both views by simply rotating its inner sleeve. CONCLUSIONS: The developed 3-D endoscope can provide the front and front-diagonal view by simply rotating the inner sleeve, therefore the risk of damage to fragile body tissues can be significantly decreased.

Real-time in situ three-dimensional integral videography and surgical navigation using augmented reality: a pilot study.

To evaluate the feasibility and accuracy of a three-dimensional augmented reality system incorporating integral videography for imaging oral and maxillofacial regions, based on preoperative computed tomography data. Three-dimensional surface models of the jawbones, based on the computed tomography data, were used to create the integral videography images of a subject's maxillofacial area. The three-dimensional augmented reality system (integral videography display, computed tomography, a position tracker and a computer) was used to generate a three-dimensional overlay that was projected on the surgical site via a half-silvered mirror. Thereafter, a feasibility study was performed on a volunteer. The accuracy of this system was verified on a solid model while simulating bone resection. Positional registration was attained by identifying and tracking the patient/surgical instrument's position. Thus, integral videography images of jawbones, teeth and the surgical tool were superimposed in the correct position. Stereoscopic images viewed from various angles were accurately displayed. Change in the viewing angle did not negatively affect the surgeon's ability to simultaneously observe the three-dimensional images and the patient, without special glasses. The difference in three-dimensional position of each measuring point on the solid model and augmented reality navigation was almost negligible (<1 mm); this indicates that the system was highly accurate. This augmented reality system was highly accurate and effective for surgical navigation and for overlaying a three-dimensional computed tomography image on a patient's surgical area, enabling the surgeon to understand the positional relationship between the preoperative image and the actual surgical site, with the naked eye.

Surgical tool alignment guidance by drawing two cross-sectional laser-beam planes.

Conventional surgical navigation requires for surgeons to move their sight and conscious off the surgical field when checking surgical tool's positions shown on the display panel. Since that takes high risks of surgical exposure possibilities to the patient's body, we propose a novel method for guiding surgical tool position and orientation directly in the surgical field by a laser beam. In our navigation procedure, two cross-sectional planar laser beams are emitted from the two laser devices attached onto both sides of an optical localizer, and show surgical tool's entry position on the patient's body surface and its orientation on the side face of the surgical tool. In the experiments, our method gave the surgeons precise and accurate surgical tool adjusting and showed the feasibility to apply to both of open and percutaneous surgeries.

Developing essential rigid-flexible outer sheath to enable novel multi-piercing surgery.

We have developed a new generation device called rigid-flexible outer sheath with multi-piercing surgery (MPS) to solve the issues of tissue closure, triangulation, and platform stability in natural orifice transluminal endoscopic surgery (NOTES), and the problems of restricted visual field, organ damage, and removing a resected organ from body in needlescopic surgery (NS). The shape of the flexible outer sheath can be selectively locked by a novel pneumatic shapelocking mechanism. Major features include four directional flexion at the distal end, four working channels, and suction and water jet functions. The insertion part of the prototype is 330 mm long with a 25 mm maximum outer diameter. The outer sheath system has successfully preformed in vivo experiment using a swine on partial gastrectomy. The advanced outer sheath system has shown great promise for solving NOTES and NS issues.

Autostereoscopic 3D Display with Long Visualization Depth Using Referential Viewing Area-Based Integral Photography.

We developed an autostereoscopic display for distant viewing of 3D computer graphics (CG) images without using special viewing glasses or tracking devices. The images are created by employing referential viewing area-based CG image generation and pixel distribution algorithm for integral photography (IP) and integral videography (IV) imaging. CG image rendering is used to generate IP/IV elemental images. The images can be viewed from each viewpoint within a referential viewing area and the elemental images are reconstructed from rendered CG images by pixel redistribution and compensation method. The elemental images are projected onto a screen that is placed at the same referential viewing distance from the lens array as in the image rendering. Photographic film is used to record the elemental images through each lens. The method enables 3D images with a long visualization depth to be viewed from relatively long distances without any apparent influence from deviated or distorted lenses in the array. We succeeded in creating an actual autostereoscopic images with an image depth of several meters in front of and behind the display that appear to have 3D even when viewed from a distance.