Design and validation of a high-speed hyperspectral laparoscopic imaging system.

Significance: Minimally invasive surgery (MIS) has shown vast improvement over open surgery by reducing post-operative stays, intraoperative blood loss, and infection rates. However, in spite of these improvements, there are still prevalent issues surrounding MIS that may be addressed through hyperspectral imaging (HSI). We present a laparoscopic HSI system to further advance the field of MIS. Aim: We present an imaging system that integrates high-speed HSI technology with a clinical laparoscopic setup and validate the system's accuracy and functionality. Different configurations that cover the visible (VIS) to near-infrared (NIR) range of electromagnetism are assessed by gauging the spectral fidelity and spatial resolution of each hyperspectral camera. Approach: Standard Spectralon reflectance tiles were used to provide ground truth spectral footprints to compare with those acquired by our system using the root mean squared error (RMSE). Demosaicing techniques were investigated and used to measure and improve spatial resolution, which was assessed with a USAF resolution test target. A perception-based image quality evaluator was used to assess the demosaicing techniques we developed. Two configurations of the system were developed for evaluation. The functionality of the system was investigated in a phantom study and by imaging ex vivo tissues. Results: Multiple configurations of our system were tested, each covering different spectral ranges, including VIS (460 to 600 nm), red/NIR (RNIR) (610 to 850 nm), and NIR (665 to 950 nm). Each configuration is capable of achieving real-time imaging speeds of up to 20 frames per second. RMSE values of 3.51 ± 2.03 % , 3.43 ± 0.84 % , and 3.47% were achieved for the VIS, RNIR, and NIR systems, respectively. We obtained sub-millimeter resolution using our demosaicing techniques. Conclusions: We developed and validated a high-speed hyperspectral laparoscopic imaging system. The HSI system can be used as an intraoperative imaging tool for tissue classification during laparoscopic surgery.

An augmented reality and high-speed optical tracking system for laparoscopic surgery.

While minimally invasive laparoscopic surgery can help reduce blood loss, reduce hospital time, and shorten recovery time compared to open surgery, it has the disadvantages of limited field of view and difficulty in locating subsurface targets. Our proposed solution applies an augmented reality (AR) system to overlay pre-operative images, such as those from magnetic resonance imaging (MRI), onto the target organ in the user's real-world environment. Our system can provide critical information regarding the location of subsurface lesions to guide surgical procedures in real time. An infrared motion tracking camera system was employed to obtain real-time position data of the patient and surgical instruments. To perform hologram registration, fiducial markers were used to track and map virtual coordinates to the real world. In this study, phantom models of each organ were constructed to test the reliability and accuracy of the AR-guided laparoscopic system. Root mean square error (RMSE) was used to evaluate the targeting accuracy of the laparoscopic interventional procedure. Our results demonstrated a registration error of 2.42 ± 0.79 mm and a procedural targeting error of 4.17 ± 1.63 mm using our AR-guided laparoscopic system that will be further refined for potential clinical procedures.

Dense surface reconstruction using a learning-based monocular vSLAM model for laparoscopic surgery.

Augmented reality (AR) has seen increased interest and attention for its application in surgical procedures. AR-guided surgical systems can overlay segmented anatomy from pre-operative imaging onto the user's environment to delineate hard-to-see structures and subsurface lesions intraoperatively. While previous works have utilized pre-operative imaging such as computed tomography or magnetic resonance images, registration methods still lack the ability to accurately register deformable anatomical structures without fiducial markers across modalities and dimensionalities. This is especially true of minimally invasive abdominal surgical techniques, which often employ a monocular laparoscope, due to inherent limitations. Surgical scene reconstruction is a critical component towards accurate registrations needed for AR-guided surgery and other downstream AR applications such as remote assistance or surgical simulation. In this work, we utilize a state-of-the-art (SOTA) deep-learning-based visual simultaneous localization and mapping (vSLAM) algorithm to generate a dense 3D reconstruction with camera pose estimations and depth maps from video obtained with a monocular laparoscope. The proposed method can robustly reconstruct surgical scenes using real-time data and provide camera pose estimations without stereo or additional sensors, which increases its usability and is less intrusive. We also demonstrate a framework to evaluate current vSLAM algorithms on non-Lambertian, low-texture surfaces and explore using its outputs on downstream tasks. We expect these evaluation methods can be utilized for the continual refinement of newer algorithms for AR-guided surgery.

A hyperspectral surgical microscope with super-resolution reconstruction for intraoperative image guidance.

Hyperspectral imaging (HSI) is an emerging imaging modality in medical applications, especially for intraoperative image guidance. A surgical microscope improves surgeons' visualization with fine details during surgery. The combination of HSI and surgical microscope can provide a powerful tool for surgical guidance. However, to acquire high-resolution hyperspectral images, the long integration time and large image file size can be a burden for intraoperative applications. Super-resolution reconstruction allows acquisition of low-resolution hyperspectral images and generates high-resolution HSI. In this work, we developed a hyperspectral surgical microscope and employed our unsupervised super-resolution neural network, which generated high-resolution hyperspectral images with fine textures and spectral characteristics of tissues. The proposed method can reduce the acquisition time and save storage space taken up by hyperspectral images without compromising image quality, which will facilitate the adaptation of hyperspectral imaging technology in intraoperative image guidance.

Dual-camera laparoscopic imaging with super-resolution reconstruction for intraoperative hyperspectral image guidance.

Laparoscopic and robotic surgery, as one type of minimally invasive surgery (MIS), has gained popularity due to the improved surgeon ergonomics, instrument precision, operative time, and postoperative recovery. Hyperspectral imaging (HSI) is an emerging medical imaging modality, which has proved useful for intraoperative image guidance. Snapshot hyperspectral cameras are ideal for intraoperative laparoscopic imaging because of their compact size and light weight, but low spatial resolution can be a limitation. In this work, we developed a dual-camera laparoscopic imaging system that consists of a high-resolution color camera and a snapshot hyperspectral camera, and we employed super-resolution reconstruction to fuse the images from both cameras to generate high-resolution hyperspectral images. The experimental results show that our method can significantly improve the resolution of hyperspectral images without compromising the image quality or spectral signatures. The proposed super-resolution reconstruction method is promising to promote the employment of high-speed hyperspectral imaging in laparoscopic surgery.

Optimization of a real-time LED based spectral imaging system.

We are designing a real-time spectral imaging system using micro-LEDs and a high-speed micro-camera for potential endoscopic applications. Currently, gastrointestinal (GI) endoscopic imaging has largely been limited to white light imaging (WLI), while other endoscopic approaches have seen advancements in imaging techniques including fluorescence imaging, narrow-band imaging, and stereoscopic visualization. To further advance GI endoscopic imaging, we are working towards a high-speed spectral imaging system that can be implemented with a chip-on-tip design for a flexible endoscope. Hyperspectral imaging has potential applications in a variety of imaging procedures with its ability to discern spectral footprints of the imaging field in a series of two-dimensional images at different wavelengths. For investigating the feasibility of a real-time LED-based hyperspectral imaging system for endoscopic applications we designed and developed a large-scale prototype using through-hole LEDs, which is further analyzed as we design our future system. For high quality imaging, the LED array must be designed with the specific illumination patterns and intensities of each LED considered. We present our work in optimizing our current LED array through optical simulations performed in silico. Damped least squares and orthogonal descent algorithms are implemented to maximize irradiance power and improve illumination homogeneity at several working distances by adjusting radial distances of each LED from the camera. With our prototype LED-based hyperspectral imaging system, the simulation and optimization approach achieved an average increase of 8.36 ± 8.79% in irradiance and a 4.3% decrease in standard deviation at multiple working distances and field-of-views for each LED, as compared to the original design, leading to improved image quality and maintained acquisition speeds. This work highlights the value of in silico optical simulation and provides a framework for improved optical system design and will inform design decisions in future works.

A dual-camera hyperspectral laparoscopic imaging system.

Minimally invasive surgery (MIS) has expanded broadly in the field of abdominal and pelvic surgery. However, there are still prevalent issues surrounding intracorporeal surgery, such as iatrogenic injury, anastomotic leakage, or the presence of positive tumor margins after resection. Current approaches to address these issues and advance laparoscopic imaging techniques often involve fluorescence imaging agents, such as indocyanine green (ICG), to improve visualization, but these have drawbacks. Hyperspectral imaging (HSI) is an emerging optical imaging modality that takes advantage of spectral characteristics of different tissues. Various applications include tissue classification and digital pathology. In this study, we developed a dual-camera system for high-speed hyperspectral imaging. This includes the development of a custom application interface and corresponding hardware setup. Characterization of the system was performed, including spectral accuracy and spatial resolution, showing little sacrifice in speed for the approximate doubling of the covered spectral range, with our system acquiring 29 spectral images from 460-850 nm. Reference color tiles with various reflectance profiles were imaged and a RMSE of 3.56 ± 1.36% was achieved. Sub-millimeter resolution was shown at 7 cm working distance for both hyperspectral cameras. Finally, we image ex vivo tissues, including porcine stomach, liver, intestine, and kidney with our system and use a high-resolution, radiometrically calibrated spectrometer for comparison and evaluation of spectral fidelity. The dual-camera hyperspectral laparoscopic imaging system can have immediate applications in various surgeries.

A High-Resolution Hyperspectral Imaging System for the Retina.

In this study, we developed an imaging system that can acquire and produce high-resolution hyperspectral images of the retina. Our system combines the view from a high-resolution RGB camera and a snapshot hyperspectral camera together. The method is fast and can be constructed into a compact imaging device. We tested our system by imaging a calibrated color chart, biological tissues ex vivo, and a phantom of the human retina. By using image pansharpening methods, we were able to produce a high-resolution hyperspectral image. The images from the hyperspectral camera alone have a spatial resolution of 0.2 mm/pixel, whereas the pansharpened images have a spatial resolution of 0.1 mm/pixel, a 2x increase in spatial resolution. Our method has the potential to capture images of the retina rapidly. Our method preserves both the spatial and spectral fidelity, as shown by comparing the original hyperspectral images with the pansharpened images. The high-resolution hyperspectral imaging device can have a variety of applications in retina examinations.

A High-Speed Hyperspectral Laparoscopic Imaging System.

Minimally invasive surgery (MIS) has expanded broadly in the field of abdominal and pelvic surgery. Laparoscopic and robotic surgery has improved surgeon ergonomics, instrument precision, operative time, and postoperative recovery across various abdominal procedures. The goal of this study is to establish the feasibility of implementing high-speed hyperspectral imaging into a standard laparoscopic setup and exploring its benefit to common intracorporeal procedures. A hyperspectral laparoscopic imaging system was constructed using a customized hyperspectral camera alongside a standard rigid laparoscope and was validated for both spectral and spatial accuracy. Demosaicing methods were investigated for improved full-resolution visualization. Hyperspectral cameras with different spectral ranges were considered and compared with one another alongside two different light sources to determine the most effective configuration. Finally, different porcine tissues were imaged ex-vivo to test the capabilities of the system and spectral footprints of the various tissues were extracted. The tissue was also imaged in a phantom to simulate the system's use in MIS. The results demonstrated a hyperspectral laparoscopic imaging system that could provide quantitative, diagnostic information while not disrupting normal workflow nor adding excessive weight to the laparoscopic setup. The high-speed hyperspectral laparoscopic imaging system can have immediate applications in image-guided surgery.