Stereo matching of binocular laparoscopic images with improved densely connected neural architecture search.

PURPOSE: Stereo matching is a crucial technology in the binocular laparoscopic-based surgical navigation systems. In recent years, neural networks have been widely applied to stereo matching and demonstrated outstanding performance. however, this method heavily relies on manual feature engineering meaning that professionals must be involved in the feature extraction and matching. This process is both time-consuming and demands specific expertise. METHODS: This paper introduces a novel stereo matching framework DCStereo that realizes a fully automatic neural architecture design for the stereo matching of binocular laparoscopic images. The proposed framework utilizes a densely connected search space which enables a more flexible and diverse architecture composition. Furthermore, the proposed algorithm leverages the channel and path sampling strategies to reduce memory consumption during searching. RESULTS: Empirically, our searched DCStereo on the SCARED training dataset achieves a mean absolute error of 3.589 mm on the test dataset, which outperforms hand-crafted stereo matching methods and other approaches. Furthermore, when directly testing on the SERV-CT dataset, our DCStereo demonstrates better generalization ability than other methods. CONCLUSION: Our proposed approach leverages the neural architecture search technique and a densely connected search space for automatic neural architecture design in stereo matching of binocular laparoscopic images. Our method delivers advanced performance on the SCARED dataset and promising results on the SERV-CT dataset. These findings demonstrate the potential of our approach for improving clinical surgical navigation systems.

A novel intrarenal multimodal 2D/3D registration algorithm and preliminary phantom study.

Retrograde intrarenal surgery (RIRS) is a widely utilized diagnostic and therapeutic tool for multiple upper urinary tract pathologies. The image-guided navigation system can assist the surgeon to perform precise surgery by providing the relative position between the lesion and the instrument after the intraoperative image is registered with the preoperative model. However, due to the structural complexity and diversity of multi-branched organs such as kidneys, bronchi, etc., the consistency of the intensity distribution of virtual and real images will be challenged, which makes the classical pure intensity registration method prone to bias and random results in a wide search domain. In this paper, we propose a structural feature similarity-based method combined with a semantic style transfer network, which significantly improves the registration accuracy when the initial state deviation is obvious. Furthermore, multi-view constraints are introduced to compensate for the collapse of spatial depth information and improve the robustness of the algorithm. Experimental studies were conducted on two models generated from patient data to evaluate the performance of the method and competing algorithms. The proposed method obtains mean target error (mTRE) of 0.971 ± 0.585 mm and 1.266 ± 0.416 mm respectively, with better accuracy and robustness overall. Experimental results demonstrate that the proposed method has the potential to be applied to RIRS and extended to other organs with similar structures.

Shape estimation of the anterior part of a flexible ureteroscope for intraoperative navigation.

PURPOSE: Flexible ureteroscopy (FURS) plays an important role in the diagnosis and treatment of urological diseases. However, manipulating a flexible ureteroscope to the target quickly and safely may be challenging because of the tortuous lumen or poor visibility. Thus, information on the shape of the anterior part of a flexible ureteroscope in addition to the real-time pose is needed to perform accurate maneuvering in the lumen with minimal impingement on the inner renal wall and resulting tissue damage in FURS. METHODS: An adaptive mixed-order Bézier curve fitting algorithm and electromagnetic tracking (EMT) technique were developed for shape estimation utilizing the length of the anterior part, kinematic constraints and the pose information provided by two electromagnetic (EM) sensors mounted at the tip and base of the anterior part. A series of experiments were performed to qualitatively and quantitatively verify the validity of our method. Moreover, algorithm threshold conditions with reference significance under various shape cases were studied. RESULTS: The performance of our method was evaluated based on 19 representative planar bending shapes that often appear in FURS and eight non-planar shapes, yielding an average error (AE) of 1.0 mm. Moreover, the experiments proved the feasibility of applying our method in cases in which large bending angles (near 270 degrees) occur. CONCLUSION: Based on data from two EM sensors mounted at the tip and base of the anterior part of a flexible ureteroscope, the proposed algorithm adaptively selects a cubic or quartic Bézier curve to fit the shape of the anterior part. Experimental results prove the feasibility of our shape estimation method over a broad bending range. The proposed method demonstrates significant potential for use in ureteroscopic navigation systems and robot-assisted surgery.

A Novel Central Camera Calibration Method Recording Point-to-Point Distortion for Vision-Based Human Activity Recognition.

The camera is the main sensor of vison-based human activity recognition, and its high-precision calibration of distortion is an important prerequisite of the task. Current studies have shown that multi-parameter model methods achieve higher accuracy than traditional methods in the process of camera calibration. However, these methods need hundreds or even thousands of images to optimize the camera model, which limits their practical use. Here, we propose a novel point-to-point camera distortion calibration method that requires only dozens of images to get a dense distortion rectification map. We have designed an objective function based on deformation between the original images and the projection of reference images, which can eliminate the effect of distortion when optimizing camera parameters. Dense features between the original images and the projection of the reference images are calculated by digital image correlation (DIC). Experiments indicate that our method obtains a comparable result with the multi-parameter model method using a large number of pictures, and contributes a 28.5% improvement to the reprojection error over the polynomial distortion model.

Deep learning for gastroscopic images: computer-aided techniques for clinicians.

Gastric disease is a major health problem worldwide. Gastroscopy is the main method and the gold standard used to screen and diagnose many gastric diseases. However, several factors, such as the experience and fatigue of endoscopists, limit its performance. With recent advancements in deep learning, an increasing number of studies have used this technology to provide on-site assistance during real-time gastroscopy. This review summarizes the latest publications on deep learning applications in overcoming disease-related and nondisease-related gastroscopy challenges. The former aims to help endoscopists find lesions and characterize them when they appear in the view shed of the gastroscope. The purpose of the latter is to avoid missing lesions due to poor-quality frames, incomplete inspection coverage of gastroscopy, etc., thus improving the quality of gastroscopy. This study aims to provide technical guidance and a comprehensive perspective for physicians to understand deep learning technology in gastroscopy. Some key issues to be handled before the clinical application of deep learning technology and the future direction of disease-related and nondisease-related applications of deep learning to gastroscopy are discussed herein.

Visual-electromagnetic system: A novel fusion-based monocular localization, reconstruction, and measurement for flexible ureteroscopy.

BACKGROUND: During flexible ureteroscopy (FURS), surgeons may lose orientation due to intrarenal structural similarities and complex shape of the pyelocaliceal cavity. Decision-making required after initially misjudging stone size will also increase the operative time and risk of severe complications. METHODS: A intraoperative navigation system based on electromagnetic tracking (EMT) and simultaneous localization and mapping (SLAM) was proposed to track the tip of the ureteroscope and reconstruct a dense intrarenal three-dimensional (3D) map. Furthermore, the contour lines of stones were segmented to measure the size. RESULTS: Our system was evaluated on a kidney phantom, achieving an absolute trajectory accuracy root mean square error (RMSE) of 0.6 mm. The median error of the longitudinal and transversal measurements was 0.061 and 0.074 mm, respectively. The in vivo experiment also demonstrated the effectiveness. CONCLUSION: The proposed system worked effectively in tracking and measurement. Further, this system can be extended to other surgical applications involving cavities, branches and intelligent robotic surgery.

Clinically Available Optical Imaging Technologies in Endoscopic Lesion Detection: Current Status and Future Perspective.

Endoscopic optical imaging technologies for the detection and evaluation of dysplasia and early cancer have made great strides in recent decades. With the capacity of in vivo early detection of subtle lesions, they allow modern endoscopists to provide accurate and effective optical diagnosis in real time. This review mainly analyzes the current status of clinically available endoscopic optical imaging techniques, with emphasis on the latest updates of existing techniques. We summarize current coverage of these technologies in major hospital departments such as gastroenterology, urology, gynecology, otolaryngology, pneumology, and laparoscopic surgery. In order to promote a broader understanding, we further cover the underlying principles of these technologies and analyze their performance. Moreover, we provide a brief overview of future perspectives in related technologies, such as computer-assisted diagnosis (CAD) algorithms dealing with exploring endoscopic video data. We believe all these efforts will benefit the healthcare of the community, help endoscopists improve the accuracy of diagnosis, and relieve patients' suffering.