PitSurgRT: real-time localization of critical anatomical structures in endoscopic pituitary surgery.

PURPOSE: Endoscopic pituitary surgery entails navigating through the nasal cavity and sphenoid sinus to access the sella using an endoscope. This procedure is intricate due to the proximity of crucial anatomical structures (e.g. carotid arteries and optic nerves) to pituitary tumours, and any unintended damage can lead to severe complications including blindness and death. Intraoperative guidance during this surgery could support improved localization of the critical structures leading to reducing the risk of complications. METHODS: A deep learning network PitSurgRT is proposed for real-time localization of critical structures in endoscopic pituitary surgery. The network uses high-resolution net (HRNet) as a backbone with a multi-head for jointly localizing critical anatomical structures while segmenting larger structures simultaneously. Moreover, the trained model is optimized and accelerated by using TensorRT. Finally, the model predictions are shown to neurosurgeons, to test their guidance capabilities. RESULTS: Compared with the state-of-the-art method, our model significantly reduces the mean error in landmark detection of the critical structures from 138.76 to 54.40 pixels in a 1280 × 720-pixel image. Furthermore, the semantic segmentation of the most critical structure, sella, is improved by 4.39% IoU. The inference speed of the accelerated model achieves 298 frames per second with floating-point-16 precision. In the study of 15 neurosurgeons, 88.67% of predictions are considered accurate enough for real-time guidance. CONCLUSION: The results from the quantitative evaluation, real-time acceleration, and neurosurgeon study demonstrate the proposed method is highly promising in providing real-time intraoperative guidance of the critical anatomical structures in endoscopic pituitary surgery.

Complete region of interest reconstruction by fusing multiview deformable three-dimensional transesophageal echocardiography images.

BACKGROUND: While three-dimensional transesophageal echocardiography (3D TEE) has been increasingly used for assessing cardiac anatomy and function, it still suffers from a limited field of view (FoV) of the ultrasound transducer. Therefore, it is difficult to examine a complete region of interest without moving the transducer. Existing methods extend the FoV of 3D TEE images by mosaicing multiview static images, which requires synchronization between 3D TEE images and electrocardiogram (ECG) signal to avoid deformations in the images and can only get the widened image at a specific phase. PURPOSE: This work aims to develop a novel multiview nonrigid registration and fusion method to extend the FoV of 3D TEE images at different cardiac phases, avoiding the bias toward the specifically chosen phase. METHODS: A multiview nonrigid registration and fusion method is proposed to enlarge the FoV of 3D TEE images by fusing dynamic images captured from different viewpoints sequentially. The deformation field for registering images is defined by a collection of affine transformations organized in a graph structure and is estimated by a direct (intensity-based) method. The accuracy of the proposed method is evaluated by comparing it with two B-spline-based methods, two Demons-based methods, and one learning-based method VoxelMorph. Twenty-nine sequences of in vivo 3D TEE images captured from four patients are used for the comparative experiments. Four performance metrics including checkerboard volumes, signed distance, mean absolute distance (MAD), and Dice similarity coefficient (DSC) are used jointly to evaluate the accuracy of the results. Additionally, paired t-tests are performed to examine the significance of the results. RESULTS: The qualitative results show that the proposed method can align images more accurately and obtain the fused images with higher quality than the other five methods. Additionally, in the evaluation of the segmented left atrium (LA) walls for the pairwise registration and sequential fusion experiments, the proposed method achieves the MAD of (0.07 ± 0.03) mm for pairwise registration and (0.19 ± 0.02) mm for sequential fusion. Paired t-tests indicate that the results obtained from the proposed method are more accurate than those obtained by the state-of-the-art VoxelMorph and the diffeomorphic Demons methods at the significance level of 0.05. In the evaluation of left ventricle (LV) segmentations for the sequential fusion experiments, the proposed method achieves a DSC of (0.88 ± 0.08), which is also significantly better than diffeomorphic Demons at the 0.05 level. The FoVs of the final fused 3D TEE images obtained by our method are enlarged around two times compared with the original images. CONCLUSIONS: Without selecting the static (ECG-gated) images from the same cardiac phase, this work addressed the problem of limited FoV of 3D TEE images in the deformable scenario, obtaining the fused images with high accuracy and good quality. The proposed method could provide an alternative to the conventional fusion methods that are biased toward the specifically chosen phase.

Direct 3D ultrasound fusion for transesophageal echocardiography.

BACKGROUND: Real-time three-dimensional transesophageal echocardiography (3D TEE) has been increasingly used in clinic for fast 3D analysis of cardiac anatomy and function. However, 3D TEE still suffers from the limited field of view (FoV). It is challenging to adopt conventional multi-view fusion methods to 3D TEE images because feature-based registration methods tend to fail in the ultrasound scenario, and conventional intensity-based methods have poor convergence properties and require an iterative coarse-to-fine strategy. METHODS: A novel multi-view registration and fusion method is proposed to enlarge the FoV of 3D TEE images efficiently. A direct method is proposed to solve the registration problem in the Lie algebra space. Fast implementation is realized by searching voxels on three orthogonal planes between two volumes. Besides, a weighted-average 3D fusion method is proposed to fuse the aligned images seamlessly. For a sequence of 3D TEE images, they are fused incrementally. RESULTS: Qualitative and quantitative results of in-vivo experiments indicate that the proposed registration algorithm outperforms a state-of-the-art PCA-based registration method in terms of accuracy and efficiency. Image registration and fusion performed on 76 in-vivo 3D TEE volumes from nine patients show apparent enlargement of FoV (enlarged around two times) in the obtained fused images. CONCLUSIONS: The proposed methods can fuse 3D TEE images efficiently and accurately so that the whole Region of Interest (ROI) can be seen in a single frame. This research shows good potential to assist clinical diagnosis, preoperative planning, and future intraoperative guidance with 3D TEE.