M-VAAL: Multimodal Variational Adversarial Active Learning for Downstream Medical Image Analysis Tasks.

Acquiring properly annotated data is expensive in the medical field as it requires experts, time-consuming protocols, and rigorous validation. Active learning attempts to minimize the need for large annotated samples by actively sampling the most informative examples for annotation. These examples contribute significantly to improving the performance of supervised machine learning models, and thus, active learning can play an essential role in selecting the most appropriate information in deep learning-based diagnosis, clinical assessments, and treatment planning. Although some existing works have proposed methods for sampling the best examples for annotation in medical image analysis, they are not task-agnostic and do not use multimodal auxiliary information in the sampler, which has the potential to increase robustness. Therefore, in this work, we propose a Multimodal Variational Adversarial Active Learning (M-VAAL) method that uses auxiliary information from additional modalities to enhance the active sampling. We applied our method to two datasets: i) brain tumor segmentation and multi-label classification using the BraTS2018 dataset, and ii) chest X-ray image classification using the COVID-QU-Ex dataset. Our results show a promising direction toward data-efficient learning under limited annotations.

Comparative Analysis of Non-Rigid Registration Techniques for Liver Surface Registration.

Non-rigid surface-based soft tissue registration is crucial for surgical navigation systems, but its adoption still faces several challenges due to the large number of degrees of freedom and the continuously varying and complex surface structures present in the intra-operative data. By employing non-rigid registration, surgeons can integrate the pre-operative images into the intra-operative guidance environment, providing real-time visualization of the patient's complex pre- and intra-operative anatomy in a common coordinate system to improve navigation accuracy. However, many of the existing registration methods, including those for liver applications, are inaccessible to the broader community. To address this limitation, we present a comparative analysis of several open-source, non-rigid surface-based liver registration algorithms, with the overall goal of contrasting their strength and weaknesses and identifying an optimal solution. We compared the robustness of three optimization-based and one data-driven nonrigid registration algorithms in response to a reduced visibility ratio (reduced partial views of the surface) and to an increasing deformation level (mean displacement), reported as the root mean square error (RMSE) between the pre-and intra-operative liver surface meshed following registration. Our results indicate that the Gaussian Mixture Model - Finite Element Model (GMM-FEM) method consistently yields a lower post-registration error than the other three tested methods in the presence of both reduced visibility ratio and increased intra-operative surface displacement, therefore offering a potentially promising solution for pre- to intra-operative nonrigid liver surface registration.

Inpainting surgical occlusion from laparoscopic video sequences for robot-assisted interventions.

Purpose: Medical technology for minimally invasive surgery has undergone a paradigm shift with the introduction of robot-assisted surgery. However, it is very difficult to track the position of the surgical tools in a surgical scene, so it is crucial to accurately detect and identify surgical tools. This task can be aided by deep learning-based semantic segmentation of surgical video frames. Furthermore, due to the limited working and viewing areas of these surgical instruments, there is a higher chance of complications from tissue injuries (e.g., tissue scars and tears). Approach: With the aid of digital inpainting algorithms, we present an application that uses image segmentation to remove surgical instruments from laparoscopic/endoscopic video. We employ a modified U-Net architecture (U-NetPlus) to segment the surgical instruments. It consists of a redesigned decoder and a pre-trained VGG11 or VGG16 encoder. The decoder was modified by substituting an up-sampling operation based on nearest-neighbor interpolation for the transposed convolution operation. Furthermore, these interpolation weights do not need to be learned to perform upsampling, which eliminates the artifacts generated by the transposed convolution. In addition, we use a very fast and adaptable data augmentation technique to further enhance performance. The instrument segmentation mask is filled in (i.e., inpainted) by the tool removal algorithms using the previously acquired tool segmentation masks and either previous instrument-containing frames or instrument-free reference frames. Results: We have shown the effectiveness of the proposed surgical tool segmentation/removal algorithms on a robotic instrument dataset from the MICCAI 2015 and 2017 EndoVis Challenge. We report a 90.20% DICE for binary segmentation, a 76.26% DICE for instrument part segmentation, and a 46.07% DICE for instrument type (i.e., all instruments) segmentation on the MICCAI 2017 challenge dataset using our U-NetPlus architecture, outperforming the results of earlier techniques used and tested on these data. In addition, we demonstrated the successful execution of the tool removal algorithm from surgical tool-free videos that contained moving surgical tools that were generated artificially. Conclusions: Our application successfully separates and eliminates the surgical tool to reveal a view of the background tissue that was otherwise hidden by the tool, producing results that are visually similar to the actual data.

Disparity refinement framework for learning-based stereo matching methods in cross-domain setting for laparoscopic images.

Purpose: Stereo matching methods that enable depth estimation are crucial for visualization enhancement applications in computer-assisted surgery. Learning-based stereo matching methods have shown great promise in making accurate predictions on laparoscopic images. However, they require a large amount of training data, and their performance may be degraded due to domain shifts. Approach: Maintaining robustness and improving the accuracy of learning-based methods are still open problems. To overcome the limitations of learning-based methods, we propose a disparity refinement framework consisting of a local disparity refinement method and a global disparity refinement method to improve the results of learning-based stereo matching methods in a cross-domain setting. Those learning-based stereo matching methods are pre-trained on a large public dataset of natural images and are tested on two datasets of laparoscopic images. Results: Qualitative and quantitative results suggest that our proposed disparity framework can effectively refine disparity maps when they are noise-corrupted on an unseen dataset, without compromising prediction accuracy when the network can generalize well on an unseen dataset. Conclusions: Our proposed disparity refinement framework could work with learning-based methods to achieve robust and accurate disparity prediction. Yet, as a large laparoscopic dataset for training learning-based methods does not exist and the generalization ability of networks remains to be improved, the incorporation of the proposed disparity refinement framework into existing networks will contribute to improving their overall accuracy and robustness associated with depth estimation.

A Deep Learning Framework to Estimate Elastic Modulus from Ultrasound Measured Displacement Fields.

Ultrasound (US) elastography is a technique that enables non-invasive quantification of material properties, such as stiffness, from ultrasound images of deforming tissue. The displacement field is measured from the US images using image matching algorithms, and then a parameter, often the elastic modulus, is inferred or subsequently measured to identify potential tissue pathologies, such as cancerous tissues. Several traditional inverse problem approaches, loosely grouped as either direct or iterative, have been explored to estimate the elastic modulus. Nevertheless, the iterative techniques are typically slow and computationally intensive, while the direct techniques, although more computationally efficient, are very sensitive to measurement noise and require the full displacement field data (i.e., both vector components). In this work, we propose a deep learning approach to solve the inverse problem and recover the spatial distribution of the elastic modulus from one component of the US measured displacement field. The neural network used here is trained using only simulated data obtained via a forward finite element (FE) model with known variations in the modulus field, thus avoiding the reliance on large measurement data sets that may be challenging to acquire. A U-net based neural network is then used to predict the modulus distribution (i.e., solve the inverse problem) using the simulated forward data as input. We quantitatively evaluated our trained model with a simulated test dataset and observed a 0.0018 mean squared error (MSE) and a 1.14% mean absolute percent error (MAPE) between the reconstructed and ground truth elastic modulus. Moreover, we also qualitatively compared the output of our U-net model to experimentally measured displacement data acquired using a US elastography tissue-mimicking calibration phantom.

Developing and Evaluating the Fidelity of Patient Specific Kidney Emulating Phantoms for Image-guided Intervention Applications.

Patient specific organ and tissue mimicking phantoms are used routinely to develop and assess new image-guided intervention tools and techniques in laboratory settings, enabling scientists to maintain acceptable anatomical relevance, while avoiding animal studies when the developed technology is still in its infancy. Gelatin phantoms, specifically, offer a cost-effective and readily available alternative to the traditional manufacturing of anatomical phantoms, and also provide the necessary versatility to mimic various stiffness properties specific to various organs or tissues. In this study, we describe the protocol to develop patient specific anthropomorphic gelatin kidney phantoms and we also assess the faithfulness of the developed phantoms against the patient specific CT images and corresponding virtual anatomical models used to generate the phantoms. We built the gelatin phantoms by first using additive manufacturing to generate a kidney mold based on patient specific CT images, into which the gelatin was poured. We then evaluated the fidelity of the phantoms (i.e., children) against the virtual kidney model generated from the patient specific CT image (i.e., parent) by comparing it to the surface model of the mold and gelatin phantoms (i.e., children) following their CT imaging post-manufacturing using various registration metrics. Our experiments showed a 0.58 ± 0.48 mm surface-to-surface distance between the phantoms and mold models following landmark-based registration, and 0.52 ± 0.40 mm surface-to-surface distance between the phantoms and the mold model following iterative closest point (ICP) registration. These experiments confirm that the described protocol provides a reliable, fast, and cost-effective method for manufacturing faithful patient specific organ emulating gelatin phantoms and can be applied or extended to other image-guided intervention applications.

A Disparity Refinement Framework for Learning-based Stereo Matching Methods in Cross-domain Setting for Laparoscopic Images.

Stereo matching methods that enable depth estimation are crucial for visualization enhancement applications in computer-assisted surgery (CAS). Learning-based stereo matching methods are promising to predict accurate results for applications involving video images. However, they require a large amount of training data, and their performance may be degraded due to domain shifts. Maintaining robustness and improving performance of learning-based methods are still open problems. To overcome the limitations of learning-based methods, we propose a disparity refinement framework consisting of a local disparity refinement method and a global disparity refinement method to improve the results of learning-based stereo matching methods in a cross-domain setting. Those learning-based stereo matching methods are pre-trained on a large public dataset of natural images and are tested on a dataset of laparoscopic images. Results from the SERV-CT dataset showed that our proposed framework can effectively refine disparity maps on an unseen dataset even when they are corrupted by noise, and without compromising correct prediction, provided the network can generalize well on unseen datasets. As such, our proposed disparity refinement framework has the potential to work with learning-based methods to achieve robust and accurate disparity prediction. Yet, as a large laparoscopic dataset for training learning-based methods does not exist and the generalization ability of networks remains to be improved, it will be beneficial to incorporate the proposed disparity refinement framework into existing networks for more accurate and robust depth estimation.

Investigating the impact of class-dependent label noise in medical image classification.

Label noise is inevitable in medical image databases developed for deep learning due to the inter-observer variability caused by the different levels of expertise of the experts annotating the images, and, in some cases, the automated methods that generate labels from medical reports. It is known that incorrect annotations or label noise can degrade the actual performance of supervised deep learning models and can bias the model's evaluation. Existing literature show that noise in one class has minimal impact on the model's performance for another class in natural image classification problems where different target classes have a relatively distinct shape and share minimal visual cues for knowledge transfer among the classes. However, it is not clear how class-dependent label noise affects the model's performance when operating on medical images, for which different output classes can be difficult to distinguish even for experts, and there is a high possibility of knowledge transfer across classes during the training period. We hypothesize that for medical image classification tasks where the different classes share a very similar shape with differences only in texture, the noisy label for one class might affect the performance across other classes, unlike the case when the target classes have different shapes and are visually distinct. In this paper, we study this hypothesis using two publicly available datasets: a 2D organ classification dataset with target organ classes being visually distinct, and a histopathology image classification dataset where the target classes look very similar visually. Our results show that the label noise in one class has a much higher impact on the model's performance on other classes for the histopathology dataset compared to the organ dataset.

Deep learning architecture for 3D image super-resolution of late gadolinium enhanced cardiac MRI.

Purpose: High-resolution late gadolinium enhanced (LGE) cardiac magnetic resonance imaging (MRI) volumes are difficult to acquire due to the limitations of the maximal breath-hold time achievable by the patient. This results in anisotropic 3D volumes of the heart with high in-plane resolution, but low-through-plane resolution. Thus, we propose a 3D convolutional neural network (CNN) approach to improve the through-plane resolution of the cardiac LGE-MRI volumes. Approach: We present a 3D CNN-based framework with two branches: a super-resolution branch to learn the mapping between low-resolution and high-resolution LGE-MRI volumes, and a gradient branch that learns the mapping between the gradient map of low-resolution LGE-MRI volumes and the gradient map of high-resolution LGE-MRI volumes. The gradient branch provides structural guidance to the CNN-based super-resolution framework. To assess the performance of the proposed CNN-based framework, we train two CNN models with and without gradient guidance, namely, dense deep back-projection network (DBPN) and enhanced deep super-resolution network. We train and evaluate our method on the 2018 atrial segmentation challenge dataset. Additionally, we also evaluate these trained models on the left atrial and scar quantification and segmentation challenge 2022 dataset to assess their generalization ability. Finally, we investigate the effect of the proposed CNN-based super-resolution framework on the 3D segmentation of the left atrium (LA) from these cardiac LGE-MRI image volumes. Results: Experimental results demonstrate that our proposed CNN method with gradient guidance consistently outperforms bicubic interpolation and the CNN models without gradient guidance. Furthermore, the segmentation results, evaluated using Dice score, obtained using the super-resolved images generated by our proposed method are superior to the segmentation results obtained using the images generated by bicubic interpolation (p<0.01) and the CNN models without gradient guidance (p<0.05). Conclusion: The presented CNN-based super-resolution method with gradient guidance improves the through-plane resolution of the LGE-MRI volumes and the structure guidance provided by the gradient branch can be useful to aid the 3D segmentation of cardiac chambers, such as LA, from the 3D LGE-MRI images.

Learning feature descriptors for pre- and intra-operative point cloud matching for laparoscopic liver registration.

PURPOSE: In laparoscopic liver surgery, preoperative information can be overlaid onto the intra-operative scene by registering a 3D preoperative model to the intra-operative partial surface reconstructed from the laparoscopic video. To assist with this task, we explore the use of learning-based feature descriptors, which, to our best knowledge, have not been explored for use in laparoscopic liver registration. Furthermore, a dataset to train and evaluate the use of learning-based descriptors does not exist. METHODS: We present the LiverMatch dataset consisting of 16 preoperative models and their simulated intra-operative 3D surfaces. We also propose the LiverMatch network designed for this task, which outputs per-point feature descriptors, visibility scores, and matched points. RESULTS: We compare the proposed LiverMatch network with a network closest to LiverMatch and a histogram-based 3D descriptor on the testing split of the LiverMatch dataset, which includes two unseen preoperative models and 1400 intra-operative surfaces. Results suggest that our LiverMatch network can predict more accurate and dense matches than the other two methods and can be seamlessly integrated with a RANSAC-ICP-based registration algorithm to achieve an accurate initial alignment. CONCLUSION: The use of learning-based feature descriptors in laparoscopic liver registration (LLR) is promising, as it can help achieve an accurate initial rigid alignment, which, in turn, serves as an initialization for subsequent non-rigid registration.

Investigating Transformer Encoding Techniques to Improve Data-Driven Volume-to-Surface Liver Registration for Image-Guided Navigation.

Due to limited direct organ visualization, minimally invasive interventions rely extensively on medical imaging and image guidance to ensure accurate surgical instrument navigation and target tissue manipulation. In the context of laparoscopic liver interventions, intra-operative video imaging only provides a limited field-of-view of the liver surface, with no information of any internal liver lesions identified during diagnosis using pre-procedural imaging. Hence, to enhance intra-procedural visualization and navigation, the registration of pre-procedural, diagnostic images and anatomical models featuring target tissues to be accessed or manipulated during surgery entails a sufficient accurate registration of the pre-procedural data into the intra-operative setting. Prior work has demonstrated the feasibility of neural network-based solutions for nonrigid volume-to-surface liver registration. However, view occlusion, lack of meaningful feature landmarks, and liver deformation between the pre- and intra-operative settings all contribute to the difficulty of this registration task. In this work, we leverage some of the state-of-the-art deep learning frameworks to implement and test various network architecture modifications toward improving the accuracy and robustness of volume-to-surface liver registration. Specifically, we focus on the adaptation of a transformer-based segmentation network for the task of better predicting the optimal displacement field for nonrigid registration. Our results suggest that one particular transformer-based network architecture-UTNet-led to significant improvements over baseline performance, yielding a mean displacement error on the order of 4 mm across a variety of datasets.

Improving Medical Image Classification in Noisy Labels Using only Self-supervised Pretraining.

Noisy labels hurt deep learning-based supervised image classification performance as the models may overfit the noise and learn corrupted feature extractors. For natural image classification training with noisy labeled data, model initialization with contrastive self-supervised pretrained weights has shown to reduce feature corruption and improve classification performance. However, no works have explored: i) how other self-supervised approaches, such as pretext task-based pretraining, impact the learning with noisy label, and ii) any self-supervised pretraining methods alone for medical images in noisy label settings. Medical images often feature smaller datasets and subtle inter-class variations, requiring human expertise to ensure correct classification. Thus, it is not clear if the methods improving learning with noisy labels in natural image datasets such as CIFAR would also help with medical images. In this work, we explore contrastive and pretext task-based selfsupervised pretraining to initialize the weights of a deep learning classification model for two medical datasets with self-induced noisy labels-NCT-CRC-HE-100K tissue histological images and COVID-QU-Ex chest X-ray images. Our results show that models initialized with pretrained weights obtained from self-supervised learning can effectively learn better features and improve robustness against noisy labels.

A Feature-based Affine Registration Method for Capturing Background Lung Tissue Deformation for Ground Glass Nodule Tracking.

Lung nodule tracking assessment relies on cross-sectional measurements of the largest lesion profile depicted in initial and follow-up computed tomography (CT) images. However, apparent changes in nodule size assessed via simple image-based measurements may also be compromised by the effect of the background lung tissue deformation on the GGN between the initial and follow-up images, leading to erroneous conclusions about nodule changes due to disease. To compensate for the lung deformation and enable consistent nodule tracking, here we propose a feature-based affine registration method and study its performance vis-a-vis several other registration methods. We implement and test each registration method using both a lung- and a lesion-centered region of interest on ten patient CT datasets featuring twelve nodules, including both benign and malignant GGO lesions containing pure GGNs, part-solid, or solid nodules. We evaluate each registration method according to the target registration error (TRE) computed across 30 - 50 homologous fiducial landmarks surrounding the lesions and selected by expert radiologists in both the initial and follow-up patient CT images. Our results show that the proposed feature-based affine lesion-centered registration yielded a 1.1 ± 1.2 mm TRE, while a Symmetric Normalization deformable registration yielded a 1.2 ± 1.2 mm TRE, and a least-square fit registration of the 30-50 validation fiducial landmark set yielded a 1.5 ± 1.2 mm TRE. Although the deformable registration yielded a slightly higher registration accuracy than the feature-based affine registration, it is significantly more computationally efficient, eliminates the need for ambiguous segmentation of GGNs featuring ill-defined borders, and reduces the susceptibility of artificial deformations introduced by the deformable registration, which may lead to increased similarity between the registered initial and follow-up images, over-compensating for the background lung tissue deformation, and, in turn, compromising the true disease-induced nodule change assessment. We also assessed the registration qualitatively, by visual inspection of the subtraction images, and conducted a pilot pre-clinical study that showed the proposed feature-based lesion-centered affine registration effectively compensates for the background lung tissue deformation between the initial and follow-up images and also serves as a reliable baseline registration method prior to assessing lung nodule changes due to disease.

SPIE Medical Imaging 50th anniversary: historical review of the Image-Guided Procedures, Robotic Interventions, and Modeling conference.

Purpose: Among the conferences comprising the Medical Imaging Symposium is the MI104 conference currently titled Image-Guided Procedures, Robotic Interventions, and Modeling, although its name has evolved through at least nine iterations over the last 30 years. Here, we discuss the important role that this forum has presented for researchers in the field during this time. Approach: The origins of the conference are traced from its roots in Image Capture and Display in the late 1980s, and some of the major themes for which the conference and its proceedings have provided a valuable forum are highlighted. Results: These major themes include image display/visualization, surgical tracking/navigation, surgical robotics, interventional imaging, image registration, and modeling. Exceptional work from the conference is highlighted by summarizing keynote lectures, the top 50 most downloaded proceedings papers over the last 30 years, the most downloaded paper each year, and the papers earning student paper and young scientist awards. Conclusions: Looking forward and considering the burgeoning technologies, algorithms, and markets related to image-guided and robot-assisted interventions, we anticipate growth and ever increasing quality of the conference as well as increased interaction with sister conferences within the symposium.

A 3D Convolutional Neural Network with Gradient Guidance for Image Super-Resolution of Late Gadolinium Enhanced Cardiac MRI.

In this paper, we describe a 3D convolutional neural network (CNN) framework to compute and generate super-resolution late gadolinium enhanced (LGE) cardiac magnetic resonance imaging (MRI) images. The proposed CNN framework consists of two branches: a super-resolution branch with a 3D dense deep back-projection network (DBPN) as the backbone to learn the mapping of low-resolution LGE cardiac volumes to high-resolution LGE cardiac volumes, and a gradient branch that learns the mapping of the gradient map of low resolution LGE cardiac volumes to the gradient map of their high-resolution counterparts. The gradient branch of the CNN provides additional cardiac structure information to the super-resolution branch to generate structurally more accurate super-resolution LGE MRI images. We conducted our experiments on the 2018 atrial segmentation challenge dataset. The proposed CNN framework achieved a mean peak signal-to-noise ratio (PSNR) of 30.91 and 25.66 and a mean structural similarity index measure (SSIM) of 0.91 and 0.75 on training the model on low-resolution images downsamp led by a scale factor of 2 and 4, respectively.

Joint Segmentation and Uncertainty Estimation of Ventricular Structures from Cardiac MRI using a Bayesian CondenseUNet.

While convolutional neural networks (CNNs) have shown potential in segmenting cardiac structures from magnetic resonance (MR) images, their clinical applications still fall short of providing reliable cardiac segmentation. As a result, it is critical to quantify segmentation uncertainty in order to identify which segmentations might be troublesome. Moreover, quantifying uncertainty is critical in real-world scenarios, where input distributions are frequently moved from the training distribution due to sample bias and non-stationarity. Therefore, well-calibrated uncertainty estimates provide information on whether a model's output should (or should not) be trusted in such situations. In this work, we used a Bayesian version of our previously proposed CondenseUNet [1] framework featuring both a learned group structure and a regularized weight-pruner to reduce the computational cost in volumetric image segmentation and help quantify predictive uncertainty. Our study further showcases the potential of our deep-learning framework to evaluate the correlation between the uncertainty and the segmentation errors for a given model. The proposed model was trained and tested on the Automated Cardiac Diagnosis Challenge (ACDC) dataset featuring 150 cine cardiac MRI patient dataset for the segmentation and uncertainty estimation of the left ventricle (LV), right ventricle (RV), and myocardium (Myo) at end-diastole (ED) and end-systole (ES) phases.

Investigating uncertainty in augmented reality enhanced renal navigation using in vitro patient-specific tissue-mimicking phantoms.

To improve the outcome of minimally invasive renal interventions, traditional video-guided needle navigation can be enhanced by tracking the needle, guiding the needle using video imaging, and augmenting the surgical scene with pre-procedural images or models of the anatomy. In our previous work we studied, both through simulations and in vitro experiments, the uncertainty associated with the model-to-phantom registration, as well as the camera-tracker calibration and video-guided navigation. In this work, we characterize the overall navigation uncertainty using tissue emulating patient-specific kidney phantoms featuring both virtual and physical internal targets. Pre-procedural models of the kidney phantoms and internal targets are generated from cone-beam CT images, and are registered to their intra-operative physical counter-parts. The user then guides the needle insertion to reach the internal targets using video-based imaging augmented with a virtual representation of the needle tracked in real time. Following navigation, we acquire post-procedural cone-beam CT images of the phantoms and inserted needles. These images are used to determine the ground truth needle navigation accuracy (i.e., needle to target distance) against which the intra-operative navigation accuracy (i.e., intra-op needle tip to target distance) is assessed. We also explore a method to update the pre-procedural model to physical phantom registration intra-operatively using tracked video imaging, with the overall goal to improve overall navigation accuracy in the event of sub-optimal initial image-to-phantom registration. Our results showed a navigation error of less than 3.5 mm in gelatin phantoms and less than 6.5 mm in PVA phantoms. Following registration correction intra-operatively, we showed an overall improvement in navigation from roughly 6 mm RMS to approximately 2 mm RMS error, which is acceptable given the inherent tracking, 3D printing and phantom manufacturing limitations.

Endoscope Localization and Dense Surgical Scene Reconstruction for Stereo Endoscopy by Unsupervised Optical Flow and Kanade-Lucas-Tomasi Tracking.

In image-guided surgery, endoscope tracking and surgical scene reconstruction are critical, yet equally challenging tasks. We present a hybrid visual odometry and reconstruction framework for stereo endoscopy that leverages unsupervised learning-based and traditional optical flow methods to enable concurrent endoscope tracking and dense scene reconstruction. More specifically, to reconstruct texture-less tissue surfaces, we use an unsupervised learning-based optical flow method to estimate dense depth maps from stereo images. Robust 3D landmarks are selected from the dense depth maps and tracked via the Kanade-Lucas-Tomasi tracking algorithm. The hybrid visual odometry also benefits from traditional visual odometry modules, such as keyframe insertion and local bundle adjustment. We evaluate the proposed framework on endoscopic video sequences openly available via the SCARED dataset against both ground truth data, as well as two other state-of-the-art methods - ORB-SLAM2 and Endo-depth. Our proposed method achieved comparable results in terms of both RMS Absolute Trajectory Error and Cloud-to-Mesh RMS Error, suggesting its potential to enable accurate endoscope tracking and scene reconstruction.

A Weakly Supervised Learning Approach for Surgical Instrument Segmentation from Laparoscopic Video Sequences.

Fully supervised learning approaches for surgical instrument segmentation from video images usually require a time-consuming process of generating accurate ground truth segmentation masks. We propose an alternative way of labeling surgical instruments for binary segmentation that first commences with rough, scribble-like annotations of the surgical instruments using a disc-shaped brush. We then present a framework that starts with a graph-model-based method for generating initial segmentation labels based on the user-annotated paint-brush scribbles and then proceeds with a deep learning model that learns from the noisy, initial segmentation labels. Experiments conducted on the 2017 MICCAI EndoVis Robotic Instrument Segmentation Challenge have shown that the proposed framework achieved a 76.82% IoU and 85.70% Dice score on binary instrument segmentation. Based on these metrics, the proposed method out-performs other weakly supervised techniques and achieves a close performance to that achieved via fully supervised networks, but eliminates the need for ground truth segmentation masks.