PolypMixNet: Enhancing semi-supervised polyp segmentation with polyp-aware augmentation.

BACKGROUND: AI-assisted polyp segmentation in colonoscopy plays a crucial role in enabling prompt diagnosis and treatment of colorectal cancer. However, the lack of sufficient annotated data poses a significant challenge for supervised learning approaches. Existing semi-supervised learning methods also suffer from performance degradation, mainly due to task-specific characteristics, such as class imbalance in polyp segmentation. PURPOSE: The purpose of this work is to develop an effective semi-supervised learning framework for accurate polyp segmentation in colonoscopy, addressing limited annotated data and class imbalance challenges. METHODS: We proposed PolypMixNet, a semi-supervised framework, for colorectal polyp segmentation, utilizing novel augmentation techniques and a Mean Teacher architecture to improve model performance. PolypMixNet introduces the polyp-aware mixup (PolypMix) algorithm and incorporates dual-level consistency regularization. PolypMix addresses the class imbalance in colonoscopy datasets and enhances the diversity of training data. By performing a polyp-aware mixup on unlabeled samples, it generates mixed images with polyp context along with their artificial labels. A polyp-directed soft pseudo-labeling (PDSPL) mechanism was proposed to generate high-quality pseudo labels and eliminate the dilution of lesion features caused by mixup operations. To ensure consistency in the training phase, we introduce the PolypMix prediction consistency (PMPC) loss and PolypMix attention consistency (PMAC) loss, enforcing consistency at both image and feature levels. Code is available at https://github.com/YChienHung/PolypMix. RESULTS: PolypMixNet was evaluated on four public colonoscopy datasets, achieving 88.97% Dice and 88.85% mIoU on the benchmark dataset of Kvasir-SEG. In scenarios where the labeled training data is limited to 15%, PolypMixNet outperforms the state-of-the-art semi-supervised approaches with a 2.88-point improvement in Dice. It also shows the ability to reach performance comparable to the fully supervised counterpart. Additionally, we conducted extensive ablation studies to validate the effectiveness of each module and highlight the superiority of our proposed approach. CONCLUSION: PolypMixNet effectively addresses the challenges posed by limited annotated data and unbalanced class distributions in polyp segmentation. By leveraging unlabeled data and incorporating novel augmentation and consistency regularization techniques, our method achieves state-of-the-art performance. We believe that the insights and contributions presented in this work will pave the way for further advancements in semi-supervised polyp segmentation and inspire future research in the medical imaging domain.

A photoacoustics-enhanced drilling probe for radiation-free pedicle screw implantation in spinal surgery.

This article proposes a novel intra-operative navigation and sensing system that optimizes the functional accuracy of spinal pedicle screw implantation. It does so by incorporating radiation-free and multi-scale macroscopic 3D ultrasound (US) imaging and local tissue-awareness from in situ photoacoustic (PA) sensing at a clinically relevant mesoscopic scale. More specifically, 3D US imaging is employed for online status updates of spinal segment posture to determine the appropriate entry point and coarse drilling path once non-negligible or relative patient motion occurs between inter-vertebral segments in the intra-operative phase. Furthermore, a sophisticated sensor-enhanced drilling probe has been developed to facilitate fine-grained local navigation that integrates a PA endoscopic imaging component for in situ tissue sensing. The PA signals from a sideways direction to differentiate cancellous bone from harder cortical bone, or to indicate weakened osteoporotic bone within the vertebrae. In so doing it prevents cortical breaches, strengthens implant stability, and mitigates iatrogenic injuries of the neighboring artery and nerves. To optimize this PA-enhanced endoscopic probe design, the light absorption spectrum of cortical bone and cancellous bone are measured in vitro, and the associated PA signals are characterized. Ultimately, a pilot study is performed on an ex vivo bovine spine to validate our developed multi-scale navigation and sensing system. The experimental results demonstrate the clinical feasibility, and hence the great potential, for functionally accurate screw implantation in complex spinal stabilization interventions.

Design and Control of a Magnetically-Actuated Capsule Robot With Biopsy Function.

OBJECTIVE: Wireless capsule endoscopy has been well used for gastrointestinal (GI) tract diagnosis. However, it can only obtain images and cannot take samples of GI tract tissues. In this study, we designed a magnetically-actuated biopsy capsule (MABC) robot for GI tract diagnosis. METHODS: The proposed robot can achieve locomotion and biopsy functions under the control of external electromagnetic actuation (EMA) system. Two types of active locomotion can be achieved, plane motion refers to the robot rolling on the surface of the GI tract with a rotating uniform magnetic field. 3D motion refers to the robot moving in 3D space under the control of the EMA system. After reaching the target position, the biopsy needle can be sprung out for sampling and then retracted under a gradient magnetic field. RESULTS: A pill-shaped robot prototype ( ϕ15 mm×32 mm) has been fabricated and tested with phantom experiments. The average motion control error is 0.32 mm in vertical direction, 3.3 mm in horizontal direction, and the maximum sampling error is about 5.0 mm. The average volume of the sampled tissue is about 0.35 mm 3. CONCLUSION: We designed a MABC robot and proposed a control framework which enables planar and 3D spatial locomotion and biopsy sampling. SIGNIFICANCE: The untethered MABC robot can be remotely controlled to achieve accurate sampling in multiple directions without internal power sources, paving the way towards precision sampling techniques for GI diseases in clinical procedures.

A Virtual Scanning Framework for Robotic Spinal Sonography with Automatic Real-time Recognition of Standard Views.

Ultrasound (US) imaging is widely used to assist in the diagnosis and intervention of the spine, but the manual scanning process would bring heavy physical and cognitive burdens on the sonographers. Robotic US acquisitions can provide an alternative to the standard handheld technique to reduce operator workload and avoid direct patient contact. However, the real-time interpretation of the acquired images is rarely addressed in existing robotic US systems. Therefore, we envision a robotic system that can automatically scan the spine and search for the standard views like an expert sonographer. In this work, we propose a virtual scanning framework based on real-world US data acquired by a robotic system to simulate the autonomous robotic spinal sonography, and incorporate automatic real-time recognition of the standard views of the spine based on a multi-scale fusion approach and deep convolutional neural networks. Our method can accurately classify 96.71% of the standard views of the spine in the test set, and the simulated clinical application preliminarily demonstrates the potential of our method.

APRNet: Alternative Prediction Refinement Network for Polyp Segmentation.

Colorectal cancer has become the second leading cause of cancer-related death, attracting considerable interest for automatic polyp segmentation in polyp screening system. Accurate segmentation of polyps from colonoscopy is a challenging task as the polyps diverse in color, size and texture while the boundary between polyp and background is sometimes ambiguous. We propose a novel alternative prediction refinement network (APRNet) to more accurately segment polyps. Based on the UNet architecture, our APRNet aims at exploiting all-level features by alternatively leveraging features from encoder and decoder branch. Specifically, a series of prediction residual refinement modules (PRR) learn the residual and progressively refine the segmentation at various resolution. The proposed APRNet is evaluated on two benchmark datasets and achieves new state-of-the-art performance with a dice of 91.33% and an accuracy of 97.31% on the Kvasir-SEG dataset, and a dice of 86.33% and an accuracy of 97.12% on the EndoScene dataset.Clinical relevance- This work proposes an automatic and accurate polyp segmentation algorithm that achieves new state- of-the-art performance, which can potentially act as an observer pointing out polyps in colonoscopy procedure.

Tip estimation approach for concentric tube robots using 2D ultrasound images and kinematic model.

Concentric tube robot (CTR) is an efficient approach for minimally invasive surgery (MIS) and diagnosis due to its small size and high dexterity. To manipulate the robot accurately and safely inside the human body, tip position and shape information need to be well measured. In this paper, we propose a tip estimation method based on 2D ultrasound images with the help of the forward kinematic model of CTR. The forward kinematic model can help to provide a fast ultrasound scanning path and narrow the region of interest in ultrasound images. For each tube, only three scan positions are needed by combining the kinematic model prediction as prior knowledge. After that, the curve fitting method is used for its shape reconstruction, while its tip position can be estimated based on the constraints of its structure and length.7 This method provides the advantage that only three scan positions are needed for estimating the tip of each telescoping section. Moreover, no structure modification is needed on the robot, which makes it an appropriate approach for existing flexible surgical robots. Experimental results verified the feasibility of the proposed method and the tip estimation error is 0.59 mm. Graphical abstract In this paper, we propose a tip estimation method based on 2D Ultrasound images with the help of the forward kinematic model of CTR. The forward kinematic model can help to provide a fast Ultrasound scanning path and narrow the region of interest in Ultrasound images. For each tube, only three scan positions are needed by combining the kinematic model prediction as prior knowledge. After that, the curve fitting method is used for its shape reconstruction, while its tip position can be estimated based on the constraints of its structure and length.

General first-order target registration error model considering a coordinate reference frame in an image-guided surgical system.

Point-based rigid registration (PBRR) techniques are widely used in many aspects of image-guided surgery (IGS). Accurately estimating target registration error (TRE) statistics is of essential value for medical applications such as optically surgical tool-tip tracking and image registration. For example, knowing the TRE distribution statistics of surgical tool tip can help the surgeon make right decisions during surgery. In the meantime, the pose of a surgical tool is usually reported relative to a second rigid body whose local frame is called coordinate reference frame (CRF). In an n-ocular tracking system, fiducial localization error (FLE) should be considered inhomogeneous, that means FLE is different between fiducials, and anisotropic that indicates FLE is different in all directions. In this paper, we extend the TRE estimation algorithm relative to a CRF from homogeneous and anisotropic to heterogeneous FLE cases. Arbitrary weightings can be assumed in solving the registration problems in the proposed TRE estimation algorithm. Monte Carlo simulation results demonstrate the proposed algorithm's effectiveness for both homogeneous and inhomogeneous FLE distributions. The results are further compared with those using the other two algorithms. When FLE distribution is anisotropic and homogeneous, the proposed TRE estimation algorithm's performance is comparable with that of the first one. When FLE distribution is heterogeneous, proposed TRE estimation algorithm outperforms the other two classical algorithms in all test cases when ideal weighting scheme is adopted in solving two registrations. Possible clinical applications include the online estimation of surgical tool-tip tracking error with respect to a CRF in IGS. Graphical Abstract This paper provides the target registration error model considering a coordinate reference frame in surgical navigation.

Design of a multi-arm concentric-tube robot system for transnasal surgery.

Concentric tube robot (CTR) has gradually attracted the attention of researchers on the basis of its small size and curved shape control ability. However, most of current experimental prototypes of CTR are single-arm structure, which can only carry out simple operation such as drug delivery or monitoring. In this paper, design and analysis of a three-arm CTR system is proposed. It has a four-DOF vision arm and two six-DOF manipulator arms, which equipped with special end effectors to achieve different surgical operations. Finally, a mean motion accuracy of 0.33 mm has been obtained quantitatively through teleoperation experiments. Moreover, tissue excision experiment in skull model is carried out to prove the effectiveness and feasibility of the proposed CTR system in nasopharyngeal carcinoma surgery. Graphical Abstract Platform of the proposed Multi-Arm Concentric Tube Robot system. (a) Configuration of the end-effectors with the CTR system. (b) The setup of the tissue removal experiment in a skull model.

Pilot Study of Trans-oral Robotic-Assisted Needle Direct Tracheostomy Puncture in Patients Requiring Prolonged Mechanical Ventilation.

COVID-19 can induce severe respiratory problems that need prolonged mechanical ventilation in the intensive care unit. While Open Tracheostomy (OT) is the preferred technique due to the excellent visualization of the surgical field and structures, Percutaneous Tracheostomy (PT) has proven to be a feasible minimally invasive alternative. However, PT's limitation relates to the inability to precisely enter the cervical trachea at the exact spot since the puncture is often performed based on crude estimation from anatomical laryngeal surface landmarks. Besides, there is no absolute control of the trajectory and force required to make the percutaneous puncture into the trachea, resulting in inadvertent injury to the cricoid ring, cervical esophagus, and vessels in the neck. Therefore, we hypothesize that a flexible mini-robotic system, incorporating the robotic needling technology, can overcome these challenges by allowing the trans-oral robotic instrument of the cervical trachea. This approach promises to improve current PT technology by making the initial trachea puncture from an "inside-out" approach, rather than an "outside-in" manner, fraught with several technical uncertainties.

Augmented Bladder Tumor Detection Using Deep Learning.

Adequate tumor detection is critical in complete transurethral resection of bladder tumor (TURBT) to reduce cancer recurrence, but up to 20% of bladder tumors are missed by standard white light cystoscopy. Deep learning augmented cystoscopy may improve tumor localization, intraoperative navigation, and surgical resection of bladder cancer. We aimed to develop a deep learning algorithm for augmented cystoscopic detection of bladder cancer. Patients undergoing cystoscopy/TURBT were recruited and white light videos were recorded. Video frames containing histologically confirmed papillary urothelial carcinoma were selected and manually annotated. We constructed CystoNet, an image analysis platform based on convolutional neural networks, for automated bladder tumor detection using a development dataset of 95 patients for algorithm training and five patients for testing. Diagnostic performance of CystoNet was validated prospectively in an additional 54 patients. In the validation dataset, per-frame sensitivity and specificity were 90.9% (95% confidence interval [CI], 90.3-91.6%) and 98.6% (95% CI, 98.5-98.8%), respectively. Per-tumor sensitivity was 90.9% (95% CI, 90.3-91.6%). CystoNet detected 39 of 41 papillary and three of three flat bladder cancers. With high sensitivity and specificity, CystoNet may improve the diagnostic yield of cystoscopy and efficacy of TURBT. PATIENT SUMMARY: Conventional cystoscopy has recognized shortcomings in bladder cancer detection, with implications for recurrence. Cystoscopy augmented with artificial intelligence may improve cancer detection and resection.

Preliminary study on magnetic tracking-based planar shape sensing and navigation for flexible surgical robots in transoral surgery: methods and phantom experiments.

PURPOSE: Flexible surgical robot can work in confined and complex environments, which makes it a good option for minimally invasive surgery. In order to utilize flexible manipulators in complicated and constrained surgical environments, it is of great significance to monitor the position and shape of the curvilinear manipulator in real time during the procedures. In this paper, we propose a magnetic tracking-based planar shape sensing and navigation system for flexible surgical robots in the transoral surgery. The system can provide the real-time tip position and shape information of the robot during the operation. METHODS: We use wire-driven flexible robot to serve as the manipulator. It has three degrees of freedom. A permanent magnet is mounted at the distal end of the robot. Its magnetic field can be sensed with a magnetic sensor array. Therefore, position and orientation of the tip can be estimated utilizing a tracking method. A shape sensing algorithm is then carried out to estimate the real-time shape based on the tip pose. With the tip pose and shape display in the 3D reconstructed CT model, navigation can be achieved. RESULTS: Using the proposed system, we carried out planar navigation experiments on a skull phantom to touch three different target positions under the navigation of the skull display interface. During the experiments, the real-time shape has been well monitored and distance errors between the robot tip and the targets in the skull have been recorded. The mean navigation error is [Formula: see text] mm, while the maximum error is 3.2 mm. CONCLUSION: The proposed method provides the advantages that no sensors are needed to mount on the robot and no line-of-sight problem. Experimental results verified the feasibility of the proposed method.

Automatic Polyp Detection via a Novel Unified Bottom-Up and Top-Down Saliency Approach.

In this paper, we propose a novel automatic computer-aided method to detect polyps for colonoscopy videos. To capture perceptually and semantically meaningful salient polyp regions, we first segment images into multilevel superpixels. Each level corresponds to different sizes of superpixels. Rather than adopting hand-designed features to describe these superpixels in images, we employ sparse autoencoder (SAE) to learn discriminative features in an unsupervised way. Then, a novel unified bottom-up and top-down saliency method is proposed to detect polyps. In the first stage, we propose a weak bottom-up (WBU) saliency map by fusing the contrast-based saliency and object-center-based saliency together. The contrast-based saliency map highlights image parts that show different appearances compared with surrounding areas, whereas the object-center-based saliency map emphasizes the center of the salient object. In the second stage, a strong classifier with multiple kernel boosting is learned to calculate the strong top-down (STD) saliency map based on samples directly from the obtained multilevel WBU saliency maps. We finally integrate these two-stage saliency maps from all levels together to highlight polyps. Experiment results achieve 0.818 recall for saliency calculation, validating the effectiveness of our method. Extensive experiments on public polyp datasets demonstrate that the proposed saliency algorithm performs better compared with state-of-the-art saliency methods to detect polyps.

Estimation of surgical tool-tip tracking error distribution in coordinate reference frame involving pivot calibration uncertainty.

Accurate understanding of surgical tool-tip tracking error is important for decision making in image-guided surgery. In this Letter, the authors present a novel method to estimate/model surgical tool-tip tracking error in which they take pivot calibration uncertainty into consideration. First, a new type of error that is referred to as total target registration error (TTRE) is formally defined in a single-rigid registration. Target localisation error (TLE) in two spaces to be registered is considered in proposed TTRE formulation. With first-order approximation in fiducial localisation error (FLE) or TLE magnitude, TTRE statistics (mean, covariance matrix and root-mean-square (RMS)) are then derived. Second, surgical tool-tip tracking error in optical tracking system (OTS) frame is formulated using TTRE when pivot calibration uncertainty is considered. Finally, TTRE statistics of tool-tip in OTS frame are then propagated relative to a coordinate reference frame (CRF) rigid-body. Monte Carlo simulations are conducted to validate the proposed error model. The percentage passing statistical tests that there is no difference between simulated and theoretical mean and covariance matrix of tool-tip tracking error in CRF space is more than 90% in all test cases. The RMS percentage difference between simulated and theoretical tool-tip tracking error in CRF space is within 5% in all test cases.

Gastrointestinal bleeding detection in wireless capsule endoscopy images using handcrafted and CNN features.

Gastrointestinal (GI) bleeding detection plays an essential role in wireless capsule endoscopy (WCE) examination. In this paper, we present a new approach for WCE bleeding detection that combines handcrafted (HC) features and convolutional neural network (CNN) features. Compared with our previous work, a smaller-scale CNN architecture is constructed to lower the computational cost. In experiments, we show that the proposed strategy is highly capable when training data is limited, and yields comparable or better results than the latest methods.

Deep learning for polyp recognition in wireless capsule endoscopy images.

PURPOSE: Wireless capsule endoscopy (WCE) enables physicians to examine the digestive tract without any surgical operations, at the cost of a large volume of images to be analyzed. In the computer-aided diagnosis of WCE images, the main challenge arises from the difficulty of robust characterization of images. This study aims to provide discriminative description of WCE images and assist physicians to recognize polyp images automatically. METHODS: We propose a novel deep feature learning method, named stacked sparse autoencoder with image manifold constraint (SSAEIM), to recognize polyps in the WCE images. Our SSAEIM differs from the traditional sparse autoencoder (SAE) by introducing an image manifold constraint, which is constructed by a nearest neighbor graph and represents intrinsic structures of images. The image manifold constraint enforces that images within the same category share similar learned features and images in different categories should be kept far away. Thus, the learned features preserve large intervariances and small intravariances among images. RESULTS: The average overall recognition accuracy (ORA) of our method for WCE images is 98.00%. The accuracies for polyps, bubbles, turbid images, and clear images are 98.00%, 99.50%, 99.00%, and 95.50%, respectively. Moreover, the comparison results show that our SSAEIM outperforms existing polyp recognition methods with relative higher ORA. CONCLUSION: The comprehensive results have demonstrated that the proposed SSAEIM can provide descriptive characterization for WCE images and recognize polyps in a WCE video accurately. This method could be further utilized in the clinical trials to help physicians from the tedious image reading work.

A general framework for wireless capsule endoscopy study synopsis.

We present a general framework for analysis of wireless capsule endoscopy (CE) studies. The current available workstations provide a time-consuming and labor-intense work-flow for clinicians which requires the inspection of the full-length video. The development of a computer-aided diagnosis (CAD) CE workstation will have a great potential to reduce the diagnostic time and improve the accuracy of assessment. We propose a general framework based on hidden Markov models (HMMs) for study synopsis that forms the computational engine of our CAD workstation. Color, edge and texture features are first extracted and analyzed by a Support Vector Machine classifier, and then encoded as the observations for the HMM, uniquely combining the temporal information during the assessment. Experiments were performed on 13 full-length CE studies, instead of selected images previously reported. The results (e.g. 0.933 accuracy with 0.933 recall for detection of polyps) show that our framework achieved promising performance for multiple classification. We also report the patient-level CAD assessment of complete CE studies for multiple abnormalities, and the patient-level validation demonstrates the effectiveness and robustness of our methods.

Weighted locality-constrained linear coding for lesion classification in CT images.

Computed tomography is a popular imaging modality for detecting abnormalities associated with abdominal organs such as the liver, kidney and uterus. In this paper, we propose a novel weighted locality-constrained linear coding (LLC) method followed by a weighted max-pooling method to classify liver lesions into three classes: cysts, metastases, hemangiomas. We first divide the lesions into same-size patches. Then, we extract the raw features in all patches followed by Principal Components Analysis (PCA) and apply K means to obtain a single LLC dictionary. Since the interior lesion patches and the boundary patches contribute different information in the image, we assign different weights on these two types of patches to obtain the LLC codes. Moreover, a weighted max pooling approach is also proposed to further evaluate the importance of these two types of patches in feature pooling. Experiments on 109 images of liver lesions were carried out to validate the proposed method. The proposed method achieves a best lesion classification accuracy of 96.33%, which appears to be superior compared with traditional image coding methods: LLC method and Bag-of-words method (BoW) and traditional features: Local Binary Pattern (LBP) features, uniform LBP and complete LBP, demonstrating that the proposed method provides better classification.

Colored petri nets to model gene mutation and amino acids classification.

The genetic code is the triplet code based on the three-letter codons, which determines the specific amino acid sequences in proteins synthesis. Choosing an appropriate model for processing these codons is a useful method to study genetic processes in Molecular Biology. As an effective modeling tool of discrete event dynamic systems (DEDS), colored petri net (CPN) has been used for modeling several biological systems, such as metabolic pathways and genetic regulatory networks. According to the genetic code table, CPN is employed to model the process of genetic information transmission. In this paper, we propose a CPN model of amino acids classification, and further present the improved CPN model. Based on the model mentioned above, we give another CPN model to classify the type of gene mutations via contrasting the bases of DNA strands and the codons of amino acids along the polypeptide chain. This model is helpful in determining whether a certain gene mutation will cause the changes of the structures and functions of protein molecules. The effectiveness and accuracy of the presented model are illustrated by the examples in this paper.

A decision fusion strategy for polyp detection in capsule endoscopy.

Wireless capsule endoscopy (CE) is now routinely used for non-invasive diagnosis of small bowel diseases. But, it still requires manual assessment of the approximately 50,000 study images. Literature has recently investigated automated methods to detect and analyze various anomalies in CE images to improve reading efficiency and reduce variability. We propose such a computer aided diagnosis (CAD) approach to detect small bowel polyps. For supervised classification of polyps, we investigated fusing multiple statistical classifiers based on color, texture and edge features. The combined boosted classifier when evaluated using 1200 CE images outperformed all individual classifiers and achieved a ~90% classification accuracy.

Tumor recognition in wireless capsule endoscopy images using textural features and SVM-based feature selection.

Tumor in digestive tract is a common disease and wireless capsule endoscopy (WCE) is a relatively new technology to examine diseases for digestive tract especially for small intestine. This paper addresses the problem of automatic recognition of tumor for WCE images. Candidate color texture feature that integrates uniform local binary pattern and wavelet is proposed to characterize WCE images. The proposed features are invariant to illumination change and describe multiresolution characteristics of WCE images. Two feature selection approaches based on support vector machine, sequential forward floating selection and recursive feature elimination, are further employed to refine the proposed features for improving the detection accuracy. Extensive experiments validate that the proposed computer-aided diagnosis system achieves a promising tumor recognition accuracy of 92.4% in WCE images on our collected data.