Motion Decoupling Network for Intra-Operative Motion Estimation Under Occlusion.

In recent intelligent-robot-assisted surgery studies, an urgent issue is how to detect the motion of instruments and soft tissue accurately from intra-operative images. Although optical flow technology from computer vision is a powerful solution to the motion-tracking problem, it has difficulty obtaining the pixel-wise optical flow ground truth of real surgery videos for supervised learning. Thus, unsupervised learning methods are critical. However, current unsupervised methods face the challenge of heavy occlusion in the surgical scene. This paper proposes a novel unsupervised learning framework to estimate the motion from surgical images under occlusion. The framework consists of a Motion Decoupling Network to estimate the tissue and the instrument motion with different constraints. Notably, the network integrates a segmentation subnet that estimates the segmentation map of instruments in an unsupervised manner to obtain the occlusion region and improve the dual motion estimation. Additionally, a hybrid self-supervised strategy with occlusion completion is introduced to recover realistic vision clues. Extensive experiments on two surgical datasets show that the proposed method achieves accurate motion estimation for intra-operative scenes and outperforms other unsupervised methods, with a margin of 15% in accuracy. The average estimation error for tissue is less than 2.2 pixels on average for both surgical datasets.

Semi-Supervised Medical Image Segmentation Guided by Bi-Directional Constrained Dual-Task Consistency.

BACKGROUND: Medical image processing tasks represented by multi-object segmentation are of great significance for surgical planning, robot-assisted surgery, and surgical safety. However, the exceptionally low contrast among tissues and limited available annotated data makes developing an automatic segmentation algorithm for pelvic CT challenging. METHODS: A bi-direction constrained dual-task consistency model named PICT is proposed to improve segmentation quality by leveraging free unlabeled data. First, to learn more unmarked data features, it encourages the model prediction of the interpolated image to be consistent with the interpolation of the model prediction at the pixel, model, and data levels. Moreover, to constrain the error prediction of interpolation interference, PICT designs an auxiliary pseudo-supervision task that focuses on the underlying information of non-interpolation data. Finally, an effective loss algorithm for both consistency tasks is designed to ensure the complementary manner and produce more reliable predictions. RESULTS: Quantitative experiments show that the proposed PICT achieves 87.18%, 96.42%, and 79.41% mean DSC score on ACDC, CTPelvic1k, and the individual Multi-tissue Pelvis dataset with gains of around 0.8%, 0.5%, and 1% compared to the state-of-the-art semi-supervised method. Compared to the baseline supervised method, the PICT brings over 3-9% improvements. CONCLUSIONS: The developed PICT model can effectively leverage unlabeled data to improve segmentation quality of low contrast medical images. The segmentation result could improve the precision of surgical path planning and provide input for robot-assisted surgery.

Fast instruments and tissues segmentation of micro-neurosurgical scene using high correlative non-local network.

Surgical scene segmentation provides critical information for guidance in micro-neurosurgery. Segmentation of instruments and critical tissues contributes further to robot assisted surgery and surgical evaluation. However, due to the lack of relevant scene segmentation dataset, scale variation and local similarity, micro-neurosurgical segmentation faces many challenges. To address these issues, a high correlative non-local network (HCNNet), is proposed to aggregate multi-scale feature by optimized non-local mechanism. HCNNet adopts two-branch design to generate features of different scale efficiently, while the two branches share common weights in shallow layers. Several short-term dense concatenate (STDC) modules are combined as the backbone to capture both semantic and spatial information. Besides, a high correlative non-local module (HCNM) is designed to guide the upsampling process of the high-level feature by modeling global context generated from the low-level feature. It filters out confused pixels of different classes in the non-local correlation map. Meanwhile, a large segmentation dataset named NeuroSeg is constructed, which contains 15 types of instruments and 3 types of tissues that appear in meningioma resection surgery. The proposed HCNNet achieves the state-of-the-art performance on NeuroSeg, it reaches an inference speed of 54.85 FPS with the highest accuracy of 59.62% mIoU, 74.7% Dice, 70.55% mAcc and 87.12% aAcc.

Autonomous path planning for robot-assisted pelvic fracture closed reduction with collision avoidance.

BACKGROUND: Robot-assisted pelvic fracture closed reduction (RPFCR) positively contributes to patient treatment. However, the current path planning suffers from incomplete obstacle avoidance and long paths. METHOD: A collision detection method is proposed for applications in the pelvic environment to improve the safety of RPFCR surgery. Meanwhile, a defined orientation planning strategy (OPS) and linear sampling search (LSS) are coupled into the A* algorithm to optimise the reduction path. Subsequently, pelvic in vitro experimental platform is built to verify the augmented A*algorithm's feasibility. RESULTS: The augmented A* algorithm planned the shortest path for the same fracture model, and the paths planned by the A* algorithm and experience-based increased by 56.12% and 89.02%, respectively. CONCLUSIONS: The augmented A* algorithm effectively improves surgical safety and shortens the path length, which can be adopted as an effective model for developing RPFCR path planning.

Learning Skill Characteristics From Manipulations.

Percutaneous coronary intervention (PCI) has increasingly become the main treatment for coronary artery disease. The procedure requires high experienced skills and dexterous manipulations. However, there are few techniques to model PCI skill so far. In this study, a learning framework with local and ensemble learning is proposed to learn skill characteristics of different skill-level subjects from their PCI manipulations. Ten interventional cardiologists (four experts and six novices) were recruited to deliver a medical guidewire to two target arteries on a porcine model for in vivo studies. Simultaneously, translation and twist manipulations of thumb, forefinger, and wrist are acquired with electromagnetic (EM) and fiber-optic bend (FOB) sensors, respectively. These behavior data are then processed with wavelet packet decomposition (WPD) under 1-10 levels for feature extraction. The feature vectors are further fed into three candidate individual classifiers in the local learning layer. Furthermore, the local learning results from different manipulation behaviors are fused in the ensemble learning layer with three rule-based ensemble learning algorithms. In subject-dependent skill characteristics learning, the ensemble learning can achieve 100% accuracy, significantly outperforming the best local result (90%). Furthermore, ensemble learning can also maintain 73% accuracy in subject-independent schemes. These promising results demonstrate the great potential of the proposed method to facilitate skill learning in surgical robotics and skill assessment in clinical practice.

Spatio-Temporal Causal Transformer for Multi-Grained Surgical Phase Recognition.

Automatic surgical phase recognition plays a key role in surgical workflow analysis and overall optimization in clinical work. In the complicated surgical procedures, similar inter-class appearance and drastic variability in phase duration make this still a challenging task. In this paper, a spatio-temporal transformer is proposed for online surgical phase recognition with different granularity. To extract rich spatial information, a spatial transformer is used to model global spatial dependencies of each time index. To overcome the variability in phase duration, a temporal transformer captures the multi-scale temporal context of different time indexes with a dual pyramid pattern. Our method is thoroughly validated on the public Cholec80 dataset with 7 coarse-grained phases and the CATARACTS2020 dataset with 19 fine-grained phases, outperforming state-of-the-art approaches with 91.4% and 84.2% accuracy, taking only 24.5M parameters.

An automatic drug injection device with spatial micro-force perception guided by an microscopic image for robot-assisted ophthalmic surgery.

Retinal vein injection guided by microscopic image is an innovative procedure for treating retinal vein occlusion. However, the retina organization is complex, fine, and weak, and the operation scale and force are small. Surgeons' limited operation and force-sensing accuracy make it difficult to perform precise and stable drug injection operations on the retina in a magnified field of image vision. In this paper, a 3-DOF automatic drug injection mechanism was designed for microscopic image guiding robot-assisted needle delivery and automatic drug injection. Additionally, the robot-assisted real-time three-dimensional micro-force-sensing method for retinal vein injection was proposed. Based on the layout of three FBG sensors on the hollow outer wall of the nested needle tube in a circular array of nickel-titanium alloys, the real-time sensing of the contact force between the intraoperative instrument and the blood vessel was realized. The experimental data of 15 groups of porcine eyeball retinal veins with diameters of 100-200 µm showed that the piercing force of surgical instruments and blood vessels is 5.95∼12.97 mN, with an average value of 9.98 mN. Furthermore, 20 groups of experimental measurements on chicken embryo blood vessels with diameters of 150-500 µm showed that the piercing force was 4.02∼23.4 mN, with an average value of 12.05 mN.

Space Squeeze Reasoning and Low-Rank Bilinear Feature Fusion for Surgical Image Segmentation.

Surgical image segmentation is critical for surgical robot control and computer-assisted surgery. In the surgical scene, the local features of objects are highly similar, and the illumination interference is strong, which makes surgical image segmentation challenging. To address the above issues, a bilinear squeeze reasoning network is proposed for surgical image segmentation. In it, the space squeeze reasoning module is proposed, which adopts height pooling and width pooling to squeeze global contexts in the vertical and horizontal directions, respectively. The similarity between each horizontal position and each vertical position is calculated to encode long-range semantic dependencies and establish the affinity matrix. The feature maps are also squeezed from both the vertical and horizontal directions to model channel relations. Guided by channel relations, the affinity matrix is expanded to the same size as the input features. It captures long-range semantic dependencies from different directions, helping address the local similarity issue. Besides, a low-rank bilinear fusion module is proposed to enhance the model's ability to recognize similar features. This module is based on the low-rank bilinear model to capture the inter-layer feature relations. It integrates the location details from low-level features and semantic information from high-level features. Various semantics can be represented more accurately, which effectively improves feature representation. The proposed network achieves state-of-the-art performance on cataract image segmentation dataset CataSeg and robotic image segmentation dataset EndoVis 2018.

A Multilayer and Multimodal-Fusion Architecture for Simultaneous Recognition of Endovascular Manipulations and Assessment of Technical Skills.

The clinical success of the percutaneous coronary intervention (PCI) is highly dependent on endovascular manipulation skills and dexterous manipulation strategies of interventionalists. However, the analysis of endovascular manipulations and related discussion for technical skill assessment are limited. In this study, a multilayer and multimodal-fusion architecture is proposed to recognize six typical endovascular manipulations. The synchronously acquired multimodal motion signals from ten subjects are used as the inputs of the architecture independently. Six classification-based and two rule-based fusion algorithms are evaluated for performance comparisons. The recognition metrics under the determined architecture are further used to assess technical skills. The experimental results indicate that the proposed architecture can achieve the overall accuracy of 96.41%, much higher than that of a single-layer recognition architecture (92.85%). In addition, the multimodal fusion brings significant performance improvement in comparison with single-modal schemes. Furthermore, the K -means-based skill assessment can obtain an accuracy of 95% to cluster the attempts made by different skill-level groups. These hopeful results indicate the great possibility of the architecture to facilitate clinical skill assessment and skill learning.

A Unified Framework for Multi-Guidewire Endpoint Localization in Fluoroscopy Images.

OBJECTIVE: In this paper, Keypoint Localization Region-based CNN (KL R-CNN) is proposed, which can simultaneously accomplish the guidewire detection and endpoint localization in a unified model. METHODS: KL R-CNN modifies Mask R-CNN by replacing the mask branch with a novel keypoint localization branch. Besides, some settings of Mask R-CNN are also modified to generate the keypoint localization results at a higher detail level. At the same time, based on the existing metrics of Average Precision (AP) and Percentage of Correct Keypoints (PCK), a new metric named APPCK is proposed to evaluate the overall performance on the multi-guidewire endpoint localization task. Compared with existing metrics, APPCK is easy to use and its results are more intuitive. RESULTS: Compared with existing methods, KL R-CNN has better performance when the threshold is loose, reaching a mean APPCK of 90.65% when the threshold is 9 pixels. CONCLUSION: KL R-CNN achieves the state-of-the-art performance on the multi-guidewire endpoint localization task and has application potentials. SIGNIFICANCE: KL R-CNN can achieve the localization of guidewire endpoints in fluoroscopy images, which is a prerequisite for computer-assisted percutaneous coronary intervention. KL R-CNN can also be extended to other multi-instrument localization tasks.

SurgiNet: Pyramid Attention Aggregation and Class-wise Self-Distillation for Surgical Instrument Segmentation.

Surgical instrument segmentation plays a promising role in robot-assisted surgery. However, illumination issues often appear in surgical scenes, altering the color and texture of surgical instruments. Changes in visual features make surgical instrument segmentation difficult. To address illumination issues, the SurgiNet is proposed to learn pyramid attention features. The double attention module is designed to capture the semantic dependencies between locations and channels. Based on semantic dependencies, the semantic features in the disturbed area can be inferred for addressing illumination issues. Pyramid attention is aggregated to capture multi-scale features and make predictions more accurate. To perform model compression, class-wise self-distillation is proposed to enhance the representation learning of the network, which performs feature distillation within the class to eliminate interference from other classes. Top-down and multi-stage knowledge distillation is designed to distill class probability maps. By inter-layer supervision, high-level probability maps are applied to calibrate the probability distribution of low-level probability maps. Since class-wise distillation enhances the self-learning of the network, the network can get excellent performance with a lightweight backbone. The proposed network achieves the state-of-the-art performance of 89.14% mIoU on CataIS with only 1.66 GFlops and 2.05 M parameters. It also takes first place on EndoVis 2017 with 66.30% mIoU.

Real-Time Multi-Guidewire Endpoint Localization in Fluoroscopy Images.

The real-time localization of the guidewire endpoints is a stepping stone to computer-assisted percutaneous coronary intervention (PCI). However, methods for multi-guidewire endpoint localization in fluoroscopy images are still scarce. In this paper, we introduce a framework for real-time multi-guidewire endpoint localization in fluoroscopy images. The framework consists of two stages, first detecting all guidewire instances in the fluoroscopy image, and then locating the endpoints of each single guidewire instance. In the first stage, a YOLOv3 detector is used for guidewire detection, and a post-processing algorithm is proposed to refine the guidewire detection results. In the second stage, a Segmentation Attention-hourglass (SA-hourglass) network is proposed to predict the endpoint locations of each single guidewire instance. The SA-hourglass network can be generalized to the keypoint localization of other surgical instruments. In our experiments, the SA-hourglass network is applied not only on a guidewire dataset but also on a retinal microsurgery dataset, reaching the mean pixel error (MPE) of 2.20 pixels on the guidewire dataset and the MPE of 5.30 pixels on the retinal microsurgery dataset, both achieving the state-of-the-art localization results. Besides, the inference rate of our framework is at least 20FPS, which meets the real-time requirement of fluoroscopy images (6-12FPS).

Comparative validation of multi-instance instrument segmentation in endoscopy: Results of the ROBUST-MIS 2019 challenge.

Intraoperative tracking of laparoscopic instruments is often a prerequisite for computer and robotic-assisted interventions. While numerous methods for detecting, segmenting and tracking of medical instruments based on endoscopic video images have been proposed in the literature, key limitations remain to be addressed: Firstly, robustness, that is, the reliable performance of state-of-the-art methods when run on challenging images (e.g. in the presence of blood, smoke or motion artifacts). Secondly, generalization; algorithms trained for a specific intervention in a specific hospital should generalize to other interventions or institutions. In an effort to promote solutions for these limitations, we organized the Robust Medical Instrument Segmentation (ROBUST-MIS) challenge as an international benchmarking competition with a specific focus on the robustness and generalization capabilities of algorithms. For the first time in the field of endoscopic image processing, our challenge included a task on binary segmentation and also addressed multi-instance detection and segmentation. The challenge was based on a surgical data set comprising 10,040 annotated images acquired from a total of 30 surgical procedures from three different types of surgery. The validation of the competing methods for the three tasks (binary segmentation, multi-instance detection and multi-instance segmentation) was performed in three different stages with an increasing domain gap between the training and the test data. The results confirm the initial hypothesis, namely that algorithm performance degrades with an increasing domain gap. While the average detection and segmentation quality of the best-performing algorithms is high, future research should concentrate on detection and segmentation of small, crossing, moving and transparent instrument(s) (parts).

Optical coherence tomography-based deep learning algorithm for quantification of the location of the intraocular lens.

BACKGROUND: Cataract surgery has been recently developed from sight rehabilitating surgery to accurate refractive surgery. The precise concentration of intraocular lens (IOL) is crucial for postoperative high visual quanlity. The three-dimentional (3D) images of ocular anterior segment captured by optial coherence tomography (OCT) make it possible to evaluate the IOL position in 3D space, which provide insights into factors relavant to the visual quanlity and better design of new functional IOL. The deep learning algorithm potentially quantify the IOL position in an objective and efficient way. METHODS: The region-based fully convolutional network (R-FCN) was used to recogonize and delineate the IOL configuration in 3D OCT images. Scleral spur was identified automatically. Then the tilt angle of the IOL relative to the scleral spur plane along with its decentration with respect to the pupil were calculated. Repeatability and reliability of the method was evaluated by the intraclass correlation coefficient. RESULTS: After improvement, the R-FCN network recognition efficiency of IOL configuration reached 0.910. The ICC of reliability and repeatability of the method is 0.867 and 0.901. The average tilt angle of the IOL relative to scleral spur is located in 1.65±1.00 degrees. The offsets dx and dy occurring in the early X and Y directions of the IOL are 0.29±0.22 and 0.33±0.24 mm, respectively. The IOL offset distance is 0.44±0.33 mm. CONCLUSIONS: We proposed a practical method to quantify the IOL postion in 3D space based on OCT images and assisted by an algorithm.

Pyramid attention recurrent networks for real-time guidewire segmentation and tracking in intraoperative X-ray fluoroscopy.

In endovascular and cardiovascular surgery, real-time and accurate segmentation and tracking of interventional instruments can aid in reducing radiation exposure, contrast agent and processing time. Nevertheless, this task often comes with the challenges of the elongated deformable structures with low contrast in noisy X-ray fluoroscopy. To address these issues, a novel efficient network architecture, termed pyramid attention recurrent networks (PAR-Net), is proposed for real-time guidewire segmentation and tracking. The proposed PAR-Net contains three major modules, namely pyramid attention module, recurrent residual module and pre-trained MobileNetV2 encoder. Specifically, a hybrid loss function of both reinforced focal loss and dice loss is proposed to better address the issues of class imbalance and misclassified examples. Quantitative and qualitative evaluations on clinical intraoperative images demonstrate that the proposed approach significantly outperforms simpler baselines as well as the best previously published result for this task, achieving the state-of-the-art performance.

An intelligent learning approach for improving ECG signal classification and arrhythmia analysis.

The recognition of cardiac arrhythmia in minimal time is important to prevent sudden and untimely deaths. The proposed work includes a complete framework for analyzing the Electrocardiogram (ECG) signal. The three phases of analysis include 1) the ECG signal quality enhancement through noise suppression by a dedicated filter combination; 2) the feature extraction by a devoted wavelet design and 3) a proposed hidden Markov model (HMM) for cardiac arrhythmia classification into Normal (N), Right Bundle Branch Block (RBBB), Left Bundle Branch Block (LBBB), Premature Ventricular Contraction (PVC) and Atrial Premature Contraction (APC). The main features extracted in the proposed work are minimum, maximum, mean, standard deviation, and median. The experiments were conducted on forty-five ECG records in MIT BIH arrhythmia database and in MIT BIH noise stress test database. The proposed model has an overall accuracy of 99.7 % with a sensitivity of 99.7 % and a positive predictive value of 100 %. The detection error rate for the proposed model is 0.0004. This paper also includes a study of the cardiac arrhythmia recognition using an IoMT (Internet of Medical Things) approach.

Qualitative and Quantitative Assessment of Technical Skills in Percutaneous Coronary Intervention: In Vivo Porcine Studies.

OBJECTIVE: Technical skill assessment plays an important role in the professional development of an interventionalist in percutaneous coronary intervention (PCI). However, most of the traditional assessment methods are time consuming and subjective. This paper aims to develop objective assessment techniques. METHODS: In this study, a natural-behavior-based assessment framework is proposed to qualitatively and quantitatively assess technical skills in PCI. In vivo porcine studies were conducted to deliver a medical guidewire to two target coronaries of left circumflex arteries by six novice and four expert interventionalists. Simultaneously, four types of natural behaviors (i.e., hand motion, proximal force, muscle activity, and finger motion) were acquired from the subjects' dominant hand and arm. The features extracted from the behaviors of different skill-level groups were compared using the Mann-Whitney U-test for effective behavior selection. The effective ones were further applied in the Gaussian-mixture-model-based qualitative assessment and Mahalanobis-distance-based quantitative assessment. RESULTS: The qualitative assessment achieves an accuracy of 92% to distinguish the novice and expert attempts, which is significantly higher than that of using single guidewire motions. Furthermore, the quantitative assessment can assign objective and effective scores for all attempts, indicating high correlation ( R = 0.9225) to those obtained by traditional methods. CONCLUSION: The objective, effective, and comprehensive assessment of technical skills can be provided by qualitatively and quantitatively analyzing interventionalists' natural behaviors in PCI. SIGNIFICANCE: This paper suggests a novel approach for the technical skill assessment and the promising results demonstrate the great importance and effectiveness of the proposed method for promoting the development of objective assessment techniques.

Removal of Artifacts from EEG Signals: A Review.

Electroencephalogram (EEG) plays an important role in identifying brain activity and behavior. However, the recorded electrical activity always be contaminated with artifacts and then affect the analysis of EEG signal. Hence, it is essential to develop methods to effectively detect and extract the clean EEG data during encephalogram recordings. Several methods have been proposed to remove artifacts, but the research on artifact removal continues to be an open problem. This paper tends to review the current artifact removal of various contaminations. We first discuss the characteristics of EEG data and the types of different artifacts. Then, a general overview of the state-of-the-art methods and their detail analysis are presented. Lastly, a comparative analysis is provided for choosing a suitable methods according to particular application.

Analysis of Interventionalists' Natural Behaviors for Recognizing Motion Patterns of Endovascular Tools During Percutaneous Coronary Interventions.

Many robotic platforms can indeed reduce radiation exposure to clinicians during percutaneous coronary intervention (PCI), however, interventionalists' natural manipulations are rarely involved in robot-assisted PCI. This requires more attention to analyze interventionalists' natural behaviors during conventional PCI. In this study, four types of natural behavior (i.e., muscle activity, hand motion, proximal force, and finger motion) were synchronously acquired from ten subjects while performing six typical types of guidewire manipulation. These behaviors are evaluated by a hidden Markov model (HMM) based analysis framework for relevant behavior selection. Relevant behaviors are further used as the input of two HMM-based classification frameworks to recognize guidewire motion patterns. Experimental results show that under the basic classification framework (BCF), 91.01% and 93.32% recognition accuracies can be achieved by using all behaviors and relevant behaviors, respectively. Furthermore, the hierarchical classification framework can significantly enhance the recognition ability of relevant behaviors with an accuracy of 96.39%. These promising results demonstrate great potential of proposed methods for promoting the future design of human-robot interfaces in robot-assisted PCI.

RASNet: Segmentation for Tracking Surgical Instruments in Surgical Videos Using Refined Attention Segmentation Network.

Segmentation for tracking surgical instruments plays an important role in robot-assisted surgery. Segmentation of surgical instruments contributes to capturing accurate spatial information for tracking. In this paper, a novel network, Refined Attention Segmentation Network, is proposed to simultaneously segment surgical instruments and identify their categories. The U-shape network which is popular in segmentation is used. Different from previous work, an attention module is adopted to help the network focus on key regions, which can improve the segmentation accuracy. To solve the class imbalance problem, the weighted sum of the cross entropy loss and the logarithm of the Jaccard index is used as loss function. Furthermore, transfer learning is adopted in our network. The encoder is pre-trained on ImageNet. The dataset from the MICCAI EndoVis Challenge 2017 is used to evaluate our network. Based on this dataset, our network achieves state-of-the-art performance 94.65% mean Dice and 90.33% mean IOU.