Fast instruments and tissues segmentation of micro-neurosurgical scene using high correlative non-local network.

Surgical scene segmentation provides critical information for guidance in micro-neurosurgery. Segmentation of instruments and critical tissues contributes further to robot assisted surgery and surgical evaluation. However, due to the lack of relevant scene segmentation dataset, scale variation and local similarity, micro-neurosurgical segmentation faces many challenges. To address these issues, a high correlative non-local network (HCNNet), is proposed to aggregate multi-scale feature by optimized non-local mechanism. HCNNet adopts two-branch design to generate features of different scale efficiently, while the two branches share common weights in shallow layers. Several short-term dense concatenate (STDC) modules are combined as the backbone to capture both semantic and spatial information. Besides, a high correlative non-local module (HCNM) is designed to guide the upsampling process of the high-level feature by modeling global context generated from the low-level feature. It filters out confused pixels of different classes in the non-local correlation map. Meanwhile, a large segmentation dataset named NeuroSeg is constructed, which contains 15 types of instruments and 3 types of tissues that appear in meningioma resection surgery. The proposed HCNNet achieves the state-of-the-art performance on NeuroSeg, it reaches an inference speed of 54.85 FPS with the highest accuracy of 59.62% mIoU, 74.7% Dice, 70.55% mAcc and 87.12% aAcc.

Motion Decoupling Network for Intra-Operative Motion Estimation Under Occlusion.

In recent intelligent-robot-assisted surgery studies, an urgent issue is how to detect the motion of instruments and soft tissue accurately from intra-operative images. Although optical flow technology from computer vision is a powerful solution to the motion-tracking problem, it has difficulty obtaining the pixel-wise optical flow ground truth of real surgery videos for supervised learning. Thus, unsupervised learning methods are critical. However, current unsupervised methods face the challenge of heavy occlusion in the surgical scene. This paper proposes a novel unsupervised learning framework to estimate the motion from surgical images under occlusion. The framework consists of a Motion Decoupling Network to estimate the tissue and the instrument motion with different constraints. Notably, the network integrates a segmentation subnet that estimates the segmentation map of instruments in an unsupervised manner to obtain the occlusion region and improve the dual motion estimation. Additionally, a hybrid self-supervised strategy with occlusion completion is introduced to recover realistic vision clues. Extensive experiments on two surgical datasets show that the proposed method achieves accurate motion estimation for intra-operative scenes and outperforms other unsupervised methods, with a margin of 15% in accuracy. The average estimation error for tissue is less than 2.2 pixels on average for both surgical datasets.

SurgiNet: Pyramid Attention Aggregation and Class-wise Self-Distillation for Surgical Instrument Segmentation.

Surgical instrument segmentation plays a promising role in robot-assisted surgery. However, illumination issues often appear in surgical scenes, altering the color and texture of surgical instruments. Changes in visual features make surgical instrument segmentation difficult. To address illumination issues, the SurgiNet is proposed to learn pyramid attention features. The double attention module is designed to capture the semantic dependencies between locations and channels. Based on semantic dependencies, the semantic features in the disturbed area can be inferred for addressing illumination issues. Pyramid attention is aggregated to capture multi-scale features and make predictions more accurate. To perform model compression, class-wise self-distillation is proposed to enhance the representation learning of the network, which performs feature distillation within the class to eliminate interference from other classes. Top-down and multi-stage knowledge distillation is designed to distill class probability maps. By inter-layer supervision, high-level probability maps are applied to calibrate the probability distribution of low-level probability maps. Since class-wise distillation enhances the self-learning of the network, the network can get excellent performance with a lightweight backbone. The proposed network achieves the state-of-the-art performance of 89.14% mIoU on CataIS with only 1.66 GFlops and 2.05 M parameters. It also takes first place on EndoVis 2017 with 66.30% mIoU.