Motion Decoupling Network for Intra-Operative Motion Estimation Under Occlusion.

In recent intelligent-robot-assisted surgery studies, an urgent issue is how to detect the motion of instruments and soft tissue accurately from intra-operative images. Although optical flow technology from computer vision is a powerful solution to the motion-tracking problem, it has difficulty obtaining the pixel-wise optical flow ground truth of real surgery videos for supervised learning. Thus, unsupervised learning methods are critical. However, current unsupervised methods face the challenge of heavy occlusion in the surgical scene. This paper proposes a novel unsupervised learning framework to estimate the motion from surgical images under occlusion. The framework consists of a Motion Decoupling Network to estimate the tissue and the instrument motion with different constraints. Notably, the network integrates a segmentation subnet that estimates the segmentation map of instruments in an unsupervised manner to obtain the occlusion region and improve the dual motion estimation. Additionally, a hybrid self-supervised strategy with occlusion completion is introduced to recover realistic vision clues. Extensive experiments on two surgical datasets show that the proposed method achieves accurate motion estimation for intra-operative scenes and outperforms other unsupervised methods, with a margin of 15% in accuracy. The average estimation error for tissue is less than 2.2 pixels on average for both surgical datasets.

Semi-Supervised Medical Image Segmentation Guided by Bi-Directional Constrained Dual-Task Consistency.

BACKGROUND: Medical image processing tasks represented by multi-object segmentation are of great significance for surgical planning, robot-assisted surgery, and surgical safety. However, the exceptionally low contrast among tissues and limited available annotated data makes developing an automatic segmentation algorithm for pelvic CT challenging. METHODS: A bi-direction constrained dual-task consistency model named PICT is proposed to improve segmentation quality by leveraging free unlabeled data. First, to learn more unmarked data features, it encourages the model prediction of the interpolated image to be consistent with the interpolation of the model prediction at the pixel, model, and data levels. Moreover, to constrain the error prediction of interpolation interference, PICT designs an auxiliary pseudo-supervision task that focuses on the underlying information of non-interpolation data. Finally, an effective loss algorithm for both consistency tasks is designed to ensure the complementary manner and produce more reliable predictions. RESULTS: Quantitative experiments show that the proposed PICT achieves 87.18%, 96.42%, and 79.41% mean DSC score on ACDC, CTPelvic1k, and the individual Multi-tissue Pelvis dataset with gains of around 0.8%, 0.5%, and 1% compared to the state-of-the-art semi-supervised method. Compared to the baseline supervised method, the PICT brings over 3-9% improvements. CONCLUSIONS: The developed PICT model can effectively leverage unlabeled data to improve segmentation quality of low contrast medical images. The segmentation result could improve the precision of surgical path planning and provide input for robot-assisted surgery.

Fast instruments and tissues segmentation of micro-neurosurgical scene using high correlative non-local network.

Surgical scene segmentation provides critical information for guidance in micro-neurosurgery. Segmentation of instruments and critical tissues contributes further to robot assisted surgery and surgical evaluation. However, due to the lack of relevant scene segmentation dataset, scale variation and local similarity, micro-neurosurgical segmentation faces many challenges. To address these issues, a high correlative non-local network (HCNNet), is proposed to aggregate multi-scale feature by optimized non-local mechanism. HCNNet adopts two-branch design to generate features of different scale efficiently, while the two branches share common weights in shallow layers. Several short-term dense concatenate (STDC) modules are combined as the backbone to capture both semantic and spatial information. Besides, a high correlative non-local module (HCNM) is designed to guide the upsampling process of the high-level feature by modeling global context generated from the low-level feature. It filters out confused pixels of different classes in the non-local correlation map. Meanwhile, a large segmentation dataset named NeuroSeg is constructed, which contains 15 types of instruments and 3 types of tissues that appear in meningioma resection surgery. The proposed HCNNet achieves the state-of-the-art performance on NeuroSeg, it reaches an inference speed of 54.85 FPS with the highest accuracy of 59.62% mIoU, 74.7% Dice, 70.55% mAcc and 87.12% aAcc.

Autonomous path planning for robot-assisted pelvic fracture closed reduction with collision avoidance.

BACKGROUND: Robot-assisted pelvic fracture closed reduction (RPFCR) positively contributes to patient treatment. However, the current path planning suffers from incomplete obstacle avoidance and long paths. METHOD: A collision detection method is proposed for applications in the pelvic environment to improve the safety of RPFCR surgery. Meanwhile, a defined orientation planning strategy (OPS) and linear sampling search (LSS) are coupled into the A* algorithm to optimise the reduction path. Subsequently, pelvic in vitro experimental platform is built to verify the augmented A*algorithm's feasibility. RESULTS: The augmented A* algorithm planned the shortest path for the same fracture model, and the paths planned by the A* algorithm and experience-based increased by 56.12% and 89.02%, respectively. CONCLUSIONS: The augmented A* algorithm effectively improves surgical safety and shortens the path length, which can be adopted as an effective model for developing RPFCR path planning.

An automatic drug injection device with spatial micro-force perception guided by an microscopic image for robot-assisted ophthalmic surgery.

Retinal vein injection guided by microscopic image is an innovative procedure for treating retinal vein occlusion. However, the retina organization is complex, fine, and weak, and the operation scale and force are small. Surgeons' limited operation and force-sensing accuracy make it difficult to perform precise and stable drug injection operations on the retina in a magnified field of image vision. In this paper, a 3-DOF automatic drug injection mechanism was designed for microscopic image guiding robot-assisted needle delivery and automatic drug injection. Additionally, the robot-assisted real-time three-dimensional micro-force-sensing method for retinal vein injection was proposed. Based on the layout of three FBG sensors on the hollow outer wall of the nested needle tube in a circular array of nickel-titanium alloys, the real-time sensing of the contact force between the intraoperative instrument and the blood vessel was realized. The experimental data of 15 groups of porcine eyeball retinal veins with diameters of 100-200 µm showed that the piercing force of surgical instruments and blood vessels is 5.95∼12.97 mN, with an average value of 9.98 mN. Furthermore, 20 groups of experimental measurements on chicken embryo blood vessels with diameters of 150-500 µm showed that the piercing force was 4.02∼23.4 mN, with an average value of 12.05 mN.

Space Squeeze Reasoning and Low-Rank Bilinear Feature Fusion for Surgical Image Segmentation.

Surgical image segmentation is critical for surgical robot control and computer-assisted surgery. In the surgical scene, the local features of objects are highly similar, and the illumination interference is strong, which makes surgical image segmentation challenging. To address the above issues, a bilinear squeeze reasoning network is proposed for surgical image segmentation. In it, the space squeeze reasoning module is proposed, which adopts height pooling and width pooling to squeeze global contexts in the vertical and horizontal directions, respectively. The similarity between each horizontal position and each vertical position is calculated to encode long-range semantic dependencies and establish the affinity matrix. The feature maps are also squeezed from both the vertical and horizontal directions to model channel relations. Guided by channel relations, the affinity matrix is expanded to the same size as the input features. It captures long-range semantic dependencies from different directions, helping address the local similarity issue. Besides, a low-rank bilinear fusion module is proposed to enhance the model's ability to recognize similar features. This module is based on the low-rank bilinear model to capture the inter-layer feature relations. It integrates the location details from low-level features and semantic information from high-level features. Various semantics can be represented more accurately, which effectively improves feature representation. The proposed network achieves state-of-the-art performance on cataract image segmentation dataset CataSeg and robotic image segmentation dataset EndoVis 2018.

SurgiNet: Pyramid Attention Aggregation and Class-wise Self-Distillation for Surgical Instrument Segmentation.

Surgical instrument segmentation plays a promising role in robot-assisted surgery. However, illumination issues often appear in surgical scenes, altering the color and texture of surgical instruments. Changes in visual features make surgical instrument segmentation difficult. To address illumination issues, the SurgiNet is proposed to learn pyramid attention features. The double attention module is designed to capture the semantic dependencies between locations and channels. Based on semantic dependencies, the semantic features in the disturbed area can be inferred for addressing illumination issues. Pyramid attention is aggregated to capture multi-scale features and make predictions more accurate. To perform model compression, class-wise self-distillation is proposed to enhance the representation learning of the network, which performs feature distillation within the class to eliminate interference from other classes. Top-down and multi-stage knowledge distillation is designed to distill class probability maps. By inter-layer supervision, high-level probability maps are applied to calibrate the probability distribution of low-level probability maps. Since class-wise distillation enhances the self-learning of the network, the network can get excellent performance with a lightweight backbone. The proposed network achieves the state-of-the-art performance of 89.14% mIoU on CataIS with only 1.66 GFlops and 2.05 M parameters. It also takes first place on EndoVis 2017 with 66.30% mIoU.

Comparative validation of multi-instance instrument segmentation in endoscopy: Results of the ROBUST-MIS 2019 challenge.

Intraoperative tracking of laparoscopic instruments is often a prerequisite for computer and robotic-assisted interventions. While numerous methods for detecting, segmenting and tracking of medical instruments based on endoscopic video images have been proposed in the literature, key limitations remain to be addressed: Firstly, robustness, that is, the reliable performance of state-of-the-art methods when run on challenging images (e.g. in the presence of blood, smoke or motion artifacts). Secondly, generalization; algorithms trained for a specific intervention in a specific hospital should generalize to other interventions or institutions. In an effort to promote solutions for these limitations, we organized the Robust Medical Instrument Segmentation (ROBUST-MIS) challenge as an international benchmarking competition with a specific focus on the robustness and generalization capabilities of algorithms. For the first time in the field of endoscopic image processing, our challenge included a task on binary segmentation and also addressed multi-instance detection and segmentation. The challenge was based on a surgical data set comprising 10,040 annotated images acquired from a total of 30 surgical procedures from three different types of surgery. The validation of the competing methods for the three tasks (binary segmentation, multi-instance detection and multi-instance segmentation) was performed in three different stages with an increasing domain gap between the training and the test data. The results confirm the initial hypothesis, namely that algorithm performance degrades with an increasing domain gap. While the average detection and segmentation quality of the best-performing algorithms is high, future research should concentrate on detection and segmentation of small, crossing, moving and transparent instrument(s) (parts).

Optical coherence tomography-based deep learning algorithm for quantification of the location of the intraocular lens.

BACKGROUND: Cataract surgery has been recently developed from sight rehabilitating surgery to accurate refractive surgery. The precise concentration of intraocular lens (IOL) is crucial for postoperative high visual quanlity. The three-dimentional (3D) images of ocular anterior segment captured by optial coherence tomography (OCT) make it possible to evaluate the IOL position in 3D space, which provide insights into factors relavant to the visual quanlity and better design of new functional IOL. The deep learning algorithm potentially quantify the IOL position in an objective and efficient way. METHODS: The region-based fully convolutional network (R-FCN) was used to recogonize and delineate the IOL configuration in 3D OCT images. Scleral spur was identified automatically. Then the tilt angle of the IOL relative to the scleral spur plane along with its decentration with respect to the pupil were calculated. Repeatability and reliability of the method was evaluated by the intraclass correlation coefficient. RESULTS: After improvement, the R-FCN network recognition efficiency of IOL configuration reached 0.910. The ICC of reliability and repeatability of the method is 0.867 and 0.901. The average tilt angle of the IOL relative to scleral spur is located in 1.65±1.00 degrees. The offsets dx and dy occurring in the early X and Y directions of the IOL are 0.29±0.22 and 0.33±0.24 mm, respectively. The IOL offset distance is 0.44±0.33 mm. CONCLUSIONS: We proposed a practical method to quantify the IOL postion in 3D space based on OCT images and assisted by an algorithm.

Pyramid attention recurrent networks for real-time guidewire segmentation and tracking in intraoperative X-ray fluoroscopy.

In endovascular and cardiovascular surgery, real-time and accurate segmentation and tracking of interventional instruments can aid in reducing radiation exposure, contrast agent and processing time. Nevertheless, this task often comes with the challenges of the elongated deformable structures with low contrast in noisy X-ray fluoroscopy. To address these issues, a novel efficient network architecture, termed pyramid attention recurrent networks (PAR-Net), is proposed for real-time guidewire segmentation and tracking. The proposed PAR-Net contains three major modules, namely pyramid attention module, recurrent residual module and pre-trained MobileNetV2 encoder. Specifically, a hybrid loss function of both reinforced focal loss and dice loss is proposed to better address the issues of class imbalance and misclassified examples. Quantitative and qualitative evaluations on clinical intraoperative images demonstrate that the proposed approach significantly outperforms simpler baselines as well as the best previously published result for this task, achieving the state-of-the-art performance.

RASNet: Segmentation for Tracking Surgical Instruments in Surgical Videos Using Refined Attention Segmentation Network.

Segmentation for tracking surgical instruments plays an important role in robot-assisted surgery. Segmentation of surgical instruments contributes to capturing accurate spatial information for tracking. In this paper, a novel network, Refined Attention Segmentation Network, is proposed to simultaneously segment surgical instruments and identify their categories. The U-shape network which is popular in segmentation is used. Different from previous work, an attention module is adopted to help the network focus on key regions, which can improve the segmentation accuracy. To solve the class imbalance problem, the weighted sum of the cross entropy loss and the logarithm of the Jaccard index is used as loss function. Furthermore, transfer learning is adopted in our network. The encoder is pre-trained on ImageNet. The dataset from the MICCAI EndoVis Challenge 2017 is used to evaluate our network. Based on this dataset, our network achieves state-of-the-art performance 94.65% mean Dice and 90.33% mean IOU.

Preliminary study for motion scaling based control in minimally invasive vascular interventional robot.

Robot-assisted vascular interventions present promising trend for reducing the X-ray radiation to the surgeon during the operation. However, the control methods in the current vascular interventional robots only repeat the manipulation of the surgeon. While under certain circumstances, it is necessary to scale the manipulation of the surgeon to obtain a higher precision or a shorter manipulation time. A novel control method based on motion scaling for vascular interventional robot is proposed in this paper. The main idea of the method is to change the motion speed ratios between the master and the slave side. The motion scaling based control method is implemented in the vascular interventional robot we've developed before, so the operator can deliver the interventional devices under different motion scaling factors. Experiment studies verify the effectiveness of the motion scaling based control.

A 3D virtual reality simulator for training of minimally invasive surgery.

For the last decade, remarkable progress has been made in the field of cardiovascular disease treatment. However, these complex medical procedures require a combination of rich experience and technical skills. In this paper, a 3D virtual reality simulator for core skills training in minimally invasive surgery is presented. The system can generate realistic 3D vascular models segmented from patient datasets, including a beating heart, and provide a real-time computation of force and force feedback module for surgical simulation. Instruments, such as a catheter or guide wire, are represented by a multi-body mass-spring model. In addition, a realistic user interface with multiple windows and real-time 3D views are developed. Moreover, the simulator is also provided with a human-machine interaction module that gives doctors the sense of touch during the surgery training, enables them to control the motion of a virtual catheter/guide wire inside a complex vascular model. Experimental results show that the simulator is suitable for minimally invasive surgery training.

Development and clinical evaluation of medical robot assisted photodynamic therapy of port wine stains.

BACKGROUND: Port wine stains (PWS) are a kind of skin disease for which photodynamic therapy (PDT) has already achieved good results. With manual operation of clinical PDT, the laser density is uneven and laser irradiation of the lesion is arbitrary and non-uniform. In addition, lengthy manual operation tires doctors; thus a robot system has been developed to assist them. METHODS: First, a novel medical manipulator consisting of five passive joints (robot arm) and two active joints (robot wrist) was developed to automatically improve the uniformity of laser irradiation. Second, image processing of the lesion was introduced. Third, kinematics and path planning of the robot were analysed, and safety precautions were introduced. Then, accuracy tests of the robot wrist and robot system were conducted separately before clinical application. Finally, a total of 50 PWS cases were treated using the robot system. The clinical outcomes and comparison of non-parametric values were employed to evaluate the robot system. RESULTS: The accuracies of the robot wrist and robot system were shown to meet the requirements of clinical PDT treatment. The robot system performed successfully in 50 PWS cases. Doctors can devote more energy to clinical judgments during treatment with the assistance of the robot system. All the PWS have shown different degrees of improvement. The results show that the robot system is useful in assisting doctors for the PDT treatment of PWS. CONCLUSIONS: The experiments show the feasibility and usefulness of the robot system in assisting doctors giving PDT treatment for PWS. The robot system can lighten the load on doctors and improve the therapeutic effect.