An Image Information-Based Objective Assessment Method of Technical Manipulation Skills for Intravascular Interventions.

The clinical success of vascular interventional surgery relies heavily on a surgeon's catheter/guidewire manipulation skills and strategies. An objective and accurate assessment method plays a critical role in evaluating the surgeon's technical manipulation skill level. Most of the existing evaluation methods incorporate the use of information technology to find more objective assessment models based on various metrics. However, in these models, sensors are often attached to the surgeon's hands or to interventional devices for data collection, which constrains the surgeon's operational movements or exerts an influence on the motion trajectory of interventional devices. In this paper, an image information-based assessment method is proposed for the evaluation of the surgeon's manipulation skills without the requirement of attaching sensors to the surgeon or catheters/guidewires. Surgeons are allowed to use their natural bedside manipulation skills during the data collection process. Their manipulation features during different catheterization tasks are derived from the motion analysis of the catheter/guidewire in video sequences. Notably, data relating to the number of speed peaks, slope variations, and the number of collisions are included in the assessment. Furthermore, the contact forces, resulting from interactions between the catheter/guidewire and the vascular model, are sensed by a 6-DoF F/T sensor. A support vector machine (SVM) classification framework is developed to discriminate the surgeon's catheterization skill levels. The experimental results demonstrate that the proposed SVM-based assessment method can obtain an accuracy of 97.02% to distinguish between the expert and novice manipulations, which is higher than that of other existing research achievements. The proposed method has great potential to facilitate skill assessment and training of novice surgeons in vascular interventional surgery.

Low-Density sEMG-Based Pattern Recognition of Unrelated Movements Rejection for Wrist Joint Rehabilitation.

sEMG-based pattern recognition commonly assumes a limited number of target categories, and the classifiers often predict each target category depending on probability. In wrist rehabilitation training, the patients may make movements that do not belong to the target category unconsciously. However, most pattern recognition methods can only identify limited patterns and are prone to be disturbed by abnormal movement, especially for wrist joint movements. To address the above the problem, a sEMG-based rejection method for unrelated movements is proposed to identify wrist joint unrelated movements using center loss. In this paper, the sEMG signal collected by the Myo armband is used as the input of the sEMG control method. First, the sEMG signal is processed by sliding signal window and image coding. Then, the CNN with center loss and softmax loss is used to describe the spatial information from the sEMG image to extract discriminative features and target movement recognition. Finally, the deep spatial information is used to train the AE to reject unrelated movements based on the reconstruction loss. The results show that the proposed method can realize the target movements recognition and reject unrelated movements with an F-score of 93.4% and a rejection accuracy of 95% when the recall is 0.9, which reveals the effectiveness of the proposed method.

A novel catheter interaction simulating method for virtual reality interventional training systems.

Endovascular robotic systems have been applied in robot-assisted interventional surgery to improve surgical safety and reduce radiation to surgeons. However, this surgery requires surgeons to be highly skilled at operating vascular interventional surgical robot. Virtual reality (VR) interventional training systems for robot-assisted interventional surgical training have many advantages over traditional training methods. For virtual interventional radiology, simulation of the behaviors of surgical tools (here mainly refers to catheter and guidewire) is a challenging work. In this paper, we developed a novel virtual reality interventional training system. This system is an extension of the endovascular robotic system. Because the master side of this system can be used for both the endovascular robotic system and the VR interventional training system, the proposed system improves training and reduces the cost of education. Moreover, we proposed a novel method to solve catheterization modeling during the interventional simulation. Our method discretizes the catheter by the collision points. The catheter between two adjacent collision points is treated as thin torsion-free elastic rods. The deformation of the rod is mainly affected by the force applied at the collision points. Meanwhile, the virtual contact force is determined by the collision points. This simplification makes the model more stable and reduces the computational complexity, and the behavior of the surgical tools can be approximated. Therefore, we realized the catheter interaction simulation and virtual force feedback for the proposed VR interventional training system. The performance of our method is experimentally validated.

A Distinguishable Pseudo-Feature Synthesis Method for Generalized Zero-Shot Learning.

Generalized zero-shot learning (GZSL) aims to classify seen classes and unseen classes that are disjoint simultaneously. Hybrid approaches based on pseudo-feature synthesis are currently the most popular among GZSL methods. However, they suffer from problems of negative transfer and low-quality class discriminability, causing poor classification accuracy. To address them, we propose a novel GZSL method of distinguishable pseudo-feature synthesis (DPFS). The DPFS model can provide high-quality distinguishable characteristics for both seen and unseen classes. Firstly, the model is pretrained by a distance prediction loss to avoid overfitting. Then, the model only selects attributes of similar seen classes and makes sparse representations based on attributes for unseen classes, thereby overcoming negative transfer. After the model synthesizes pseudo-features for unseen classes, it disposes of the pseudo-feature outliers to improve the class discriminability. The pseudo-features are fed into a classifier of the model together with features of seen classes for GZSL classification. Experimental results on four benchmark datasets verify that the proposed DPFS has GZSL classification performance better than that in existing methods.

Dynamic Analysis and Path Planning of a Turtle-Inspired Amphibious Spherical Robot.

A dynamic path-planning algorithm based on a general constrained optimization problem (GCOP) model and a sequential quadratic programming (SQP) method with sensor input is proposed in this paper. In an unknown underwater space, the turtle-inspired amphibious spherical robot (ASR) can realise the path-planning control movement and achieve collision avoidance. Due to the special underwater environments, thrusters and diamond parallel legs (DPLs) are installed in the lower hemisphere to realise accurate motion control. A propulsion model for a novel water-jet thruster based on experimental analysis and a modified Denavit-Hartenberg (MDH) algorithm are developed for multiple degrees of freedom (MDOF) to realize high-precision and high-speed motion control. Simulations and experiments verify that the effectiveness of the GCOP and SQP algorithms can realize reasonable path planning and make it possible to improve the flexibility of underwater movement with a small estimation error.

A Deep-Learning-Based Guidewire Compliant Control Method for the Endovascular Surgery Robot.

Endovascular surgery is a high-risk operation with limited vision and intractable guidewires. At present, endovascular surgery robot (ESR) systems based on force feedback liberates surgeons' operation skills, but it lacks the ability to combine force perception with vision. In this study, a deep learning-based guidewire-compliant control method (GCCM) is proposed, which guides the robot to avoid surgical risks and improve the efficiency of guidewire operation. First, a deep learning-based model called GCCM-net is built to identify whether the guidewire tip collides with the vascular wall in real time. The experimental results in a vascular phantom show that the best accuracy of GCCM-net is 94.86 ± 0.31%. Second, a real-time operational risk classification method named GCCM-strategy is proposed. When the surgical risks occur, the GCCM-strategy uses the result of GCCM-net as damping and decreases the robot's running speed through virtual resistance. Compared with force sensors, the robot with GCCM-strategy can alleviate the problem of force position asynchrony caused by the long and soft guidewires in real-time. Experiments run by five guidewire operators show that the GCCM-strategy can reduce the average operating force by 44.0% and shorten the average operating time by 24.6%; therefore the combination of vision and force based on deep learning plays a positive role in improving the operation efficiency in ESR.

Kinetics Analysis and ADRC-Based Controller for a String-Driven Vascular Intervention Surgical Robotic System.

Vascular interventional surgery is a typical method for diagnosing and treating cardio-cerebrovascular diseases. However, a surgeon is exposed to significant X-radiation exposure when the operation is conducted for a long period of time. A vascular intervention surgical robotic system for assisting the surgeon is a promising approach to address the aforementioned issue. When developing the robotic system, a high displacement accuracy is crucial, and this can aid in enhancing operating efficiency and safety. In this study, a novel kinetics analysis and active disturbance rejection control (ADRC)-based controller is proposed to provide high accuracy for a string-driven robotic system. In this controller, kinetics analysis is initially used to improve the accuracy affected by the inner factors of the slave manipulator. Then, the ADRC controller is used to further improve the operating accuracy of the robotic system. Finally, the proposed controller is evaluated by conducting experiments on a vascular model. The results indicate maximum steady errors of 0.45 mm and 6.67°. The experimental results demonstrate that the proposed controller can satisfy the safety requirements of the string-driven robotic system.

A Home-based Tele-rehabilitation System With Enhanced Therapist-patient Remote Interaction: A Feasibility Study.

As a promising alternative to hospital-based manual therapy, robot-assisted tele-rehabilitation therapy has shown significant benefits in reducing the therapist's workload and accelerating the patient's recovery process. However, existing telerobotic systems for rehabilitation face barriers to implementing appropriate therapy treatment due to the lack of effective therapist-patient interactive capabilities. In this paper, we develop a home-based tele-rehabilitation system that implements two alternative training methods, including a haptic-enabled guided training that allows the therapist to adjust the intensity of therapeutic movements provided by the rehabilitation device and a surface electromyography (sEMG)-based supervised training that explores remote assessment of the patient's kinesthetic awareness. Preliminary experiments were conducted to demonstrate the feasibility of the proposed alternative training methods and evaluate the functionality of the developed tele-rehabilitation system. Results showed that the proposed tele-rehabilitation system enabled therapist-in-the-loop to dynamically adjust the rehabilitation intensity and provided more interactivity in therapist-patient remote interaction.

An Endovascular Catheterization Robotic System Using Collaborative Operation with Magnetically Controlled Haptic Force Feedback.

Robot-assisted technology is often used to perform endovascular catheterization surgeries, which generally depend on the flexible operability and the accurate force feedback of a robotic system. In this paper, an endovascular catheterization robotic system (ECRS) was developed to improve collaborative operation and haptic force feedback. A couple of operating handles were designed to maximize the use of the natural operations of surgeons on the master side, which is a flexible and ergonomic device. A magnetically controlled haptic force feedback structure is proposed based on hydrogel and solid magnetorheological (MR) fluid to offer a sense of haptic feedback to operators; this has potential influence on the field of force feedback. In addition, a unique tremor-reduction structure is introduced to enhance operating safety. Tracking performance experiments and in vitro experiments were conducted to evaluate the performance of the developed ECRS. According to these experimental results, the average translation-tracking error is 0.94 mm, and the average error of rotation is 0.89 degrees. Moreover, in vitro experiments demonstrated that haptic feedback has the advantage of reducing workload and shortening surgery completion time. The developed ECRS also has the benefits of inspiring other researchers to study collaborative robots and magnetically controlled feedback.

sEMG-Based Motion Recognition of Upper Limb Rehabilitation Using the Improved Yolo-v4 Algorithm.

The surface electromyography (sEMG) signal is widely used as a control source of the upper limb exoskeleton rehabilitation robot. However, the traditional way of controlling the exoskeleton robot by the sEMG signal requires one to specially extract and calculate for complex sEMG features. Moreover, due to the huge amount of calculation and individualized difference, the real-time control of the exoskeleton robot cannot be realized. Therefore, this paper proposes a novel method using an improved detection algorithm to recognize limb joint motion and detect joint angle based on sEMG images, aiming to obtain a high-security and fast-processing action recognition strategy. In this paper, MobileNetV2 combined the Ghost module as the feature extraction network to obtain the pretraining model. Then, the target detection network Yolo-V4 was used to estimate the six movement categories of the upper limb joints and to predict the joint movement angles. The experimental results showed that the proposed motion recognition methods were available. Every 100 pictures can be accurately identified in approximately 78 pictures, and the processing speed of every single picture on the PC side was 17.97 ms. For the train data, the mAP@0.5 could reach 82.3%, and mAP@0.5-0.95 could reach 0.42; for the verification data, the average recognition accuracy could reach 80.7%.

An Isomorphic Interactive Device for the Interventional Surgical Robot after In Vivo Study.

Interventional surgical robots are widely used in neurosurgery to improve surgeons' working environment and surgical safety. Based on the actual operational needs of surgeons' feedback during preliminary in vivo experiments, this paper proposed an isomorphic interactive master controller for the master-slave interventional surgical robot. The isomorphic design of the controller allows surgeons to utilize their surgical skills during remote interventional surgeries. The controller uses the catheter and guidewire as the operating handle, the same as during actual surgeries. The collaborative operational structure design and the working methods followed the clinical operational skills. The linear force feedback and torque feedback devices were designed to improve the safety of surgeries under remote operating conditions. An eccentric force compensation was conducted to achieve accurate force feedback. Several experiments were carried out, such as calibration experiments, master-slave control performance evaluation experiments, and operation comparison experiments on the novel and previously used controllers. The experimental results show that the proposed controller can perform complex operations in remote surgery applications and has the potential for further animal experiment evaluations.

A Mirror Bilateral Neuro-Rehabilitation Robot System with the sEMG-Based Real-Time Patient Active Participant Assessment.

In this paper, a novel mirror visual feedback-based (MVF) bilateral neurorehabilitation system with surface electromyography (sEMG)-based patient active force assessment was proposed for upper limb motor recovery and improvement of limb inter-coordination. A mirror visual feedback-based human-robot interface was designed to facilitate the bilateral isometric force output training task. To achieve patient active participant assessment, an sEMG signals-based elbow joint isometric force estimation method was implemented into the proposed system for real-time affected side force assessment and participation evaluation. To assist the affected side limb efficiently and precisely, a mirror bilateral control framework was presented for bilateral limb coordination. Preliminary experiments were conducted to evaluate the estimation accuracy of force estimation method and force tracking accuracy of system performance. The experimental results show the proposed force estimation method can efficiently calculate the elbow joint force in real-time, and the affected side limb of patients can be assisted to track output force of the non-paretic side limb for better limb coordination by the proposed bilateral rehabilitation system.

ADRC-Based Control Method for the Vascular Intervention Master-Slave Surgical Robotic System.

In vascular interventional surgery, surgeons operate guidewires and catheters to diagnose and treat patients with the assistance of the digital subtraction angiography (DSA). Therefore, the surgeon will be exposed to X-rays for extended periods. To protect the surgeon, the development of a robot-assisted surgical system is of great significance. The displacement tracking accuracy is the most important issue to be considered in the development of the system. In this study, the active disturbance rejection control (ADRC) method is applied to guarantee displacement tracking accuracy. First, the core contents of the proportional-integral-derivative (PID) and ADRC methods are analyzed. Second, comparative evaluation experiments for incremental PID and ADRC methods are presented. The results show that the ADRC method has better performance of than that of the incremental PID method. Finally, the calibration experiments for the ADRC control method are implemented using the master-slave robotic system. These experiments demonstrate that the maximum tracking error is 0.87 mm using the ADRC method, effectively guaranteeing surgical safety.

Performance Evaluation of a Magnetically Driven Microrobot for Targeted Drug Delivery.

Given that the current microrobot cannot achieve fixed-point and quantitative drug application in the gastrointestinal (GI) tract, a targeted drug delivery microrobot is proposed, and its principle and characteristics are studied. Through the control of an external magnetic field, it can actively move to the affected area to realize the targeted drug delivery function. The microrobot has a cam structure connected with a radially magnetized permanent magnet, which can realize two movement modes: movement and targeted drug delivery. Firstly, the magnetic actuated capsule microrobotic system (MACMS) is analyzed. Secondly, the dynamic model and quantitative drug delivery model of the targeted drug delivery microrobot driven by the spiral jet structure are established, and the motion characteristics of the targeted drug delivery microrobot are simulated and analyzed by the method of Computational Fluid Dynamics (CFD). Finally, the whole process of the targeted drug delivery task of the microrobot is simulated. The results show that the targeted drug delivery microrobot can realize basic movements such as forward, backward, fixed-point parking and drug delivery through external magnetic field control, which lays the foundation for gastrointestinal diagnosis and treatment.

A Magnetorheological Fluids-Based Robot-Assisted Catheter/Guidewire Surgery System for Endovascular Catheterization.

A teleoperated robotic catheter operating system is a solution to avoid occupational hazards caused by repeated exposure radiation of the surgeon to X-ray during the endovascular procedures. However, inadequate force feedback and collision detection while teleoperating surgical tools elevate the risk of endovascular procedures. Moreover, surgeons cannot control the force of the catheter/guidewire within a proper range, and thus the risk of blood vessel damage will increase. In this paper, a magnetorheological fluid (MR)-based robot-assisted catheter/guidewire surgery system has been developed, which uses the surgeon's natural manipulation skills acquired through experience and uses haptic cues to generate collision detection to ensure surgical safety. We present tests for the performance evaluation regarding the teleoperation, the force measurement, and the collision detection with haptic cues. Results show that the system can track the desired position of the surgical tool and detect the relevant force event at the catheter. In addition, this method can more readily enable surgeons to distinguish whether the proximal force exceeds or meets the safety threshold of blood vessels.

A Home-Based Bilateral Rehabilitation System With sEMG-based Real-Time Variable Stiffness.

Bilateral rehabilitation allows patients with hemiparesis to exploit the cooperative capabilities of both arms to promote the recovery process. Although various approaches have been proposed to facilitate synchronized robot-assisted bilateral movements, few studies have focused on addressing the varying joint stiffness resulting from dynamic motions. This paper presents a novel bilateral rehabilitation system that implements a surface electromyography (sEMG)-based stiffness control to achieve real-time stiffness adjustment based on the user's dynamic motion. An sEMG-driven musculoskeletal model that incorporates the muscle activation and muscular contraction dynamics is developed to provide reference signals for the robot's real-time stiffness control. Preliminary experiments were conducted to evaluate the system performance in tracking accuracy and comfortability, which showed the proposed rehabilitation system with sEMG-based real-time stiffness variation achieved fast adaption to the patient's dynamic movement as well as improving the comfort in robot-assisted bilateral training.

Multiple Bio-Inspired Father-Son Underwater Robot for Underwater Target Object Acquisition and Identification.

Underwater target acquisition and identification performed by manipulators having broad application prospects and value in the field of marine development. Conventional manipulators are too heavy to be used for small target objects and unsuitable for shallow sea working. In this paper, a bio-inspired Father-Son Underwater Robot System (FURS) is designed for underwater target object image acquisition and identification. Our spherical underwater robot (SUR), as the father underwater robot of the FURS, has the ability of strong dynamic balance and good maneuverability, can realize approach the target area quickly, and then cruise and surround the target object. A coiling mechanism was installed on SUR for the recycling and release of the son underwater robot. A Salamandra-inspired son underwater robot is used as the manipulator of the FURS, which is connected to the spherical underwater robot by a tether. The son underwater robot has multiple degrees of freedom and realizes both swimming and walking movement modes. The son underwater robot can move to underwater target objects. The vision system is installed to enable the FURS to acquire the image information of the target object with the aid of the camera, and also to identify the target object. Finally, verification experiments are conducted in an indoor water tank and outdoor swimming pool conditions to verify the effectiveness of the proposed in this paper.

Design and implementation of a novel wireless modular capsule robotic system in pipe.

Capsule endoscopy is a new type of technology in the diagnosis and treatment of digestive diseases, with painless and low invasive features. However, current capsule robots have many problems, such as over-sized, single function and lack of active locomotion control. This study proposed and designed a new wireless modular capsule robotic system in pipe. The modular capsule robots could move forward and backward in the pipe in the axial direction, turn in a bending environment, and achieve the rendezvous and separation action through the three-dimensional rotating magnetic field generated by the three-axis Helmholtz coils. In this paper, the drive system of the three-axis Helmholtz coils, the power supply control system, and the modular capsule robot structure were analyzed and designed respectively. Finally, a series of characterization experiments were carried out to evaluate the motion characteristics of the modular capsule robots, including the influence of the flow environment imitated to human body's gastrointestinal motility, the frequency of the input signal, and the different structure of the robots on the movement characteristics of the modular capsule robot in this study. The study also evaluated the turning characteristics of robots. Experimental results showed that under different circumstances, modular capsule robots had good motion characteristics, and the effectiveness of the modular functionality had also been verified.

Machine learning-based operation skills assessment with vascular difficulty index for vascular intervention surgery.

An accurate assessment of surgical operation skills is essential for improving the vascular intervention surgical outcome and the performance of endovascular surgery robots. In existing studies, subjective and objective assessments of surgical operation skills use a variety of indicators, such as the operation speed and operation smoothness. However, the vascular conditions of particular patients have not been considered in the assessment, leading to deviations in the evaluation. Therefore, in this paper, an operation skills assessment method including the vascular difficulty level index for catheter insertion at the aortic arch in endovascular surgery is proposed. First, the model describing the difficulty of the vascular anatomical structure is established with characteristics of different aortic arch branches based on machine learning. Afterwards, the vascular difficulty level is set as an objective index combined with operating characteristics extracted from the operations performed by surgeons to evaluate the surgical operation skills at the aortic arch using machine learning. The accuracy of the assessment improves from 86.67 to 96.67% after inclusion of the vascular difficulty as an evaluation indicator to more objectively and accurately evaluate skills. The method described in this paper can be adopted to train novice surgeons in endovascular surgery, and for studies of vascular interventional surgery robots. Graphical abstract Operation skill assessment with vascular difficulty for vascular interventional surgery.

A novel noncontact detection method of surgeon's operation for a master-slave endovascular surgery robot.

Master-slave endovascular interventional surgery (EIS) robots have brought revolutionary advantages to traditional EIS, such as avoiding X-ray radiation to the surgeon and improving surgical precision and safety. However, the master controllers of most of the current EIS robots always lead to bad human-machine interaction, because of the difference in nature between the rigid operating handle and the flexible medical catheter used in EIS. In this paper, a noncontact detection method is proposed, and a novel master controller is developed to realize real-time detection of surgeon's operation without interference to the surgeon. A medical catheter is used as the operating handle. It is enabled by using FAST corner detection algorithm and optical flow algorithm to track the corner points of the continuous markers on a designed sensing pipe. A mathematical model is established to calculate the axial and rotational motion of the sensing pipe according to the moving distance of the corner points in image coordinates. A master-slave EIS robot system is constructed by integrating the proposed master controller and a developed slave robot. Surgical task performance evaluation in an endovascular evaluator (EVE) is conducted, and the results indicate that the proposed detection method breaks through the axial measuring range limitation of the previous marker-based detection method. In addition, the rotational detection error is reduced by 92.5% compared with the previous laser-based detection method. The results also demonstrate the capability and efficiency of the proposed master controller to control the slave robot for surgical task implementation. Graphical abstract A novel master controller is developed to realize real-time noncontact detection of surgeon's operation without interference to the surgeon. The master controller is used to remotely control the slave robot to implement certain surgical tasks.