Learning-based needle tip tracking in 2D ultrasound by fusing visual tracking and motion prediction.

Visual trackers are the most commonly adopted approach for needle tip tracking in ultrasound (US)-based procedures. However, they often perform unsatisfactorily in biological tissues due to the significant background noise and anatomical occlusion. This paper presents a learning-based needle tip tracking system, which consists of not only a visual tracking module, but also a motion prediction module. In the visual tracking module, two sets of masks are designed to improve the tracker's discriminability, and a template update submodule is used to keep up to date with the needle tip's current appearance. In the motion prediction module, a Transformer network-based prediction architecture estimates the target's current position according to its historical position data to tackle the problem of target's temporary disappearance. A data fusion module then integrates the results from the visual tracking and motion prediction modules to provide robust and accurate tracking results. Our proposed tracking system showed distinct improvement against other state-of-the-art trackers during the motorized needle insertion experiments in both gelatin phantom and biological tissue environments (e.g. 78% against <60% in terms of the tracking success rate in the most challenging scenario of "In-plane-static" during the tissue experiments). Its robustness was also verified in manual needle insertion experiments under varying needle velocities and directions, and occasional temporary needle tip disappearance, with its tracking success rate being >18% higher than the second best performing tracking system. The proposed tracking system, with its computational efficiency, tracking robustness, and tracking accuracy, will lead to safer targeting during existing clinical practice of US-guided needle operations and potentially be integrated in a tissue biopsy robotic system.

A Continuum Robotic Cannula With Tip Following Capability and Distal Dexterity for Intracerebral Hemorrhage Evacuation.

OBJECTIVE: This paper aims to investigate a new continuum robot design and its motion implementation methods appropriate for a minimally invasive intracerebral hemorrhage (ICH) evacuation. METHODS: We propose a continuum robotic cannula, consisting of a precurved body and a 2-degree-of-freedom (DoF) flexible tip, monolithically fabricated. Kinematic model with cable elongation model, and a dedicated design optimization and motion planning algorithm were developed to enable the follow-the-leader (FTL) motion of the cannula. A task-dependent Jacobian-based closed loop control was also designed to track the cannula motion during the insertion and its independent tip motion. RESULTS: Comprehensive experiments were conducted to verify the kinematic model and submillimeter motion coupling between the cannula precurved body and its flexible tip. The cannula was also capable of achieving FTL motion within around 2.5 mm shape deviation and control performance within submillimeter errors. It was finally demonstrated to be capable of the nonlinear insertion and tip manipulation in the brain phantom. CONCLUSION: The new cannula design, together with the proposed algorithms, provides the unique ability to access ICH in a nonlinear trajectory and dexterous tip motion. SIGNIFICANCE: These motion capabilities of the robot in such a slender form factor will lead to more complete ICH evacuation and reduced trauma to the healthy brain tissues.

Tele-Operated Oropharyngeal Swab (TOOS) Robot Enabled by TSS Soft Hand for Safe and Effective Sampling.

The COVID-19 pandemic has imposed serious challenges in multiple perspectives of human life. To diagnose COVID-19, oropharyngeal swab (OP SWAB) sampling is generally applied for viral nucleic acid (VNA) specimen collection. However, manual sampling exposes medical staff to a high risk of infection. Robotic sampling is promising to mitigate this risk to the minimum level, but traditional robot suffers from safety, cost, and control complexity issues for wide-scale deployment. In this work, we present soft robotic technology is promising to achieve robotic OP swab sampling with excellent swab manipulability in a confined oral space and works as dexterous as existing manual approach. This is enabled by a novel Tstone soft (TSS) hand, consisting of a soft wrist and a soft gripper, designed from human sampling observation and bio-inspiration. TSS hand is in a compact size, exerts larger workspace, and achieves comparable dexterity compared to human hand. The soft wrist is capable of agile omnidirectional bending with adjustable stiffness. The terminal soft gripper is effective for disposable swab pinch and replacement. The OP sampling force is easy to be maintained in a safe and comfortable range (throat sampling comfortable region) under a hybrid motion and stiffness virtual fixture-based controller. A dedicated 3 DOFs RCM platform is used for TSS hand global positioning. Design, modeling, and control of the TSS hand are discussed in detail with dedicated experimental validations. A sampling test based on human tele-operation is processed on the oral cavity model with excellent success rate. The proposed TOOS robot demonstrates a highly promising solution for tele-operated, safe, cost-effective, and quick deployable COVID-19 OP swab sampling.

Modular FBG Bending Sensor for Continuum Neurosurgical Robot.

We present a modular sensing system to measure the deflection of a minimally invasive neurosurgical intracranial robot: MINIR-II. The MINIR-II robot is a tendon-driven continuum robot comprised of multiple spring backbone segments, which has been developed in our prior work. Due to the flexibility of the spring backbone and unique tendon routing configuration, each segment of MINIR-II can bend up to a large curvature (≥100 m-1) in multiple directions. However, the shape measurement of the robot based on tendon displacement is not precise due to friction and unknown external load/disturbance. In this regard, we propose a bending sensor module comprised of a fiber Bragg grating (FBG) fiber, a Polydimethylsiloxane (PDMS) cylinder, and a superelastic spring. The grating segment of the FBG fiber is enclosed inside a PDMS cylinder (1 mm in diameter), and the PDMS cylinder is bonded with the superelastic spring in series. The deflection or bending of the robot backbone segment is translated into an axial loading in the superelastic spring, which applies tension to the FBG; therefore, by measuring the peak wavelength shift of the FBG, the bending angle can be estimated. This paper describes the design, fabrication, and kinematic aspects of the sensor module in detail. To evaluate the proposed concept, one such sensor module has been tested and evaluated on the MINIR-II robot.

Towards Patient-Specific 3D-Printed Robotic Systems for Surgical Interventions.

Surgical robots have been extensively researched for a wide range of surgical procedures due to the advantages of improved precision, sensing capabilities, motion scaling, and tremor reduction, to name a few. Though the underlying disease condition or pathology may be the same across patients, the intervention approach to treat the condition can vary significantly across patients. This is especially true for endovascular interventions, where each case brings forth its own challenges. Hence it is critical to develop patient-specific surgical robotic systems to maximize the benefits of robot-assisted surgery. Manufacturing patient-specific robots can be challenging for complex procedures and furthermore the time required to build them can be a challenge. To overcome this challenge, additive manufacturing, namely 3D-printing, is a promising solution. 3D-printing enables fabrication of complex parts precisely and efficiently. Although 3D-printing techniques have been researched for general medical applications, patient-specific surgical robots are currently in their infancy. After reviewing the state-of-the-art in 3D-printed surgical robots, this paper discusses 3D-printing techniques that could potentially satisfy the stringent requirements for surgical interventions. We also present the accomplishments in our group in developing 3D-printed surgical robots for neurosurgical and cardiovascular interventions. Finally, we discuss the challenges in developing 3D-printed surgical robots and provide our perspectives on future research directions.

Design, Analysis, and Evaluation of a Remotely Actuated MRI-Compatible Neurosurgical Robot.

The paper presents an MRI-compatible neurosurgical robotic system that is designed to operate the head-mounted meso-scale 6-degree-of-freedom (DoF) spring-based MINIR-II. The robotic system consists of an actuation module, a transmission module, and the robot module. The transmission module consist of a switching mechanism for reducing the required number of motors by half, an innovative linkage mechanism to insert and retract the robot with minimal tendon displacement and friction loss, and a quick-connect mechanism for easy attachment of the disposable MINIR-II. Design, analysis, and development of each module are described in detail. Most of the critical components such as the robot, the quick-connect, the linkage mechanism, and various gear-pulley combinations in our design are 3-D printed. Preliminary mechanical properties characterization of the system and the capability of the underactuated system to replicate the critical functions of the 6-DoF robot are presented. The robot motion capability in a brain phantom model and its MRI compatibility in a 7-Tesla magnet were verified.

Active Stiffness Tuning of a Spring-based Continuum Robot for MRI-Guided Neurosurgery.

Deep intracranial tumor removal can be achieved if the neurosurgical robot has sufficient flexibility and stability. Towards achieving this goal, we have developed a spring-based continuum robot, namely a Minimally Invasive Neurosurgical Intracranial Robot (MINIR-II) with novel tendon routing and tunable stiffness for use in a magnetic resonance imaging (MRI) environment. The robot consists of a pair of springs in parallel, i.e., an inner inter-connected spring that promotes flexibility with decoupled segment motion and an outer spring that maintains its smooth curved shape during its interaction with the tissue. We propose a shape memory alloy (SMA) spring backbone that provides local stiffness control and a tendon routing configuration that enables independent segment locking. In this work, we also present a detailed local stiffness analysis of the SMA backbone and model the relationship between the resistive force at the robot tip and the tension in the tendon. We also demonstrate through experiments, the validity of our local stiffness model of the SMA backbone and the correlation between the tendon tension and the resistive force. We also performed MRI compatibility studies of the 3-segment MINIR-II robot by attaching it to a robotic platform that consists of SMA spring actuators with integrated water cooling modules.

Toward the Development of a Flexible Mesoscale MRI-compatible Neurosurgical Continuum Robot.

Brain tumor, be it primary or metastatic, is usually life threatening for a person of any age. Primary surgical resection which is one of the most effective ways of treating brain tumors can have tremendously increased success rate if the appropriate imaging modality is used for complete tumor resection. Magnetic resonance imaging (MRI) is the imaging modality of choice for brain tumor imaging because of its excellent soft-tissue contrast. MRI combined with continuum soft robotics has immense potential to be the next major technological breakthrough in the field of brain cancer diagnosis and therapy. In this work, we present the design, kinematic, and force analysis of a flexible spring-based minimally invasive neurosurgical intracranial robot (MINIR-II). It is comprised of an inter-connected inner spring and an outer spring and is connected to actively cooled shape memory alloy spring actuators via tendon driven mechanism. Our robot has three serially connected 2-DoF segments which can be independently controlled due to the central tendon routing configuration. The kinematic and force analysis of the robot and the independent segment control were verified by experiments. Robot motion under forced cooling of SMA springs was evaluated as well as the MRI compatibility of the robot and its motion capability in brainlike gelatin environment.

New Actuation Mechanism for Actively Cooled SMA Springs in a Neurosurgical Robot.

The paper presents the use of shape memory alloy (SMA) spring actuators with real-time cooling to control the motion of the MINIR-II robot. A new actuation mechanism involving the passage of water as the cooling medium and air as the medium to drive out the water has been developed to facilitate real-time control of the springs. Control parameters, such as current, water flow rates, SMA pre-displacement, and gauge pressure of the compressed air, are identified from the SMA thermal model and from the actuation mechanism. In depth modeling and characterization have been performed regarding these parameters to optimize the robot motion speed. Forced water cooling has also been compared with forced air cooling and proved to be the superior method to achieve higher robot speed. An improved robot design and an MRI-compatible experimental platform have been developed for the implementation of the actuation mechanism.

Modeling and characterization of shape memory alloy springs with water cooling strategy in a neurosurgical robot.

Since shape memory alloy (SMA) has high power density and is magnetic resonance imaging (MRI) compatible, it has been chosen as the actuator for the meso-scale minimally invasive neurosurgical intracranial robot (MINIR-II) that is envisioned to be operated under continuous MRI guidance. We have devised a water cooling strategy to improve its actuation frequency by threading a silicone tube through the spring coils to form a compact cooling module-integrated actuator. To create active bi-directional motion in each robot joint, we configured the SMA springs in an antagonistic way. We modeled the antagonistic SMA spring behavior and provided the detailed steps to simulate its motion for a complete cycle. We investigated heat transfer during the resistive heating and water cooling processes. Characterization experiments were performed to determine the parameters used in both models, which were then verified by comparing the experimental and simulated data. The actuation frequency of the antagonistic SMAs was evaluated for several motion amplitudes and we could achieve a maximum actuation frequency of 0.143 Hz for a sinusoidal trajectory with 2 mm amplitude. Lastly, we developed a robotic system to implement the actuators on the MINIR-II to move its end segment back and forth for approximately ±25°.

Design, development, and evaluation of an MRI-guided SMA spring-actuated neurosurgical robot.

In this paper, we present our work on the development of a magnetic resonance imaging (MRI)-compatible Minimally Invasive Neurosurgical Intracranial Robot (MINIR) comprising of shape memory alloy (SMA) spring actuators and tendon-sheath mechanism. We present the detailed modeling and analysis along with experimental results of the characterization of SMA spring actuators. Furthermore, to demonstrate image-feedback control, we used the images obtained from a camera to control the motion of the robot so that eventually continuous MR images could be used in the future to control the robot motion. Since the image tracking algorithm may fail in some situations, we also developed a temperature feedback control scheme which served as a backup controller for the robot. Experimental results demonstrated that both image feedback and temperature feedback can be used to control the motion of MINIR. A series of MRI compatibility tests were performed on the robot and the experimental results demonstrated that the robot is MRI compatible and no significant visual image distortion was observed in the MR images during robot operation.