In Vivo Feasibility Study: Evaluating Autonomous Data-Driven Robotic Needle Trajectory Correction in MRI-Guided Transperineal Procedures.

This study addresses the targeting challenges in MRI-guided transperineal needle placement for prostate cancer (PCa) diagnosis and treatment, a procedure where accuracy is crucial for effective outcomes. We introduce a parameter-agnostic trajectory correction approach incorporating a data-driven closed-loop strategy by radial displacement and an FBG-based shape sensing to enable autonomous needle steering. In an animal study designed to emulate clinical complexity and assess MRI compatibility through a PCa mock biopsy procedure, our approach demonstrated a significant improvement in targeting accuracy (p<0.05), with mean target error of only 2.2 ± 1.9 mm on first insertion attempts, without needle reinsertions. To the best of our knowledge, this work represents the first in vivo evaluation of robotic needle steering with FBG-sensor feedback, marking a significant step towards its clinical translation.

Towards Autonomous Retinal Microsurgery Using RGB-D Images.

Retinal surgery is a challenging procedure requiring precise manipulation of the fragile retinal tissue, often at the scale of tens-of-micrometers. Its difficulty has motivated the development of robotic assistance platforms to enable precise motion, and more recently, novel sensors such as microscope integrated optical coherence tomography (OCT) for RGB-D view of the surgical workspace. The combination of these devices opens new possibilities for robotic automation of tasks such as subretinal injection (SI), a procedure that involves precise needle insertion into the retina for targeted drug delivery. Motivated by this opportunity, we develop a framework for autonomous needle navigation during SI. We develop a system which enables the surgeon to specify waypoint goals in the microscope and OCT views, and the system autonomously navigates the needle to the desired subretinal space in real-time. Our system integrates OCT and microscope images with convolutional neural networks (CNNs) to automatically segment the surgical tool and retinal tissue boundaries, and model predictive control that generates optimal trajectories that respect kinematic constraints to ensure patient safety. We validate our system by demonstrating 30 successful SI trials on pig eyes. Preliminary comparisons to a human operator in robot-assisted mode highlight the enhanced safety and performance of our system.

Robotic navigation with deep reinforcement learning in transthoracic echocardiography.

PURPOSE: The search for heart components in robotic transthoracic echocardiography is a time-consuming process. This paper proposes an optimized robotic navigation system for heart components using deep reinforcement learning to achieve an efficient and effective search technique for heart components. METHOD: The proposed method introduces (i) an optimized search behavior generation algorithm that avoids multiple local solutions and searches for the optimal solution and (ii) an optimized path generation algorithm that minimizes the search path, thereby realizing short search times. RESULTS: The mitral valve search with the proposed method reaches the optimal solution with a probability of 74.4%, the mitral valve confidence loss rate when the local solution stops is 16.3% on average, and the inspection time with the generated path is 48.6 s on average, which is 56.6% of the time cost of the conventional method. CONCLUSION: The results indicate that the proposed method improves the search efficiency, and the optimal location can be searched in many cases with the proposed method, and the loss rate of the confidence in the mitral valve was low even when a local solution rather than the optimal solution was reached. It is suggested that the proposed method enables accurate and quick robotic navigation to find heart components.

Flexible endoscope manipulating robot using quad-roller friction mechanism.

A robotic system for manipulating a flexible endoscope in surgery can provide enhanced accuracy and usability compared to manual operation. However, previous studies require large-scale, complex hardware systems to implement the rotational and translational motions of the soft endoscope cable. The conventional control of the endoscope by actuating the endoscope handle also leads to undesired slack between the endoscope tip and the handle, which becomes more problematic with long endoscopes such as a colonoscope. This study proposes a compact quad-roller friction mechanism that enables rotational and translational motions triggered not from the endoscope handle but at the endoscope tip. Controlling two pairs of tilted rollers achieves both types of motion within a small space. The proposed system also introduces an unsynchronized motion strategy between the handle and tip parts to minimize the robot's motion near the patient by employing the slack positively as a control index. Experiments indicate that the proposed system achieves accurate rotational and translational motions, and the unsynchronized control method reduces the total translational motion by up to 88% compared to the previous method.

Design and Motion Control of Master-Slave Control Endotracheal Intubation Robot.

BACKGROUND: Master-slave remote control technology allows patients to be treated promptly during transport and also reduces the risk of contagious infections. Endotracheal intubation, guided by endoscopy and a master-slave system, enables doctors to perform the procedure efficiently and accurately. METHODS: In this paper, we present the development of a master-slave controlled endotracheal intubation robot (EIR). It is based on operation incremental mapping, a weighted recursive average filtering method to reduce vibration, and a virtual fixture designed to reduce mishandling in minimally invasive surgery. RESULTS: Simulation analysis of the master-slave control demonstrates that the weighted recursive average filtering method effectively reduces vibration, while the virtual fixture assists in confining the operator's movement within a delimited area. Experimental validation confirms the validity of the robot's structural design and control method. CONCLUSIONS: The developed robot successfully achieves the necessary motion for endotracheal intubation surgery through master-slave control.

Simulation-based Flexible Needle Control with Single-core FBG Feedback for Spinal Injections.

Objective: We present a general framework of simultaneous needle shape reconstruction and control input generation for robot-assisted spinal injection procedures, without continuous imaging feedback. Methods: System input-output mapping is generated with a real-time needle-tissue interaction simulation, and single-core FBG sensor readings are used as local needle shape feedback within the same simulation framework. FBG wavelength shifts due to temperature variation is removed by exploiting redundancy in fiber arrangement. Results: Targeting experiments performed on both plastisol lumbar phantoms as well as an ex vivo porcine lumbar section achieved in-plane tip errors of 0.6 ± 0.3 mm and 1.6 ± 0.9 mm , and total tip errors of 0.9 ± 0.7 mm and 2.1 ± 0.8 mm for the two testing environments. Significance: Our clinically inspired control strategy and workflow is self-contained and not dependent on the modality of imaging guidance. The generalizability of the proposed approach can be applied to other needle-based interventions where medical imaging cannot be reliably utilized as part of a closed-loop control system for needle guidance.

A multi-chamber soft robot for transesophageal echocardiography: continuous kinematic matching control of soft medical robots.

OBJECTIVES: This research investigates designing a continuum soft robot and proposing a kinematic matching control to enable the robot to perform a specified medical task, which in this paper is the transesophageal echocardiography (TEE). METHODS: A multi-chamber soft robot was designed and fabricated based on the molding of separate layers. The method of transformation matrices was used to develop the kinematic models, and a control method using Jacobian matrices was proposed to manipulate the robot. RESULTS: A prototype was made based on a multi-chamber multi-layer design. The system contains three segments that can be actuated independently to mimic the active bending part of the respective probe. Kinematic models were developed. Negative pressure (vacuum) was used as actuation input. An open-loop controller inspired by a redundancy resolution technique was proposed to make the soft robot tip follow the desired path, i.e. the path of the rigid ultrasound probe. CONCLUSIONS: It is concluded that the soft solution can perform the required task as the reachable points of the TEE tip cover the proposed robot workspace and the proposed control can be used for maneuvering in arbitrary trajectories.

Investigating Haptic Feedback in Vision-Deficient Millirobot Telemanipulation.

The evolution of magnetically actuated millirobots gives rise to unique teleoperation challenges due to their non-traditional kinematic and dynamic architectures, as well as their frequent use of suboptimal imaging modalities. Recent investigations into haptic interfaces for millirobots have shown promise but lack the clinically motivated task scenarios necessary to justify future development. In this work, we investigate the utility of haptic feedback on bilateral teleoperation of a magnetically actuated millirobot in visually deficient conditions. We conducted an N=23 user study in an aneurysm coiling inspired procedure, which required participants to navigate the robot through a maze in near total darkness to manipulate beads to a target under simulated fluoroscopy. We hypothesized that users will be better able to complete the telemanipulation task with haptic feedback while reducing excess forces on their surroundings compared to the no feedback conditions. Our results showed an over 40% improvement in participants' bead scoring, a nearly 10% reduction in mean force, and 13% reduction in maximum force with haptic feedback, as well as significant improvements in other metrics. Results highlight that benefits of haptic feedback are retained when haptic feedback is removed. These findings suggest that haptic feedback has the potential to significantly improve millirobot telemanipulation and control in traditionally vision deficient tasks.

Continuum Robots and Magnetic Soft Robots: From Models to Interdisciplinary Challenges for Medical Applications.

This article explores the challenges of continuum and magnetic soft robotics for medical applications, extending from model development to an interdisciplinary perspective. First, we established a unified model framework based on algebra and geometry. The research progress and challenges in principle models, data-driven, and hybrid modeling were then analyzed in depth. Simultaneously, a numerical analysis framework for the principle model was constructed. Furthermore, we expanded the model framework to encompass interdisciplinary research and conducted a comprehensive analysis, including an in-depth case study. Current challenges and the need to address meta-problems were identified through discussion. Overall, this review provides a novel perspective on understanding the challenges and complexities of continuum and magnetic soft robotics in medical applications, paving the way for interdisciplinary researchers to assimilate knowledge in this domain rapidly.

Autonomous Needle Navigation in Subretinal Injections via iOCT.

Subretinal injection is an effective method for direct delivery of therapeutic agents to treat prevalent subretinal diseases. Among the challenges for surgeons are physiological hand tremor, difficulty resolving single-micron scale depth perception, and lack of tactile feedback. The recent introduction of intraoperative Optical Coherence Tomography (iOCT) enables precise depth information during subretinal surgery. However, even when relying on iOCT, achieving the required micron-scale precision remains a significant surgical challenge. This work presents a robot-assisted workflow for high-precision autonomous needle navigation for subretinal injection. The workflow includes online registration between robot and iOCT coordinates; tool-tip localization in iOCT coordinates using a Convolutional Neural Network (CNN); and tool-tip planning and tracking system using real-time Model Predictive Control (MPC). The proposed workflow is validated using a silicone eye phantom and ex vivo porcine eyes. The experimental results demonstrate that the mean error to reach the user-defined target and the mean procedure duration are within an acceptable precision range. The proposed workflow achieves a 100% success rate for subretinal injection, while maintaining scleral forces at the scleral insertion point below 15mN throughout the navigation procedures.

Integrated Sensors for Soft Medical Robotics.

Minimally invasive procedures assisted by soft robots for surgery, diagnostics, and drug delivery have unprecedented benefits over traditional solutions from both patient and surgeon perspectives. However, the translation of such technology into commercialization remains challenging. The lack of perception abilities is one of the obstructive factors paramount for a safe, accurate and efficient robot-assisted intervention. Integrating different types of miniature sensors onto robotic end-effectors is a promising trend to compensate for the perceptual deficiencies in soft robots. For example, haptic feedback with force sensors helps surgeons to control the interaction force at the tool-tissue interface, impedance sensing of tissue electrical properties can be used for tumor detection. The last decade has witnessed significant progress in the development of multimodal sensors built on the advancement in engineering, material science and scalable micromachining technologies. This review article provides a snapshot on common types of integrated sensors for soft medical robots. It covers various sensing mechanisms, examples for practical and clinical applications, standard manufacturing processes, as well as insights on emerging engineering routes for the fabrication of novel and high-performing sensing devices.

Autonomous System for Tumor Resection (ASTR) - Dual-Arm Robotic Midline Partial Glossectomy.

Head and neck cancers are the seventh most common cancers worldwide, with squamous cell carcinoma being the most prevalent histologic subtype. Surgical resection is a primary treatment modality for many patients with head and neck squamous cell carcinoma, and accurately identifying tumor boundaries and ensuring sufficient resection margins are critical for optimizing oncologic outcomes. This study presents an innovative autonomous system for tumor resection (ASTR) and conducts a feasibility study by performing supervised autonomous midline partial glossectomy for pseudotumor with millimeter accuracy. The proposed ASTR system consists of a dual-camera vision system, an electrosurgical instrument, a newly developed vacuum grasping instrument, two 6-DOF manipulators, and a novel autonomous control system. The letter introduces an ontology-based research framework for creating and implementing a complex autonomous surgical workflow, using the glossectomy as a case study. Porcine tongue tissues are used in this study, and marked using color inks and near-infrared fluorescent (NIRF) markers to indicate the pseudotumor. ASTR actively monitors the NIRF markers and gathers spatial and color data from the samples, enabling planning and execution of robot trajectories in accordance with the proposed glossectomy workflow. The system successfully performs six consecutive supervised autonomous pseudotumor resections on porcine specimens. The average surface and depth resection errors measure 0.73±0.60 mm and 1.89±0.54 mm, respectively, with no positive tumor margins detected in any of the six resections. The resection accuracy is demonstrated to be on par with manual pseudotumor glossectomy performed by an experienced otolaryngologist.

Robotic system for nasopharyngeal swab sampling based on remote center of motion mechanism.

PURPOSE: In this study, a robotic system is proposed for nasopharyngeal (NP) swab sampling with high safety and efficiency. Most existing swab-sampling robots have more than six degrees of freedom (DOFs). However, not all six DOFs are necessarily required for NP swab sampling. A high number of DOFs can cause safety problems, such as collisions between the robot and patient. METHOD: We developed a new type of robot with four DOFs for NP swab sampling that consists of a two DOFs remote center of motion (RCM) mechanism, a two DOFs insertion mechanism, and a nostril support unit. With the nostril support unit, the robot no longer needs to adjust the insertion position of the swab. The proposed robot enables the insertion orientation and depth to be adjusted according to different postures or facial shapes of the subject. For intuitive and precise remote control of the robot, a dedicated master device for the RCM and a visual feedback system were developed. RESULT: The effectiveness of the robotic system was demonstrated by repeatability, RCM accuracy, tracking accuracy, and in vitro phantom experiments. The average tracking error between the master device and the robot was less than 2 mm. The contact force exerted on the swab prior to reaching the nasopharynx was less than 0.04 N, irrespective of the phantom's pose. CONCLUSION: This study confirmed that the RCM-based robotic system is effective and safe for NP swab sampling while using minimal DOFs.

Telerobotic Transcatheter Delivery System for Mitral Valve Implant.

Mitral regurgitation (MR) is the most common type of valvular heart disease, affecting over 2% of the world population, and the gold-standard treatment is surgical mitral valve repair/replacement. Compared to open-heart surgeries, minimally invasive surgeries (MIS) using transcatheter approaches have become popular because of their notable benefits such as less postoperative pain, shorter hospital stay, and faster recovery time. However, commercially available catheters are manually actuated, causing over-exposure of clinical staff to radiation and increased risk of human error during medical interventions. To tackle this problem, in this letter, we propose a telerobotic transcatheter delivery system, which consists of a robotic catheter (5.7 mm OD), a reinforced guide tube (1.11m length), and an actuation system. We present the robotic system design, fabrication of key components, and static model of reinforced quadlumen tube. The robot interface design enables the user to intuitively control the robot. We demonstrate the effectiveness of the telerobotic transcatheter delivery system and reinforced quadlumen tube in a realistic human cardiovascular phantom for preclinical evaluation.

Robotic Navigation Autonomy for Subretinal Injection via Intelligent Real-Time Virtual iOCT Volume Slicing.

In the last decade, various robotic platforms have been introduced that could support delicate retinal surgeries. Concurrently, to provide semantic understanding of the surgical area, recent advances have enabled microscope-integrated intraoperative Optical Coherent Tomography (iOCT) with high-resolution 3D imaging at near video rate. The combination of robotics and semantic understanding enables task autonomy in robotic retinal surgery, such as for subretinal injection. This procedure requires precise needle insertion for best treatment outcomes. However, merging robotic systems with iOCT introduces new challenges. These include, but are not limited to high demands on data processing rates and dynamic registration of these systems during the procedure. In this work, we propose a framework for autonomous robotic navigation for subretinal injection, based on intelligent real-time processing of iOCT volumes. Our method consists of an instrument pose estimation method, an online registration between the robotic and the iOCT system, and trajectory planning tailored for navigation to an injection target. We also introduce intelligent virtual B-scans, a volume slicing approach for rapid instrument pose estimation, which is enabled by Convolutional Neural Networks (CNNs). Our experiments on ex-vivo porcine eyes demonstrate the precision and repeatability of the method. Finally, we discuss identified challenges in this work and suggest potential solutions to further the development of such systems.

Real-time intraoperative surgical guidance system in the da Vinci surgical robot based on transrectal ultrasound/photoacoustic imaging with photoacoustic markers: an ex vivo demonstration.

This paper introduces the first integrated real-time intraoperative surgical guidance system, in which an endoscope camera of da Vinci surgical robot and a transrectal ultrasound (TRUS) transducer are co-registered using photoacoustic markers that are detected in both fluorescence (FL) and photoacoustic (PA) imaging. The co-registered system enables the TRUS transducer to track the laser spot illuminated by a pulsed-laser-diode attached to the surgical instrument, providing both FL and PA images of the surgical region-of-interest (ROI). As a result, the generated photoacoustic marker is visualized and localized in the da Vinci endoscopic FL images, and the corresponding tracking can be conducted by rotating the TRUS transducer to display the PA image of the marker. A quantitative evaluation revealed that the average registration and tracking errors were 0.84 mm and 1.16°, respectively. This study shows that the co-registered photoacoustic marker tracking can be effectively deployed intraoperatively using TRUS+PA imaging providing functional guidance of the surgical ROI.

A review on model-based and model-free approaches to control soft actuators and their potentials in colonoscopy.

Colorectal cancer (CRC) is the third most common cancer worldwide and responsible for approximately 1 million deaths annually. Early screening is essential to increase the chances of survival, and it can also reduce the cost of treatments for healthcare centres. Colonoscopy is the gold standard for CRC screening and treatment, but it has several drawbacks, including difficulty in manoeuvring the device, patient discomfort, and high cost. Soft endorobots, small and compliant devices thatcan reduce the force exerted on the colonic wall, offer a potential solution to these issues. However, controlling these soft robots is challenging due to their deformable materials and the limitations of mathematical models. In this Review, we discuss model-free and model-based approaches for controlling soft robots that can potentially be applied to endorobots for colonoscopy. We highlight the importance of selecting appropriate control methods based on various parameters, such as sensor and actuator solutions. This review aims to contribute to the development of smart control strategies for soft endorobots that can enhance the effectiveness and safety of robotics in colonoscopy. These strategies can be defined based on the available information about the robot and surrounding environment, control demands, mechanical design impact and characterization data based on calibration.

The Objective Dementia Severity Scale Based on MRI with Contrastive Learning: A Whole Brain Neuroimaging Perspective.

In the clinical treatment of Alzheimer's disease, one of the most important tasks is evaluating its severity for diagnosis and therapy. However, traditional testing methods are deficient, such as their susceptibility to subjective factors, incomplete evaluation, low accuracy, or insufficient granularity, resulting in unreliable evaluation scores. To address these issues, we propose an objective dementia severity scale based on MRI (ODSS-MRI) using contrastive learning to automatically evaluate the neurological function of patients. The approach utilizes a deep learning framework and a contrastive learning strategy to mine relevant information from structural magnetic resonance images to obtain the patient's neurological function level score. Given that the model is driven by the patient's whole brain imaging data, but without any possible biased manual intervention or instruction from the physician or patient, it provides a comprehensive and objective evaluation of the patient's neurological function. We conducted experiments on the Alzheimer's disease Neuroimaging Initiative (ADNI) dataset, and the results showed that the proposed ODSS-MRI was correlated with the stages of AD 88.55% better than all existing methods. This demonstrates its efficacy to describe the neurological function changes of patients during AD progression. It also outperformed traditional psychiatric rating scales in discriminating different stages of AD, which is indicative of its superiority for neurological function evaluation.

Flexible Needle Bending Model for Spinal Injection Procedures.

An in situ needle manipulation technique used by physicians when performing spinal injections is modeled to study its effect on needle shape and needle tip position. A mechanics-based model is proposed and solved using finite element method. A test setup is presented to mimic the needle manipulation motion. Tissue phantoms made from plastisol as well as porcine skeletal muscle samples are used to evaluate the model accuracy against medical images. The effect of different compression models as well as model parameters on model accuracy is studied, and the effect of needle-tissue interaction on the needle remote center of motion is examined. With the correct combination of compression model and model parameters, the model simulation is able to predict needle tip position within submillimeter accuracy.