Autonomous Needle Navigation in Subretinal Injections via iOCT.

Subretinal injection is an effective method for direct delivery of therapeutic agents to treat prevalent subretinal diseases. Among the challenges for surgeons are physiological hand tremor, difficulty resolving single-micron scale depth perception, and lack of tactile feedback. The recent introduction of intraoperative Optical Coherence Tomography (iOCT) enables precise depth information during subretinal surgery. However, even when relying on iOCT, achieving the required micron-scale precision remains a significant surgical challenge. This work presents a robot-assisted workflow for high-precision autonomous needle navigation for subretinal injection. The workflow includes online registration between robot and iOCT coordinates; tool-tip localization in iOCT coordinates using a Convolutional Neural Network (CNN); and tool-tip planning and tracking system using real-time Model Predictive Control (MPC). The proposed workflow is validated using a silicone eye phantom and ex vivo porcine eyes. The experimental results demonstrate that the mean error to reach the user-defined target and the mean procedure duration are within an acceptable precision range. The proposed workflow achieves a 100% success rate for subretinal injection, while maintaining scleral forces at the scleral insertion point below 15mN throughout the navigation procedures.

Autonomous Needle Navigation in Retinal Microsurgery: Evaluation in ex vivo Porcine Eyes.

Important challenges in retinal microsurgery include prolonged operating time, inadequate force feedback, and poor depth perception due to a constrained top-down view of the surgery. The introduction of robot-assisted technology could potentially deal with such challenges and improve the surgeon's performance. Motivated by such challenges, this work develops a strategy for autonomous needle navigation in retinal microsurgery aiming to achieve precise manipulation, reduced end-to-end surgery time, and enhanced safety. This is accomplished through real-time geometry estimation and chance-constrained Model Predictive Control (MPC) resulting in high positional accuracy while keeping scleral forces within a safe level. The robotic system is validated using both open-sky and intact (with lens and partial vitreous removal) ex vivo porcine eyes. The experimental results demonstrate that the generation of safe control trajectories is robust to small motions associated with head drift. The mean navigation time and scleral force for MPC navigation experiments are 7.208 s and 11.97 mN, which can be considered efficient and well within acceptable safe limits. The resulting mean errors along lateral directions of the retina are below 0.06 mm, which is below the typical hand tremor amplitude in retinal microsurgery.

Towards Autonomous Eye Surgery by Combining Deep Imitation Learning with Optimal Control.

During retinal microsurgery, precise manipulation of the delicate retinal tissue is required for positive surgical outcome. However, accurate manipulation and navigation of surgical tools remain difficult due to a constrained workspace and the top-down view during the surgery, which limits the surgeon's ability to estimate depth. To alleviate such difficulty, we propose to automate the tool-navigation task by learning to predict relative goal position on the retinal surface from the current tool-tip position. Given an estimated target on the retina, we generate an optimal trajectory leading to the predicted goal while imposing safety-related physical constraints aimed to minimize tissue damage. As an extended task, we generate goal predictions to various points across the retina to localize eye geometry and further generate safe trajectories within the estimated confines. Through experiments in both simulation and with several eye phantoms, we demonstrate that our framework can permit navigation to various points on the retina within 0.089mm and 0.118mm in xy error which is less than the human's surgeon mean tremor at the tool-tip of 0.180mm. All safety constraints were fulfilled and the algorithm was robust to previously unseen eyes as well as unseen objects in the scene. Live video demonstration is available here: https://youtu.be/n5j5jCCelXk.

Real Time Prediction of Sclera Force with LSTM Neural Networks in Robot-Assisted Retinal Surgery.

Retinal microsurgery is one of the most technically demanding surgeries, during which the surgical tool needs to be inserted into the eyeball and is constantly constrained by the sclerotomy port. During the surgery, any unexpected manipulation could cause extreme tool-sclera contact force leading to sclera damage. Although, a robot assistant could reduce hand tremor and improve the tool positioning accuracy, it cannot prevent or alarm the surgeon about the upcoming danger caused by surgeon's misoperations, i.e., applying excessive force on the sclera. In this paper, we present a new method based on a Long Short Term Memory recurrent neural network for predicting the user behavior, i.e., the contact force between the tool and sclera (sclera force) and the insertion depth of the tool from sclera contact point (insertion depth) in real time (40Hz). The predicted force information is provided to the user through auditory feedback to alarm any unexpected sclera force. The user behavior data is collected in a mock retinal surgical operation on a dry eye phantom with Steady Hand Eye Robot and a novel multi-function sensing tool. The Long Short Term Memory recurrent neural network is trained on the collected time series of sclera force and insertion depth. The network can predict the sclera force and insertion depth 100 milliseconds in the future with 95.29% and 96.57% accuracy, respectively, and can help reduce the fraction of unsafe sclera forces from 40.19% to 15.43%.

Toward Safe Retinal Microsurgery: Development and Evaluation of an RNN-Based Active Interventional Control Framework.

OBJECTIVE: Robotics-assisted retinal microsurgery provides several benefits including improvement of manipulation precision. The assistance provided to the surgeons by current robotic frameworks is, however, a "passive" support, e.g., by damping hand tremor. Intelligent assistance and active guidance are, however, lacking in the existing robotic frameworks. In this paper, an active interventional control framework (AICF) has been presented to increase operation safety by actively intervening the operation to avoid exertion of excessive forces to the sclera. METHODS: AICF consists of the following four components: first, the steady-hand eye robot as the robotic module; second, a sensorized tool to measure tool-to-sclera forces; third, a recurrent neural network to predict occurrence of undesired events based on a short history of time series of sensor measurements; and finally, a variable admittance controller to command the robot away from the undesired instances. RESULTS: A set of user studies were conducted involving 14 participants (with four surgeons). The users were asked to perform a vessel-following task on an eyeball phantom with the assistance of AICF as well as other two benchmark approaches, i.e., auditory feedback (AF) and real-time force feedback (RF). Statistical analysis shows that AICF results in a significant reduction of proportion of undesired instances to about 2.5%, compared with 38.4% and 26.2% using AF and RF, respectively. CONCLUSION: AICF can effectively predict excessive-force instances and augment performance of the user to avoid undesired events during robot-assisted microsurgical tasks. SIGNIFICANCE: The proposed system may be extended to other fields of microsurgery and may potentially reduce tissue injury.

Autonomously Navigating a Surgical Tool Inside the Eye by Learning from Demonstration.

A fundamental challenge in retinal surgery is safely navigating a surgical tool to a desired goal position on the retinal surface while avoiding damage to surrounding tissues, a procedure that typically requires tens-of-microns accuracy. In practice, the surgeon relies on depth-estimation skills to localize the tool-tip with respect to the retina in order to perform the tool-navigation task, which can be prone to human error. To alleviate such uncertainty, prior work has introduced ways to assist the surgeon by estimating the tool-tip distance to the retina and providing haptic or auditory feedback. However, automating the tool-navigation task itself remains unsolved and largely unexplored. Such a capability, if reliably automated, could serve as a building block to streamline complex procedures and reduce the chance for tissue damage. Towards this end, we propose to automate the tool-navigation task by learning to mimic expert demonstrations of the task. Specifically, a deep network is trained to imitate expert trajectories toward various locations on the retina based on recorded visual servoing to a given goal specified by the user. The proposed autonomous navigation system is evaluated in simulation and in physical experiments using a silicone eye phantom. We show that the network can reliably navigate a needle surgical tool to various desired locations within 137 µm accuracy in physical experiments and 94 µm in simulation on average, and generalizes well to unseen situations such as in the presence of auxiliary surgical tools, variable eye backgrounds, and brightness conditions.

Preliminary study of an RNN-based active interventional robotic system (AIRS) in retinal microsurgery.

PURPOSE: Retinal microsurgery requires highly dexterous and precise maneuvering of instruments inserted into the eyeball through the sclerotomy port. During such procedures, the sclera can potentially be injured from extreme tool-to-sclera contact force caused by surgeon's unintentional misoperations. METHODS: We present an active interventional robotic system to prevent such iatrogenic accidents by enabling the robotic system to actively counteract the surgeon's possible unsafe operations in advance of their occurrence. Relying on a novel force sensing tool to measure and collect scleral forces, we construct a recurrent neural network with long short-term memory unit to oversee surgeon's operation and predict possible unsafe scleral forces up to the next 200 ms. We then apply a linear admittance control to actuate the robot to reduce the undesired scleral force. The system is implemented using an existing "steady hand" eye robot platform. The proposed method is evaluated on an artificial eye phantom by performing a "vessel following" mock retinal surgery operation. RESULTS: Empirical validation over multiple trials indicates that the proposed active interventional robotic system could help to reduce the number of unsafe manipulation events. CONCLUSIONS: We develop an active interventional robotic system to actively prevent surgeon's unsafe operations in retinal surgery. The result of the evaluation experiments shows that the proposed system can improve the surgeon's performance.

Enabling Technology for Safe Robot-Assisted Retinal Surgery: Early Warning for Unsafe Scleral Force.

Retinal microsurgery is technically demanding and requires high surgical skill with very little room for manipulation error. During surgery the tool needs to be inserted into the eyeball while maintaining constant contact with the sclera. Any unexpected manipulation could cause extreme tool-sclera contact force (scleral force) thus damage the sclera. The introduction of robotic assistance could enhance and expand the surgeon's manipulation capabilities during surgery. However, the potential intra-operative danger from surgeon's misoperations remains difficult to detect and prevent by existing robotic systems. Therefore, we propose a method to predict imminent unsafe manipulation in robot-assisted retinal surgery and generate feedback to the surgeon via auditory substitution. The surgeon could then react to the possible unsafe events in advance. This work specifically focuses on minimizing sclera damage using a force-sensing tool calibrated to measure small scleral forces. A recurrent neural network is designed and trained to predict the force safety status up to 500 milliseconds in the future. The system is implemented using an existing "steady hand" eye robot. A vessel following manipulation task is designed and performed on a dry eye phantom to emulate the retinal surgery and to analyze the proposed method. Finally, preliminary validation experiments are performed by five users, the results of which indicate that the proposed early warning system could help to reduce the number of unsafe manipulation events.

Adaptive Control of Sclera Force and Insertion Depth for Safe Robot-Assisted Retinal Surgery.

One of the significant challenges of moving from manual to robot-assisted retinal surgery is the loss of perception of forces applied to the sclera (sclera forces) by the surgical tools. This damping of force feedback is primarily due to the stiffness and inertia of the robot. The diminished perception of tool-to-eye interactions might put the eye tissue at high risk of injury due to excessive sclera forces or extreme insertion of the tool into the eye. In the present study therefore a 1-dimensional adaptive control method is customized for 3-dimensional control of sclera force components and tool insertion depth and then implemented on the velocity-controlled Johns Hopkins Steady-Hand Eye Robot. The control method enables the robot to perform autonomous motions to make the sclera force and/or insertion depth of the tool tip to follow pre-defined desired and safe trajectories when they exceed safe bounds. A robotic light pipe holding application in retinal surgery is also investigated using the adaptive control method. The implementation results indicate that the adaptive control is able to achieve the imposed safety margins and prevent sclera forces and insertion depth from exceeding safe boundaries.

Sclera Force Control in Robot-assisted Eye Surgery: Adaptive Force Control vs. Auditory Feedback.

Surgeon hand tremor limits human capability during microsurgical procedures such as those that treat the eye. In contrast, elimination of hand tremor through the introduction of microsurgical robots diminishes the surgeons tactile perception of useful and familiar tool-to-sclera forces. While the large mass and inertia of eye surgical robot prevents surgeon microtremor, loss of perception of small scleral forces may put the sclera at risk of injury. In this paper, we have applied and compared two different methods to assure the safety of sclera tissue during robot-assisted eye surgery. In the active control method, an adaptive force control strategy is implemented on the Steady-Hand Eye Robot in order to control the magnitude of scleral forces when they exceed safe boundaries. This autonomous force compensation is then compared to a passive force control method in which the surgeon performs manual adjustments in response to the provided audio feedback proportional to the magnitude of sclera force. A pilot study with three users indicate that the active control method is potentially more efficient.

Toward Sclera-Force-Based Robotic Assistance for Safe Micromanipulation in Vitreoretinal Surgery.

In vitreoretinal surgery instruments are inserted through the sclera to perform precise surgical maneuvers inside the eyeball, which exceeds typical human capabilities. Robotic assistance can enhance the skills of a novice surgeon, provide guidance during tool manipulation based on the desired behavior defined by expert surgeons' maneuvers, and consequently improve the surgical outcome. This paper presents an experimental study characterizing the safe/desired magnitude of forces between the surgical instrument and the sclera insertion port as a function of the tool insertion depth. We explore two types of regressions, a polynomial and a sum of sines fit, to describe the observed user behavior during our one-user pilot study, based on which a variable admittance control scheme can be implemented to robotically guide other users towards this desired behavior for a safe operation.