Visualization of micro-agents and surroundings by real-time multicolor fluorescence microscopy.

Optical microscopy techniques are a popular choice for visualizing micro-agents. They generate images with relatively high spatiotemporal resolution but do not reveal encoded information for distinguishing micro-agents and surroundings. This study presents multicolor fluorescence microscopy for rendering color-coded identification of mobile micro-agents and dynamic surroundings by spectral unmixing. We report multicolor microscopy performance by visualizing the attachment of single and cluster micro-agents to cancer spheroids formed with HeLa cells as a proof-of-concept for targeted drug delivery demonstration. A microfluidic chip is developed to immobilize a single spheroid for the attachment, provide a stable environment for multicolor microscopy, and create a 3D tumor model. In order to confirm that multicolor microscopy is able to visualize micro-agents in vascularized environments, in vitro vasculature network formed with endothelial cells and ex ovo chicken chorioallantoic membrane are employed as experimental models. Full visualization of our models is achieved by sequential excitation of the fluorophores in a round-robin manner and synchronous individual image acquisition from three-different spectrum bands. We experimentally demonstrate that multicolor microscopy spectrally decomposes micro-agents, organic bodies (cancer spheroids and vasculatures), and surrounding media utilizing fluorophores with well-separated spectrum characteristics and allows image acquisition with 1280 [Formula: see text] 1024 pixels up to 15 frames per second. Our results display that real-time multicolor microscopy provides increased understanding by color-coded visualization regarding the tracking of micro-agents, morphology of organic bodies, and clear distinction of surrounding media.

Serial imaging of micro-agents and cancer cell spheroids in a microfluidic channel using multicolor fluorescence microscopy.

Multicolor fluorescence microscopy is a powerful technique to fully visualize many biological phenomena by acquiring images from different spectrum channels. This study expands the scope of multicolor fluorescence microscopy by serial imaging of polystyrene micro-beads as surrogates for drug carriers, cancer spheroids formed using HeLa cells, and microfluidic channels. Three fluorophores with different spectral characteristics are utilized to perform multicolor microscopy. According to the spectrum analysis of the fluorophores, a multicolor widefield fluorescence microscope is developed. Spectral crosstalk is corrected by exciting the fluorophores in a round-robin manner and synchronous emitted light collection. To report the performance of the multicolor microscopy, a simplified 3D tumor model is created by placing beads and spheroids inside a channel filled with the cell culture medium is imaged at varying exposure times. As a representative case and a method for bio-hybrid drug carrier fabrication, a spheroid surface is coated with beads in a channel utilizing electrostatic forces under the guidance of multicolor microscopy. Our experiments show that multicolor fluorescence microscopy enables crosstalk-free and spectrally-different individual image acquisition of beads, spheroids, and channels with the minimum exposure time of 5.5 ms. The imaging technique has the potential to monitor drug carrier transportation to cancer cells in real-time.

Real-Time Multi-Modal Sensing and Feedback for Catheterization in Porcine Tissue.

OBJECTIVE: In this study, we introduce a multi-modal sensing and feedback framework aimed at assisting clinicians during endovascular surgeries and catheterization procedures. This framework utilizes state-of-the-art imaging and sensing sub-systems to produce a 3D visualization of an endovascular catheter and surrounding vasculature without the need for intra-operative X-rays. METHODS: The catheterization experiments within this study are conducted inside a porcine limb undergoing motions. A hybrid position-force controller of a robotically-actuated ultrasound (US) transducer for uneven porcine tissue surfaces is introduced. The tissue, vasculature, and catheter are visualized by integrated real-time US images, 3D surface imaging, and Fiber Bragg Grating (FBG) sensors. RESULTS: During externally-induced limb motions, the vasculature and catheter can be reliably reconstructed at mean accuracies of 1.9±0.3 mm and 0.82±0.21 mm, respectively. CONCLUSIONS: The conventional use of intra-operative X-ray imaging to visualize instruments and vasculature in the human body can be reduced by employing improved diagnostic technologies that do not operate via ionizing radiation or nephrotoxic contrast agents. SIGNIFICANCE: The presented multi-modal framework enables the radiation-free and accurate reconstruction of significant tissues and instruments involved in catheterization procedures.

A Contactless and Biocompatible Approach for 3D Active Microrobotic Targeted Drug Delivery.

As robotic tools are becoming a fundamental part of present day surgical interventions, microrobotic surgery is steadily approaching clinically-relevant scenarios. In particular, minimally invasive microrobotic targeted drug deliveries are reaching the grasp of the current state-of-the-art technology. However, clinically-relevant issues, such as lack of biocompatibility and dexterity, complicate the clinical application of the results obtained in controlled environments. Consequently, in this work we present a proof-of-concept fully contactless and biocompatible approach for active targeted delivery of a drug-model. In order to achieve full biocompatiblity and contacless actuation, magnetic fields are used for motion control, ultrasound is used for imaging, and induction heating is used for active drug-model release. The presented system is validated in a three-dimensional phantom of human vessels, performing ten trials that mimic targeted drug delivery using a drug-coated microrobot. The system is capable of closed-loop motion control with average velocity and positioning error of 0.3 mm/s and 0.4 mm, respectively. Overall, our findings suggest that the presented approach could augment the current capabilities of microrobotic tools, helping the development of clinically-relevant approaches for active in-vivo targeted drug delivery.

Hybrid control algorithm for flexible needle steering: Demonstration in phantom and human cadaver.

Needles are commonly used in the clinic for percutaneous procedures. The outcome of such procedures heavily depends on accurate placement of the needle. There are two main challenges to achieve high accuracy: First, aligning the needle with the targeted lesion, and second, compensating for the deflection of the needle in the tissue. In order to address these challenges, scientists have developed several robotic setups for needle steering. However, the subject is still under research and reliable implementations which can be used in clinical practice are not yet available. In this paper, we have taken some steps in order to bring needle steering closer to practice. A new hybrid control algorithm is developed, which enables us to control a flexible needle by combing base-manipulation and beveled-tip steering methods. A pre-operative path planner is developed which considers the clinical requirements. The proposed method is tested in the lung of a fresh-frozen human cadaver. The work-flow of the experiments are similar to the current clinical practice. Three experimental cases are used to evaluate the proposed steering algorithm. Experimental Case I shows that using the proposed steering algorithm controllability of the needle is increased. In Case II and Case III, the needle is steered in a gelatin phantom and a human cadaver, respectively. The targeting accuracy of 1.35±0.49mm in gelatin phantom and 1.97±0.89mm in cadave is achieved. A feasibility study is performed, in which a fine needle aspiration (FNA) needle is steered in the lungs of a human cadaver under computed tomography guidance. The targeting error for the feasibility study is 2.89±0.22mm. The results suggest that such a robotic system can be beneficial for clinical use and the patient receives less x-ray radiation.

Design, characterization and control of thermally-responsive and magnetically-actuated micro-grippers at the air-water interface.

The design and control of untethered microrobotic agents has drawn a lot of attention in recent years. This technology truly possesses the potential to revolutionize the field of minimally invasive surgery and microassembly. However, miniaturization and reliable actuation of micro-fabricated grippers are still challenging at sub-millimeter scale. In this study, we design, manufacture, characterize, and control four similarly-structured semi-rigid thermoresponsive micro-grippers. Furthermore, we develop a closed loop-control algorithm to demonstrate and compare the performance of the said grippers when moving in hard-to-reach and unpredictable environments. Finally, we analyze the grasping characteristics of three of the presented designs. Overall, not only does the study demonstrate motion control in unstructured dynamic environments-at velocities up to 3.4, 2.9, 3.3, and 1 body-lengths/s with 980, 750, 250, and 100 µm-sized grippers, respectively-but it also aims to provide quantitative data and considerations to help a targeted design of magnetically-controlled thin micro-grippers.

Autonomous planning and control of soft untethered grippers in unstructured environments.

The use of small, maneuverable, untethered and reconfigurable robots could provide numerous advantages in various micromanipulation tasks. Examples include microassembly, pick-and-place of fragile micro-objects for lab-on-a-chip applications, assisted hatching for in-vitro fertilization and minimally invasive surgery. This study assesses the potential of soft untethered magnetic grippers as alternatives or complements to conventional tethered or rigid micromanipulators. We demonstrate closed-loop control of untethered grippers and automated pick-and-place of biological material on porcine tissue in an unstructured environment. We also demonstrate the ability of the soft grippers to recognize and sort non-biological micro-scale objects. The fully autonomous nature of the experiments is made possible by the integration of planning and decision-making algorithms, as well as by closed-loop temperature and electromagnetic motion control. The grippers are capable of completing pick-and-place tasks of biological material at an average velocity of 1.8 ±0.71 mm/s and a drop-off error of 0.62 ±0.22 mm. Color-sensitive sorting of three micro-scale objects is completed at a velocity of 1.21 ±0.68 mm/s and a drop-off error of 0.85 ±0.41 mm. Our findings suggest that improved autonomous untethered grippers could augment the capabilities of current soft-robotic instruments especially in advancedtasks involving manipulation.

Computed tomography (CT)-compatible remote center of motion needle steering robot: Fusing CT images and electromagnetic sensor data.

Lung cancer is the most common cause of cancer-related death, and early detection can reduce the mortality rate. Patients with lung nodules greater than 10 mm usually undergo a computed tomography (CT)-guided biopsy. However, aligning the needle with the target is difficult and the needle tends to deflect from a straight path. In this work, we present a CT-compatible robotic system, which can both position the needle at the puncture point and also insert and rotate the needle. The robot has a remote-center-of-motion arm which is achieved through a parallel mechanism. A new needle steering scheme is also developed where CT images are fused with electromagnetic (EM) sensor data using an unscented Kalman filter. The data fusion allows us to steer the needle using the real-time EM tracker data. The robot design and the steering scheme are validated using three experimental cases. Experimental Case I and II evaluate the accuracy and CT-compatibility of the robot arm, respectively. In experimental Case III, the needle is steered towards 5 real targets embedded in an anthropomorphic gelatin phantom of the thorax. The mean targeting error for the 5 experiments is 1.78 ± 0.70 mm. The proposed robotic system is shown to be CT-compatible with low targeting error. Small nodule size and large needle diameter are two risk factors that can lead to complications in lung biopsy. Our results suggest that nodules larger than 5 mm in diameter can be targeted using our method which may result in lower complication rate.

Model predictive control of a robotically actuated delivery sheath for beating heart compensation.

Minimally invasive surgery (MIS) during cardiovascular interventions reduces trauma and enables the treatment of high-risk patients who were initially denied surgery. However, restricted access, reduced visibility and control of the instrument at the treatment locations limits the performance and capabilities of such interventions during MIS. Therefore, the demand for technology such as steerable sheaths or catheters that assist the clinician during the procedure is increasing. In this study, we present and evaluate a robotically actuated delivery sheath (RADS) capable of autonomously and accurately compensating for beating heart motions by using a model-predictive control (MPC) strategy. We develop kinematic models and present online ultrasound segmentation of the RADS that are integrated with the MPC strategy. As a case study, we use pre-operative ultrasound images from a patient to extract motion profiles of the aortic heart valve (AHV). This allows the MPC strategy to anticipate for AHV motions. Further, mechanical hysteresis in the steering mechanism is compensated for in order to improve tip positioning accuracy. The novel integrated system is capable of controlling the articulating tip of the RADS to assist the clinician during cardiovascular surgery. Experiments demonstrate that the RADS follows the AHV motion with a mean positioning error of 1.68 mm. The presented modelling, imaging and control framework could be adapted and applied to a range of continuum-style robots and catheters for various cardiovascular interventions.

Stimuli-Responsive Soft Untethered Grippers for Drug Delivery and Robotic Surgery.

Untethered microtools that can be precisely navigated into deep in vivo locations are important for clinical procedures pertinent to minimally invasive surgery and targeted drug delivery. In this mini-review, untethered soft grippers are discussed, with an emphasis on a class of autonomous stimuli-responsive gripping soft tools that can be used to excise tissues and release drugs in a controlled manner. The grippers are composed of polymers and hydrogels and are thus compliant to soft tissues. They can be navigated using magnetic fields and controlled by robotic path-planning strategies to carry out tasks like pick-and-place of microspheres and biological materials either with user assistance, or in a fully autonomous manner. It is envisioned that the use of these untethered soft grippers will translate from laboratory experiments to clinical scenarios and the challenges that need to be overcome to make this transition are discussed.

Segmentation and three-dimensional reconstruction of lesions using the automated breast volume scanner (ABVS).

BACKGROUND: Ultrasound is an effective tool for breast cancer diagnosis. However, its relatively low image quality makes small lesion analysis challenging. This promotes the development of tools to help clinicians in the diagnosis. METHODS: We propose a method for segmentation and three-dimensional (3D) reconstruction of lesions from ultrasound images acquired using the automated breast volume scanner (ABVS). Segmentation and reconstruction algorithms are applied to obtain the lesion's 3D geometry. A total of 140 artificial lesions with different sizes and shapes are reconstructed in gelatin-based phantoms and biological tissue. Dice similarity coefficient (DSC) is used to evaluate the reconstructed shapes. The algorithm is tested using a human breast phantom and clinical data from six patients. RESULTS: DSC values are 0.86 ± 0.06 and 0.86 ± 0.05 for gelatin-based phantoms and biological tissue, respectively. The results are validated by a specialized clinician. CONCLUSIONS: Evaluation metrics show that the algorithm accurately segments and reconstructs various lesions. Copyright © 2016 John Wiley & Sons, Ltd.

Experimental evaluation of co-manipulated ultrasound-guided flexible needle steering.

BACKGROUND: A teleoperation system for bevel-tipped flexible needle steering has been evaluated. Robotic systems have been exploited as the main tool to achieve high accuracy and reliability. However, for reasons of safety and acceptance by the surgical community, keeping the physician tightly in the loop is preferable. METHODS: The system uses ultrasound imaging, path planning, and control to compute the desired needle orientation during the insertion and intuitively passes this information to the operator, who teleoperates the motion of the needle's tip. Navigation cues about the computed orientation are provided through haptic and visual feedback to the operator to steer the needle. RESULTS: The targeting accuracy of several co-manipulation strategies were studied in four sets of experiments involving human subjects with clinical backgrounds. CONCLUSIONS: Experimental results show that receiving feedback regarding the desired needle orientation improves the targeting accuracy by a factor of 9 with respect to manual insertions. Copyright © 2015 John Wiley & Sons, Ltd.

Design and evaluation of a computed tomography (CT)-compatible needle insertion device using an electromagnetic tracking system and CT images.

PURPOSE: Percutaneous needle insertion procedures are commonly used for diagnostic and therapeutic purposes. Although current technology allows accurate localization of lesions, they cannot yet be precisely targeted. Lung cancer is the most common cause of cancer-related death, and early detection reduces the mortality rate. Therefore, suspicious lesions are tested for diagnosis by performing needle biopsy. METHODS: In this paper, we have presented a novel computed tomography (CT)-compatible needle insertion device (NID). The NID is used to steer a flexible needle (φ0.55 mm) with a bevel at the tip in biological tissue. CT images and an electromagnetic (EM) tracking system are used in two separate scenarios to track the needle tip in three-dimensional space during the procedure. Our system uses a control algorithm to steer the needle through a combination of insertion and minimal number of rotations. RESULTS: Noise analysis of CT images has demonstrated the compatibility of the device. The results for three experimental cases (case 1: open-loop control, case 2: closed-loop control using EM tracking system and case 3: closed-loop control using CT images) are presented. Each experimental case is performed five times, and average targeting errors are 2.86 ± 1.14, 1.11 ± 0.14 and 1.94 ± 0.63 mm for case 1, case 2 and case 3, respectively. CONCLUSIONS: The achieved results show that our device is CT-compatible and it is able to steer a bevel-tipped needle toward a target. We are able to use intermittent CT images and EM tracking data to control the needle path in a closed-loop manner. These results are promising and suggest that it is possible to accurately target the lesions in real clinical procedures in the future.

Closed-loop asymmetric-tip needle steering under continuous intraoperative MRI guidance.

Magnetic resonance imaging (MRI) provides excellent image contrast for various types of tissues, making it a suitable choice over other imaging modalities for various image-guided needle interventions. Furthermore, robot-assistance is maturing for surgical procedures such as percutaneous prostate and brain interventions. Although MRI-guided, robot-assisted needle interventions are approaching clinical usage, they are still typically open-loop in nature due to the lack of continuous intraoperative needle tracking. Closed-loop needle-based procedures can improve the accuracy of needle tip placement by correcting the needle trajectory during insertion. This paper proposes a system for robot-assisted, flexible asymmetric-tipped needle interventions under continuous intraoperative MRI guidance. A flexible needle's insertion depth and rotation angle are manipulated by an MRI-compatible robot in the bore of the MRI scanner during continuous multi-planar image acquisition to reach a desired target location. Experiments are performed on gelatin phantoms to assess the accuracy of needle placement into the target location. The system was able to successfully utilize live MR imaging to guide the path of the needle, and results show an average total targeting error of 2.5±0.47mm, with an average in-plane error of 2.09±0.33mm.

Biomechanics-Based Curvature Estimation for Ultrasound-guided Flexible Needle Steering in Biological Tissues.

Needle-based procedures are commonly performed during minimally invasive surgery for treatment and diagnosis. Accurate needle tip placement is important for the success of the procedures. Misplacement of the needle tip might cause unsuccessful treatment or misdiagnosis. Robot-assisted needle insertion systems have been developed in order to steer flexible bevel-tipped needles. However, current systems depend on the information of maximum needle curvature, which is estimated by performing prior insertions. This work presents a new three-dimensional flexible needle steering system which integrates an optimal steering control, ultrasound-based needle tracking system, needle deflection model, online needle curvature estimation and offline curvature estimation based on biomechanics properties. The online and the offline curvature estimations are used to update the steering control in real time. The system is evaluated by experiments in gelatin phantoms and biological tissues (chicken breast tissues). The average targeting error in gelatin phantoms is 0.42 ± 0.17 mm, and in biological tissues is 1.63 ± 0.29 mm. The system is able to accurately steer a flexible needle in multi-layer phantoms and biological tissues without performing prior insertions to estimate the maximum needle curvature.

Ultrasound-guided three-dimensional needle steering in biological tissue with curved surfaces.

In this paper, we present a system capable of automatically steering a bevel-tipped flexible needle under ultrasound guidance toward a physical target while avoiding a physical obstacle embedded in gelatin phantoms and biological tissue with curved surfaces. An ultrasound pre-operative scan is performed for three-dimensional (3D) target localization and shape reconstruction. A controller based on implicit force control is developed to align the transducer with curved surfaces to assure the maximum contact area, and thus obtain an image of sufficient quality. We experimentally investigate the effect of needle insertion system parameters such as insertion speed, needle diameter and bevel angle on target motion to adjust the parameters that minimize the target motion during insertion. A fast sampling-based path planner is used to compute and periodically update a feasible path to the target that avoids obstacles. We present experimental results for target reconstruction and needle insertion procedures in gelatin-based phantoms and biological tissue. Mean targeting errors of 1.46±0.37 mm, 1.29±0.29 mm and 1.82±0.58 mm are obtained for phantoms with inclined, curved and combined (inclined and curved) surfaces, respectively, for insertion distance of 86-103 mm. The achieved targeting errors suggest that our approach is sufficient for targeting lesions of 3mm radius that can be detected using clinical ultrasound imaging systems.

Teleoperation of steerable flexible needles by combining kinesthetic and vibratory feedback.

Needle insertion in soft-tissue is a minimally invasive surgical procedure that demands high accuracy. In this respect, robotic systems with autonomous control algorithms have been exploited as the main tool to achieve high accuracy and reliability. However, for reasons of safety and responsibility, autonomous robotic control is often not desirable. Therefore, it is necessary to focus also on techniques enabling clinicians to directly control the motion of the surgical tools. In this work, we address that challenge and present a novel teleoperated robotic system able to steer flexible needles. The proposed system tracks the position of the needle using an ultrasound imaging system and computes needle's ideal position and orientation to reach a given target. The master haptic interface then provides the clinician with mixed kinesthetic-vibratory navigation cues to guide the needle toward the computed ideal position and orientation. Twenty participants carried out an experiment of teleoperated needle insertion into a soft-tissue phantom, considering four different experimental conditions. Participants were provided with either mixed kinesthetic-vibratory feedback or mixed kinesthetic-visual feedback. Moreover, we considered two different ways of computing ideal position and orientation of the needle: with or without set-points. Vibratory feedback was found more effective than visual feedback in conveying navigation cues, with a mean targeting error of 0.72 mm when using set-points, and of 1.10 mm without set-points.

Experimental evaluation of ultrasound-guided 3D needle steering in biological tissue.

PURPOSE: In this paper, we present a system capable of automatically steering bevel tip flexible needles under ultrasound guidance toward stationary and moving targets in gelatin phantoms and biological tissue while avoiding stationary and moving obstacles. We use three-dimensional (3D) ultrasound to track the needle tip during the procedure. METHODS: Our system uses a fast sampling-based path planner to compute and periodically update a feasible path to the target that avoids obstacles. We then use a novel control algorithm to steer the needle along the path in a manner that reduces the number of needle rotations, thus reducing tissue damage. We present experimental results for needle insertion procedures for both stationary and moving targets and obstacles for up to 90 mm of needle insertion. RESULTS: We obtained a mean targeting error of [Formula: see text] and [Formula: see text] mm in gelatin-based phantom and biological tissue, respectively. CONCLUSIONS: The achieved submillimeter accuracy suggests that our approach is sufficient to target the smallest lesions ([Formula: see text] 2 mm) that can be detected using state-of-the-art ultrasound imaging systems.

Needle path planning and steering in a three-dimensional non-static environment using two-dimensional ultrasound images.

Needle insertion is commonly performed in minimally invasive medical procedures such as biopsy and radiation cancer treatment. During such procedures, accurate needle tip placement is critical for correct diagnosis or successful treatment. Accurate placement of the needle tip inside tissue is challenging, especially when the target moves and anatomical obstacles must be avoided. We develop a needle steering system capable of autonomously and accurately guiding a steerable needle using two-dimensional (2D) ultrasound images. The needle is steered to a moving target while avoiding moving obstacles in a three-dimensional (3D) non-static environment. Using a 2D ultrasound imaging device, our system accurately tracks the needle tip motion in 3D space in order to estimate the tip pose. The needle tip pose is used by a rapidly exploring random tree-based motion planner to compute a feasible needle path to the target. The motion planner is sufficiently fast such that replanning can be performed repeatedly in a closed-loop manner. This enables the system to correct for perturbations in needle motion, and movement in obstacle and target locations. Our needle steering experiments in a soft-tissue phantom achieves maximum targeting errors of 0.86 ± 0.35 mm (without obstacles) and 2.16 ± 0.88 mm (with a moving obstacle).