Real-Time Endomicroscopic Image Mosaicking with an EKF-based Sensor Fusion Approach.

This study presents a real-time sensor fusion framework based on the extended Kalman filter (EKF) for accurate and robust endomicroscopic image mosaicking. The sensor fusion framework incorporates an optical tracking system that can track 6-DOF pose of the imaging probe with high accuracy in real time in conjunction with 2D local image registration from image feature matching between two consecutive frames. We evaluated the performance of the real-time image mosaicking based on the sensor fusion compared with the image or tracker only approach. As a result, it could retain the microscopic level of image detail from the image-based approach and also achieve a robust image mosaic without any drift by using the accurate optical tracking system.

NO Synthesis but Not Apoptosis, Mitosis or Inflammation Can Explain Correlations between Flow Directionality and Paracellular Permeability of Cultured Endothelium.

Haemodynamic wall shear stress varies from site to site within the arterial system and is thought to cause local variation in endothelial permeability to macromolecules. Our aim was to investigate mechanisms underlying the changes in paracellular permeability caused by different patterns of shear stress in long-term culture. We used the swirling well system and a substrate-binding tracer that permits visualisation of transport at the cellular level. Permeability increased in the centre of swirled wells, where flow is highly multidirectional, and decreased towards the edge, where flow is more uniaxial, compared to static controls. Overall, there was a reduction in permeability. There were also decreases in early- and late-stage apoptosis, proliferation and mitosis, and there were significant correlations between the first three and permeability when considering variation from the centre to the edge under flow. However, data from static controls did not fit the same relation, and a cell-by-cell analysis showed that <5% of uptake under shear was associated with each of these events. Nuclear translocation of NF-κB p65 increased and then decreased with the duration of applied shear, as did permeability, but the spatial correlation between them was not significant. Application of an NO synthase inhibitor abolished the overall decrease in permeability caused by chronic shear and the difference in permeability between the centre and the edge of the well. Hence, shear and paracellular permeability appear to be linked by NO synthesis and not by apoptosis, mitosis or inflammation. The effect was mediated by an increase in transport through tricellular junctions.

In Vivo Cellular-Level 3D Imaging of Peripheral Nerves Using a Dual-Focusing Technique for Intra-Neural Interface Implantation.

In vivo volumetric imaging of the microstructural changes of peripheral nerves with an inserted electrode could be key for solving the chronic implantation failure of an intra-neural interface necessary to provide amputated patients with natural motion and sensation. Thus far, no imaging devices can provide a cellular-level three-dimensional (3D) structural images of a peripheral nerve in vivo. In this study, an optical coherence tomography-based peripheral nerve imaging platform that employs a newly proposed depth of focus extension technique is reported. A point spread function with the finest transverse resolution of 1.27 µm enables the cellular-level volumetric visualization of the metal wire and microstructural changes in a rat sciatic nerve with the metal wire inserted in vivo. Further, the feasibility of applying the imaging platform to large animals for a preclinical study is confirmed through in vivo rabbit sciatic nerve imaging. It is expected that new possibilities for the successful chronic implantation of an intra-neural interface will open up by providing the 3D microstructural changes of nerves around the inserted electrode.

Honeycomb Artifact Removal Using Convolutional Neural Network for Fiber Bundle Imaging.

We present a new deep learning framework for removing honeycomb artifacts yielded by optical path blocking of cladding layers in fiber bundle imaging. The proposed framework, HAR-CNN, provides an end-to-end mapping from a raw fiber bundle image to an artifact-free image via a convolution neural network (CNN). The synthesis of honeycomb patterns on ordinary images allows conveniently learning and validating the network without the enormous ground truth collection by extra hardware setups. As a result, HAR-CNN shows significant performance improvement in honeycomb pattern removal and also detailed preservation for the 1961 USAF chart sample, compared with other conventional methods. Finally, HAR-CNN is GPU-accelerated for real-time processing and enhanced image mosaicking performance.

Evaluation of Applied Force During Nasopharyngeal Swab Sampling Using Handheld Sensorized Instrument.

Nasopharyngeal swab is the most widely used diagnostic test for COVID-19 detection. However, enormous tests have posed a high risk of infection to medical professionals due to close contact with patients and substantial health burden. While automation of the nasopharyngeal swab is regarded as a potential solution to address these problems, the quantitative study of force for safe and effective control has not been widely performed yet. Hence, this study presents applied force during the standard nasopharyngeal swab sampling procedure using a handheld sensorized instrument. The sensorized instrument can simultaneously measure multi-axis forces and 6-DOF hand motion while allowing natural hand motion as is used in the standard swab sampling. To accurately measure force from the handheld instrument, the compensation of gravity bias is accomplished online while estimating the orientation of the hand with an embedded IMU sensor. As a result, the instrument can measure all three-axes forces by an error below 5 mN. A simulated test on a phantom model using the sensorized instrument shows that how the forces vary during the sampling sequences.

One-step Implantation of a 3D Neural Microelectrode Array.

This paper presents a neurosurgical device called NEIT 2 (Nerve Electrode Insertion Tool) to implant a 3D microelectrode array into a peripheral nervous system. Using an elastomer-made nerve holder, the device is able to stable target a flexible nerve, and then safely inserts an electrode array into the fixed nerve. Finally, a nerve containment assembly is made at once. We conducted animal experiments to evaluate the proposed scenario using a 3D printed prototype and commercial microelectrodes. The results show that microelectrodes are successfully implanted into sciatic nerves of rats and neural signals are recorded through the chronically implanted electrodes.

Ultra-high curvature sensors for multi-bend structures using fiber Bragg gratings.

Measuring high curvatures is essential in various applications such as structural engineering, medical treatment, and robotics. Herein, we present a novel ultra-high curvature sensor with an ultra-thin and highly flexible structure incorporating fiber Bragg gratings (FBGs). The sensor can measure curvature of bidirectional bending up to 200 m-1. In addition, the multi-bend curvature sensor in conjunction with the proposed calibration scheme enables the accurate reconstruction of a curve with varying curvature. The sensitivity and the accuracy of the curvature sensor are investigated for different sensor designs. Finally, we demonstrate the accurate shape sensing of various 2-D patterns using the multi-bend curvatures sensor.

Towards Finger Motion Capture System Using FBG Sensors.

This paper introduces a novel finger motion capture system using FBG (fiber Bragg grating) optical sensors. We develop two types of sensors to seamlessly reconstruct finger motion from strains induced in the FBGs. First, the shape sensor incorporates three optical fibers with multiple FBGs to reconstruct the position and orientation of a finger joint in 3D. In addition, the angle sensor is designed to measure the high curvature of bending on the finger joints. By deploying the two types of sensors on the fingers, we can reconstruct various finger motion in real time without drift over time. The accuracies of the fabricated FBG sensors are evaluated, resulting in an average error of 1.49 mm for the shape sensor at the distal tip (1.9% for the full length of the sensor) and 0.21° error for the angle sensor. We finally demonstrate finger motion tracking with the FBG sensors in real time, while measuring the multi-DOF motion at the carpometacarpal joint of the thumb and also the high curvatures of bending motion at the metacarpophalangeal and interphalangeal joints of the thumb and the index finger.

Techniques for robot-aided intraocular surgery using monocular vision.

This paper presents techniques for robot-aided intraocular surgery using monocular vision in order to overcome erroneous stereo reconstruction in an intact eye. We propose a new retinal surface estimation method based on a structured-light approach. A handheld robot known as the Micron enables automatic scanning of a laser probe, creating projected beam patterns on the retinal surface. Geometric analysis of the patterns then allows planar reconstruction of the surface. To realize automated surgery in an intact eye, monocular hybrid visual servoing is accomplished through a scheme that incorporates surface reconstruction and partitioned visual servoing. We investigate the sensitivity of the estimation method according to relevant parameters and also evaluate its performance in both dry and wet conditions. The approach is validated through experiments for automated laser photocoagulation in a realistic eye phantom in vitro. Finally, we present the first demonstration of automated intraocular laser surgery in porcine eyes ex vivo.

EyeSLAM: Real-time simultaneous localization and mapping of retinal vessels during intraocular microsurgery.

BACKGROUND: Fast and accurate mapping and localization of the retinal vasculature is critical to increasing the effectiveness and clinical utility of robot-assisted intraocular microsurgery such as laser photocoagulation and retinal vessel cannulation. METHODS: The proposed EyeSLAM algorithm delivers 30 Hz real-time simultaneous localization and mapping of the human retina and vasculature during intraocular surgery, combining fast vessel detection with 2D scan-matching techniques to build and localize a probabilistic map of the vasculature. RESULTS: In the harsh imaging environment of retinal surgery with high magnification, quick shaky motions, textureless retina background, variable lighting and tool occlusion, EyeSLAM can map 75% of the vessels within two seconds of initialization and localize the retina in real time with a root mean squared (RMS) error of under 5.0 pixels (translation) and 1° (rotation). CONCLUSIONS: EyeSLAM robustly provides retinal maps and registration that enable intelligent surgical micromanipulators to aid surgeons in simulated retinal vessel tracing and photocoagulation tasks.

Toward Monocular Camera-Guided Retinal Vein Cannulation with an Actively Stabilized Handheld Robot.

In this paper we describe work towards retinal vessel cannulation using an actively stabilized handheld robot, guided by monocular vision. We employ a previously developed monocular camera based surface reconstruction method using automated laser beam scanning over the retina. We use the reconstructed plane to find a coordinate transform between the 2D image plane coordinate system and the global 3D frame. Within a hemispherical region around the target, we use motion scaling for higher precision. The contribution of this work is the homography matrix estimation using monocular vision and application of the previously developed laser surface reconstruction to Micron guided vein cannulation. Experiments are conducted in a wet eye phantom to show the higher accuracy of the surface reconstruction as compared to standard stereo reconstruction. Further, experiments to show the increased surgical accuracy due to motion scaling are also carried out.

Visualization of three pathways for macromolecule transport across cultured endothelium and their modification by flow.

Transport of macromolecules across vascular endothelium and its modification by fluid mechanical forces are important for normal tissue function and in the development of atherosclerosis. However, the routes by which macromolecules cross endothelium, the hemodynamic stresses that maintain endothelial physiology or trigger disease, and the dependence of transendothelial transport on hemodynamic stresses are controversial. We visualized pathways for macromolecule transport and determined the effect on these pathways of different types of flow. Endothelial monolayers were cultured under static conditions or on an orbital shaker producing different flow profiles in different parts of the wells. Fluorescent tracers that bound to the substrate after crossing the endothelium were used to identify transport pathways. Maps of tracer distribution were compared with numerical simulations of flow to determine effects of different shear stress metrics on permeability. Albumin-sized tracers dominantly crossed the cultured endothelium via junctions between neighboring cells, high-density lipoprotein-sized tracers crossed at tricellular junctions, and low-density lipoprotein-sized tracers crossed through cells. Cells aligned close to the angle that minimized shear stresses across their long axis. The rate of paracellular transport under flow correlated with the magnitude of these minimized transverse stresses, whereas transport across cells was uniformly reduced by all types of flow. These results contradict the long-standing two-pore theory of solute transport across microvessel walls and the consensus view that endothelial cells align with the mean shear vector. They suggest that endothelial cells minimize transverse shear, supporting its postulated proatherogenic role. Preliminary data show that similar tracer techniques are practicable in vivo.NEW & NOTEWORTHY Solutes of increasing size crossed cultured endothelium through intercellular junctions, through tricellular junctions, or transcellularly. Cells aligned to minimize the shear stress acting across their long axis. Paracellular transport correlated with the level of this minimized shear, but transcellular transport was reduced uniformly by flow regardless of the shear profile.

Comparative Evaluation of Handheld Robot-Aided Intraocular Laser Surgery.

This paper presents robot-aided intraocular laser surgery using a handheld robot known as Micron. The micromanipulator incorporated in Micron enables visual servoing of a laser probe, while maintaining a constant distance of the tool tip from the retinal surface. The comparative study was conducted with various control methods for evaluation of robot-aided intraocular laser surgery.

Hybrid position/force control of an active handheld micromanipulator for membrane peeling.

BACKGROUND: Peeling procedures in retinal surgery require micron-scale manipulation and control of sub-tactile forces. METHODS: Hybrid position/force control of an actuated handheld microsurgical instrument is presented as a means for simultaneously improving positioning accuracy and reducing forces to prevent avoidable trauma to tissue. The system response was evaluated, and membrane-peeling trials were performed by four test subjects in both artificial and animal models. RESULTS: Maximum force was reduced by 56% in both models compared with position control. No statistically significant effect on procedure duration was observed. CONCLUSIONS: A hybrid position/force control system has been implemented that successfully attenuates forces and minimizes unwanted excursions during microsurgical procedures such as membrane peeling. Results also suggest that improvements in safety using this technique may be attained without increasing the duration of the procedure.

Handheld-automated microsurgical instrumentation for intraocular laser surgery.

BACKGROUND AND OBJECTIVE: Laser photocoagulation is a mainstay or adjuvant treatment for a variety of common retinal diseases. Automated laser photocoagulation during intraocular surgery has not yet been established. The authors introduce an automated laser photocoagulation system for intraocular surgery, based on a novel handheld instrument. The goals of the system are to enhance accuracy and efficiency and improve safety. MATERIALS AND METHODS: Triple-ring patterns are introduced as a typical arrangement for the treatment of proliferative retinopathy and registered to a preoperative fundus image. In total, 32 target locations are specified along the circumferences of three rings having diameters of 1, 2, and 3 mm, with a burn spacing of 600 µm. Given the initial system calibration, the retinal surface is reconstructed using stereo vision, and the targets specified on the preoperative image are registered with the control system. During automated operation, the laser probe attached to the manipulator of the active handheld instrument is deflected as needed via visual servoing in order to correct the error between the aiming beam and a specified target, regardless of any erroneous handle motion by the surgeon. A constant distance of the laser probe from the retinal surface is maintained in order to yield consistent size of burns and ensure safety during operation. Real-time tracking of anatomical features enables compensation for any movement of the eye. A graphical overlay system within operating microscope provides the surgeon with guidance cues for automated operation. Two retinal surgeons performed automated and manual trials in an artificial model of the eye, with each trial repeated three times. For the automated trials, various targeting thresholds (50-200 µm) were used to automatically trigger laser firing. In manual operation, fixed repetition rates were used, with frequencies of 1.0-2.5 Hz. The power of the 532 nm laser was set at 3.0 W with a duration of 20 ms. After completion of each trial, the speed of operation and placement error of burns were measured. The performance of the automated laser photocoagulation was compared with manual operation, using interpolated data for equivalent firing rates from 1.0 to 1.75 Hz. RESULTS: In automated trials, average error increased from 45 ± 27 to 60 ± 37 µm as the targeting threshold varied from 50 to 200 µm, while average firing rate significantly increased from 0.69 to 1.71 Hz. The average error in the manual trials increased from 102 ± 67 to 174 ± 98 µm as firing rate increased from 1.0 to 2.5 Hz. Compared to the manual trials, the average error in the automated trials was reduced by 53.0-56.4%, resulting in statistically significant differences (P ≤ 10(-20) ) for all equivalent frequencies (1.0-1.75 Hz). The depth of the laser tip in the automated trials was consistently maintained within 18 ± 2 µm root-mean-square (RMS) of its initial position, whereas it significantly varied in the manual trials, yielding an error of 296 ± 30 µm RMS. At high firing rates in manual trials, such as at 2.5 Hz, laser photocoagulation is marginally attained, yielding failed burns of 30% over the entire pattern, whereas no failed burns are found in automated trials. Relatively regular burn sizes are attained in the automated trials by the depth servoing of the laser tip, while burn sizes in the manual trials vary considerably. Automated avoidance of blood vessels was also successfully demonstrated, utilizing the retina-tracking feature to identify avoidance zones. CONCLUSION: Automated intraocular laser surgery can improve the accuracy of photocoagulation while ensuring safety during operation. This paper provides an initial demonstration of the technique under reasonably realistic laboratory conditions; development of a clinically applicable system requires further work.

Manipulator Design and Operation for a Six-Degree-of-Freedom Handheld Tremor-Canceling Microsurgical Instrument.

This paper presents the design and actuation of a six-degree-of-freedom (6-DOF) manipulator for a handheld instrument, known as "Micron," which performs active tremor compensation during microsurgery. The design incorporates a Gough-Stewart platform based on piezoelectric linear motor, with a specified minimum workspace of a cylinder 4 mm long and 4 mm in diameter at the end-effector. Given the stall force of the motors and the loading typically encountered in vitreoretinal microsurgery, the dimensions of the manipulator are optimized to tolerate a transverse load of 0.2 N on a remote center of motion near the midpoint of the tool shaft. The optimization yields a base diameter of 23 mm and a height of 37 mm. The fully handheld instrument includes a custom-built optical tracking system for control feedback, and an ergonomic housing to serve as a handle. The manipulation performance was investigated in both clamped and handheld conditions. In positioning experiments with varying side loads, the manipulator tolerates side load up to 0.25 N while tracking a sinusoidal target trajectory with less than 20 µm error. Physiological hand tremor is reduced by about 90% in a pointing task, and error less than 25 µm is achieved in handheld circle-tracing.

Design of endoscopic micro-robotic end effectors: safety and performance evaluation based on physical intestinal tissue damage characteristics.

During the last several years, legged locomotive mechanism has been considered as one of the main self-propelling mechanisms for future endoscopic microrobots due to its superior propulsion efficiency of an endoscopic microrobot inside the intestinal track. Nevertheless, its clinical application has been largely limited since the legged locomotive mechanism utilizes an end effector which has a sharp tip to generate sufficient traction by physically penetrating and interlocking with the intestinal tissue. This can cause excessive physical tissue damage or even complete perforation of the intestinal wall that can lead to abdominal inflammation. Hence, in this work two types of new end effectors, penetration-limited end effector (PLEE) and bi-material structured end effector (BMEE) were specially designed to acquire high medical safety as well as effective traction generation performance. The microscopic end effector specimens were fabricated with micro-wire electric discharge machining process. Traction generation performance of the end effectors was evaluated by direct measurement of resistance forces during contact-sliding tests using a custom-built contact-sliding tester. The safety of the end effector design was evaluated by examination of microscopic intestinal tissue damage using a scanning electron microscope (SEM). Physical damage characteristics of the intestinal tissue and related contact physics of the end effectors were discussed. From the results, the end effectors were evaluated with respect to their prospects in future medical applications as safe end effectors as well as micro-surgical tools.

Toward Automated Intraocular Laser Surgery Using a Handheld Micromanipulator.

This paper presents a technique for automated intraocular laser surgery using a handheld micromanipulator known as Micron. The novel handheld manipulator enables the automated scanning of a laser probe within a cylinder of 4 mm long and 4 mm in diameter. For the automation, the surface of the retina is reconstructed using a stereomicroscope, and then preplanned targets are placed on the surface. The laser probe is precisely located on the target via visual servoing of the aiming beam, while maintaining a specific distance above the surface. In addition, the system is capable of tracking the surface of the eye in order to compensate for any eye movement introduced during the operation. We compared the performance of the automated scanning using various control thresholds, in order to find the most effective threshold in terms of accuracy and speed. Given the selected threshold, we conducted the handheld operation above a fixed target surface. The average error and execution time are reduced by 63.6% and 28.5%, respectively, compared to the unaided trials. Finally, the automated laser photocoagulation was demonstrated also in an eye phantom, including compensation for the eye movement.

Performance of a 6-degree-of-freedom active microsurgical manipulator in handheld tasks.

This paper presents the first experimental results from human users of a new 6-degree-of-freedom handheld micromanipulator. This is the latest prototype of a fully-handheld system, known as "Micron," which performs active compensation of hand tremor for microsurgery. The manipulator is a miniature Gough-Stewart platform incorporating linear ultrasonic motors that provide a cylindrical workspace 4 mm long and 4 mm wide. In addition, the platform allows the possibility of imposing a remote center of motion for controlling motion not only at the tip but also at the entry point in the sclera of the eye. We demonstrate hand tremor reduction in both static and dynamic micromanipulation tasks on a rubber pad. The handheld performance is also evaluated in an artificial eye model while imposing a remote center of motion. In all cases, hand tremor is significantly reduced.

Improvement of optical coherence tomography using active handheld micromanipulator in vitreoretinal surgery.

An active handheld micromanipulator has been developed to cancel hand tremor during microsurgery. The micromanipulator is also applicable in optical coherence tomography to improve the quality of scanning and minimize surgical risks during the scans. The manipulator can maneuver the tool tip with six degrees of freedom within a cylindrical workspace 4 mm in diameter and 4 mm high. The imaging system is equipped with a 25-gauge Fourier-domain common-path OCT probe. This paper introduces the handheld OCT imaging system and techniques involved and presents stabilized OCT images of A-mode and M-mode scans in air and live rabbit eyes. We show the first demonstration of OCT imaging using the active handheld micromanipulator in vivo.