Convolutional neural network-based common-path optical coherence tomography A-scan boundary-tracking training and validation using a parallel Monte Carlo synthetic dataset.

We present a parallel Monte Carlo (MC) simulation platform for rapidly generating synthetic common-path optical coherence tomography (CP-OCT) A-scan image dataset for image-guided needle insertion. The computation time of the method has been evaluated on different configurations and 100000 A-scan images are generated based on 50 different eye models. The synthetic dataset is used to train an end-to-end convolutional neural network (Ascan-Net) to localize the Descemet's membrane (DM) during the needle insertion. The trained Ascan-Net has been tested on the A-scan images collected from the ex-vivo human and porcine cornea as well as simulated data and shows improved tracking accuracy compared to the result by using the Canny-edge detector.

Deep point cloud landmark localization for fringe projection profilometry.

Point clouds have been widely used due to their information being richer than images. Fringe projection profilometry (FPP) is one of the camera-based point cloud acquisition techniques that is being developed as a vision system for robotic surgery. For semi-autonomous robotic suturing, fluorescent fiducials were previously used on a target tissue as suture landmarks. This not only increases system complexity but also imposes safety concerns. To address these problems, we propose a numerical landmark localization algorithm based on a convolutional neural network (CNN) and a conditional random field (CRF). A CNN is applied to regress landmark heatmaps from the four-channel image data generated by the FPP. A CRF leveraging both local and global shape constraints is developed to better tune the landmark coordinates, reject extra landmarks, and recover missing landmarks. The robustness of the proposed method is demonstrated through ex vivo porcine intestine landmark localization experiments.

Molecular Radiative Energy Shifts under Strong Oscillating Fields.

Coherent manipulation of light-matter interactions is pivotal to the advancement of nanophotonics. Conventionally, the non-resonant optical Stark effect is harnessed for band engineering by intense laser pumping. However, this method is hindered by the transient Stark shifts and the high-energy laser pumping which, by itself, is precluded as a nanoscale optical source due to light diffraction. As an analog of photons in a laser, surface plasmons are uniquely positioned to coherently interact with matter through near-field coupling, thereby, providing a potential source of electric fields. Herein, the first demonstration of plasmonic Stark effect is reported and attributed to a newly uncovered energy-bending mechanism. As a complementary approach to the optical Stark effect, it is envisioned that the plasmonic Stark effect will advance fundamental understanding of coherent light-matter interactions and will also provide new opportunities for advanced optoelectronic tools, such as ultrafast all-optical switches and biological nanoprobes at lower light power levels.

Stabilizing the phase of swept-source optical coherence tomography by a wrapped Gaussian mixture model.

The phase of an optical coherence tomography (OCT) signal carries critical information about particle micro-displacements. However, swept-source OCT (SSOCT) suffers from phase instability problems due to trigger jitters from the swept source. In this Letter, a wrapped Gaussian mixture model (WGMM) is proposed to stabilize the phase of SSOCT systems. A closed-form iteration solution of the WGMM is derived using the expectation-maximization algorithm. Necessary approximations are made for real-time graphic processing unit implementation. The performance of the proposed method is demonstrated through ex vivo, in vivo, and flow phantom experiments. The results show the robustness of the method in different application scenarios.

Optical flow optical coherence tomography for determining accurate velocity fields.

Determining micron-scale fluid flow velocities using optical coherence tomography (OCT) is important in both biomedical research and clinical diagnosis. Numerous methods have been explored to quantify the flow information, which can be divided into either phase-based or amplitude-based methods. However, phase-based methods, such as Doppler methods, are less sensitive to transverse velocity components and suffer from wrapped phase and phase instability problems for axial velocity components. On the other hand, amplitude-based methods, such as speckle variance OCT, correlation mapping OCT and split-spectrum amplitude-decorrelation angiography, focus more on segmenting flow areas than quantifying flow velocities. In this paper, we propose optical flow OCT (OFOCT) to quantify accurate velocity fields. The equivalence between optical flow and real velocity fields is validated in OCT imaging. The sensitivity fall-off of a Fourier-domain OCT (FDOCT) system is considered in the modified optical flow continuity constraint. Spatial-temporal smoothness constraints are used to make the optical flow problem well-posed and reduce noises in the velocity fields. An iteration solution to the optical flow problem is implemented in a graphics processing unit (GPU) for real-time processing. The accuracy of the velocity fields is verified through phantom flow experiments by using a diluted milk powder solution as a scattering medium. Velocity fields are then used to detect flow turbulence and reconstruct flow trajectory. The results show that OFOCT is accurate in determining velocity fields and applicable to research concerning fluid dynamics.

High-performance concealment of defective pixel clusters in infrared imagers.

Defective pixel concealment is a necessary procedure in infrared image processing and is widely used. However, current approaches are mainly focused on the concealment of isolated pixels and small defective pixel clusters. Consequently, these approaches cannot meet the requirements when applied to infrared detectors with large defective pixel clusters. In this paper, we present a novel and comprehensive approach to processing the image data acquired from infrared imagers with large and small defective pixel clusters. Our approach consists of preprocessing, coarse concealment, high dynamic range enhancement, and fine concealment by generative adversarial networks. Experiments using mid-wave infrared and long-wave infrared images demonstrated that the proposed approach achieves better results than the best conventional approach, known as transforming image completion, with the peak signal-to-noise ratio and structural similarity metrics improved by 2.7063 dB (16.3%) and 0.1951 dB (34.1%), respectively.

Real-time infrared image detail enhancement based on fast guided image filter and plateau equalization.

Image detail enhancement is critical to the performance of infrared imaging systems because the original images generally suffer from low contrast and a low signal-to-noise ratio. Although conventional decomposition-based methods have advantages in enhancing image details, they also have clear disadvantages, which include intensive computations, over-enhanced noise, and gradient reversal artifacts. In this paper, we propose to accelerate enhancement processing by using a fast guided filter and plateau equalization. Our method consists of image decomposition, base and detail layers processing, and projection of the enhanced image to an 8-bit dynamic range. Experimental results demonstrated that our proposed method achieves a good balance among detail enhancement performance, noise and gradient reversal artifacts suppression, and computational cost, with a frame rate around 30 fps for 640×512 infrared images.

Demonstration of Optical Coherence Tomography Guided Big Bubble Technique for Deep Anterior Lamellar Keratoplasty (DALK).

Deep anterior lamellar keratoplasty (DALK) is a highly challenging procedure for cornea transplant that involves removing the corneal layers above Descemet's membrane (DM). This is achieved by a "big bubble" technique where a needle is inserted into the stroma of the cornea down to DM and the injection of either air or liquid. DALK has important advantages over penetrating keratoplasty (PK) including lower rejection rate, less endothelial cell loss, and increased graft survival. In this paper, we successfully designed and evaluated the optical coherence tomography (OCT) distal sensor integrated needle for a precise big bubble technique. We successfully used this sensor for micro-control of a robotic DALK device termed AUTO-DALK for autonomous big bubble needle insertion. The OCT distal sensor was integrated inside a 25-gauge needle, which was used for pneumo-dissection. The AUTO-DALK device is built on a manual trephine platform which includes a vacuum ring to fix the device on the eye and add a needle driver at an angle of 60 degrees from vertical. During the test on five porcine eyes with a target depth of 90%, the measured insertion depth as a percentage of cornea thickness for the AUTO-DALK device was 90 . 05 % ± 2 . 33 % without any perforation compared to 79 . 16 % ± 5 . 68 % for unassisted free-hand insertion and 86 . 20 % ± 5 . 31 % for assisted free-hand insertion. The result showed a higher precision and consistency of the needle placement with AUTO-DALK, which could lead to better visual outcomes and fewer complications.

Motorized Micro-Forceps with Active Motion Guidance based on Common-Path SSOCT for Epiretinal Membranectomy.

In this study, we built and tested a handheld motion-guided micro-forceps system using common-path swept source optical coherence tomography (CP-SSOCT) for highly accurate depth controlled epiretinal membranectomy. A touch sensor and two motors were used in the forceps design to minimize the inherent motion artifact while squeezing the tool handle to actuate the tool and grasp, and to independently control the depth of the tool-tip. A smart motion monitoring and a guiding algorithm were devised to provide precise and intuitive freehand control. We compared the involuntary tool-tip motion occurring while grasping with a standard manual micro-forceps and our touch sensor activated micro-forceps. The results showed that our touch-sensor-based and motor-actuated tool can significantly attenuate the motion artifact during grasping (119.81 µm with our device versus 330.73 µm with the standard micro-forceps). By activating the CP-SSOCT based depth locking feature, the erroneous tool-tip motion can be further reduced down to 5.11µm. We evaluated the performance of our device in comparison to the standard instrument in terms of the elapsed time, the number of grasping attempts, and the maximum depth of damage created on the substrate surface while trying to pick up small pieces of fibers (Ø 125 µm) from a soft polymer surface. The results indicate that all metrics were significantly improved when using our device; of note, the average elapsed time, the number of grasping attempts, and the maximum depth of damage were reduced by 25%, 31%, and 75%, respectively.

Experimental assessment of a 3-D plenoptic endoscopic imaging system.

An endoscopic imaging system using a plenoptic technique to reconstruct 3-D information is demonstrated and analyzed in this Letter. The proposed setup integrates a clinical surgical endoscope with a plenoptic camera to achieve a depth accuracy error of about 1 mm and a precision error of about 2 mm, within a 25 mm × 25 mm field of view, operating at 11 frames per second.

Ghost reduction in CP-SSOCT having multiple references using Fourier-domain Shift and Sum.

We propose, test and validate a novel Fourier-domain based method for ghost image artifacts reduction in a common-path SSOCT system having multiple adjacent reference planes. Common-path probes with imaging systems containing high-index sapphire ball or other lenses produce multiple fixed references due to Fresnel reflections from the lens surfaces. The multiple reference planes produce multiple and overlapping OCT images. Since such ghost artifacts are the result of the superposition of multiple identical images having different amplitudes and spatial shifts, one can correctly shift and sum the images in the Fourier-domain once the relative amplitude and lateral position between the reference planes are known. This theory and numerical testing are presented to elucidate our method. We then validate the potential effectiveness using OCT imaging experiments.

Extravascular optical coherence tomography: evaluation of carotid atherosclerosis and pravastatin therapy.

BACKGROUND AND PURPOSE: Extravascular optical coherence tomography (OCT), as a noninvasive imaging methodology with micrometer resolution, was evaluated in a murine model of carotid atherosclerosis by way of assessing the efficacy of pravastatin therapy. METHODS: An OCT device was engineered for extravascular plaque imaging. Wild-type mice and apolipoprotein E-deficient (ApoE(-/-)) mice were randomized to 3 treatment groups: (1) wild-type on a diet of standard rodent chow (n=13); (2) ApoE(-/-) on a high-fat, atherosclerotic diet (HFD; n=13); and (3) ApoE(-/-) on a HFD given daily pravastatin (n=13). Mice were anesthetized and the left common carotid was surgically exposed. Three-dimensional (3D; 2 spatial dimensions+time) and 4D (3 spatial dimensions+time) OCT images of the vessel lumen patency were evaluated. After perfusion, in situ OCT imaging was performed for statistical comparison with the in vivo results and final histology. RESULTS: Intraoperative OCT imaging positively identified carotid plaque in 100% of ApoE(-/-) mice on HFD. ApoE(-/-) mice on HFD had a significantly decreased lumen patency when compared with that in wild-type mice (P<0.001). Pravastatin therapy was found to increase lumen patency significantly in ApoE(-/-) mice on HFD (P<0.01; compared with ApoE(-/-) on HFD). The findings were confirmed with OCT imaging after perfusion and histology. CONCLUSIONS: OCT imaging offers the potential for real-time, detailed vessel lumen evaluation, potentially improving surgical accuracy and outcomes during cerebrovascular neurosurgical procedures. Pravastatin significantly increases vessel lumen patency in the ApoE(-/-) mouse on HFD.

GPU-accelerated non-uniform fast Fourier transform-based compressive sensing spectral domain optical coherence tomography.

We implemented the graphics processing unit (GPU) accelerated compressive sensing (CS) non-uniform in k-space spectral domain optical coherence tomography (SD OCT). Kaiser-Bessel (KB) function and Gaussian function are used independently as the convolution kernel in the gridding-based non-uniform fast Fourier transform (NUFFT) algorithm with different oversampling ratios and kernel widths. Our implementation is compared with the GPU-accelerated modified non-uniform discrete Fourier transform (MNUDFT) matrix-based CS SD OCT and the GPU-accelerated fast Fourier transform (FFT)-based CS SD OCT. It was found that our implementation has comparable performance to the GPU-accelerated MNUDFT-based CS SD OCT in terms of image quality while providing more than 5 times speed enhancement. When compared to the GPU-accelerated FFT based-CS SD OCT, it shows smaller background noise and less side lobes while eliminating the need for the cumbersome k-space grid filling and the k-linear calibration procedure. Finally, we demonstrated that by using a conventional desktop computer architecture having three GPUs, real-time B-mode imaging can be obtained in excess of 30 fps for the GPU-accelerated NUFFT based CS SD OCT with frame size 2048(axial) × 1,000(lateral).

Real-time compressive sensing spectral domain optical coherence tomography.

We developed and demonstrated real-time compressive sensing (CS) spectral domain optical coherence tomography (SD-OCT) B-mode imaging at excess of 70 fps. The system was implemented using a conventional desktop computer architecture having three graphics processing units. This result shows speed gain of 459 and 112 times compared to the best CS implementations based on the MATLAB and C++, respectively, and that real-time CS SD-OCT imaging can finally be realized.

Random transverse motion-induced spatial compounding for fiber bundle imaging.

We propose and demonstrate a novel fiber bundle imaging based on spatial compounding induced by random transverse motion to remove the pixelation effect, to improve resolution, and to increase image quality. The experimental results using a USAF target and pyramidal neuron cell showed that 20-frame compounding improved image quality (contrast-to-noise ratio by >9 dB, global SNR by >6 dB, equivalent number of looks by >1.8 times, and 1/ß by >1.5 times), resolution by better than 2 µm, and completely eliminated pixelation artifact.

Real-time dispersion-compensated image reconstruction for compressive sensing spectral domain optical coherence tomography.

In this work, we propose a novel dispersion compensation method that enables real-time compressive sensing (CS) spectral domain optical coherence tomography (SD OCT) image reconstruction. We show that dispersion compensation can be incorporated into CS SD OCT by multiplying the dispersion-correcting terms by the undersampled spectral data before CS reconstruction. High-quality SD OCT imaging with dispersion compensation was demonstrated at a speed in excess of 70 frames per s using 40% of the spectral measurements required by the well-known Shannon/Nyquist theory. The data processing and image display were performed on a conventional workstation having three graphics processing units.

High-resolution in vivo 4D-OCT fish-eye imaging using 3D-UNet with multi-level residue decoder.

Optical coherence tomography (OCT) allows high-resolution volumetric imaging of biological tissues in vivo. However, 3D-image acquisition often suffers from motion artifacts due to slow frame rates and involuntary and physiological movements of living tissue. To solve these issues, we implement a real-time 4D-OCT system capable of reconstructing near-distortion-free volumetric images based on a deep learning-based reconstruction algorithm. The system initially collects undersampled volumetric images at a high speed and then upsamples the images in real-time by a convolutional neural network (CNN) that generates high-frequency features using a deep learning algorithm. We compare and analyze both dual-2D- and 3D-UNet-based networks for the OCT 3D high-resolution image reconstruction. We refine the network architecture by incorporating multi-level information to accelerate convergence and improve accuracy. The network is optimized by utilizing the 16-bit floating-point precision for network parameters to conserve GPU memory and enhance efficiency. The result shows that the refined and optimized 3D-network is capable of retrieving the tissue structure more precisely and enable real-time 4D-OCT imaging at a rate greater than 10 Hz with a root mean square error (RMSE) of ∼0.03.

Towards Autonomous Retinal Microsurgery Using RGB-D Images.

Retinal surgery is a challenging procedure requiring precise manipulation of the fragile retinal tissue, often at the scale of tens-of-micrometers. Its difficulty has motivated the development of robotic assistance platforms to enable precise motion, and more recently, novel sensors such as microscope integrated optical coherence tomography (OCT) for RGB-D view of the surgical workspace. The combination of these devices opens new possibilities for robotic automation of tasks such as subretinal injection (SI), a procedure that involves precise needle insertion into the retina for targeted drug delivery. Motivated by this opportunity, we develop a framework for autonomous needle navigation during SI. We develop a system which enables the surgeon to specify waypoint goals in the microscope and OCT views, and the system autonomously navigates the needle to the desired subretinal space in real-time. Our system integrates OCT and microscope images with convolutional neural networks (CNNs) to automatically segment the surgical tool and retinal tissue boundaries, and model predictive control that generates optimal trajectories that respect kinematic constraints to ensure patient safety. We validate our system by demonstrating 30 successful SI trials on pig eyes. Preliminary comparisons to a human operator in robot-assisted mode highlight the enhanced safety and performance of our system.

Design and Evaluation of an Eye Mountable AutoDALK Robot for Deep Anterior Lamellar Keratoplasty.

Partial-thickness corneal transplants using a deep anterior lamellar keratoplasty (DALK) approach has demonstrated better patient outcomes than a full-thickness cornea transplant. However, despite better clinical outcomes from the DALK procedure, adoption of the technique has been limited because the accurate insertion of the needle into the deep stroma remains technically challenging. In this work, we present a novel hands-free eye mountable robot for automatic needle placement in the cornea, AutoDALK, that has the potential to simplify this critical step in the DALK procedure. The system integrates dual light-weight linear piezo motors, an OCT A-scan distance sensor, and a vacuum trephine-inspired design to enable the safe, consistent, and controllable insertion of a needle into the cornea for the pneumodissection of the anterior cornea from the deep posterior cornea and Descemet's membrane. AutoDALK was designed with feedback from expert corneal surgeons and performance was evaluated by finite element analysis simulation, benchtop testing, and ex vivo experiments to demonstrate the feasibility of the system for clinical applications. The mean open-loop positional deviation was 9.39 µm, while the system repeatability and accuracy were 39.48 µm and 43.18 µm, respectively. The maximum combined thrust of the system was found to be 1.72 N, which exceeds the clinical penetration force of the cornea. In a head-to-head ex vivo comparison against an expert surgeon using a freehand approach, AutoDALK achieved more consistent needle depth, which resulted in fewer perforations of Descemet's membrane and significantly deeper pneumodissection of the stromal tissue. The results of this study indicate that robotic needle insertion has the potential to simplify the most challenging task of the DALK procedure, enable more consistent surgical outcomes for patients, and standardize partial-thickness corneal transplants as the gold standard of care if demonstrated to be more safe and more effective than penetrating keratoplasty.

Arc-to-line frame registration method for ultrasound and photoacoustic image-guided intraoperative robot-assisted laparoscopic prostatectomy.

PURPOSE: To achieve effective robot-assisted laparoscopic prostatectomy, the integration of transrectal ultrasound (TRUS) imaging system which is the most widely used imaging modality in prostate imaging is essential. However, manual manipulation of the ultrasound transducer during the procedure will significantly interfere with the surgery. Therefore, we propose an image co-registration algorithm based on a photoacoustic marker (PM) method, where the ultrasound/photoacoustic (US/PA) images can be registered to the endoscopic camera images to ultimately enable the TRUS transducer to automatically track the surgical instrument. METHODS: An optimization-based algorithm is proposed to co-register the images from the two different imaging modalities. The principle of light propagation and an uncertainty in PM detection were assumed in this algorithm to improve the stability and accuracy of the algorithm. The algorithm is validated using the previously developed US/PA image-guided system with a da Vinci surgical robot. RESULTS: The target-registration-error (TRE) is measured to evaluate the proposed algorithm. In both simulation and experimental demonstration, the proposed algorithm achieved a sub-centimeter accuracy which is acceptable in practical clinics (i.e., 1.15 ± 0.29 mm from the experimental evaluation). The result is also comparable with our previous approach (i.e., 1.05 ± 0.37 mm), and the proposed method can be implemented with a normal white light stereo camera and does not require highly accurate localization of the PM. CONCLUSION: The proposed frame registration algorithm enabled a simple yet efficient integration of commercial US/PA imaging system into laparoscopic surgical setting by leveraging the characteristic properties of acoustic wave propagation and laser excitation, contributing to automated US/PA image-guided surgical intervention applications.