Rupture Detection During Needle Insertion Using Complex OCT Data and CNNs.

OBJECTIVE: Soft tissue deformation and ruptures complicate needle placement. However, ruptures at tissue interfaces also contain information which helps physicians to navigate through different layers. This navigation task can be challenging, whenever ultrasound (US) image guidance is hard to align and externally sensed forces are superimposed by friction. METHODS: We propose an experimental setup for reproducible needle insertions, applying optical coherence tomography (OCT) directly at the needle tip as well as external US and force measurements. Processing the complex OCT data is challenging as the penetration depth is limited and the data can be difficult to interpret. Using a machine learning approach, we show that ruptures can be detected in the complex OCT data without additional external guidance or measurements after training with multi-modal ground-truth from US and force. RESULTS: We can detect ruptures with accuracies of 0.94 and 0.91 on homogeneous and inhomogeneous phantoms, respectively, and 0.71 for ex-situ tissues. CONCLUSION: We propose an experimental setup and deep learning based rupture detection for the complex OCT data in front of the needle tip, even in deeper tissue structures without the need for US or force sensor guiding. SIGNIFICANCE: This study promises a suitable approach to complement a robust robotic needle placement.

Deep learning with 4D spatio-temporal data representations for OCT-based force estimation.

Estimating the forces acting between instruments and tissue is a challenging problem for robot-assisted minimally-invasive surgery. Recently, numerous vision-based methods have been proposed to replace electro-mechanical approaches. Moreover, optical coherence tomography (OCT) and deep learning have been used for estimating forces based on deformation observed in volumetric image data. The method demonstrated the advantage of deep learning with 3D volumetric data over 2D depth images for force estimation. In this work, we extend the problem of deep learning-based force estimation to 4D spatio-temporal data with streams of 3D OCT volumes. For this purpose, we design and evaluate several methods extending spatio-temporal deep learning to 4D which is largely unexplored so far. Furthermore, we provide an in-depth analysis of multi-dimensional image data representations for force estimation, comparing our 4D approach to previous, lower-dimensional methods. Also, we analyze the effect of temporal information and we study the prediction of short-term future force values, which could facilitate safety features. For our 4D force estimation architectures, we find that efficient decoupling of spatial and temporal processing is advantageous. We show that using 4D spatio-temporal data outperforms all previously used data representations with a mean absolute error of 10.7 mN. We find that temporal information is valuable for force estimation and we demonstrate the feasibility of force prediction.

Spatio-temporal deep learning methods for motion estimation using 4D OCT image data.

PURPOSE: Localizing structures and estimating the motion of a specific target region are common problems for navigation during surgical interventions. Optical coherence tomography (OCT) is an imaging modality with a high spatial and temporal resolution that has been used for intraoperative imaging and also for motion estimation, for example, in the context of ophthalmic surgery or cochleostomy. Recently, motion estimation between a template and a moving OCT image has been studied with deep learning methods to overcome the shortcomings of conventional, feature-based methods. METHODS: We investigate whether using a temporal stream of OCT image volumes can improve deep learning-based motion estimation performance. For this purpose, we design and evaluate several 3D and 4D deep learning methods and we propose a new deep learning approach. Also, we propose a temporal regularization strategy at the model output. RESULTS: Using a tissue dataset without additional markers, our deep learning methods using 4D data outperform previous approaches. The best performing 4D architecture achieves an correlation coefficient (aCC) of 98.58% compared to 85.0% of a previous 3D deep learning method. Also, our temporal regularization strategy at the output further improves 4D model performance to an aCC of 99.06%. In particular, our 4D method works well for larger motion and is robust toward image rotations and motion distortions. CONCLUSIONS: We propose 4D spatio-temporal deep learning for OCT-based motion estimation. On a tissue dataset, we find that using 4D information for the model input improves performance while maintaining reasonable inference times. Our regularization strategy demonstrates that additional temporal information is also beneficial at the model output.

Concept for Markerless 6D Tracking Employing Volumetric Optical Coherence Tomography.

Optical tracking systems are widely used, for example, to navigate medical interventions. Typically, they require the presence of known geometrical structures, the placement of artificial markers, or a prominent texture on the target's surface. In this work, we propose a 6D tracking approach employing volumetric optical coherence tomography (OCT) images. OCT has a micrometer-scale resolution and employs near-infrared light to penetrate few millimeters into, for example, tissue. Thereby, it provides sub-surface information which we use to track arbitrary targets, even with poorly structured surfaces, without requiring markers. Our proposed system can shift the OCT's field-of-view in space and uses an adaptive correlation filter to estimate the motion at multiple locations on the target. This allows one to estimate the target's position and orientation. We show that our approach is able to track translational motion with root-mean-squared errors below 0 . 25 m m and in-plane rotations with errors below 0 . 3 ∘ . For out-of-plane rotations, our prototypical system can achieve errors around 0 . 6 ∘ .

In-Vitro MPI-guided IVOCT catheter tracking in real time for motion artifact compensation.

PURPOSE: Using 4D magnetic particle imaging (MPI), intravascular optical coherence tomography (IVOCT) catheters are tracked in real time in order to compensate for image artifacts related to relative motion. Our approach demonstrates the feasibility for bimodal IVOCT and MPI in-vitro experiments. MATERIAL AND METHODS: During IVOCT imaging of a stenosis phantom the catheter is tracked using MPI. A 4D trajectory of the catheter tip is determined from the MPI data using center of mass sub-voxel strategies. A custom built IVOCT imaging adapter is used to perform different catheter motion profiles: no motion artifacts, motion artifacts due to catheter bending, and heart beat motion artifacts. Two IVOCT volume reconstruction methods are compared qualitatively and quantitatively using the DICE metric and the known stenosis length. RESULTS: The MPI-tracked trajectory of the IVOCT catheter is validated in multiple repeated measurements calculating the absolute mean error and standard deviation. Both volume reconstruction methods are compared and analyzed whether they are capable of compensating the motion artifacts. The novel approach of MPI-guided catheter tracking corrects motion artifacts leading to a DICE coefficient with a minimum of 86% in comparison to 58% for a standard reconstruction approach. CONCLUSIONS: IVOCT catheter tracking with MPI in real time is an auspicious method for radiation free MPI-guided IVOCT interventions. The combination of MPI and IVOCT can help to reduce motion artifacts due to catheter bending and heart beat for optimized IVOCT volume reconstructions.

Optimizing robot motion for robotic ultrasound-guided radiation therapy.

An important aspect of robotic radiation therapy is active compensation of target motion. Recently, ultrasound has been proposed to obtain real-time volumetric images of abdominal organ motion. One approach to realize flexible probe placement throughout the treatment fraction is based on a robotic arm holding the ultrasound probe. However, the probe and the robot holding it may obstruct some of the beams with a potentially adverse effect on the plan quality. This can be mitigated by using a kinematically redundant robot, which allows maintaining a steady pose of the ultrasound probe while moving its elbow in order to minimize beam blocking. Ultimately, the motion of both the beam source carrying and the ultrasound probe holding robot contributes to the overall treatment time, i.e. beam delivery and robot motion. We propose an approach to optimize the motion and coordination of both robots based on a generalized traveling salesman problem. Furthermore, we study an application of the model to a prostate treatment scenario. Because the underlying optimization problem is hard, we compare results from a state-of-the-art heuristic solver and an approximation scheme with low computational effort. Our results show that integration of the robot holding the ultrasound probe is feasible with acceptable overhead in overall treatment time. For clinically realistic velocities of the robots, the overhead is less than 4% which is a small cost for the added benefit of continuous, volumetric, and non-ionizing tracking of organ motion over periodic x-ray-based tracking.

Analysis and optimization of the robot setup for robotic-ultrasound-guided radiation therapy.

PURPOSE: Robotic ultrasound promises continuous, volumetric, and non-ionizing tracking of organ motion during radiation therapy. However, placement of the robot is critical because it is radio-opaque and might severely influence the achievable dose distribution. METHODS: We propose two heuristic optimization strategies for automatic placement of an ultrasound robot around a patient. Considering a kinematically redundant robot arm, we compare a generic approach based on stochastic search and a more problem-specific segmentwise construction approach. The former allows for multiple elbow configurations while the latter is deterministic. Additionally, we study different objective functions guiding the search. Our evaluation is based on data for ten actual prostate cancer cases and we compare the resulting plan quality for both methods to manually chosen robot configurations previously proposed. RESULTS: The mean improvements in the treatment planning objective value with respect to the best manually selected robot position and a single elbow configuration range from 8.2 to 32.8% and 8.5 to 15.5% for segmentwise construction and stochastic search, respectively. Considering three different elbow configurations, the stochastic search results in better objective values in 80% of the cases, with 30% being significantly better. The optimization strategies are robust with respect to beam sampling and transducer orientation and using previous optimization results as starting point for stochastic search typically results in better solutions compared to random starting points. CONCLUSION: We propose a robust and generic optimization scheme, which can be used to optimize the robot placement for robotic ultrasound guidance in radiation therapy. The automatic optimization further mitigates the impact of robotic ultrasound on the treatment plan quality.

Bimodal intravascular volumetric imaging combining OCT and MPI.

PURPOSE: Intravascular optical coherence tomography (IVOCT) is a catheter-based image modality allowing for high-resolution imaging of vessels. It is based on a fast sequential acquisition of A-scans with an axial spatial resolution in the range of 5-10 µm, that is, one order of magnitude higher than in conventional methods like intravascular ultrasound or computed tomography angiography. However, position and orientation of the catheter in patient coordinates cannot be obtained from the IVOCT measurements alone. Hence, the pose of the catheter needs to be established to correctly reconstruct the three-dimensional vessel shape. Magnetic particle imaging (MPI) is a three-dimensional tomographic, tracer-based, and radiation-free image modality providing high temporal resolution with unlimited penetration depth. Volumetric MPI images are angiographic and hence suitable to complement IVOCT as a comodality. We study simultaneous bimodal IVOCT MPI imaging with the goal of estimating the IVOCT pullback path based on the 3D MPI data. METHODS: We present a setup to study and evaluate simultaneous IVOCT and MPI image acquisition of differently shaped vessel phantoms. First, the influence of the MPI tracer concentration on the optical properties required for IVOCT is analyzed. Second, using a concentration allowing for simultaneous imaging, IVOCT and MPI image data are acquired sequentially and simultaneously. Third, the luminal centerline is established from the MPI image volumes and used to estimate the catheter pullback trajectory for IVOCT image reconstruction. The image volumes are compared to the known shape of the phantoms. RESULTS: We were able to identify a suitable MPI tracer concentration of 2.5 mmol/L with negligible influence on the IVOCT signal. The pullback trajectory estimated from MPI agrees well with the centerline of the phantoms. Its mean absolute error ranges from 0.27 to 0.28 mm and from 0.25 mm to 0.28 mm for sequential and simultaneous measurements, respectively. Likewise, reconstructing the shape of the vessel phantoms works well with mean absolute errors for the diameter ranging from 0.11 to 0.21 mm and from 0.06 to 0.14 mm for sequential and simultaneous measurements, respectively. CONCLUSIONS: Magnetic particle imaging can be used in combination with IVOCT to estimate the catheter trajectory and the vessel shape with high precision and without ionizing radiation.

Optimizing Configurations for 7-DoF Robotic Ultrasound Guidance in Radiotherapy of the Prostate.

Robotic ultrasound guidance is promising for tracking of organ motion during radiotherapy treatments, but the radio-opaque robot and probe interfere with beam delivery. The effect on treatment plan quality can be mitigated by the use of a robot arm with kinematic redundancy, such that the robot is able to elude delivered beams during treatment by changing its configuration. However, these changes require robot motion close to the patient, lead to an increased treatment time, and require coordination with the beam delivery. We propose an optimization workflow which integrates the problem of selecting suitable robot configurations into a linear-programming-based workflow for treatment plan optimization. Starting with a large set of candidate configurations, a minimal subset is determined which provides equivalent plan quality. Our results show that, typically, six configurations are sufficient for this purpose. Furthermore, we show that optimal configurations can be reused for dose planning of subsequent patients.

Towards OCT-Navigated Tissue Ablation with a Picosecond Infrared Laser (PIRL) and Mass-Spectrometric Analysis.

Medical lasers are commonly used in interventions to ablate tumor tissue. Recently, the picosecond infrared laser has been introduced, which greatly decreases damaging of surrounding healthy tissue. Further, its ablation plume contains intact biomolecules which can be collected and analyzed by mass spectrometry. This allows for a specific chracterization of the tissue. For a precise treatment, however, a suitable guidance is needed. Further, spatial information is required if the tissue is to be characterized at different parts in the ablated area. Therefore, we propose a system which employs optical coherence tomography as the guiding imaging modality. We describe a prototypical system which provides automatic ablation of areas defined in the image data. For this purpose, we use a calibration with a robot which drives the laser fiber and collects the arising plume. We demonstrate our system on porcine tissue samples.

A deep learning approach for pose estimation from volumetric OCT data.

Tracking the pose of instruments is a central problem in image-guided surgery. For microscopic scenarios, optical coherence tomography (OCT) is increasingly used as an imaging modality. OCT is suitable for accurate pose estimation due to its micrometer range resolution and volumetric field of view. However, OCT image processing is challenging due to speckle noise and reflection artifacts in addition to the images' 3D nature. We address pose estimation from OCT volume data with a new deep learning-based tracking framework. For this purpose, we design a new 3D convolutional neural network (CNN) architecture to directly predict the 6D pose of a small marker geometry from OCT volumes. We use a hexapod robot to automatically acquire labeled data points which we use to train 3D CNN architectures for multi-output regression. We use this setup to provide an in-depth analysis on deep learning-based pose estimation from volumes. Specifically, we demonstrate that exploiting volume information for pose estimation yields higher accuracy than relying on 2D representations with depth information. Supporting this observation, we provide quantitative and qualitative results that 3D CNNs effectively exploit the depth structure of marker objects. Regarding the deep learning aspect, we present efficient design principles for 3D CNNs, making use of insights from the 2D deep learning community. In particular, we present Inception3D as a new architecture which performs best for our application. We show that our deep learning approach reaches errors at our ground-truth label's resolution. We achieve a mean average error of 14.89 ±â€¯9.3 µm and 0.096 ±â€¯0.072° for position and orientation learning, respectively.

Influence of 17beta-estradiol and insulin on type II collagen and protein synthesis of articular chondrocytes.

Clinical observations have suggested a relationship between osteoarthritis and a changed estrogen metabolism in menopausal women. Type II collagen is one main structural protein of articular cartilage matrix and its synthesis is increased by insulin in growth plate cartilage. Therefore, it was investigated if [(3)H]-proline incorporation and type II collagen synthesis (immunocytochemistry, ELISA) in female bovine articular chondrocytes are affected by 17beta-estradiol and/or insulin. Articular chondrocytes were cultured in monolayers at 5% O(2) in medium containing serum for 5-9 days, followed by application of 10(-13) to 10(-9) M estradiol or 5 microg/ml insulin during a serum-free culture phase of 2-3 days. Immunostaining for type II collagen was strong in the serum-free culture phase whereas it was negative for type I collagen, indicating that cells did not dedifferentiate to fibroblast-like cells during culture in serum-free medium. Whereas insulin raised the proline incorporation and the type II collagen synthesis significantly, physiological doses of estradiol did not show significant effects. The stimulating effect of insulin on the [(3)H]-proline incorporation or the type II collagen synthesis was significantly suppressed after preincubation of cells with 10(-11) to 10(-9) M estradiol resembling an unfavorable effect for articular cartilage. The suppression was reversed if cells were incubated with 10(-11) to 10(-7) M tamoxifen or ICI 182,780 combined with 10(-11) or 10(-9) M estradiol followed by incubation with 5 microg/ml insulin, indicating an estrogen receptor-mediated process. Because the articular cartilage of diabetic patients is biomechanically less stable, further experiments are needed to clarify the role of estradiol and insulin in the metabolism of articular chondrocytes.