Needle tracking in low-resolution ultrasound volumes using deep learning.

PURPOSE: Clinical needle insertion into tissue, commonly assisted by 2D ultrasound imaging for real-time navigation, faces the challenge of precise needle and probe alignment to reduce out-of-plane movement. Recent studies investigate 3D ultrasound imaging together with deep learning to overcome this problem, focusing on acquiring high-resolution images to create optimal conditions for needle tip detection. However, high-resolution also requires a lot of time for image acquisition and processing, which limits the real-time capability. Therefore, we aim to maximize the US volume rate with the trade-off of low image resolution. We propose a deep learning approach to directly extract the 3D needle tip position from sparsely sampled US volumes. METHODS: We design an experimental setup with a robot inserting a needle into water and chicken liver tissue. In contrast to manual annotation, we assess the needle tip position from the known robot pose. During insertion, we acquire a large data set of low-resolution volumes using a 16  ×  16 element matrix transducer with a volume rate of 4 Hz. We compare the performance of our deep learning approach with conventional needle segmentation. RESULTS: Our experiments in water and liver show that deep learning outperforms the conventional approach while achieving sub-millimeter accuracy. We achieve mean position errors of 0.54 mm in water and 1.54 mm in liver for deep learning. CONCLUSION: Our study underlines the strengths of deep learning to predict the 3D needle positions from low-resolution ultrasound volumes. This is an important milestone for real-time needle navigation, simplifying the alignment of needle and ultrasound probe and enabling a 3D motion analysis.

Tissue Classification During Needle Insertion Using Self-Supervised Contrastive Learning and Optical Coherence Tomography.

Needle positioning is essential for various medical applications such as epidural anaesthesia. Physicians rely on their instincts while navigating the needle in epidural spaces. Thereby, identifying the tissue structures may be helpful to the physician as they can provide additional feedback in the needle insertion process. To this end, we propose a deep neural network that classifies the tissues from the phase and intensity data of complex OCT signals acquired at the needle tip. We investigate the performance of the deep neural network in a limited labelled dataset scenario and propose a novel contrastive pretraining strategy that learns invariant representation for phase and intensity data. We show that with 10% of the training set, our proposed pretraining strategy helps the model achieve an F1 score of 0.84±0.10 whereas the model achieves an F1 score of 0.60±0.07 without it. Further, we analyse the importance of phase and intensity individually towards tissue classification.

A Miniature Dual-Fiber Probe for Quantitative Optical Coherence Elastography.

OBJECTIVE: Optical coherence elastography (OCE) allows for high resolution analysis of elastic tissue properties. However, due to the limited penetration of light into tissue, miniature probes are required to reach structures inside the body, e.g., vessel walls. Shear wave elastography relates shear wave velocities to quantitative estimates of elasticity. Generally, this is achieved by measuring the runtime of waves between two or multiple points. For miniature probes, optical fibers have been integrated and the runtime between the point of excitation and a single measurement point has been considered. This approach requires precise temporal synchronization and spatial calibration between excitation and imaging. METHODS: We present a miniaturized dual-fiber OCE probe of 1 mm diameter allowing for robust shear wave elastography. Shear wave velocity is estimated between two optics and hence independent of wave propagation between excitation and imaging. We quantify the wave propagation by evaluating either a single or two measurement points. Particularly, we compare both approaches to ultrasound elastography. RESULTS: Our experimental results demonstrate that quantification of local tissue elasticities is feasible. For homogeneous soft tissue phantoms, we obtain mean deviations of 0.15 ms-1 and 0.02 ms-1 for single-fiber and dual-fiber OCE, respectively. In inhomogeneous phantoms, we measure mean deviations of up to 0.54 ms-1 and 0.03 ms-1 for single-fiber and dual-fiber OCE, respectively. CONCLUSION: We present a dual-fiber OCE approach that is much more robust in inhomogeneous tissues. Moreover, we demonstrate the feasibility of elasticity quantification in ex-vivo coronary arteries. SIGNIFICANCE: This study introduces an approach for robust elasticity quantification from within the tissue.

Ultrasound Shear Wave Elasticity Imaging With Spatio-Temporal Deep Learning.

Ultrasound shear wave elasticity imaging is a valuable tool for quantifying the elastic properties of tissue. Typically, the shear wave velocity is derived and mapped to an elasticity value, which neglects information such as the shape of the propagating shear wave or push sequence characteristics. We present 3D spatio-temporal CNNs for fast local elasticity estimation from ultrasound data. This approach is based on retrieving elastic properties from shear wave propagation within small local regions. A large training data set is acquired with a robot from homogeneous gelatin phantoms ranging from 17.42 kPa to 126.05 kPa with various push locations. The results show that our approach can estimate elastic properties on a pixelwise basis with a mean absolute error of 5.01(437) kPa. Furthermore, we estimate local elasticity independent of the push location and can even perform accurate estimates inside the push region. For phantoms with embedded inclusions, we report a 53.93% lower MAE (7.50 kPa) and on the background of 85.24% (1.64 kPa) compared to a conventional shear wave method. Overall, our method offers fast local estimations of elastic properties with small spatio-temporal window sizes.

Rupture Detection During Needle Insertion Using Complex OCT Data and CNNs.

OBJECTIVE: Soft tissue deformation and ruptures complicate needle placement. However, ruptures at tissue interfaces also contain information which helps physicians to navigate through different layers. This navigation task can be challenging, whenever ultrasound (US) image guidance is hard to align and externally sensed forces are superimposed by friction. METHODS: We propose an experimental setup for reproducible needle insertions, applying optical coherence tomography (OCT) directly at the needle tip as well as external US and force measurements. Processing the complex OCT data is challenging as the penetration depth is limited and the data can be difficult to interpret. Using a machine learning approach, we show that ruptures can be detected in the complex OCT data without additional external guidance or measurements after training with multi-modal ground-truth from US and force. RESULTS: We can detect ruptures with accuracies of 0.94 and 0.91 on homogeneous and inhomogeneous phantoms, respectively, and 0.71 for ex-situ tissues. CONCLUSION: We propose an experimental setup and deep learning based rupture detection for the complex OCT data in front of the needle tip, even in deeper tissue structures without the need for US or force sensor guiding. SIGNIFICANCE: This study promises a suitable approach to complement a robust robotic needle placement.

In-Vitro MPI-guided IVOCT catheter tracking in real time for motion artifact compensation.

PURPOSE: Using 4D magnetic particle imaging (MPI), intravascular optical coherence tomography (IVOCT) catheters are tracked in real time in order to compensate for image artifacts related to relative motion. Our approach demonstrates the feasibility for bimodal IVOCT and MPI in-vitro experiments. MATERIAL AND METHODS: During IVOCT imaging of a stenosis phantom the catheter is tracked using MPI. A 4D trajectory of the catheter tip is determined from the MPI data using center of mass sub-voxel strategies. A custom built IVOCT imaging adapter is used to perform different catheter motion profiles: no motion artifacts, motion artifacts due to catheter bending, and heart beat motion artifacts. Two IVOCT volume reconstruction methods are compared qualitatively and quantitatively using the DICE metric and the known stenosis length. RESULTS: The MPI-tracked trajectory of the IVOCT catheter is validated in multiple repeated measurements calculating the absolute mean error and standard deviation. Both volume reconstruction methods are compared and analyzed whether they are capable of compensating the motion artifacts. The novel approach of MPI-guided catheter tracking corrects motion artifacts leading to a DICE coefficient with a minimum of 86% in comparison to 58% for a standard reconstruction approach. CONCLUSIONS: IVOCT catheter tracking with MPI in real time is an auspicious method for radiation free MPI-guided IVOCT interventions. The combination of MPI and IVOCT can help to reduce motion artifacts due to catheter bending and heart beat for optimized IVOCT volume reconstructions.

Towards an Optical Biopsy during Visceral Surgical Interventions.

BACKGROUND: Cancer will replace cardiovascular diseases as the most frequent cause of death. Therefore, the goals of cancer treatment are prevention strategies and early detection by cancer screening and ideal stage therapy. From an oncological point of view, complete tumor resection is a significant prognostic factor. Optical coherence tomography (OCT) and confocal laser microscopy (CLM) are two techniques that have the potential to complement intraoperative frozen section analysis as in vivo and real-time optical biopsies. SUMMARY: In this review we present both procedures and review the progress of evaluation for intraoperative application in visceral surgery. For visceral surgery, there are promising studies evaluating OCT and CLM; however, application during routine visceral surgical interventions is still lacking. KEY MESSAGE: OCT and CLM are not competing but complementary approaches of tissue analysis to intraoperative frozen section analysis. Although intraoperative application of OCT and CLM is at an early stage, they are two promising techniques of intraoperative in vivo and real-time tissue examination. Additionally, deep learning strategies provide a significant supplement for automated tissue detection.

Quantitative Analysis of 3D Artery Volume Reconstructions Using Biplane Angiography and Intravascular OCT Imaging.

Diameter and volume are frequently used parameters for cardiovascular diagnosis, e.g., to identify a stenosis of the coronary arteries. Intra-vascular OCT imaging has a high spatial resolution and promises accurate estimates of the vessel diameter. However, the actual images are reconstructed from A-scans relative to the catheter tip and imaging is subject to rotational artifacts. We study the impact of different volume reconstruction approaches on the accuracy of the vessel shape estimate. Using X-ray angiography we obtain the 3D vessel centerline and the 3D catheter trajectory, and we propose to align the A-scans using both. For comparison we consider reconstruction along a straight line and along the centerline. All methods are evaluated based on an experimental setup using a clinical angiography system and a vessel phantom with known shape. Our results illustrate potential pitfalls in the estimation of the vessel shape, particularly when the vessel is curved. We demonstrate that the conventional reconstruction approaches may result in an overestimate of the cross-section and that the proposed approach results in a good shape agreement in general and for curver segments, with DICE coefficients of approximately 0.96 and 0.98, respectively.

Bimodal intravascular volumetric imaging combining OCT and MPI.

PURPOSE: Intravascular optical coherence tomography (IVOCT) is a catheter-based image modality allowing for high-resolution imaging of vessels. It is based on a fast sequential acquisition of A-scans with an axial spatial resolution in the range of 5-10 µm, that is, one order of magnitude higher than in conventional methods like intravascular ultrasound or computed tomography angiography. However, position and orientation of the catheter in patient coordinates cannot be obtained from the IVOCT measurements alone. Hence, the pose of the catheter needs to be established to correctly reconstruct the three-dimensional vessel shape. Magnetic particle imaging (MPI) is a three-dimensional tomographic, tracer-based, and radiation-free image modality providing high temporal resolution with unlimited penetration depth. Volumetric MPI images are angiographic and hence suitable to complement IVOCT as a comodality. We study simultaneous bimodal IVOCT MPI imaging with the goal of estimating the IVOCT pullback path based on the 3D MPI data. METHODS: We present a setup to study and evaluate simultaneous IVOCT and MPI image acquisition of differently shaped vessel phantoms. First, the influence of the MPI tracer concentration on the optical properties required for IVOCT is analyzed. Second, using a concentration allowing for simultaneous imaging, IVOCT and MPI image data are acquired sequentially and simultaneously. Third, the luminal centerline is established from the MPI image volumes and used to estimate the catheter pullback trajectory for IVOCT image reconstruction. The image volumes are compared to the known shape of the phantoms. RESULTS: We were able to identify a suitable MPI tracer concentration of 2.5 mmol/L with negligible influence on the IVOCT signal. The pullback trajectory estimated from MPI agrees well with the centerline of the phantoms. Its mean absolute error ranges from 0.27 to 0.28 mm and from 0.25 mm to 0.28 mm for sequential and simultaneous measurements, respectively. Likewise, reconstructing the shape of the vessel phantoms works well with mean absolute errors for the diameter ranging from 0.11 to 0.21 mm and from 0.06 to 0.14 mm for sequential and simultaneous measurements, respectively. CONCLUSIONS: Magnetic particle imaging can be used in combination with IVOCT to estimate the catheter trajectory and the vessel shape with high precision and without ionizing radiation.

Automatic Plaque Detection in IVOCT Pullbacks Using Convolutional Neural Networks.

Coronary heart disease is a common cause of death despite being preventable. To treat the underlying plaque deposits in the arterial walls, intravascular optical coherence tomography can be used by experts to detect and characterize the lesions. In clinical routine, hundreds of images are acquired for each patient, which require automatic plaque detection for fast and accurate decision support. So far, automatic approaches rely on classic machine learning methods and deep learning solutions have rarely been studied. Given the success of deep learning methods with other imaging modalities, a thorough understanding of deep learning-based plaque detection for future clinical decision support systems is required. We address this issue with a new data set consisting of in vivo patient images labeled by three trained experts. Using this data set, we employ the state-of-the-art deep learning models that directly learn plaque classification from the images. For improved performance, we study different transfer learning approaches. Furthermore, we investigate the use of Cartesian and polar image representations and employ data augmentation techniques tailored to each representation. We fuse both representations in a multi-path architecture for more effective feature exploitation. Last, we address the challenge of plaque differentiation in addition to detection. Overall, we find that our combined model performs best with an accuracy of 91.7%, a sensitivity of 90.9%, and a specificity of 92.4%. Our results indicate that building a deep learning-based clinical decision support system for plaque detection is feasible.