Enhancement of instrumented ultrasonic tracking images using deep learning.

PURPOSE: Instrumented ultrasonic tracking provides needle localisation during ultrasound-guided minimally invasive percutaneous procedures. Here, a post-processing framework based on a convolutional neural network (CNN) is proposed to improve the spatial resolution of ultrasonic tracking images. METHODS: The custom ultrasonic tracking system comprised a needle with an integrated fibre-optic ultrasound (US) transmitter and a clinical US probe for receiving those transmissions and for acquiring B-mode US images. For post-processing of tracking images reconstructed from the received fibre-optic US transmissions, a recently-developed framework based on ResNet architecture, trained with a purely synthetic dataset, was employed. A preliminary evaluation of this framework was performed with data acquired from needle insertions in the heart of a fetal sheep in vivo. The axial and lateral spatial resolution of the tracking images were used as performance metrics of the trained network. RESULTS: Application of the CNN yielded improvements in the spatial resolution of the tracking images. In three needle insertions, in which the tip depth ranged from 23.9 to 38.4 mm, the lateral resolution improved from 2.11 to 1.58 mm, and the axial resolution improved from 1.29 to 0.46 mm. CONCLUSION: The results provide strong indications of the potential of CNNs to improve the spatial resolution of ultrasonic tracking images and thereby to increase the accuracy of needle tip localisation. These improvements could have broad applicability and impact across multiple clinical fields, which could lead to improvements in procedural efficiency and reductions in risk of complications.

A patient-specific multi-modality abdominal aortic aneurysm imaging phantom.

PURPOSE: Multimodality imaging of the vascular system is a rapidly growing area of innovation and research, which is increasing with awareness of the dangers of ionizing radiation. Phantom models that are applicable across multiple imaging modalities facilitate testing and comparisons in pre-clinical studies of new devices. Additionally, phantom models are of benefit to surgical trainees for gaining experience with new techniques. We propose a temperature-stable, high-fidelity method for creating complex abdominal aortic aneurysm phantoms that are compatible with both radiation-based, and ultrasound-based imaging modalities, using low cost materials. METHODS: Volumetric CT data of an abdominal aortic aneurysm were acquired. Regions of interest were segmented to form a model compatible with 3D printing. The novel phantom fabrication method comprised a hybrid approach of using 3D printing of water-soluble materials to create wall-less, patient-derived vascular structures embedded within tailored tissue-mimicking materials to create realistic surrounding tissues. A non-soluble 3-D printed spine was included to provide a radiological landmark. RESULTS: The phantom was found to provide realistic appearances with intravascular ultrasound, computed tomography and transcutaneous ultrasound. Furthermore, the utility of this phantom as a training model was demonstrated during a simulated endovascular aneurysm repair procedure with image fusion. CONCLUSION: With the hybrid fabrication method demonstrated here, complex multimodality imaging patient-derived vascular phantoms can be successfully fabricated. These have potential roles in the benchtop development of emerging imaging technologies, refinement of novel minimally invasive surgical techniques and as clinical training tools.

Neural Network Kalman Filtering for 3-D Object Tracking From Linear Array Ultrasound Data.

Many interventional surgical procedures rely on medical imaging to visualize and track instruments. Such imaging methods not only need to be real time capable but also provide accurate and robust positional information. In ultrasound (US) applications, typically, only 2-D data from a linear array are available, and as such, obtaining accurate positional estimation in three dimensions is nontrivial. In this work, we first train a neural network, using realistic synthetic training data, to estimate the out-of-plane offset of an object with the associated axial aberration in the reconstructed US image. The obtained estimate is then combined with a Kalman filtering approach that utilizes positioning estimates obtained in previous time frames to improve localization robustness and reduce the impact of measurement noise. The accuracy of the proposed method is evaluated using simulations, and its practical applicability is demonstrated on experimental data obtained using a novel optical US imaging setup. Accurate and robust positional information is provided in real time. Axial and lateral coordinates for out-of-plane objects are estimated with a mean error of 0.1 mm for simulated data and a mean error of 0.2 mm for experimental data. The 3-D localization is most accurate for elevational distances larger than 1 mm, with a maximum distance of 6 mm considered for a 25-mm aperture.

Deep Learning for Instrumented Ultrasonic Tracking: From Synthetic Training Data to In Vivo Application.

Instrumented ultrasonic tracking is used to improve needle localization during ultrasound guidance of minimally invasive percutaneous procedures. Here, it is implemented with transmitted ultrasound pulses from a clinical ultrasound imaging probe, which is detected by a fiber-optic hydrophone integrated into a needle. The detected transmissions are then reconstructed to form the tracking image. Two challenges are considered with the current implementation of ultrasonic tracking. First, tracking transmissions are interleaved with the acquisition of B-mode images, and thus, the effective B-mode frame rate is reduced. Second, it is challenging to achieve an accurate localization of the needle tip when the signal-to-noise ratio is low. To address these challenges, we present a framework based on a convolutional neural network (CNN) to maintain spatial resolution with fewer tracking transmissions and enhance signal quality. A major component of the framework included the generation of realistic synthetic training data. The trained network was applied to unseen synthetic data and experimental in vivo tracking data. The performance of needle localization was investigated when reconstruction was performed with fewer (up to eightfold) tracking transmissions. CNN-based processing of conventional reconstructions showed that the axial and lateral spatial resolutions could be improved even with an eightfold reduction in tracking transmissions. The framework presented in this study will significantly improve the performance of ultrasonic tracking, leading to faster image acquisition rates and increased localization accuracy.

CuInS2 Quantum Dot and Polydimethylsiloxane Nanocomposites for All-Optical Ultrasound and Photoacoustic Imaging.

Dual-modality imaging employing complementary modalities, such as all-optical ultrasound and photoacoustic imaging, is emerging as a well-suited technique for guiding minimally invasive surgical procedures. Quantum dots are a promising material for use in these dual-modality imaging devices as they can provide wavelength-selective optical absorption. The first quantum dot nanocomposite engineered for co-registered laser-generated ultrasound and photoacoustic imaging is presented. The nanocomposites developed, comprising CuInS2 quantum dots and medical-grade polydimethylsiloxane (CIS-PDMS), are applied onto the distal ends of miniature optical fibers. The films exhibit wavelength-selective optical properties, with high optical absorption (> 90%) at 532 nm for ultrasound generation, and low optical absorption (< 5%) at near-infrared wavelengths greater than 700 nm. Under pulsed laser irradiation, the CIS-PDMS films generate ultrasound with pressures exceeding 3.5 MPa, with a corresponding bandwidth of 18 MHz. An ultrasound transducer is fabricated by pairing the coated optical fiber with a Fabry-Pérot (FP) fiber optic sensor. The wavelength-selective nature of the film is exploited to enable co-registered all-optical ultrasound and photoacoustic imaging of an ink-filled tube phantom. This work demonstrates the potential for quantum dots as wavelength-selective absorbers for all-optical ultrasound generation.

Refractive Two-View Reconstruction for Underwater 3D Vision.

Recovering 3D geometry from cameras in underwater applications involves the Refractive Structure-from-Motion problem where the non-linear distortion of light induced by a change of medium density invalidates the single viewpoint assumption. The pinhole-plus-distortion camera projection model suffers from a systematic geometric bias since refractive distortion depends on object distance. This leads to inaccurate camera pose and 3D shape estimation. To account for refraction, it is possible to use the axial camera model or to explicitly consider one or multiple parallel refractive interfaces whose orientations and positions with respect to the camera can be calibrated. Although it has been demonstrated that the refractive camera model is well-suited for underwater imaging, Refractive Structure-from-Motion remains particularly difficult to use in practice when considering the seldom studied case of a camera with a flat refractive interface. Our method applies to the case of underwater imaging systems whose entrance lens is in direct contact with the external medium. By adopting the refractive camera model, we provide a succinct derivation and expression for the refractive fundamental matrix and use this as the basis for a novel two-view reconstruction method for underwater imaging. For validation we use synthetic data to show the numerical properties of our method and we provide results on real data to demonstrate its practical application within laboratory settings and for medical applications in fluid-immersed endoscopy. We demonstrate our approach outperforms classic two-view Structure-from-Motion method relying on the pinhole-plus-distortion camera model.

Patient-Specific Polyvinyl Alcohol Phantom Fabrication with Ultrasound and X-Ray Contrast for Brain Tumor Surgery Planning.

Phantoms are essential tools for clinical training, surgical planning and the development of novel medical devices. However, it is challenging to create anatomically accurate head phantoms with realistic brain imaging properties because standard fabrication methods are not optimized to replicate any patient-specific anatomical detail and 3D printing materials are not optimized for imaging properties. In order to test and validate a novel navigation system for use during brain tumor surgery, an anatomically accurate phantom with realistic imaging and mechanical properties was required. Therefore, a phantom was developed using real patient data as input and 3D printing of molds to fabricate a patient-specific head phantom comprising the skull, brain and tumor with both ultrasound and X-ray contrast. The phantom also had mechanical properties that allowed the phantom tissue to be manipulated in a similar manner to how human brain tissue is handled during surgery. The phantom was successfully tested during a surgical simulation in a virtual operating room. The phantom fabrication method uses commercially available materials and is easy to reproduce. The 3D printing files can be readily shared, and the technique can be adapted to encompass many different types of tumor.

A simple, realistic walled phantom for intravascular and intracardiac applications.

PURPOSE: This work aims to develop a simple, anatomically and haptically realistic vascular phantom, compatible with intravascular and intracardiac ultrasound. The low-cost, dual-layered phantom bridges the gap between traditional wall-only and wall-less phantoms by showing both the vessel wall and surrounding tissue in ultrasound imaging. This phantom can better assist clinical tool training, testing of intravascular devices, blood flow studies, and validation of algorithms for intravascular and intracardiac surgical systems. METHODS: Polyvinyl alcohol cryogel (PVA-c) incorporating a scattering agent was used to obtain vessel and tissue-mimicking materials. Our specific design targeted the inferior vena cava and renal bifurcations which were modelled using CAD software. A custom mould and container were 3D-printed for shaping the desired vessel wall. Three phantoms were prepared by varying both the concentrations of scattering agent as well as the number of freeze-thaw cycles to which the phantom layers were subjected during the manufacturing process. Each phantom was evaluated using ultrasound imaging using the Foresight™ ICE probe. Geometrical validation was provided by comparing CAD design to a CT scan of the phantom. RESULTS: The desired vascular phantom was constructed using 2.5% and 0.05% scattering agent concentration in the vessel and tissue-mimicking layers, respectively. Imaging of the three phantoms showed that increasing the number of freeze-thaw cycles did not significantly enhance the image contrast. Comparison of the US images with their CT equivalents resulted in an average error of 0.9[Formula: see text] for the lumen diameter. CONCLUSION: The phantom is anatomically realistic when imaged with intracardiac ultrasound and provides a smooth lumen for the ultrasound probe and catheter to manoeuvre. The vascular phantom enables validation of intravascular and intracardiac image guidance systems. The simple construction technique also provides a workflow for designing complex, multi-layered arterial phantoms.

Ultrasound 3D reconstruction of malignant masses in robotic-assisted partial nephrectomy using the PAF rail system: a comparison study.

PURPOSE: In robotic-assisted partial nephrectomy (RAPN), the use of intraoperative ultrasound (IOUS) helps to localise and outline the tumours as well as the blood vessels within the kidney. The aim of this work is to evaluate the use of the pneumatically attachable flexible (PAF) rail system for US 3D reconstruction of malignant masses in RAPN. The PAF rail system is a novel device developed and previously presented by the authors to enable track-guided US scanning. METHODS: We present a comparison study between US 3D reconstruction of masses based on: the da Vinci Surgical System kinematics, single- and stereo-camera tracking of visual markers embedded on the probe. An US-realistic kidney phantom embedding a mass is used for testing. A new design for the US probe attachment to enhance the performance of the kinematic approach is presented. A feature extraction algorithm is proposed to detect the margins of the targeted mass in US images. RESULTS: To evaluate the performance of the investigated approaches the resulting 3D reconstructions have been compared to a CT scan of the phantom. The data collected indicates that single camera reconstruction outperformed the other approaches, reconstructing with a sub-millimetre accuracy the targeted mass. CONCLUSIONS: This work demonstrates that the PAF rail system provides a reliable platform to enable accurate US 3D reconstruction of masses in RAPN procedures. The proposed system has also the potential to be employed in other surgical procedures such as hepatectomy or laparoscopic liver resection.

Photoacoustic imaging of the human placental vasculature.

Minimally invasive fetal interventions require accurate imaging from inside the uterine cavity. Twin-to-twin transfusion syndrome (TTTS), a condition considered in this study, occurs from abnormal vascular anastomoses in the placenta that allow blood to flow unevenly between the fetuses. Currently, TTTS is treated fetoscopically by identifying the anastomosing vessels, and then performing laser photocoagulation. However, white light fetoscopy provides limited visibility of placental vasculature, which can lead to missed anastomoses or incomplete photocoagulation. Photoacoustic (PA) imaging is an alternative imaging method that provides contrast for hemoglobin, and in this study, two PA systems were used to visualize chorionic (fetal) superficial and subsurface vasculature in human placentas. The first system comprised an optical parametric oscillator for PA excitation and a 2D Fabry-Pérot cavity ultrasound sensor; the second, light emitting diode arrays and a 1D clinical linear-array ultrasound imaging probe. Volumetric photoacoustic images were acquired from ex vivo normal term and TTTS-treated placentas. It was shown that superficial and subsurface branching blood vessels could be visualized to depths of approximately 7 mm, and that ablated tissue yielded negative image contrast. This study demonstrated the strong potential of PA imaging to guide minimally invasive fetal therapies.

Video-rate all-optical ultrasound imaging.

All-optical ultrasound imaging, where ultrasound is generated and detected using light, has recently been demonstrated as a viable modality that is inherently insensitive to electromagnetic interference and exhibits wide bandwidths. High-quality 2D and 3D all-optical ultrasound images of tissues have previously been presented; however, to date, long acquisition times (ranging from minutes to hours) have hindered clinical application. Here, we present the first all-optical ultrasound imaging system capable of video-rate, real-time two-dimensional imaging of biological tissue. This was achieved using a spatially extended nano-composite optical ultrasound generator, a highly sensitive fibre-optic acoustic receiver, and eccentric illumination resulting in an acoustic source exhibiting optimal directivity. This source was scanned across a one-dimensional source aperture using a fast galvo mirror, thus enabling the dynamic synthesis of source arrays comprising spatially overlapping sources at non-uniform source separation distances. The resulting system achieved a sustained frame rate of 15 Hz, a dynamic range of 30 dB, a penetration depth of at least 6 mm, a resolution of 75 µm (axial) by 100 µm (lateral), and enabled the dynamics of a pulsating ex vivo carotid artery to be captured.

Handheld Real-Time LED-Based Photoacoustic and Ultrasound Imaging System for Accurate Visualization of Clinical Metal Needles and Superficial Vasculature to Guide Minimally Invasive Procedures.

Ultrasound imaging is widely used to guide minimally invasive procedures, but the visualization of the invasive medical device and the procedure’s target is often challenging. Photoacoustic imaging has shown great promise for guiding minimally invasive procedures, but clinical translation of this technology has often been limited by bulky and expensive excitation sources. In this work, we demonstrate the feasibility of guiding minimally invasive procedures using a dual-mode photoacoustic and ultrasound imaging system with excitation from compact arrays of light-emitting diodes (LEDs) at 850 nm. Three validation experiments were performed. First, clinical metal needles inserted into biological tissue were imaged. Second, the imaging depth of the system was characterized using a blood-vessel-mimicking phantom. Third, the superficial vasculature in human volunteers was imaged. It was found that photoacoustic imaging enabled needle visualization with signal-to-noise ratios that were 1.2 to 2.2 times higher than those obtained with ultrasound imaging, over insertion angles of 26 to 51 degrees. With the blood vessel mimicking phantom, the maximum imaging depth was 38 mm. The superficial vasculature of a human middle finger and a human wrist were clearly visualized in real-time. We conclude that the LED-based system is promising for guiding minimally invasive procedures with peripheral tissue targets.

Gel wax-based tissue-mimicking phantoms for multispectral photoacoustic imaging.

Tissue-mimicking phantoms are widely used for the calibration, evaluation and standardisation of medical imaging systems, and for clinical training. For photoacoustic imaging, tissue-mimicking materials (TMMs) that have tuneable optical and acoustic properties, high stability, and mechanical robustness are highly desired. In this study, gel wax is introduced as a TMM that satisfies these criteria for developing photoacoustic imaging phantoms. The reduced scattering and optical absorption coefficients were independently tuned with the addition of TiO2 and oil-based inks. The frequency-dependent acoustic attenuation obeyed a power law; for native gel wax, it varied from 0.71 dB/cm at 3 MHz to 9.93 dB/cm at 12 MHz. The chosen oil-based inks, which have different optical absorption spectra in the range of 400 to 900 nm, were found to have good photostability under pulsed illumination with photoacoustic excitation light. Optically heterogeneous phantoms that comprised of inclusions with different concentrations of carbon black and coloured inks were fabricated, and multispectral photoacoustic imaging was performed with an optical parametric oscillator and a planar Fabry-Pérot sensor. We conclude that gel wax is well suited as a TMM for multispectral photoacoustic imaging.

Anatomically realistic ultrasound phantoms using gel wax with 3D printed moulds.

Here we describe methods for creating tissue-mimicking ultrasound phantoms based on patient anatomy using a soft material called gel wax. To recreate acoustically realistic tissue properties, two additives to gel wax were considered: paraffin wax to increase acoustic attenuation, and solid glass spheres to increase backscattering. The frequency dependence of ultrasound attenuation was well described with a power law over the measured range of 3-10 MHz. With the addition of paraffin wax in concentrations of 0 to 8 w/w%, attenuation varied from 0.72 to 2.91 dB cm-1 at 3 MHz and from 6.84 to 26.63 dB cm-1 at 10 MHz. With solid glass sphere concentrations in the range of 0.025-0.9 w/w%, acoustic backscattering consistent with a wide range of ultrasonic appearances was achieved. Native gel wax maintained its integrity during compressive deformations up to 60%; its Young's modulus was 17.4 ± 1.4 kPa. The gel wax with additives was shaped by melting and pouring it into 3D printed moulds. Three different phantoms were constructed: a nerve and vessel phantom for peripheral nerve blocks, a heart atrium phantom, and a placental phantom for minimally-invasive fetal interventions. In the first, nerves and vessels were represented as hyperechoic and hypoechoic tubular structures, respectively, in a homogeneous background. The second phantom comprised atria derived from an MRI scan of a patient with an intervening septum and adjoining vena cavae. The third comprised the chorionic surface of a placenta with superficial fetal vessels derived from an image of a post-partum human placenta. Gel wax is a material with widely tuneable ultrasound properties and mechanical characteristics that are well suited for creating patient-specific ultrasound phantoms in several clinical disciplines.

Haptic Guidance Based on All-Optical Ultrasound Distance Sensing for Safer Minimally Invasive Fetal Surgery.

By intervening during the early stage of gestation, fetal surgeons aim to correct or minimize the effects of congenital disorders. As compared to postnatal treatment of these disorders, such early interventions can often actually save the life of the fetus and also improve the quality of life of the newborn. However, fetal surgery is considered one of the most challenging disciplines within Minimally Invasive Surgery (MIS), owing to factors such as the fragility of the anatomic features, poor visibility, limited manoeuvrability, and extreme requirements in terms of instrument handling with precise positioning. This work is centered on a fetal laser surgery procedure treating placental disorders. It proposes the use of haptic guidance to enhance the overall safety of this procedure and to simplify instrument handling. A method is described that provides effective guidance by installing a forbidden region virtual fixture over the placenta, thereby safeguarding adequate clearance between the instrument tip and the placenta. With a novel application of all-optical ultrasound distance sensing in which transmission and reception are performed with fibre optics, this method can be used with a sole reliance on intraoperatively acquired data. The added value of the guidance approach, in terms of safety and performance, is demonstrated in a series of experiments with a robotic platform.

A Continuum Robot and Control Interface for Surgical Assist in Fetoscopic Interventions.

Twin-twin transfusion syndrome requires interventional treatment using a fetoscopically introduced laser to sever the shared blood supply between the fetuses. This is a delicate procedure relying on small instrumentation with limited articulation to guide the laser tip and a narrow field of view to visualize all relevant vascular connections. In this letter, we report on a mechatronic design for a comanipulated instrument that combines concentric tube actuation to a larger manipulator constrained by a remote centre of motion. A stereoscopic camera is mounted at the distal tip and used for imaging. Our mechanism provides enhanced dexterity and stability of the imaging device. We demonstrate that the imaging system can be used for computing geometry and enhancing the view at the operating site. Results using electromagnetic sensors for verification and comparison to visual odometry from the distal sensor show that our system is promising and can be developed further for multiple clinical needs in fetoscopic procedures.

From medical imaging data to 3D printed anatomical models.

Anatomical models are important training and teaching tools in the clinical environment and are routinely used in medical imaging research. Advances in segmentation algorithms and increased availability of three-dimensional (3D) printers have made it possible to create cost-efficient patient-specific models without expert knowledge. We introduce a general workflow that can be used to convert volumetric medical imaging data (as generated by Computer Tomography (CT)) to 3D printed physical models. This process is broken up into three steps: image segmentation, mesh refinement and 3D printing. To lower the barrier to entry and provide the best options when aiming to 3D print an anatomical model from medical images, we provide an overview of relevant free and open-source image segmentation tools as well as 3D printing technologies. We demonstrate the utility of this streamlined workflow by creating models of ribs, liver, and lung using a Fused Deposition Modelling 3D printer.

Interventional Photoacoustic Imaging of the Human Placenta with Ultrasonic Tracking for Minimally Invasive Fetal Surgeries.

Image guidance plays a central role in minimally invasive fetal surgery such as photocoagulation of inter-twin placental anastomosing vessels to treat twin-to-twin transfusion syndrome (TTTS). Fetoscopic guidance provides insufficient sensitivity for imaging the vasculature that lies beneath the fetal placental surface due to strong light scattering in biological tissues. Incomplete photocoagulation of anastamoses is associated with postoperative complications and higher perinatal mortality. In this study, we investigated the use of multi-spectral photoacoustic (PA) imaging for better visualization of the placental vasculature. Excitation light was delivered with an optical fiber with dimensions that are compatible with the working channel of a fetoscope. Imaging was performed on an ex vivo normal term human placenta collected at Caesarean section birth. The photoacoustically-generated ultrasound signals were received by an external clinical linear array ultrasound imaging probe. A vein under illumination on the fetal placenta surface was visualized with PA imaging, and good correspondence was obtained between the measured PA spectrum and the optical absorption spectrum of deoxygenated blood. The delivery fiber had an attached fiber optic ultrasound sensor positioned directly adjacent to it, so that its spatial position could be tracked by receiving transmissions from the ultrasound imaging probe. This study provides strong indications that PA imaging in combination with ultrasonic tracking could be useful for detecting the human placental vasculature during minimally invasive fetal surgery.