Bridging the simulation-to-real gap for AI-based needle and target detection in robot-assisted ultrasound-guided interventions.

BACKGROUND: Artificial intelligence (AI)-powered, robot-assisted, and ultrasound (US)-guided interventional radiology has the potential to increase the efficacy and cost-efficiency of interventional procedures while improving postsurgical outcomes and reducing the burden for medical personnel. METHODS: To overcome the lack of available clinical data needed to train state-of-the-art AI models, we propose a novel approach for generating synthetic ultrasound data from real, clinical preoperative three-dimensional (3D) data of different imaging modalities. With the synthetic data, we trained a deep learning-based detection algorithm for the localization of needle tip and target anatomy in US images. We validated our models on real, in vitro US data. RESULTS: The resulting models generalize well to unseen synthetic data and experimental in vitro data making the proposed approach a promising method to create AI-based models for applications of needle and target detection in minimally invasive US-guided procedures. Moreover, we show that by one-time calibration of the US and robot coordinate frames, our tracking algorithm can be used to accurately fine-position the robot in reach of the target based on 2D US images alone. CONCLUSIONS: The proposed data generation approach is sufficient to bridge the simulation-to-real gap and has the potential to overcome data paucity challenges in interventional radiology. The proposed AI-based detection algorithm shows very promising results in terms of accuracy and frame rate. RELEVANCE STATEMENT: This approach can facilitate the development of next-generation AI algorithms for patient anatomy detection and needle tracking in US and their application to robotics. KEY POINTS: • AI-based methods show promise for needle and target detection in US-guided interventions. • Publicly available, annotated datasets for training AI models are limited. • Synthetic, clinical-like US data can be generated from magnetic resonance or computed tomography data. • Models trained with synthetic US data generalize well to real in vitro US data. • Target detection with an AI model can be used for fine positioning of the robot.

Learning metric volume estimation of fruits and vegetables from short monocular video sequences.

We present a novel approach for extracting metric volume information of fruits and vegetables from short monocular video sequences and associated inertial data recorded with a hand-held smartphone. Estimated segmentation masks from a pre-trained object detector are fused with the predicted change in relative pose obtained from the inertial data to predict the class and volume of the objects of interest. Our approach works with simple RGB video frames and inertial data which are readily available from modern smartphones. It does not require reference objects of known size in the video frames. Using a balanced validation dataset, we achieve a classification accuracy of 95% and a mean absolute percentage error for the volume prediction of 16% on untrained objects, which is comparable to state-of-the-art results requiring more elaborated data recording setups. A very accurate estimation of the model uncertainty is achieved through ensembling and the use of Gaussian negative log-likelihood loss. The dataset used in our experiments including ground-truth volume information is available at https://sst.aau.at/cns/datasets.

Automated Data Annotation for 6-DoF AI-Based Navigation Algorithm Development.

Accurately estimating the six degree of freedom (6-DoF) pose of objects in images is essential for a variety of applications such as robotics, autonomous driving, and autonomous, AI, and vision-based navigation for unmanned aircraft systems (UAS). Developing such algorithms requires large datasets; however, generating those is tedious as it requires annotating the 6-DoF relative pose of each object of interest present in the image w.r.t. to the camera. Therefore, this work presents a novel approach that automates the data acquisition and annotation process and thus minimizes the annotation effort to the duration of the recording. To maximize the quality of the resulting annotations, we employ an optimization-based approach for determining the extrinsic calibration parameters of the camera. Our approach can handle multiple objects in the scene, automatically providing ground-truth labeling for each object and taking into account occlusion effects between different objects. Moreover, our approach can not only be used to generate data for 6-DoF pose estimation and corresponding 3D-models but can be also extended to automatic dataset generation for object detection, instance segmentation, or volume estimation for any kind of object.

Measuring the Uncertainty of Predictions in Deep Neural Networks with Variational Inference.

We present a novel approach for training deep neural networks in a Bayesian way. Compared to other Bayesian deep learning formulations, our approach allows for quantifying the uncertainty in model parameters while only adding very few additional parameters to be optimized. The proposed approach uses variational inference to approximate the intractable a posteriori distribution on basis of a normal prior. By representing the a posteriori uncertainty of the network parameters per network layer and depending on the estimated parameter expectation values, only very few additional parameters need to be optimized compared to a non-Bayesian network. We compare our approach to classical deep learning, Bernoulli dropout and Bayes by Backprop using the MNIST dataset. Compared to classical deep learning, the test error is reduced by 15%. We also show that the uncertainty information obtained can be used to calculate credible intervals for the network prediction and to optimize network architecture for the dataset at hand. To illustrate that our approach also scales to large networks and input vector sizes, we apply it to the GoogLeNet architecture on a custom dataset, achieving an average accuracy of 0.92. Using 95% credible intervals, all but one wrong classification result can be detected.

Serial time-resolved crystallography of photosystem II using a femtosecond X-ray laser.

Photosynthesis, a process catalysed by plants, algae and cyanobacteria converts sunlight to energy thus sustaining all higher life on Earth. Two large membrane protein complexes, photosystem I and II (PSI and PSII), act in series to catalyse the light-driven reactions in photosynthesis. PSII catalyses the light-driven water splitting process, which maintains the Earth's oxygenic atmosphere. In this process, the oxygen-evolving complex (OEC) of PSII cycles through five states, S0 to S4, in which four electrons are sequentially extracted from the OEC in four light-driven charge-separation events. Here we describe time resolved experiments on PSII nano/microcrystals from Thermosynechococcus elongatus performed with the recently developed technique of serial femtosecond crystallography. Structures have been determined from PSII in the dark S1 state and after double laser excitation (putative S3 state) at 5 and 5.5 Å resolution, respectively. The results provide evidence that PSII undergoes significant conformational changes at the electron acceptor side and at the Mn4CaO5 core of the OEC. These include an elongation of the metal cluster, accompanied by changes in the protein environment, which could allow for binding of the second substrate water molecule between the more distant protruding Mn (referred to as the 'dangler' Mn) and the Mn3CaOx cubane in the S2 to S3 transition, as predicted by spectroscopic and computational studies. This work shows the great potential for time-resolved serial femtosecond crystallography for investigation of catalytic processes in biomolecules.

Structure of a photosynthetic reaction centre determined by serial femtosecond crystallography.

Serial femtosecond crystallography is an X-ray free-electron-laser-based method with considerable potential to have an impact on challenging problems in structural biology. Here we present X-ray diffraction data recorded from microcrystals of the Blastochloris viridis photosynthetic reaction centre to 2.8 Å resolution and determine its serial femtosecond crystallography structure to 3.5 Å resolution. Although every microcrystal is exposed to a dose of 33 MGy, no signs of X-ray-induced radiation damage are visible in this integral membrane protein structure.

Optimal mapping of x-ray laser diffraction patterns into three dimensions using routing algorithms.

Coherent diffractive imaging with x-ray free-electron lasers (XFEL) promises high-resolution structure determination of noncrystalline objects. Randomly oriented particles are exposed to XFEL pulses for acquisition of two-dimensional (2D) diffraction snapshots. The knowledge of their orientations enables 3D imaging by multiview reconstruction, combining 2D diffraction snapshots in different orientations. Here we introduce a globally optimal algorithm that can infer these orientations. We apply it to experimental XFEL data of nanoparticles and so determine their 3D electron density.

Sensing the wavefront of x-ray free-electron lasers using aerosol spheres.

Characterizing intense, focused x-ray free electron laser (FEL) pulses is crucial for their use in diffractive imaging. We describe how the distribution of average phase tilts and intensities on hard x-ray pulses with peak intensities of 10(21) W/m(2) can be retrieved from an ensemble of diffraction patterns produced by 70 nm-radius polystyrene spheres, in a manner that mimics wavefront sensors. Besides showing that an adaptive geometric correction may be necessary for diffraction data from randomly injected sample sources, our paper demonstrates the possibility of collecting statistics on structured pulses using only the diffraction patterns they generate and highlights the imperative to study its impact on single-particle diffractive imaging.

Natively inhibited Trypanosoma brucei cathepsin B structure determined by using an X-ray laser.

The Trypanosoma brucei cysteine protease cathepsin B (TbCatB), which is involved in host protein degradation, is a promising target to develop new treatments against sleeping sickness, a fatal disease caused by this protozoan parasite. The structure of the mature, active form of TbCatB has so far not provided sufficient information for the design of a safe and specific drug against T. brucei. By combining two recent innovations, in vivo crystallization and serial femtosecond crystallography, we obtained the room-temperature 2.1 angstrom resolution structure of the fully glycosylated precursor complex of TbCatB. The structure reveals the mechanism of native TbCatB inhibition and demonstrates that new biomolecular information can be obtained by the "diffraction-before-destruction" approach of x-ray free-electron lasers from hundreds of thousands of individual microcrystals.

Toward unsupervised single-shot diffractive imaging of heterogeneous particles using X-ray free-electron lasers.

Single shot diffraction imaging experiments via X-ray free-electron lasers can generate as many as hundreds of thousands of diffraction patterns of scattering objects. Recovering the real space contrast of a scattering object from these patterns currently requires a reconstruction process with user guidance in a number of steps, introducing severe bottlenecks in data processing. We present a series of measures that replace user guidance with algorithms that reconstruct contrasts in an unsupervised fashion. We demonstrate the feasibility of automating the reconstruction process by generating hundreds of contrasts obtained from soot particle diffraction experiments.

High-resolution protein structure determination by serial femtosecond crystallography.

Structure determination of proteins and other macromolecules has historically required the growth of high-quality crystals sufficiently large to diffract x-rays efficiently while withstanding radiation damage. We applied serial femtosecond crystallography (SFX) using an x-ray free-electron laser (XFEL) to obtain high-resolution structural information from microcrystals (less than 1 micrometer by 1 micrometer by 3 micrometers) of the well-characterized model protein lysozyme. The agreement with synchrotron data demonstrates the immediate relevance of SFX for analyzing the structure of the large group of difficult-to-crystallize molecules.

Femtosecond free-electron laser x-ray diffraction data sets for algorithm development.

We describe femtosecond X-ray diffraction data sets of viruses and nanoparticles collected at the Linac Coherent Light Source. The data establish the first large benchmark data sets for coherent diffraction methods freely available to the public, to bolster the development of algorithms that are essential for developing this novel approach as a useful imaging technique. Applications are 2D reconstructions, orientation classification and finally 3D imaging by assembling 2D patterns into a 3D diffraction volume.

Anti-contamination device for cryogenic soft X-ray diffraction microscopy.

Cryogenic microscopy allows one to view frozen hydrated biological and soft matter specimens with good structural preservation and a high degree of stability against radiation damage. We describe a liquid nitrogen-cooled anti-contamination device for cryogenic X-ray diffraction microscopy. The anti-contaminator greatly reduces the buildup of ice layers on the specimen due to condensation of residual water vapor in the experimental vacuum chamber. We show by coherent X-ray diffraction measurements that this leads to fivefold reduction of background scattering, which is important for far-field X-ray diffraction microscopy of biological specimens.

Incorrect support and missing center tolerances of phasing algorithms.

In x-ray diffraction microscopy, iterative algorithms retrieve reciprocal space phase information, and a real space image, from an object's coherent diffraction intensities through the use of a priori information such as a finite support constraint. In many experiments, the object's shape or support is not well known, and the diffraction pattern is incompletely measured. We describe here computer simulations to look at the effects of both of these possible errors when using several common reconstruction algorithms. Overly tight object supports prevent successful convergence; however, we show that this can often be recognized through pathological behavior of the phase retrieval transfer function. Dynamic range limitations often make it difficult to record the central speckles of the diffraction pattern. We show that this leads to increasing artifacts in the image when the number of missing central speckles exceeds about 10, and that the removal of unconstrained modes from the reconstructed image is helpful only when the number of missing central speckles is less than about 50. This simulation study helps in judging the reconstructability of experimentally recorded coherent diffraction patterns.

Data preparation and evaluation techniques for x-ray diffraction microscopy.

The post-experiment processing of X-ray Diffraction Microscopy data is often time-consuming and difficult. This is mostly due to the fact that even if a preliminary result has been reconstructed, there is no definitive answer as to whether or not a better result with more consistently retrieved phases can still be obtained. We show here that the first step in data analysis, the assembly of two-dimensional diffraction patterns from a large set of raw diffraction data, is crucial to obtaining reconstructions of highest possible consistency. We have developed software that automates this process and results in consistently accurate diffraction patterns. We have furthermore derived some criteria of validity for a tool commonly used to assess the consistency of reconstructions, the phase retrieval transfer function, and suggest a modified version that has improved utility for judging reconstruction quality.

High-resolution x-ray diffraction microscopy of specifically labeled yeast cells.

X-ray diffraction microscopy complements other x-ray microscopy methods by being free of lens-imposed radiation dose and resolution limits, and it allows for high-resolution imaging of biological specimens too thick to be viewed by electron microscopy. We report here the highest resolution (11-13 nm) x-ray diffraction micrograph of biological specimens, and a demonstration of molecular-specific gold labeling at different depths within cells via through-focus propagation of the reconstructed wavefield. The lectin concanavalin A conjugated to colloidal gold particles was used to label the alpha-mannan sugar in the cell wall of the yeast Saccharomyces cerevisiae. Cells were plunge-frozen in liquid ethane and freeze-dried, after which they were imaged whole using x-ray diffraction microscopy at 750 eV photon energy.

Signal-to-noise and radiation exposure considerations in conventional and diffraction x-ray microscopy.

Using a signal-to-noise ratio estimation based on correlations between multiple simulated images, we compare the dose efficiency of two soft x-ray imaging systems: incoherent brightfield imaging using zone plate optics in a transmission x-ray microscope (TXM), and x-ray diffraction microscopy (XDM) where an image is reconstructed from the far-field coherent diffraction pattern. In XDM one must computationally phase weak diffraction signals; in TXM one suffers signal losses due to the finite numerical aperture and efficiency of the optics. In simulations with objects representing isolated cells such as yeast, we find that XDM has the potential for delivering equivalent resolution images using fewer photons. This can be an important advantage for studying radiation-sensitive biological and soft matter specimens.

Soft X-ray diffraction microscopy of a frozen hydrated yeast cell.

We report the first image of an intact, frozen hydrated eukaryotic cell using x-ray diffraction microscopy, or coherent x-ray diffraction imaging. By plunge freezing the specimen in liquid ethane and maintaining it below -170 degrees C, artifacts due to dehydration, ice crystallization, and radiation damage are greatly reduced. In this example, coherent diffraction data using 520 eV x rays were recorded and reconstructed to reveal a budding yeast cell at a resolution better than 25 nm. This demonstration represents an important step towards high resolution imaging of cells in their natural, hydrated state, without limitations imposed by x-ray optics.