Mitigation of motion-induced artifacts in cone beam computed tomography using deep convolutional neural networks.

BACKGROUND: Cone beam computed tomography (CBCT) is often employed on radiation therapy treatment devices (linear accelerators) used in image-guided radiation therapy (IGRT). For each treatment session, it is necessary to obtain the image of the day in order to accurately position the patient and to enable adaptive treatment capabilities including auto-segmentation and dose calculation. Reconstructed CBCT images often suffer from artifacts, in particular those induced by patient motion. Deep-learning based approaches promise ways to mitigate such artifacts. PURPOSE: We propose a novel deep-learning based approach with the goal to reduce motion induced artifacts in CBCT images and improve image quality. It is based on supervised learning and includes neural network architectures employed as pre- and/or post-processing steps during CBCT reconstruction. METHODS: Our approach is based on deep convolutional neural networks which complement the standard CBCT reconstruction, which is performed either with the analytical Feldkamp-Davis-Kress (FDK) method, or with an iterative algebraic reconstruction technique (SART-TV). The neural networks, which are based on refined U-net architectures, are trained end-to-end in a supervised learning setup. Labeled training data are obtained by means of a motion simulation, which uses the two extreme phases of 4D CT scans, their deformation vector fields, as well as time-dependent amplitude signals as input. The trained networks are validated against ground truth using quantitative metrics, as well as by using real patient CBCT scans for a qualitative evaluation by clinical experts. RESULTS: The presented novel approach is able to generalize to unseen data and yields significant reductions in motion induced artifacts as well as improvements in image quality compared with existing state-of-the-art CBCT reconstruction algorithms (up to +6.3 dB and +0.19 improvements in peak signal-to-noise ratio, PSNR, and structural similarity index measure, SSIM, respectively), as evidenced by validation with an unseen test dataset, and confirmed by a clinical evaluation on real patient scans (up to 74% preference for motion artifact reduction over standard reconstruction). CONCLUSIONS: For the first time, it is demonstrated, also by means of clinical evaluation, that inserting deep neural networks as pre- and post-processing plugins in the existing 3D CBCT reconstruction and trained end-to-end yield significant improvements in image quality and reduction of motion artifacts.

Reducing residual-motion artifacts in iterative 3D CBCT reconstruction in image-guided radiation therapy.

PURPOSE: Recent evaluations of a 3D iterative cone-beam computed tomography (iCBCT) reconstruction method available on Varian radiation treatment devices demonstrated that iCBCT provides superior image quality when compared to analytical Feldkamp-Davis-Kress (FDK) method. However, iCBCT employs statistical penalized likelihood (PL) that is known to be highly sensitive to inconsistencies due to physiological motion occurring during the acquisition. We propose a computationally inexpensive extension of iCBCT addressing this deficiency. METHODS: During the iterative process, the gradients of PL are modified to avoid the generation of motion-related artifacts. To assess the impact of this modification, we propose a motion simulation generating CBCT projections of a moving anatomy together with artifact-free images used as ground truth. Contrast-to-noise ratio and power spectra of difference images are computed to quantify the impact of the motion on reconstructed CBCT volumes as well as the effect of the proposed modification. RESULTS: Using both simulated and clinical data, it is shown that the motion of patient's abdominal wall during the acquisition results in artifacts that can be quantified as low-frequency components in volumes reconstructed with iCBCT. Further, a quantitative evaluation demonstrates that the proposed modification of PL reduces these low-frequency components. While preserving the advantages of PL, it effectively suppresses the propagation of motion-related artifacts into clinically important regions, thus increasing the motion resiliency of iCBCT. CONCLUSIONS: The proposed modified iterative reconstruction method significantly improves the quality of CBCT images of anatomies suffering from residual motion.

Biomechanical modelling of probe to tissue interaction during ultrasound scanning.

PURPOSE: Biomechanical simulation of anatomical deformations caused by ultrasound probe pressure is of outstanding importance for several applications, from the testing of robotic acquisition systems to multi-modal image fusion and development of ultrasound training platforms. Different approaches can be exploited for modelling the probe-tissue interaction, each achieving different trade-offs among accuracy, computation time and stability. METHODS: We assess the performances of different strategies based on the finite element method for modelling the interaction between the rigid probe and soft tissues. Probe-tissue contact is modelled using (i) penalty forces, (ii) constraint forces, and (iii) by prescribing the displacement of the mesh surface nodes. These methods are tested in the challenging context of ultrasound scanning of the breast, an organ undergoing large nonlinear deformations during the procedure. RESULTS: The obtained results are evaluated against those of a non-physically based method. While all methods achieve similar accuracy, performance in terms of stability and speed shows high variability, especially for those methods modelling the contacts explicitly. Overall, prescribing surface displacements is the approach with best performances, but it requires prior knowledge of the contact area and probe trajectory. CONCLUSIONS: In this work, we present different strategies for modelling probe-tissue interaction, each able to achieve different compromises among accuracy, speed and stability. The choice of the preferred approach highly depends on the requirements of the specific clinical application. Since the presented methodologies can be applied to describe general tool-tissue interactions, this work can be seen as a reference for researchers seeking the most appropriate strategy to model anatomical deformation induced by the interaction with medical tools.

Elastic Registration Based on Compliance Analysis and Biomechanical Graph Matching.

An automatic elastic registration method suited for vascularized organs is proposed. The vasculature in both the preoperative and intra-operative images is represented as a graph. A typical application of this method is the fusion of pre-operative information onto the organ during surgery, to compensate for the limited details provided by the intra-operative imaging modality (e.g. cone beam CT) and to cope with changes in the shape of the organ. Due to image modalities differences and organ deformation, each graph has a different topology and shape. The adaptive compliance graph matching (ACGM) method presented does not require any manual initialization, handles intra-operative nonrigid deformations of up to 65 mm and computes a complete displacement field over the organ from only the matched vasculature. ACGM is better than the previous biomechanical graph matching method (Garcia Guevara et al. IJCARS, 2018) (BGM) because it uses an efficient biomechanical vascularized liver model to compute the organ's transformation and the vessels bifurcations compliance. This allows to efficiently find the best graph matches with a novel compliance-based adaptive search. These contributions are evaluated on 10 realistic synthetic and 2 porcine automatically segmented datasets. ACGM obtains better target registration error (TRE) than BGM, with an average TRE in the real datasets of 4.2 mm compared to 6.5 mm, respectively. It also is up to one order of magnitude faster, less dependent on the parameters used and more robust to noise.

Correction to: Position-based modeling of lesion displacement in ultrasound-guided breast biopsy.

The original version of this article unfortunately contained a mistake.

Position-based modeling of lesion displacement in ultrasound-guided breast biopsy.

PURPOSE: Although ultrasound (US) images represent the most popular modality for guiding breast biopsy, malignant regions are often missed by sonography, thus preventing accurate lesion localization which is essential for a successful procedure. Biomechanical models can support the localization of suspicious areas identified on a preoperative image during US scanning since they are able to account for anatomical deformations resulting from US probe pressure. We propose a deformation model which relies on position-based dynamics (PBD) approach to predict the displacement of internal targets induced by probe interaction during US acquisition. METHODS: The PBD implementation available in NVIDIA FleX is exploited to create an anatomical model capable of deforming online. Simulation parameters are initialized on a calibration phantom under different levels of probe-induced deformations; then, they are fine-tuned by minimizing the localization error of a US-visible landmark of a realistic breast phantom. The updated model is used to estimate the displacement of other internal lesions due to probe-tissue interaction. RESULTS: The localization error obtained when applying the PBD model remains below 11 mm for all the tumors even for input displacements in the order of 30 mm. This proposed method obtains results aligned with FE models with faster computational performance, suitable for real-time applications. In addition, it outperforms rigid model used to track lesion position in US-guided breast biopsies, at least halving the localization error for all the displacement ranges considered. CONCLUSION: Position-based dynamics approach has proved to be successful in modeling breast tissue deformations during US acquisition. Its stability, accuracy and real-time performance make such model suitable for tracking lesions displacement during US-guided breast biopsy.

FiloGen: A Model-Based Generator of Synthetic 3-D Time-Lapse Sequences of Single Motile Cells With Growing and Branching Filopodia.

The existence of diverse image datasets accompanied by reference annotations is a crucial prerequisite for an objective benchmarking of bioimage analysis methods. Nevertheless, such a prerequisite is hard to satisfy for time lapse, multidimensional fluorescence microscopy image data, manual annotations of which are laborious and often impracticable. In this paper, we present a simulation system capable of generating 3-D time-lapse sequences of single motile cells with filopodial protrusions of user-controlled structural and temporal attributes, such as the number, thickness, length, level of branching, and lifetime of filopodia, accompanied by inherently generated reference annotations. The proposed simulation system involves three globally synchronized modules, each being responsible for a separate task: the evolution of filopodia on a molecular level, linear elastic deformation of the entire cell with filopodia, and the synthesis of realistic, time-coherent cell texture. Its flexibility is demonstrated by generating multiple synthetic 3-D time-lapse sequences of single lung cancer cells of two different phenotypes, qualitatively and quantitatively resembling their real counterparts acquired using a confocal fluorescence microscope.

Biomechanics-based graph matching for augmented CT-CBCT.

PURPOSE: Augmenting intraoperative cone beam computed tomography (CBCT) images with preoperative computed tomography data in the context of image-guided liver therapy is proposed. The expected benefit is an improved visualization of tumor(s), vascular system and other internal structures of interest. METHOD: An automatic elastic registration based on matching of vascular trees extracted from both the preoperative and intraoperative images is presented. Although methods dedicated to nonrigid graph matching exist, they are not efficient when large intraoperative deformations of tissues occur, as is the case during the liver surgery. The contribution is an extension of the graph matching algorithm using Gaussian process regression (GPR) (Serradell et al. in IEEE Trans Pattern Anal Mach Intell 37(3):625-638, 2015): First, an improved GPR matching is introduced by imposing additional constraints during the matching when the number of hypothesis is large; like the original algorithm, this extended version does not require a manual initialization of matching. Second, a fast biomechanical model is employed to make the method capable of handling large deformations. RESULTS: The proposed automatic intraoperative augmentation is evaluated on both synthetic and real data. It is demonstrated that the algorithm is capable of handling large deformations, thus being more robust and reliable than previous approaches. Moreover, the time required to perform the elastic registration is compatible with the intraoperative navigation scenario. CONCLUSION: A biomechanics-based graph matching method, which can handle large deformations and augment intraoperative CBCT, is presented and evaluated.

Fast elastic registration of soft tissues under large deformations.

A fast and accurate fusion of intra-operative images with a pre-operative data is a key component of computer-aided interventions which aim at improving the outcomes of the intervention while reducing the patient's discomfort. In this paper, we focus on the problematic of the intra-operative navigation during abdominal surgery, which requires an accurate registration of tissues undergoing large deformations. Such a scenario occurs in the case of partial hepatectomy: to facilitate the access to the pathology, e.g. a tumor located in the posterior part of the right lobe, the surgery is performed on a patient in lateral position. Due to the change in patient's position, the resection plan based on the pre-operative CT scan acquired in the supine position must be updated to account for the deformations. We suppose that an imaging modality, such as the cone-beam CT, provides the information about the intra-operative shape of an organ, however, due to the reduced radiation dose and contrast, the actual locations of the internal structures necessary to update the planning are not available. To this end, we propose a method allowing for fast registration of the pre-operative data represented by a detailed 3D model of the liver and its internal structure and the actual configuration given by the organ surface extracted from the intra-operative image. The algorithm behind the method combines the iterative closest point technique with a biomechanical model based on a co-rotational formulation of linear elasticity which accounts for large deformations of the tissue. The performance, robustness and accuracy of the method is quantitatively assessed on a control semi-synthetic dataset with known ground truth and a real dataset composed of nine pairs of abdominal CT scans acquired in supine and flank positions. It is shown that the proposed surface-matching method is capable of reducing the target registration error evaluated of the internal structures of the organ from more than 40 mm to less then 10 mm. Moreover, the control data is used to demonstrate the compatibility of the method with intra-operative clinical scenario, while the real datasets are utilized to study the impact of parametrization on the accuracy of the method. The method is also compared to a state-of-the art intensity-based registration technique in terms of accuracy and performance.

Non-Rigid Contour-Based Registration of Cell Nuclei in 2-D Live Cell Microscopy Images Using a Dynamic Elasticity Model.

The analysis of the pure motion of subnuclear structures without influence of the cell nucleus motion and deformation is essential in live cell imaging. In this paper, we propose a 2-D contour-based image registration approach for compensation of nucleus motion and deformation in fluorescence microscopy time-lapse sequences. The proposed approach extends our previous approach, which uses a static elasticity model to register cell images. Compared with that scheme, the new approach employs a dynamic elasticity model for the forward simulation of nucleus motion and deformation based on the motion of its contours. The contour matching process is embedded as a constraint into the system of equations describing the elastic behavior of the nucleus. This results in better performance in terms of the registration accuracy. Our approach was successfully applied to real live cell microscopy image sequences of different types of cells including image data that was specifically designed and acquired for evaluation of cell image registration methods. An experimental comparison with the existing contour-based registration methods and an intensity-based registration method has been performed. We also studied the dependence of the results on the choice of method parameters.

3D Physics-Based Registration of 2D Dynamic MRI Data.

We present a method allowing for intra-operative targeting of a specific anatomical feature. The method is based on a registration of 3D pre-operative data to 2D intra-operative images. Such registration is performed using an elastic model reconstructed from the 3D images, in combination with sliding constraints imposed via Lagrange multipliers. We register the pre-operative data, where the feature is clearly detectable, to intra-operative dynamic images where such feature is no more visible. Despite the lack of visibility on the 2D MRI images, we are able both to determine the location of the target as well as follow its displacement due to respiratory motion.

Patient-Specific Biomechanical Modeling for Guidance During Minimally-Invasive Hepatic Surgery.

During the minimally-invasive liver surgery, only the partial surface view of the liver is usually provided to the surgeon via the laparoscopic camera. Therefore, it is necessary to estimate the actual position of the internal structures such as tumors and vessels from the pre-operative images. Nevertheless, such task can be highly challenging since during the intervention, the abdominal organs undergo important deformations due to the pneumoperitoneum, respiratory and cardiac motion and the interaction with the surgical tools. Therefore, a reliable automatic system for intra-operative guidance requires fast and reliable registration of the pre- and intra-operative data. In this paper we present a complete pipeline for the registration of pre-operative patient-specific image data to the sparse and incomplete intra-operative data. While the intra-operative data is represented by a point cloud extracted from the stereo-endoscopic images, the pre-operative data is used to reconstruct a biomechanical model which is necessary for accurate estimation of the position of the internal structures, considering the actual deformations. This model takes into account the patient-specific liver anatomy composed of parenchyma, vascularization and capsule, and is enriched with anatomical boundary conditions transferred from an atlas. The registration process employs the iterative closest point technique together with a penalty-based method. We perform a quantitative assessment based on the evaluation of the target registration error on synthetic data as well as a qualitative assessment on real patient data. We demonstrate that the proposed registration method provides good results in terms of both accuracy and robustness w.r.t. the quality of the intra-operative data.

Preoperative trajectory planning for percutaneous procedures in deformable environments.

In image-guided percutaneous interventions, a precise planning of the needle path is a key factor to a successful intervention. In this paper we propose a novel method for computing a patient-specific optimal path for such interventions, accounting for both the deformation of the needle and soft tissues due to the insertion of the needle in the body. To achieve this objective, we propose an optimization method for estimating preoperatively a curved trajectory allowing to reach a target even in the case of tissue motion and needle bending. Needle insertions are simulated and regarded as evaluations of the objective function by the iterative planning process. In order to test the planning algorithm, it is coupled with a fast needle insertion simulation involving a flexible needle model and soft tissue finite element modeling, and experimented on the use-case of thermal ablation of liver tumors. Our algorithm has been successfully tested on twelve datasets of patient-specific geometries. Fast convergence to the actual optimal solution has been shown. This method is designed to be adapted to a wide range of percutaneous interventions.

Impact of Soft Tissue Heterogeneity on Augmented Reality for Liver Surgery.

This paper presents a method for real-time augmented reality of internal liver structures during minimally invasive hepatic surgery. Vessels and tumors computed from pre-operative CT scans can be overlaid onto the laparoscopic view for surgery guidance. Compared to current methods, our method is able to locate the in-depth positions of the tumors based on partial three-dimensional liver tissue motion using a real-time biomechanical model. This model permits to properly handle the motion of internal structures even in the case of anisotropic or heterogeneous tissues, as it is the case for the liver and many anatomical structures. Experimentations conducted on phantom liver permits to measure the accuracy of the augmentation while real-time augmentation on in vivo human liver during real surgery shows the benefits of such an approach for minimally invasive surgery.

Atlas-based transfer of boundary conditions for biomechanical simulation.

An environment composed of different types of living tissues (such as the abdominal cavity) reveals a high complexity of boundary conditions, which are the attachments (e.g. connective tissues, ligaments) connecting different anatomical structures. Together with the material properties, the boundary conditions have a significant influence on the mechanical response of the organs, however corresponding correct mechanical modeling remains a challenging task, as the connective structures are difficult to identify in certain standard imaging modalities. In this paper, we present a method for automatic modeling of boundary conditions in deformable anatomical structures, which is an important step in patient-specific biomechanical simulations. The method is based on a statistical atlas which gathers data defining the connective structures attached to the organ of interest. In order to transfer the information stored in the atlas to a specific patient, the atlas is registered to the patient data using a physics-based technique and the resulting boundary conditions are defined according to the mean position and variance available in the atlas. The method is evaluated using abdominal scans of ten patients. The results show that the atlas provides a sufficient information about the boundary conditions which can be reliably transferred to a specific patient. The boundary conditions obtained by the atlas-based transfer show a good match both with actual segmented boundary conditions and in terms of mechanical response of deformable organs.

Constraint-based simulation for non-rigid real-time registration.

In this paper we propose a method to address the problem of non-rigid registration in real-time. We use Lagrange multipliers and soft sliding constraints to combine data acquired from dynamic image sequence and a biomechanical model of the structure of interest. The biomechanical model plays a role of regularization to improve the robustness and the flexibility of the registration. We apply our method to a pre-operative 3D CT scan of a porcine liver that is registered to a sequence of 2D dynamic MRI slices during the respiratory motion. The finite element simulation provides a full 3D representation (including heterogeneities such as vessels, tumor,...) of the anatomical structure in real-time.

Complete Real-Time Liver Model Including Glisson's Capsule, Vascularization and Parenchyma.

Accurate biomechanical modeling of liver is of paramount interest in pre-operative planning or computer-aided per-operative guidance. Since the liver is an organ composed of three different components (parenchyma, vascularization and Glisson's capsule), an efficient and realistic simulation of its behaviour is a challenging task. In this paper we propose a complete model of liver where each component is modelled with different type of finite elements chosen according to the nature and mechanical properties of the component. The elements of different types are coupled via mechanical mapping encoded in the global stiffness matrix. In the result section, we first focus on simulation of Glisson's capsule using constant-strain triangular elements: we compare the model to a detailed non-real-time model and also reproduce previously published aspiration test showing the importance of the capsule. Finally, we demonstrate that the proposed complete liver model can be used in a real-time simulation.

Modeling and real-time simulation of a vascularized liver tissue.

In Europe only, about 100,000 deaths per year are related to cirrhosis or liver cancer. While surgery remains the option that offers the foremost success rate against such pathologies, several limitations still hinder its widespread development. Among the limiting factors is the lack of accurate planning systems, which has been a motivation for several recent works, aiming at better resection planning and training systems, relying on pre-operative imaging, anatomical and biomechanical modelling. While the vascular network in the liver plays a key role in defining the operative strategy, its influence at a biomechanical level has not been taken into account. In the paper we propose a real-time model of vascularized organs such as the liver. The model takes into account separate constitutive laws for the parenchyma and vessels, and defines a coupling mechanism between these two entities. In the evaluation section, we present results of in vitro porcine liver experiments that indicate a significant influence of vascular structures on the mechanical behaviour of tissue. We confirm the values obtained in the experiments by computer simulation using standard FEM. Finally, we show that the conventional modelling approach can be efficiently approximated with the proposed composite model capable of real-time calculations.

Sound-speed image reconstruction in sparse-aperture 3-D ultrasound transmission tomography.

The paper is focused on sound-speed image reconstruction in 3-D ultrasound transmission tomography. Along with ultrasound reflectivity and the attenuation coefficient, sound speed is an important parameter which is related to the type and pathological state of the imaged tissue. This is important in the intended application, breast cancer diagnosis. In contrast to 2-D ultrasound transmission tomography systems, a 3-D system can provide an isotropic spatial resolution in the x-, y-, and z-directions in reconstructed 3-D images of ultrasound parameters. Several challenges must, however, be addressed for 3-D systems-namely, a sparse transducer distribution, low signal-to-noise ratio, and higher computational complexity. These issues are addressed in terms of sound-speed image reconstruction, using edge-preserving regularized algebraic reconstruction in combination with synthetic aperture focusing. The critical points of the implementation are also discussed, because they are crucial to enable a complete 3-D image reconstruction. The methods were tested on a synthetic data set and on data sets measured with the Karlsruhe 3-D ultrasound computer tomography (USCT) I prototype using phantoms. The sound-speed estimates in the reconstructed volumes agreed with the reference values. The breast-phantom outlines and the lesion-mimicking objects were also detectable in the resulting sound-speed volumes.

Modified time-of-flight based calibration approach for ultrasonic computed tomography.

In the previous paper [11], a method for geometrical and transducer-time-delay auto-calibration of an ultrasonic computed tomography (USCT) system has been described, aiming at calibration of individual ultrasonic (US) transducer positions. The present contribution describes a novel modification of the method utilizing the particular USCT system concept: the exactly known spatial relations among transducers grouped in each of the transducer array systems (TASes). The algorithms used for the calibration remain based on the principles similar to the global positioning system (GPS) navigation, however, the positions and orientations of complete TASes are calibrated, rather than individual positions of transducers. This way, the number of unknowns is substantially reduced while the number of available equations remains unchanged. Consequently, a solution substantially more robust with respect to measurement noise can be obtained based on this highly overdetermined equation system. The method is capable of calibrating the individual positions of all ultrasonic transducers via their positions in TASes as well as their individual time delays at once during sc. empty measurement, without a need for any particular arrangements, e.g. calibration phantoms.