Deep-learning-based reconstruction of undersampled MRI to reduce scan times: a multicentre, retrospective, cohort study.

BACKGROUND: The extended acquisition times required for MRI limit its availability in resource-constrained settings. Consequently, accelerating MRI by undersampling k-space data, which is necessary to reconstruct an image, has been a long-standing but important challenge. We aimed to develop a deep convolutional neural network (dCNN) optimisation method for MRI reconstruction and to reduce scan times and evaluate its effect on image quality and accuracy of oncological imaging biomarkers. METHODS: In this multicentre, retrospective, cohort study, MRI data from patients with glioblastoma treated at Heidelberg University Hospital (775 patients and 775 examinations) and from the phase 2 CORE trial (260 patients, 1083 examinations, and 58 institutions) and the phase 3 CENTRIC trial (505 patients, 3147 examinations, and 139 institutions) were used to develop, train, and test dCNN for reconstructing MRI from highly undersampled single-coil k-space data with various acceleration rates (R=2, 4, 6, 8, 10, and 15). Independent testing was performed with MRIs from the phase 2/3 EORTC-26101 trial (528 patients with glioblastoma, 1974 examinations, and 32 institutions). The similarity between undersampled dCNN-reconstructed and original MRIs was quantified with various image quality metrics, including structural similarity index measure (SSIM) and the accuracy of undersampled dCNN-reconstructed MRI on downstream radiological assessment of imaging biomarkers in oncology (automated artificial intelligence-based quantification of tumour burden and treatment response) was performed in the EORTC-26101 test dataset. The public NYU Langone Health fastMRI brain test dataset (558 patients and 558 examinations) was used to validate the generalisability and robustness of the dCNN for reconstructing MRIs from available multi-coil (parallel imaging) k-space data. FINDINGS: In the EORTC-26101 test dataset, the median SSIM of undersampled dCNN-reconstructed MRI ranged from 0·88 to 0·99 across different acceleration rates, with 0·92 (95% CI 0·92-0·93) for 10-times acceleration (R=10). The 10-times undersampled dCNN-reconstructed MRI yielded excellent agreement with original MRI when assessing volumes of contrast-enhancing tumour (median DICE for spatial agreement of 0·89 [95% CI 0·88 to 0·89]; median volume difference of 0·01 cm3 [95% CI 0·00 to 0·03] equalling 0·21%; p=0·0036 for equivalence) or non-enhancing tumour or oedema (median DICE of 0·94 [95% CI 0·94 to 0·95]; median volume difference of -0·79 cm3 [95% CI -0·87 to -0·72] equalling -1·77%; p=0·023 for equivalence) in the EORTC-26101 test dataset. Automated volumetric tumour response assessment in the EORTC-26101 test dataset yielded an identical median time to progression of 4·27 months (95% CI 4·14 to 4·57) when using 10-times-undersampled dCNN-reconstructed or original MRI (log-rank p=0·80) and agreement in the time to progression in 374 (95·2%) of 393 patients with data. The dCNN generalised well to the fastMRI brain dataset, with significant improvements in the median SSIM when using multi-coil compared with single-coil k-space data (p<0·0001). INTERPRETATION: Deep-learning-based reconstruction of undersampled MRI allows for a substantial reduction of scan times, with a 10-times acceleration demonstrating excellent image quality while preserving the accuracy of derived imaging biomarkers for the assessment of oncological treatment response. Our developments are available as open source software and hold considerable promise for increasing the accessibility to MRI, pending further prospective validation. FUNDING: Deutsche Forschungsgemeinschaft (German Research Foundation) and an Else Kröner Clinician Scientist Endowed Professorship by the Else Kröner Fresenius Foundation.

Periprosthetic radiolucency in total wrist arthroplasty: a radiographic study.

This retrospective study investigates the incidences of periprosthetic radiolucency and the position of the prosthesis in patients who underwent total wrist arthroplasty. A total of 50 patients with a mean age of 58 years (SD 11) were included. The available dorsopalmar and lateral radiographs were categorized into the following groups: immediately postoperative and 1, 2, 3, 5 and beyond 6 years postoperatively. The findings of this study indicate that periprosthetic radiolucency is a progressive phenomenon that originates at the bone adjacent to the joint line, possibly due to stress shielding. The size of the periprosthetic radiolucency showed no correlation with any clinical parameter, nor can its size be predicted by intraoperative implant positioning. However, a significant correlation was observed between a reduced implant-middle finger carpometacarpal distance and higher postoperative pain levels as well as patient dissatisfaction. Revision surgery after total wrist arthroplasty should not be solely guided by radiological signs of periprosthetic radiolucency. Instead, this study suggests that consideration for revision surgery should be reserved for symptomatic patients experiencing persistent pain and swelling accompanied by radiographic evidence of carpal implant subsidence.Level of evidence: IV.

Optical Flow-Guided Cine MRI Segmentation With Learned Corrections.

In cardiac cine magnetic resonance imaging (MRI), the heart is repeatedly imaged at numerous time points during the cardiac cycle. Frequently, the temporal evolution of a certain region of interest such as the ventricles or the atria is highly relevant for clinical diagnosis. In this paper, we devise a novel approach that allows for an automatized propagation of an arbitrary region of interest (ROI) along the cardiac cycle from respective annotated ROIs provided by medical experts at two different points in time, most frequently at the end-systolic (ES) and the end-diastolic (ED) cardiac phases. At its core, a 3D TV- L1 -based optical flow algorithm computes the apparent motion of consecutive MRI images in forward and backward directions. Subsequently, the given terminal annotated masks are propagated by this bidirectional optical flow in 3D, which results, however, in improper initial estimates of the segmentation masks due to numerical inaccuracies. These initially propagated segmentation masks are then refined by a 3D U-Net-based convolutional neural network (CNN), which was trained to enforce consistency with the forward and backward warped masks using a novel loss function. Moreover, a penalization term in the loss function controls large deviations from the initial segmentation masks. This method is benchmarked both on a new dataset with annotated single ventricles containing patients with severe heart diseases and on a publicly available dataset with different annotated ROIs. We emphasize that our novel loss function enables fine-tuning the CNN on a single patient, thereby yielding state-of-the-art results along the complete cardiac cycle.

Reduction of Gadolinium-Based Contrast Agents in MRI Using Convolutional Neural Networks and Different Input Protocols: Limited Interchangeability of Synthesized Sequences With Original Full-Dose Images Despite Excellent Quantitative Performance.

OBJECTIVES: The purpose of this study was to implement a state-of-the-art convolutional neural network used to synthesize artificial T1-weighted (T1w) full-dose images from corresponding noncontrast and low-dose images (using various settings of input sequences) and test its performance on a patient population acquired prospectively. MATERIALS AND METHODS: In this monocentric, institutional review board-approved study, a total of 138 participants were included who received an adapted imaging protocol with acquisition of a T1w low dose after administration of 10% of the standard dose and acquisition of a T1w full dose after administration of the remaining 90% of the standard dose of a gadolinium-containing contrast agent. A total of 83 participants formed the training sample (51.7 ± 16.5 years, 36 women), 25 the validation sample (55.3 ± 16.4 years, 11 women), and 30 the test sample (55.0 ± 15.0 years, 9 women). Four input settings were differentiated: only the T1w noncontrast and T1w low-dose images (standard setting), only the T1w noncontrast and T1w low-dose images with a prolonged postinjection time of 5 minutes (5-minute setting), multiple noncontrast sequences (T1w, T2w, diffusion) and the T1w low-dose images (extended setting), and only noncontrast sequences (T1w, T2w, diffusion) were used (zero-dose setting). For each setting, a deep neural network was trained to synthesize artificial T1w full-dose images, which were assessed on the test sample using an objective evaluation based on quantitative metrics and a subjective evaluation through a reader-based study. Three readers scored the overall image quality, the interchangeability in regard to the clinical conclusion compared with the true T1w full-dose sequence, the contrast enhancement of lesions, and their conformity to the respective references in the true T1w full dose. RESULTS: Quantitative analysis of the artificial T1w full-dose images of the standard setting provided a peak signal-to-noise ratio of 33.39 ± 0.62 (corresponding to an average improvement of the low-dose sequences of 5.2 dB) and a structural similarity index measure of 0.938 ± 0.005. In the 4-fold cross-validation, the extended setting yielded similar performance to the standard setting in terms of peak signal-to-noise ratio ( P = 0.20), but a slight improvement in structural similarity index measure ( P < 0.0001). For all settings, the reader study found comparable overall image quality between the original and artificial T1w full-dose images. The proportion of scans scored as fully or mostly interchangeable was 55%, 58%, 43%, and 3% and the average counts of false positives per case were 0.42 ± 0.83, 0.34 ± 0.71, 0.82 ± 1.15, and 2.00 ± 1.07 for the standard, 5-minute, extended, and zero-dose setting, respectively. Using a 5-point Likert scale (0 to 4, 0 being the worst), all settings of synthesized full-dose images showed significantly poorer contrast enhancement of lesions compared with the original full-dose sequence (difference of average degree of contrast enhancement-standard: -0.97 ± 0.83, P = <0.001; 5-minute: -0.93 ± 0.91, P = <0.001; extended: -0.96 ± 0.97, P = <0.001; zero-dose: -2.39 ± 1.14, P = <0.001). The average scores of conformity of the lesions compared with the original full-dose sequence were 2.25 ± 1.21, 2.22 ± 1.27, 2.24 ± 1.25, and 0.73 ± 0.93 for the standard, 5-minute, extended, and zero-dose setting, respectively. CONCLUSIONS: The tested deep learning algorithm for synthesis of artificial T1w full-dose sequences based on images after administration of only 10% of the standard dose of a gadolinium-based contrast agent showed very good quantitative performance. Despite good image quality in all settings, both false-negative and false-positive signals resulted in significantly limited interchangeability of the synthesized sequences with the original full-dose sequences.

Artificial Contrast: Deep Learning for Reducing Gadolinium-Based Contrast Agents in Neuroradiology.

ABSTRACT: Deep learning approaches are playing an ever-increasing role throughout diagnostic medicine, especially in neuroradiology, to solve a wide range of problems such as segmentation, synthesis of missing sequences, and image quality improvement. Of particular interest is their application in the reduction of gadolinium-based contrast agents, the administration of which has been under cautious reevaluation in recent years because of concerns about gadolinium deposition and its unclear long-term consequences. A growing number of studies are investigating the reduction (low-dose approach) or even complete substitution (zero-dose approach) of gadolinium-based contrast agents in diverse patient populations using a variety of deep learning methods. This work aims to highlight selected research and discusses the advantages and limitations of recent deep learning approaches, the challenges of assessing its output, and the progress toward clinical applicability distinguishing between the low-dose and zero-dose approach.

Constrained and unconstrained deep image prior optimization models with automatic regularization.

Deep Image Prior (DIP) is currently among the most efficient unsupervised deep learning based methods for ill-posed inverse problems in imaging. This novel framework relies on the implicit regularization provided by representing images as the output of generative Convolutional Neural Network (CNN) architectures. So far, DIP has been shown to be an effective approach when combined with classical and novel regularizers. Unfortunately, to obtain appropriate solutions, all the models proposed up to now require an accurate estimate of the regularization parameter. To overcome this difficulty, we consider a locally adapted regularized unconstrained model whose local regularization parameters are automatically estimated for additively separable regularizers. Moreover, we propose a novel constrained formulation in analogy to Morozov's discrepancy principle which enables the application of a broader range of regularizers. Both the unconstrained and the constrained models are solved via the proximal gradient descent-ascent method. Numerical results demonstrate the robustness with respect to image content, noise levels and hyperparameters of the proposed models on both denoising and deblurring of simulated as well as real natural and medical images.

A joint alignment and reconstruction algorithm for electron tomography to visualize in-depth cell-to-cell interactions.

Electron tomography allows one to obtain 3D reconstructions visualizing a tissue's ultrastructure from a series of 2D projection images. An inherent problem with this imaging technique is that its projection images contain unwanted shifts, which must be corrected for to achieve reliable reconstructions. Commonly, the projection images are aligned with each other by means of fiducial markers prior to the reconstruction procedure. In this work, we propose a joint alignment and reconstruction algorithm that iteratively solves for both the unknown reconstruction and the unintentional shift and does not require any fiducial markers. We evaluate the approach first on synthetic phantom data where the focus is not only on the reconstruction quality but more importantly on the shift correction. Subsequently, we apply the algorithm to healthy C57BL/6J mice and then compare it with non-obese diabetic (NOD) mice, with the aim of visualizing the attack of immune cells on pancreatic beta cells within type 1 diabetic mice at a more profound level through 3D analysis. We empirically demonstrate that the proposed algorithm is able to compute the shift with a remaining error at only the sub-pixel level and yields high-quality reconstructions for the limited-angle inverse problem. By decreasing labour and material costs, the algorithm facilitates further research directed towards investigating the immune system's attacks in pancreata of NOD mice for numerous samples at different stages of type 1 diabetes.

Bayesian Uncertainty Estimation of Learned Variational MRI Reconstruction.

Recent deep learning approaches focus on improving quantitative scores of dedicated benchmarks, and therefore only reduce the observation-related (aleatoric) uncertainty. However, the model-immanent (epistemic) uncertainty is less frequently systematically analyzed. In this work, we introduce a Bayesian variational framework to quantify the epistemic uncertainty. To this end, we solve the linear inverse problem of undersampled MRI reconstruction in a variational setting. The associated energy functional is composed of a data fidelity term and the total deep variation (TDV) as a learned parametric regularizer. To estimate the epistemic uncertainty we draw the parameters of the TDV regularizer from a multivariate Gaussian distribution, whose mean and covariance matrix are learned in a stochastic optimal control problem. In several numerical experiments, we demonstrate that our approach yields competitive results for undersampled MRI reconstruction. Moreover, we can accurately quantify the pixelwise epistemic uncertainty, which can serve radiologists as an additional resource to visualize reconstruction reliability.

Total Deep Variation: A Stable Regularization Method for Inverse Problems.

Various problems in computer vision and medical imaging can be cast as inverse problems. A frequent method for solving inverse problems is the variational approach, which amounts to minimizing an energy composed of a data fidelity term and a regularizer. Classically, handcrafted regularizers are used, which are commonly outperformed by state-of-the-art deep learning approaches. In this work, we combine the variational formulation of inverse problems with deep learning by introducing the data-driven general-purpose total deep variation regularizer. In its core, a convolutional neural network extracts local features on multiple scales and in successive blocks. This combination allows for a rigorous mathematical analysis including an optimal control formulation of the training problem in a mean-field setting and a stability analysis with respect to the initial values and the parameters of the regularizer. In addition, we experimentally verify the robustness against adversarial attacks and numerically derive upper bounds for the generalization error. Finally, we achieve state-of-the-art results for several imaging tasks.

Image Morphing in Deep Feature Spaces: Theory and Applications.

This paper combines image metamorphosis with deep features. To this end, images are considered as maps into a high-dimensional feature space and a structure-sensitive, anisotropic flow regularization is incorporated in the metamorphosis model proposed by Miller and Younes (Int J Comput Vis 41(1):61-84, 2001) and Trouvé and Younes (Found Comput Math 5(2):173-198, 2005). For this model, a variational time discretization of the Riemannian path energy is presented and the existence of discrete geodesic paths minimizing this energy is demonstrated. Furthermore, convergence of discrete geodesic paths to geodesic paths in the time continuous model is investigated. The spatial discretization is based on a finite difference approximation in image space and a stable spline approximation in deformation space; the fully discrete model is optimized using the iPALM algorithm. Numerical experiments indicate that the incorporation of semantic deep features is superior to intensity-based approaches.

Accelerating Prostate Diffusion-weighted MRI Using a Guided Denoising Convolutional Neural Network: Retrospective Feasibility Study.

PURPOSE: To investigate the feasibility of accelerating prostate diffusion-weighted imaging (DWI) by reducing the number of acquired averages and denoising the resulting image using a proposed guided denoising convolutional neural network (DnCNN). MATERIALS AND METHODS: Raw data from the prostate DWI scans were retrospectively gathered between July 2018 and July 2019 from six single-vendor MRI scanners. There were 103 datasets used for training (median age, 64 years; interquartile range [IQR], 11), 15 for validation (median age, 68 years; IQR, 12), and 37 for testing (median age, 64 years; IQR, 12). High b-value diffusion-weighted (hb DW) data were reconstructed into noisy images using two averages and reference images using all 16 averages. A conventional DnCNN was modified into a guided DnCNN, which uses the low b-value DW image as a guidance input. Quantitative and qualitative reader evaluations were performed on the denoised hb DW images. A cumulative link mixed regression model was used to compare the readers' scores. The agreement between the apparent diffusion coefficient (ADC) maps (denoised vs reference) was analyzed using Bland-Altman analysis. RESULTS: Compared with the original DnCNN, the guided DnCNN produced denoised hb DW images with higher peak signal-to-noise ratio (32.79 ± 3.64 [standard deviation] vs 33.74 ± 3.64), higher structural similarity index (0.92 ± 0.05 vs 0.93 ± 0.04), and lower normalized mean square error (3.9% ± 10 vs 1.6% ± 1.5) (P < .001 for all). Compared with the reference images, the denoised images received higher image quality scores from the readers (P < .0001). The ADC values based on the denoised hb DW images were in good agreement with the reference ADC values (mean ADC difference ranged from -0.04 to 0.02 × 10-3 mm2/sec). CONCLUSION: Accelerating prostate DWI by reducing the number of acquired averages and denoising the resulting image using the proposed guided DnCNN is technically feasible. Supplemental material is available for this article. © RSNA, 2020.

Variational Networks: An Optimal Control Approach to Early Stopping Variational Methods for Image Restoration.

We investigate a well-known phenomenon of variational approaches in image processing, where typically the best image quality is achieved when the gradient flow process is stopped before converging to a stationary point. This paradox originates from a tradeoff between optimization and modeling errors of the underlying variational model and holds true even if deep learning methods are used to learn highly expressive regularizers from data. In this paper, we take advantage of this paradox and introduce an optimal stopping time into the gradient flow process, which in turn is learned from data by means of an optimal control approach. After a time discretization, we obtain variational networks, which can be interpreted as a particular type of recurrent neural networks. The learned variational networks achieve competitive results for image denoising and image deblurring on a standard benchmark data set. One of the key theoretical results is the development of first- and second-order conditions to verify optimal stopping time. A nonlinear spectral analysis of the gradient of the learned regularizer gives enlightening insights into the different regularization properties.

SparseCT: System concept and design of multislit collimators.

PURPOSE: SparseCT, an undersampling scheme for compressed sensing (CS) computed tomography (CT), has been proposed to reduce radiation dose by acquiring undersampled projection data from clinical CT scanners (Koesters et al. in, SparseCT: Interrupted-Beam Acquisition and Sparse Reconstruction for Radiation Dose Reduction; 2017). SparseCT partially blocks the x-ray beam with a multislit collimator (MSC) to perform a multidimensional undersampling along the view and detector row dimensions. SparseCT undersamples the projection data within each view and moves the MSC along the z-direction during gantry rotation to change the undersampling pattern. It enables reconstruction of images from undersampled data using CS algorithms. The purpose of this work is to design the spacing and width of the MSC slits and the MSC motion patterns based on beam separation, undersampling efficiency, and image quality. The development and testing of a SparseCT prototype with the designed MSC will be described in a following paper. METHODS: We chose a few initial MSC designs based on the guidance from two metrics: beam separation and undersampling efficiency. Both beam separation and undersampling efficiency were measured from numerically simulated photon distribution with MSC taken into consideration. Beam separation measures the separation between x-ray beams from consecutive slits, taking into account penumbra effects on both sides of each slit. Undersampling efficiency measures the dose-weighted similarity between penumbra undersampling and binary undersampling, in other words, the effective contribution of the incident dose to the signal to noise ratio of the projection data. We then compared the initially chosen MSC designs in terms of their reconstruction image quality. SparseCT projections were simulated from fully sampled patient projection data according to the MSC design and motion pattern, reconstructed iteratively using a sparsity-enforcing penalized weighted least squares cost function with ordered subsets/momentum algorithm, and compared visually and quantitatively. RESULTS: Simulated photon distributions indicate that the size of the penumbra is dominated by the size of the focal spot. Therefore, a wider MSC slit and a smaller focal spot lead to increased beam separation and undersampling efficiency. For fourfold undersampling with a 1.2 mm focal spot, a minimum MSC slit width of three detector rows (projected to the detector surface) is needed for beam separation; for threefold undersampling, a minimum slit width of four detector rows is needed. Simulations of SparseCT projection and reconstruction indicate that the motion pattern of the MSC does not have a visible impact on image quality. An MSC slit width of three or four detector rows yields similar image quality. CONCLUSION: The MSC is the key component of the SparseCT method. Simulations of MSC designs incorporating x-ray beam penumbra effects showed that for threefold and fourfold dose reductions, an MSC slit width of four detector rows provided reasonable beam separation, undersampling efficiency, and image quality.

Joint reconstruction and classification of tumor cells and cell interactions in melanoma tissue sections with synthesized training data.

PURPOSE: Cancers are almost always diagnosed by morphologic features in tissue sections. In this context, machine learning tools provide new opportunities to describe tumor immune cell interactions within the tumor microenvironment and thus provide phenotypic information that might be predictive for the response to immunotherapy. METHODS: We develop a machine learning approach using variational networks for joint image denoising and classification of tissue sections for melanoma, which is an established model tumor for immuno-oncology research. The manual annotation of real training data would require substantial user interaction of experienced pathologists for each single training image, and the training of larger networks would rely on a very large number of such data sets with ground truth annotation. To overcome this bottleneck, we synthesize training data together with a proper tissue structure classification. To this end, a stochastic data generation process is used to mimic cell morphology, cell distribution and tissue architecture in the tumor microenvironment. Particular components of this tool are random placement and rotation of a large number of patches for presegmented cell nuclei, a stochastic fast marching approach to mimic the geometry of cells and texture generation based on a color covariance analysis of real data. Here, the generated training data reflect a large range of interaction patterns. RESULTS: In several applications to histological tissue sections, we analyze the efficiency and accuracy of the proposed approach. As a result, depending on the scenario considered, almost all cells and nuclei which ought to be detected are actually marked as classified and hardly any misclassifications occur. CONCLUSIONS: The proposed method allows for a computer-aided screening of histological tissue sections utilizing variational networks with a particular emphasis on tumor immune cell interactions and on the robust cell nuclei classification.

Assessment of the generalization of learned image reconstruction and the potential for transfer learning.

PURPOSE: Although deep learning has shown great promise for MR image reconstruction, an open question regarding the success of this approach is the robustness in the case of deviations between training and test data. The goal of this study is to assess the influence of image contrast, SNR, and image content on the generalization of learned image reconstruction, and to demonstrate the potential for transfer learning. METHODS: Reconstructions were trained from undersampled data using data sets with varying SNR, sampling pattern, image contrast, and synthetic data generated from a public image database. The performance of the trained reconstructions was evaluated on 10 in vivo patient knee MRI acquisitions from 2 different pulse sequences that were not used during training. Transfer learning was evaluated by fine-tuning baseline trainings from synthetic data with a small subset of in vivo MR training data. RESULTS: Deviations in SNR between training and testing led to substantial decreases in reconstruction image quality, whereas image contrast was less relevant. Trainings from heterogeneous training data generalized well toward the test data with a range of acquisition parameters. Trainings from synthetic, non-MR image data showed residual aliasing artifacts, which could be removed by transfer learning-inspired fine-tuning. CONCLUSION: This study presents insights into the generalization ability of learned image reconstruction with respect to deviations in the acquisition settings between training and testing. It also provides an outlook for the potential of transfer learning to fine-tune trainings to a particular target application using only a small number of training cases.

Learning a variational network for reconstruction of accelerated MRI data.

PURPOSE: To allow fast and high-quality reconstruction of clinical accelerated multi-coil MR data by learning a variational network that combines the mathematical structure of variational models with deep learning. THEORY AND METHODS: Generalized compressed sensing reconstruction formulated as a variational model is embedded in an unrolled gradient descent scheme. All parameters of this formulation, including the prior model defined by filter kernels and activation functions as well as the data term weights, are learned during an offline training procedure. The learned model can then be applied online to previously unseen data. RESULTS: The variational network approach is evaluated on a clinical knee imaging protocol for different acceleration factors and sampling patterns using retrospectively and prospectively undersampled data. The variational network reconstructions outperform standard reconstruction algorithms, verified by quantitative error measures and a clinical reader study for regular sampling and acceleration factor 4. CONCLUSION: Variational network reconstructions preserve the natural appearance of MR images as well as pathologies that were not included in the training data set. Due to its high computational performance, that is, reconstruction time of 193 ms on a single graphics card, and the omission of parameter tuning once the network is trained, this new approach to image reconstruction can easily be integrated into clinical workflow. Magn Reson Med 79:3055-3071, 2018. © 2017 International Society for Magnetic Resonance in Medicine.