Determining individual glomerular proteinuria and periglomerular infiltration in a cleared murine kidney by a 3-dimensional fast marching algorithm.

Three-dimensional (3D) imaging has advanced basic research and clinical medicine. However, limited resolution and imperfections of real-world 3D image material often preclude algorithmic image analysis. Here, we present a methodologic framework for such imaging and analysis for functional and spatial relations in experimental nephritis. First, optical tissue-clearing protocols were optimized to preserve fluorescence signals for light sheet fluorescence microscopy and compensated attenuation effects using adjustable 3D correction fields. Next, we adapted the fast marching algorithm to conduct backtracking in 3D environments and developed a tool to determine local concentrations of extractable objects. As a proof-of-concept application, we used this framework to determine in a glomerulonephritis model the individual proteinuria and periglomerular immune cell infiltration for all glomeruli of half a mouse kidney. A correlation between these parameters surprisingly did not support the intuitional assumption that the most inflamed glomeruli are the most proteinuric. Instead, the spatial density of adjacent glomeruli positively correlated with the proteinuria of a given glomerulus. Because proteinuric glomeruli appear clustered, this suggests that the exact location of a kidney biopsy may affect the observed severity of glomerular damage. Thus, our algorithmic pipeline described here allows analysis of various parameters of various organs composed of functional subunits, such as the kidney, and can theoretically be adapted to processing other image modalities.

Optical Flow-Guided Cine MRI Segmentation With Learned Corrections.

In cardiac cine magnetic resonance imaging (MRI), the heart is repeatedly imaged at numerous time points during the cardiac cycle. Frequently, the temporal evolution of a certain region of interest such as the ventricles or the atria is highly relevant for clinical diagnosis. In this paper, we devise a novel approach that allows for an automatized propagation of an arbitrary region of interest (ROI) along the cardiac cycle from respective annotated ROIs provided by medical experts at two different points in time, most frequently at the end-systolic (ES) and the end-diastolic (ED) cardiac phases. At its core, a 3D TV- L1 -based optical flow algorithm computes the apparent motion of consecutive MRI images in forward and backward directions. Subsequently, the given terminal annotated masks are propagated by this bidirectional optical flow in 3D, which results, however, in improper initial estimates of the segmentation masks due to numerical inaccuracies. These initially propagated segmentation masks are then refined by a 3D U-Net-based convolutional neural network (CNN), which was trained to enforce consistency with the forward and backward warped masks using a novel loss function. Moreover, a penalization term in the loss function controls large deviations from the initial segmentation masks. This method is benchmarked both on a new dataset with annotated single ventricles containing patients with severe heart diseases and on a publicly available dataset with different annotated ROIs. We emphasize that our novel loss function enables fine-tuning the CNN on a single patient, thereby yielding state-of-the-art results along the complete cardiac cycle.

Digital Twinning of Cardiac Electrophysiology Models From the Surface ECG: A Geodesic Backpropagation Approach.

The eikonal equation has become an indispensable tool for modeling cardiac electrical activation accurately and efficiently. In principle, by matching clinically recorded and eikonal-based electrocardiograms (ECGs), it is possible to build patient-specific models of cardiac electrophysiology in a purely non-invasive manner. Nonetheless, the fitting procedure remains a challenging task. The present study introduces a novel method, Geodesic-BP, to solve the inverse eikonal problem. Geodesic-BP is well-suited for GPU-accelerated machine learning frameworks, allowing us to optimize the parameters of the eikonal equation to reproduce a given ECG. We show that Geodesic-BP can reconstruct a simulated cardiac activation with high accuracy in a synthetic test case, even in the presence of modeling inaccuracies. Furthermore, we apply our algorithm to a publicly available dataset of a biventricular rabbit model, with promising results. Given the future shift towards personalized medicine, Geodesic-BP has the potential to help in future functionalizations of cardiac models meeting clinical time constraints while maintaining the physiological accuracy of state-of-the-art cardiac models.

Combining CNNs and Markov-like Models for Facial Landmark Detection with Spatial Consistency Estimates.

The accurate localization of facial landmarks is essential for several tasks, including face recognition, head pose estimation, facial region extraction, and emotion detection. Although the number of required landmarks is task-specific, models are typically trained on all available landmarks in the datasets, limiting efficiency. Furthermore, model performance is strongly influenced by scale-dependent local appearance information around landmarks and the global shape information generated by them. To account for this, we propose a lightweight hybrid model for facial landmark detection designed specifically for pupil region extraction. Our design combines a convolutional neural network (CNN) with a Markov random field (MRF)-like process trained on only 17 carefully selected landmarks. The advantage of our model is the ability to run different image scales on the same convolutional layers, resulting in a significant reduction in model size. In addition, we employ an approximation of the MRF that is run on a subset of landmarks to validate the spatial consistency of the generated shape. This validation process is performed against a learned conditional distribution, expressing the location of one landmark relative to its neighbor. Experimental results on popular facial landmark localization datasets such as 300 w, WFLW, and HELEN demonstrate the accuracy of our proposed model. Furthermore, our model achieves state-of-the-art performance on a well-defined robustness metric. In conclusion, the results demonstrate the ability of our lightweight model to filter out spatially inconsistent predictions, even with significantly fewer training landmarks.

Reduction of Gadolinium-Based Contrast Agents in MRI Using Convolutional Neural Networks and Different Input Protocols: Limited Interchangeability of Synthesized Sequences With Original Full-Dose Images Despite Excellent Quantitative Performance.

OBJECTIVES: The purpose of this study was to implement a state-of-the-art convolutional neural network used to synthesize artificial T1-weighted (T1w) full-dose images from corresponding noncontrast and low-dose images (using various settings of input sequences) and test its performance on a patient population acquired prospectively. MATERIALS AND METHODS: In this monocentric, institutional review board-approved study, a total of 138 participants were included who received an adapted imaging protocol with acquisition of a T1w low dose after administration of 10% of the standard dose and acquisition of a T1w full dose after administration of the remaining 90% of the standard dose of a gadolinium-containing contrast agent. A total of 83 participants formed the training sample (51.7 ± 16.5 years, 36 women), 25 the validation sample (55.3 ± 16.4 years, 11 women), and 30 the test sample (55.0 ± 15.0 years, 9 women). Four input settings were differentiated: only the T1w noncontrast and T1w low-dose images (standard setting), only the T1w noncontrast and T1w low-dose images with a prolonged postinjection time of 5 minutes (5-minute setting), multiple noncontrast sequences (T1w, T2w, diffusion) and the T1w low-dose images (extended setting), and only noncontrast sequences (T1w, T2w, diffusion) were used (zero-dose setting). For each setting, a deep neural network was trained to synthesize artificial T1w full-dose images, which were assessed on the test sample using an objective evaluation based on quantitative metrics and a subjective evaluation through a reader-based study. Three readers scored the overall image quality, the interchangeability in regard to the clinical conclusion compared with the true T1w full-dose sequence, the contrast enhancement of lesions, and their conformity to the respective references in the true T1w full dose. RESULTS: Quantitative analysis of the artificial T1w full-dose images of the standard setting provided a peak signal-to-noise ratio of 33.39 ± 0.62 (corresponding to an average improvement of the low-dose sequences of 5.2 dB) and a structural similarity index measure of 0.938 ± 0.005. In the 4-fold cross-validation, the extended setting yielded similar performance to the standard setting in terms of peak signal-to-noise ratio ( P = 0.20), but a slight improvement in structural similarity index measure ( P < 0.0001). For all settings, the reader study found comparable overall image quality between the original and artificial T1w full-dose images. The proportion of scans scored as fully or mostly interchangeable was 55%, 58%, 43%, and 3% and the average counts of false positives per case were 0.42 ± 0.83, 0.34 ± 0.71, 0.82 ± 1.15, and 2.00 ± 1.07 for the standard, 5-minute, extended, and zero-dose setting, respectively. Using a 5-point Likert scale (0 to 4, 0 being the worst), all settings of synthesized full-dose images showed significantly poorer contrast enhancement of lesions compared with the original full-dose sequence (difference of average degree of contrast enhancement-standard: -0.97 ± 0.83, P = <0.001; 5-minute: -0.93 ± 0.91, P = <0.001; extended: -0.96 ± 0.97, P = <0.001; zero-dose: -2.39 ± 1.14, P = <0.001). The average scores of conformity of the lesions compared with the original full-dose sequence were 2.25 ± 1.21, 2.22 ± 1.27, 2.24 ± 1.25, and 0.73 ± 0.93 for the standard, 5-minute, extended, and zero-dose setting, respectively. CONCLUSIONS: The tested deep learning algorithm for synthesis of artificial T1w full-dose sequences based on images after administration of only 10% of the standard dose of a gadolinium-based contrast agent showed very good quantitative performance. Despite good image quality in all settings, both false-negative and false-positive signals resulted in significantly limited interchangeability of the synthesized sequences with the original full-dose sequences.

Artificial Contrast: Deep Learning for Reducing Gadolinium-Based Contrast Agents in Neuroradiology.

ABSTRACT: Deep learning approaches are playing an ever-increasing role throughout diagnostic medicine, especially in neuroradiology, to solve a wide range of problems such as segmentation, synthesis of missing sequences, and image quality improvement. Of particular interest is their application in the reduction of gadolinium-based contrast agents, the administration of which has been under cautious reevaluation in recent years because of concerns about gadolinium deposition and its unclear long-term consequences. A growing number of studies are investigating the reduction (low-dose approach) or even complete substitution (zero-dose approach) of gadolinium-based contrast agents in diverse patient populations using a variety of deep learning methods. This work aims to highlight selected research and discusses the advantages and limitations of recent deep learning approaches, the challenges of assessing its output, and the progress toward clinical applicability distinguishing between the low-dose and zero-dose approach.

Bayesian Uncertainty Estimation of Learned Variational MRI Reconstruction.

Recent deep learning approaches focus on improving quantitative scores of dedicated benchmarks, and therefore only reduce the observation-related (aleatoric) uncertainty. However, the model-immanent (epistemic) uncertainty is less frequently systematically analyzed. In this work, we introduce a Bayesian variational framework to quantify the epistemic uncertainty. To this end, we solve the linear inverse problem of undersampled MRI reconstruction in a variational setting. The associated energy functional is composed of a data fidelity term and the total deep variation (TDV) as a learned parametric regularizer. To estimate the epistemic uncertainty we draw the parameters of the TDV regularizer from a multivariate Gaussian distribution, whose mean and covariance matrix are learned in a stochastic optimal control problem. In several numerical experiments, we demonstrate that our approach yields competitive results for undersampled MRI reconstruction. Moreover, we can accurately quantify the pixelwise epistemic uncertainty, which can serve radiologists as an additional resource to visualize reconstruction reliability.

Total Deep Variation: A Stable Regularization Method for Inverse Problems.

Various problems in computer vision and medical imaging can be cast as inverse problems. A frequent method for solving inverse problems is the variational approach, which amounts to minimizing an energy composed of a data fidelity term and a regularizer. Classically, handcrafted regularizers are used, which are commonly outperformed by state-of-the-art deep learning approaches. In this work, we combine the variational formulation of inverse problems with deep learning by introducing the data-driven general-purpose total deep variation regularizer. In its core, a convolutional neural network extracts local features on multiple scales and in successive blocks. This combination allows for a rigorous mathematical analysis including an optimal control formulation of the training problem in a mean-field setting and a stability analysis with respect to the initial values and the parameters of the regularizer. In addition, we experimentally verify the robustness against adversarial attacks and numerically derive upper bounds for the generalization error. Finally, we achieve state-of-the-art results for several imaging tasks.

Renal Denervation Exacerbates LPS- and Antibody-induced Acute Kidney Injury, but Protects from Pyelonephritis in Mice.

BACKGROUND: Renal denervation (RDN) is an invasive intervention to treat drug-resistant arterial hypertension. Its therapeutic value is contentious. Here we examined the effects of RDN on inflammatory and infectious kidney disease models in mice. METHODS: Mice were unilaterally or bilaterally denervated, or sham operated, then three disease models were induced: nephrotoxic nephritis (NTN, a model for crescentic GN), pyelonephritis, and acute endotoxemic kidney injury (as a model for septic kidney injury). Analytical methods included measurement of renal glomerular filtration, proteinuria, flow cytometry of renal immune cells, immunofluorescence microscopy, and three-dimensional imaging of optically cleared kidney tissue by light-sheet fluorescence microscopy followed by algorithmic analysis. RESULTS: Unilateral RDN increased glomerular filtration in denervated kidneys, but decreased it in the contralateral kidneys. In the NTN model, more nephritogenic antibodies were deposited in glomeruli of denervated kidneys, resulting in stronger inflammation and injury in denervated compared with contralateral nondenervated kidneys. Also, intravenously injected LPS increased neutrophil influx and inflammation in the denervated kidneys, both after unilateral and bilateral RDN. When we induced pyelonephritis in bilaterally denervated mice, both kidneys contained less bacteria and neutrophils. In unilaterally denervated mice, pyelonephritis was attenuated and intrarenal neutrophil numbers were lower in the denervated kidneys. The nondenervated contralateral kidneys harbored more bacteria, even compared with sham-operated mice, and showed the strongest influx of neutrophils. CONCLUSIONS: Our data suggest that the increased perfusion and filtration in denervated kidneys can profoundly influence concomitant inflammatory diseases. Renal deposition of circulating nephritic material is higher, and hence antibody- and endotoxin-induced kidney injury was aggravated in mice. Pyelonephritis was attenuated in denervated murine kidneys, because the higher glomerular filtration facilitated better flushing of bacteria with the urine, at the expense of contralateral, nondenervated kidneys after unilateral denervation.

GEASI: Geodesic-based earliest activation sites identification in cardiac models.

The identification of the initial ventricular activation sequence is a critical step for the correct personalization of patient-specific cardiac models. In healthy conditions, the Purkinje network is the main source of the electrical activation, but under pathological conditions the so-called earliest activation sites (EASs) are possibly sparser and more localized. Yet, their number, location and timing may not be easily inferred from remote recordings, such as the epicardial activation or the 12-lead electrocardiogram (ECG), due to the underlying complexity of the model. In this work, we introduce GEASI (Geodesic-based Earliest Activation Sites Identification) as a novel approach to simultaneously identify all EASs. To this end, we start from the anisotropic eikonal equation modeling cardiac electrical activation and exploit its Hamilton-Jacobi formulation to minimize a given objective function, for example, the quadratic mismatch to given activation measurements. This versatile approach can be extended to estimate the number of activation sites by means of the topological gradient, or fitting a given ECG. We conducted various experiments in 2D and 3D for in-silico models and an in-vivo intracardiac recording collected from a patient undergoing cardiac resynchronization therapy. The results demonstrate the clinical applicability of GEASI for potential future personalized models and clinical intervention.

Image Morphing in Deep Feature Spaces: Theory and Applications.

This paper combines image metamorphosis with deep features. To this end, images are considered as maps into a high-dimensional feature space and a structure-sensitive, anisotropic flow regularization is incorporated in the metamorphosis model proposed by Miller and Younes (Int J Comput Vis 41(1):61-84, 2001) and Trouvé and Younes (Found Comput Math 5(2):173-198, 2005). For this model, a variational time discretization of the Riemannian path energy is presented and the existence of discrete geodesic paths minimizing this energy is demonstrated. Furthermore, convergence of discrete geodesic paths to geodesic paths in the time continuous model is investigated. The spatial discretization is based on a finite difference approximation in image space and a stable spline approximation in deformation space; the fully discrete model is optimized using the iPALM algorithm. Numerical experiments indicate that the incorporation of semantic deep features is superior to intensity-based approaches.

Variational Networks: An Optimal Control Approach to Early Stopping Variational Methods for Image Restoration.

We investigate a well-known phenomenon of variational approaches in image processing, where typically the best image quality is achieved when the gradient flow process is stopped before converging to a stationary point. This paradox originates from a tradeoff between optimization and modeling errors of the underlying variational model and holds true even if deep learning methods are used to learn highly expressive regularizers from data. In this paper, we take advantage of this paradox and introduce an optimal stopping time into the gradient flow process, which in turn is learned from data by means of an optimal control approach. After a time discretization, we obtain variational networks, which can be interpreted as a particular type of recurrent neural networks. The learned variational networks achieve competitive results for image denoising and image deblurring on a standard benchmark data set. One of the key theoretical results is the development of first- and second-order conditions to verify optimal stopping time. A nonlinear spectral analysis of the gradient of the learned regularizer gives enlightening insights into the different regularization properties.

Joint reconstruction and classification of tumor cells and cell interactions in melanoma tissue sections with synthesized training data.

PURPOSE: Cancers are almost always diagnosed by morphologic features in tissue sections. In this context, machine learning tools provide new opportunities to describe tumor immune cell interactions within the tumor microenvironment and thus provide phenotypic information that might be predictive for the response to immunotherapy. METHODS: We develop a machine learning approach using variational networks for joint image denoising and classification of tissue sections for melanoma, which is an established model tumor for immuno-oncology research. The manual annotation of real training data would require substantial user interaction of experienced pathologists for each single training image, and the training of larger networks would rely on a very large number of such data sets with ground truth annotation. To overcome this bottleneck, we synthesize training data together with a proper tissue structure classification. To this end, a stochastic data generation process is used to mimic cell morphology, cell distribution and tissue architecture in the tumor microenvironment. Particular components of this tool are random placement and rotation of a large number of patches for presegmented cell nuclei, a stochastic fast marching approach to mimic the geometry of cells and texture generation based on a color covariance analysis of real data. Here, the generated training data reflect a large range of interaction patterns. RESULTS: In several applications to histological tissue sections, we analyze the efficiency and accuracy of the proposed approach. As a result, depending on the scenario considered, almost all cells and nuclei which ought to be detected are actually marked as classified and hardly any misclassifications occur. CONCLUSIONS: The proposed method allows for a computer-aided screening of histological tissue sections utilizing variational networks with a particular emphasis on tumor immune cell interactions and on the robust cell nuclei classification.