Predicting early breast cancer recurrence from histopathological images in the Carolina Breast Cancer Study.

Approaches for rapidly identifying patients at high risk of early breast cancer recurrence are needed. Image-based methods for prescreening hematoxylin and eosin (H&E) stained tumor slides could offer temporal and financial efficiency. We evaluated a data set of 704 1-mm tumor core H&E images (2-4 cores per case), corresponding to 202 participants (101 who recurred; 101 non-recurrent matched on age and follow-up time) from breast cancers diagnosed between 2008-2012 in the Carolina Breast Cancer Study. We leveraged deep learning to extract image information and trained a model to identify recurrence. Cross-validation accuracy for predicting recurrence was 62.4% [95% CI: 55.7, 69.1], similar to grade (65.8% [95% CI: 59.3, 72.3]) and ER status (66.3% [95% CI: 59.8, 72.8]). Interestingly, 70% (19/27) of early-recurrent low-intermediate grade tumors were identified by our image model. Relative to existing markers, image-based analyses provide complementary information for predicting early recurrence.

Charting Aging Trajectories of Knee Cartilage Thickness for Early Osteoarthritis Risk Prediction: An MRI Study from the Osteoarthritis Initiative Cohort.

Knee osteoarthritis (OA), a prevalent joint disease in the U.S., poses challenges in terms of predicting of its early progression. Although high-resolution knee magnetic resonance imaging (MRI) facilitates more precise OA diagnosis, the heterogeneous and multifactorial aspects of OA pathology remain significant obstacles for prognosis. MRI-based scoring systems, while standardizing OA assessment, are both time-consuming and labor-intensive. Current AI technologies facilitate knee OA risk scoring and progression prediction, but these often focus on the symptomatic phase of OA, bypassing initial-stage OA prediction. Moreover, their reliance on complex algorithms can hinder clinical interpretation. To this end, we make this effort to construct a computationally efficient, easily-interpretable, and state-of-the-art approach aiding in the radiographic OA (rOA) auto-classification and prediction of the incidence and progression, by contrasting an individual's cartilage thickness with a similar demographic in the rOA-free cohort. To better visualize, we have developed the toolset for both prediction and local visualization. A movie demonstrating different subtypes of dynamic changes in local centile scores during rOA progression is available at https://tli3.github.io/KneeOA/. Specifically, we constructed age-BMI-dependent reference charts for knee OA cartilage thickness, based on MRI scans from 957 radiographic OA (rOA)-free individuals from the Osteoarthritis Initiative cohort. Then we extracted local and global centiles by contrasting an individual's cartilage thickness to the rOA-free cohort with a similar age and BMI. Using traditional boosting approaches with our centile-based features, we obtain rOA classification of KLG ≤ 1 versus KLG = 2 (AUC = 0.95, F1 = 0.89), KLG ≤ 1 versus KLG ≥ 2 (AUC = 0.90, F1 = 0.82) and prediction of KLG2 progression (AUC = 0.98, F1 = 0.94), rOA incidence (KLG increasing from < 2 to ≥ 2; AUC = 0.81, F1 = 0.69) and rOA initial transition (KLG from 0 to 1; AUC = 0.64, F1 = 0.65) within a future 48-month period. Such performance in classifying KLG ≥ 2 matches that of deep learning methods in recent literature. Furthermore, its clinical interpretation suggests that cartilage changes, such as thickening in lateral femoral and anterior femoral regions and thinning in lateral tibial regions, may serve as indicators for prediction of rOA incidence and early progression. Meanwhile, cartilage thickening in the posterior medial and posterior lateral femoral regions, coupled with a reduction in the central medial femoral region, may signify initial phases of rOA transition.

Patterns of variation among baseline femoral and tibial cartilage thickness and clinical features: Data from the osteoarthritis initiative.

Objective: To employ novel methodologies to identify phenotypes in knee OA based on variation among three baseline data blocks: 1) femoral cartilage thickness, 2) tibial cartilage thickness, and 3) participant characteristics and clinical features. Methods: Baseline data were from 3321 Osteoarthritis Initiative (OAI) participants with available cartilage thickness maps (6265 knees) and 77 clinical features. Cartilage maps were obtained from 3D DESS MR images using a deep-learning based segmentation approach and an atlas-based analysis developed by our group. Angle-based Joint and Individual Variation Explained (AJIVE) was used to capture and quantify variation, both shared among multiple data blocks and individual to each block, and to determine statistical significance. Results: Three major modes of variation were shared across the three data blocks. Mode 1 reflected overall thicker cartilage among men, those with higher education, and greater knee forces; Mode 2 showed associations between worsening Kellgren-Lawrence Grade, medial cartilage thinning, and worsening symptoms; and Mode 3 contrasted lateral and medial-predominant cartilage loss associated with BMI and malalignment. Each data block also demonstrated individual, independent modes of variation consistent with the known discordance between symptoms and structure in knee OA and reflecting the importance of features such as physical function, symptoms, and comorbid conditions independent of structural damage. Conclusions: This exploratory analysis, combining the rich OAI dataset with novel methods for determining and visualizing cartilage thickness, reinforces known associations in knee OA while providing insights into the potential for data integration in knee OA phenotyping.

Optimal transport features for morphometric population analysis.

Brain pathologies often manifest as partial or complete loss of tissue. The goal of many neuroimaging studies is to capture the location and amount of tissue changes with respect to a clinical variable of interest, such as disease progression. Morphometric analysis approaches capture local differences in the distribution of tissue or other quantities of interest in relation to a clinical variable. We propose to augment morphometric analysis with an additional feature extraction step based on unbalanced optimal transport. The optimal transport feature extraction step increases statistical power for pathologies that cause spatially dispersed tissue loss, minimizes sensitivity to shifts due to spatial misalignment or differences in brain topology, and separates changes due to volume differences from changes due to tissue location. We demonstrate the proposed optimal transport feature extraction step in the context of a volumetric morphometric analysis of the OASIS-1 study for Alzheimer's disease. The results demonstrate that the proposed approach can identify tissue changes and differences that are not otherwise measurable.

A Feature-based Affine Registration Method for Capturing Background Lung Tissue Deformation for Ground Glass Nodule Tracking.

Lung nodule tracking assessment relies on cross-sectional measurements of the largest lesion profile depicted in initial and follow-up computed tomography (CT) images. However, apparent changes in nodule size assessed via simple image-based measurements may also be compromised by the effect of the background lung tissue deformation on the GGN between the initial and follow-up images, leading to erroneous conclusions about nodule changes due to disease. To compensate for the lung deformation and enable consistent nodule tracking, here we propose a feature-based affine registration method and study its performance vis-a-vis several other registration methods. We implement and test each registration method using both a lung- and a lesion-centered region of interest on ten patient CT datasets featuring twelve nodules, including both benign and malignant GGO lesions containing pure GGNs, part-solid, or solid nodules. We evaluate each registration method according to the target registration error (TRE) computed across 30 - 50 homologous fiducial landmarks surrounding the lesions and selected by expert radiologists in both the initial and follow-up patient CT images. Our results show that the proposed feature-based affine lesion-centered registration yielded a 1.1 ± 1.2 mm TRE, while a Symmetric Normalization deformable registration yielded a 1.2 ± 1.2 mm TRE, and a least-square fit registration of the 30-50 validation fiducial landmark set yielded a 1.5 ± 1.2 mm TRE. Although the deformable registration yielded a slightly higher registration accuracy than the feature-based affine registration, it is significantly more computationally efficient, eliminates the need for ambiguous segmentation of GGNs featuring ill-defined borders, and reduces the susceptibility of artificial deformations introduced by the deformable registration, which may lead to increased similarity between the registered initial and follow-up images, over-compensating for the background lung tissue deformation, and, in turn, compromising the true disease-induced nodule change assessment. We also assessed the registration qualitatively, by visual inspection of the subtraction images, and conducted a pilot pre-clinical study that showed the proposed feature-based lesion-centered affine registration effectively compensates for the background lung tissue deformation between the initial and follow-up images and also serves as a reliable baseline registration method prior to assessing lung nodule changes due to disease.

DADP: Dynamic abnormality detection and progression for longitudinal knee magnetic resonance images from the Osteoarthritis Initiative.

Osteoarthritis (OA) is the most common disabling joint disease. Magnetic resonance (MR) imaging has been commonly used to assess knee joint degeneration due to its distinct advantage in detecting morphologic cartilage changes. Although several statistical methods over conventional radiography have been developed to perform quantitative cartilage analyses, little work has been done capturing the development and progression of cartilage lesions (or abnormal regions) and how they naturally progress. There are two major challenges, including (i) the lack of building spatial-temporal correspondences and correlations in cartilage thickness and (ii) the spatio-temporal heterogeneity in abnormal regions. The goal of this work is to propose a dynamic abnormality detection and progression (DADP) framework for quantitative cartilage analysis, while addressing the two challenges. First, spatial correspondences are established on flattened 2D cartilage thickness maps extracted from 3D knee MR images both across time within each subject and across all subjects. Second, a dynamic functional mixed effects model (DFMEM) is proposed to quantify abnormality progression across time points and subjects, while accounting for the spatio-temporal heterogeneity. We systematically evaluate our DADP using simulations and real data from the Osteoarthritis Initiative (OAI). Our results show that DADP not only effectively detects subject-specific dynamic abnormal regions, but also provides population-level statistical disease mapping and subgroup analysis.

Differential Role for Hippocampal Subfields in Alzheimer's Disease Progression Revealed with Deep Learning.

Mild cognitive impairment (MCI) is often considered the precursor of Alzheimer's disease. However, MCI is associated with substantially variable progression rates, which are not well understood. Attempts to identify the mechanisms that underlie MCI progression have often focused on the hippocampus but have mostly overlooked its intricate structure and subdivisions. Here, we utilized deep learning to delineate the contribution of hippocampal subfields to MCI progression. We propose a dense convolutional neural network architecture that differentiates stable and progressive MCI based on hippocampal morphometry with an accuracy of 75.85%. A novel implementation of occlusion analysis revealed marked differences in the contribution of hippocampal subfields to the performance of the model, with presubiculum, CA1, subiculum, and molecular layer showing the most central role. Moreover, the analysis reveals that 10.5% of the volume of the hippocampus was redundant in the differentiation between stable and progressive MCI.

Perfusion Imaging: An Advection Diffusion Approach.

Perfusion imaging is of great clinical importance and is used to assess a wide range of diseases including strokes and brain tumors. Commonly used approaches for the quantitative analysis of perfusion images are based on measuring the effect of a contrast agent moving through blood vessels and into tissue. Contrast-agent free approaches, for example, based on intravoxel incoherent motion and arterial spin labeling, also exist, but are so far not routinely used clinically. Existing contrast-agent-dependent methods typically rely on the estimation of the arterial input function (AIF) to approximately model tissue perfusion. These approaches neglect spatial dependencies. Further, as reliably estimating the AIF is non-trivial, different AIF estimates may lead to different perfusion measures. In this work we therefore propose PIANO, an approach that provides additional insights into the perfusion process. PIANO estimates the velocity and diffusion fields of an advection-diffusion model best explaining the contrast dynamics without using an AIF. PIANO accounts for spatial dependencies and neither requires estimating the AIF nor relies on a particular contrast agent bolus shape. Specifically, we propose a convenient parameterization of the estimation problem, a numerical estimation approach, and extensively evaluate PIANO. Simulation experiments show the robustness and effectiveness of PIANO, along with its ability to distinguish between advection and diffusion. We further apply PIANO on a public brain magnetic resonance (MR) perfusion dataset of acute stroke patients, and demonstrate that PIANO can successfully resolve velocity and diffusion field ambiguities and results in sensitive measures for the assessment of stroke, comparing favorably to conventional measures of perfusion.

Deep-learning-based image registration and automatic segmentation of organs-at-risk in cone-beam CT scans from high-dose radiation treatment of pancreatic cancer.

PURPOSE: Accurate deformable registration between computed tomography (CT) and cone-beam CT (CBCT) images of pancreatic cancer patients treated with high biologically effective radiation doses is essential to assess changes in organ-at-risk (OAR) locations and shapes and to compute delivered dose. This study describes the development and evaluation of a deep-learning (DL) registration model to predict OAR segmentations on the CBCT derived from segmentations on the planning CT. METHODS: The DL model is trained with CT-CBCT image pairs of the same patient, on which OAR segmentations of the small bowel, stomach, and duodenum have been manually drawn. A transformation map is obtained, which serves to warp the CT image and segmentations. In addition to a regularity loss and an image similarity loss, an OAR segmentation similarity loss is also used during training, which penalizes the mismatch between warped CT segmentations and manually drawn CBCT segmentations. At test time, CBCT segmentations are not required as they are instead obtained from the warped CT segmentations. In an IRB-approved retrospective study, a dataset consisting of 40 patients, each with one planning CT and two CBCT scans, was used in a fivefold cross-validation to train and evaluate the model, using physician-drawn segmentations as reference. Images were preprocessed to remove gas pockets. Network performance was compared to two intensity-based deformable registration algorithms (large deformation diffeomorphic metric mapping [LDDMM] and multimodality free-form [MMFF]) as baseline. Evaluated metrics were Dice similarity coefficient (DSC), change in OAR volume within a volume of interest (enclosing the low-dose PTV plus 1 cm margin) from planning CT to CBCT, and maximum dose to 5 cm3 of the OAR [D(5cc)]. RESULTS: Processing time for one CT-CBCT registration with the DL model at test time was less than 5 seconds on a GPU-based system, compared to an average of 30 minutes for LDDMM optimization. For both small bowel and stomach/duodenum, the DL model yielded larger median DSC and smaller interquartile variation than either MMFF (paired t-test P < 10-4 for both type of OARs) or LDDMM (P < 10-3 and P = 0.03 respectively). Root-mean-square deviation (RMSD) of DL-predicted change in small bowel volume relative to reference was 22% less than for MMFF (P = 0.007). RMSD of DL-predicted stomach/duodenum volume change was 28% less than for LDDMM (P = 0.0001). RMSD of DL-predicted D(5cc) in small bowel was 39% less than for MMFF (P = 0.001); in stomach/duodenum, RMSD of DL-predicted D(5cc) was 18% less than for LDDMM (P < 10-3 ). CONCLUSIONS: The proposed deep network CT-to-CBCT deformable registration model shows improved segmentation accuracy compared to intensity-based algorithms and achieves an order-of-magnitude reduction in processing time.

General anaesthesia during infancy reduces white matter micro-organisation in developing rhesus monkeys.

BACKGROUND: Non-human primates are commonly used in neuroimaging research for which general anaesthesia or sedation is typically required for data acquisition. In this analysis, the cumulative effects of exposure to ketamine, Telazol® (tiletamine and zolazepam), and the inhaled anaesthetic isoflurane on early brain development were evaluated in two independent cohorts of typically developing rhesus macaques. METHODS: Diffusion MRI scans were analysed from 43 rhesus macaques (20 females and 23 males) at either 12 or 18 months of age from two separate primate colonies. RESULTS: Significant, widespread reductions in fractional anisotropy with corresponding increased axial, mean, and radial diffusivity were observed across the brain as a result of repeated anaesthesia exposures. These effects were dose dependent and remained after accounting for age and sex at time of exposure in a generalised linear model. Decreases of up to 40% in fractional anisotropy were detected in some brain regions. CONCLUSIONS: Multiple exposures to commonly used anaesthetics were associated with marked changes in white matter microstructure. This study is amongst the first to examine clinically relevant anaesthesia exposures on the developing primate brain. It will be important to examine if, or to what degree, the maturing brain can recover from these white matter changes.

JOINT AND INDIVIDUAL ANALYSIS OF BREAST CANCER HISTOLOGIC IMAGES AND GENOMIC COVARIATES.

The two main approaches in the study of breast cancer are histopathology (analyzing visual characteristics of tumors) and genomics. While both histopathology and genomics are fundamental to cancer research, the connections between these fields have been relatively superficial. We bridge this gap by investigating the Carolina Breast Cancer Study through the development of an integrative, exploratory analysis framework. Our analysis gives insights - some known, some novel - that are engaging to both pathologists and geneticists. Our analysis framework is based on Angle-based Joint and Individual Variation Explained (AJIVE) for statistical data integration and exploits Convolutional Neural Networks (CNNs) as a powerful, automatic method for image feature extraction. CNNs raise interpretability issues that we address by developing novel methods to explore visual modes of variation captured by statistical algorithms (e.g. PCA or AJIVE) applied to CNN features.

ICON: Learning Regular Maps Through Inverse Consistency.

Learning maps between data samples is fundamental. Applications range from representation learning, image translation and generative modeling, to the estimation of spatial deformations. Such maps relate feature vectors, or map between feature spaces. Well-behaved maps should be regular, which can be imposed explicitly or may emanate from the data itself. We explore what induces regularity for spatial transformations, e.g., when computing image registrations. Classical optimization-based models compute maps between pairs of samples and rely on an appropriate regularizer for well-posedness. Recent deep learning approaches have attempted to avoid using such regularizers altogether by relying on the sample population instead. We explore if it is possible to obtain spatial regularity using an inverse consistency loss only and elucidate what explains map regularity in such a context. We find that deep networks combined with an inverse consistency loss and randomized off-grid interpolation yield well behaved, approximately diffeomorphic, spatial transformations. Despite the simplicity of this approach, our experiments present compelling evidence, on both synthetic and real data, that regular maps can be obtained without carefully tuned explicit regularizers, while achieving competitive registration performance.

Computational methods for visualizing and measuring verapamil efficacy for cerebral vasospasm.

Cerebral vasospasm is a dreaded sequelae of aneurysmal subarachnoid hemorrhage (aSAH), requiring timely intervention with therapeutic goals of improving brain perfusion. There are currently no standardized real-time, objective assessments of the interventional procedures performed to treat vasospasm. Here we describe real-time techniques to quantify cerebral perfusion during interventional cerebral angiography. We retrospectively analyzed 39 consecutive cases performed to treat clinical vasospasm and quantified the changes in perfusion metrics between pre- and post- verapamil administrations. With Digital Subtraction Angiography (DSA) perfusion analysis, we are able to identify hypoperfused territories and quantify the exact changes in cerebral perfusion for each individual case and vascular territory. We demonstrate that perfusion analysis for DSA can be performed in real time. This provides clinicians with a colorized map which directly visualizes hypoperfused tissue, combined with associated perfusion statistics. Quantitative thresholds and analysis based on DSA perfusion may assist with real-time dosage estimation and help predict response to treatment, however future prospective analysis is required for validation.

A Deep Network for Joint Registration and Reconstruction of Images with Pathologies.

Registration of images with pathologies is challenging due to tissue appearance changes and missing correspondences caused by the pathologies. Moreover, mass effects as observed for brain tumors may displace tissue, creating larger deformations over time than what is observed in a healthy brain. Deep learning models have successfully been applied to image registration to offer dramatic speed up and to use surrogate information (e.g., segmentations) during training. However, existing approaches focus on learning registration models using images from healthy patients. They are therefore not designed for the registration of images with strong pathologies for example in the context of brain tumors, and traumatic brain injuries. In this work, we explore a deep learning approach to register images with brain tumors to an atlas. Our model learns an appearance mapping from images with tumors to the atlas, while simultaneously predicting the transformation to atlas space. Using separate decoders, the network disentangles the tumor mass effect from the reconstruction of quasi-normal images. Results on both synthetic and real brain tumor scans show that our approach outperforms cost function masking for registration to the atlas and that reconstructed quasi-normal images can be used for better longitudinal registrations.

VOTENET+ : AN IMPROVED DEEP LEARNING LABEL FUSION METHOD FOR MULTI-ATLAS SEGMENTATION.

In this work, we improve the performance of multi-atlas segmentation (MAS) by integrating the recently proposed VoteNet model with the joint label fusion (JLF) approach. Specifically, we first illustrate that using a deep convolutional neural network to predict atlas probabilities can better distinguish correct atlas labels from incorrect ones than relying on image intensity difference as is typical in JLF. Motivated by this finding, we propose VoteNet+, an improved deep network to locally predict the probability of an atlas label to differ from the label of the target image. Furthermore, we show that JLF is more suitable for the VoteNet framework as a label fusion method than plurality voting. Lastly, we use Platt scaling to calibrate the probabilities of our new model. Results on LPBA40 3D MR brain images show that our proposed method can achieve better performance than VoteNet.

Patient-Specific Registration of Pre-operative and Post-recurrence Brain Tumor MRI Scans.

Registering brain magnetic resonance imaging (MRI) scans containing pathologies is challenging primarily due to large deformations caused by the pathologies, leading to missing correspondences between scans. However, the registration task is important and directly related to personalized medicine, as registering between baseline pre-operative and post-recurrence scans may allow the evaluation of tumor infiltration and recurrence. While many registration methods exist, most of them do not specifically account for pathologies. Here, we propose a framework for the registration of longitudinal image-pairs of individual patients diagnosed with glioblastoma. Specifically, we present a combined image registration/reconstruction approach, which makes use of a patient-specific principal component analysis (PCA) model of image appearance to register baseline pre-operative and post-recurrence brain tumor scans. Our approach uses the post-recurrence scan to construct a patient-specific model, which then guides the registration of the pre-operative scan. Quantitative and qualitative evaluations of our framework on 10 patient image-pairs indicate that it provides excellent registration performance without requiring (1) any human intervention or (2) prior knowledge of tumor location, growth or appearance.

Fast predictive simple geodesic regression.

Deformable image registration and regression are important tasks in medical image analysis. However, they are computationally expensive, especially when analyzing large-scale datasets that contain thousands of images. Hence, cluster computing is typically used, making the approaches dependent on such computational infrastructure. Even larger computational resources are required as study sizes increase. This limits the use of deformable image registration and regression for clinical applications and as component algorithms for other image analysis approaches. We therefore propose using a fast predictive approach to perform image registrations. In particular, we employ these fast registration predictions to approximate a simplified geodesic regression model to capture longitudinal brain changes. The resulting method is orders of magnitude faster than the standard optimization-based regression model and hence facilitates large-scale analysis on a single graphics processing unit (GPU). We evaluate our results on 3D brain magnetic resonance images (MRI) from the ADNI datasets.

Multiseg pipeline: automatic tissue segmentation of brain MR images with subject-specific atlases.

Automated segmentation and labeling of individual brain anatomical regions is challenging due to individual structural variability. Although, atlas-based segmentation has shown its potential for both tissue and structure segmentation, the inherent natural variability as well as disease-related changes in MR appearance is often inappropriately represented by a single atlas image. In order to have a more accurate representation, several atlases may be used for the segmentation task in a given neuroimaging study. In this paper, we present the MultisegPipeline, it uses multiple atlases that have been visually inspected and capture the expected variability in a neonatal population. The MultisegPipeline transfers the labeled regions from each atlas to the target image using deformable registration (ANTs1 or QuickSilver2 is available for this task). Additionally, the set of labels are merged using a label fusion technique that reduces the errors produced by the registration. The final output is a single label map that combines the results produced by all atlases into a consensus solution. In our study, the MultisegPipeline is used to segment brain MR images from 31 infants, a leave-one-out strategy was used to test our framework. The average dice score coefficient was 0.89.

Image-Based Methods for Phase Estimation, Gating, and Temporal Superresolution of Cardiac Ultrasound.

OBJECTIVE: Ultrasound is an effective tool for rapid noninvasive assessment of cardiac structure and function. Determining the cardiorespiratory phases of each frame in the ultrasound video and capturing the cardiac function at a much higher temporal resolution are essential in many applications. Fulfilling these requirements is particularly challenging in preclinical studies involving small animals with high cardiorespiratory rates, requiring cumbersome and expensive specialized hardware. METHODS: We present a novel method for the retrospective estimation of cardiorespiratory phases directly from the ultrasound videos. It transforms the videos into a univariate time series preserving the evidence of periodic cardiorespiratory motion, decouples the signatures of cardiorespiratory motion with a trend extraction technique, and estimates the cardiorespiratory phases using a Hilbert transform approach. We also present a robust nonparametric regression technique for respiratory gating and a novel kernel-regression model for reconstructing images at any cardiac phase facilitating temporal superresolution. RESULTS: We validated our methods using two-dimensional echocardiography videos and electrocardiogram (ECG) recordings of six mice. Our cardiac phase estimation method provides accurate phase estimates with a mean-phase-error range of 3%-6% against ECG derived phase and outperforms three previously published methods in locating ECGs R-wave peak frames with a mean-frame-error range of 0.73-1.36. Our kernel-regression model accurately reconstructs images at any cardiac phase with a mean-normalized-correlation range of 0.81-0.85 over 50 leave-one-out-cross-validation rounds. CONCLUSION AND SIGNIFICANCE: Our methods can enable tracking of cardiorespiratory phases without additional hardware and reconstruction of respiration-free single cardiac-cycle videos at a much higher temporal resolution.

Region-specific Diffeomorphic Metric Mapping.

We introduce a region-specific diffeomorphic metric mapping (RDMM) registration approach. RDMM is non-parametric, estimating spatio-temporal velocity fields which parameterize the sought-for spatial transformation. Regularization of these velocity fields is necessary. In contrast to existing non-parametric registration approaches using a fixed spatially-invariant regularization, for example, the large displacement diffeomorphic metric mapping (LDDMM) model, our approach allows for spatially-varying regularization which is advected via the estimated spatio-temporal velocity field. Hence, not only can our model capture large displacements, it does so with a spatio-temporal regularizer that keeps track of how regions deform, which is a more natural mathematical formulation. We explore a family of RDMM registration approaches: 1) a registration model where regions with separate regularizations are pre-defined (e.g., in an atlas space or for distinct foreground and background regions), 2) a registration model where a general spatially-varying regularizer is estimated, and 3) a registration model where the spatially-varying regularizer is obtained via an end-to-end trained deep learning (DL) model. We provide a variational derivation of RDMM, showing that the model can assure diffeomorphic transformations in the continuum, and that LDDMM is a particular instance of RDMM. To evaluate RDMM performance we experiment 1) on synthetic 2D data and 2) on two 3D datasets: knee magnetic resonance images (MRIs) of the Osteoarthritis Initiative (OAI) and computed tomography images (CT) of the lung. Results show that our framework achieves comparable performance to state-of-the-art image registration approaches, while providing additional information via a learned spatio-temporal regularizer. Further, our deep learning approach allows for very fast RDMM and LDDMM estimations. Code is available at https://github.com/uncbiag/registration.