SE(3)-Equivariant and Noise-Invariant 3D Rigid Motion Tracking in Brain MRI.

Rigid motion tracking is paramount in many medical imaging applications where movements need to be detected, corrected, or accounted for. Modern strategies rely on convolutional neural networks (CNN) and pose this problem as rigid registration. Yet, CNNs do not exploit natural symmetries in this task, as they are equivariant to translations (their outputs shift with their inputs) but not to rotations. Here we propose EquiTrack, the first method that uses recent steerable SE(3)-equivariant CNNs (E-CNN) for motion tracking. While steerable E-CNNs can extract corresponding features across different poses, testing them on noisy medical images reveals that they do not have enough learning capacity to learn noise invariance. Thus, we introduce a hybrid architecture that pairs a denoiser with an E-CNN to decouple the processing of anatomically irrelevant intensity features from the extraction of equivariant spatial features. Rigid transforms are then estimated in closed-form. EquiTrack outperforms state-of-the-art learning and optimisation methods for motion tracking in adult brain MRI and fetal MRI time series. Our code is available at https://github.com/BBillot/EquiTrack.

Empirical assessment of the assumptions of ComBat with diffusion tensor imaging.

Purpose: Diffusion tensor imaging (DTI) is a magnetic resonance imaging technique that provides unique information about white matter microstructure in the brain but is susceptible to confounding effects introduced by scanner or acquisition differences. ComBat is a leading approach for addressing these site biases. However, despite its frequent use for harmonization, ComBat's robustness toward site dissimilarities and overall cohort size have not yet been evaluated in terms of DTI. Approach: As a baseline, we match N=358 participants from two sites to create a "silver standard" that simulates a cohort for multi-site harmonization. Across sites, we harmonize mean fractional anisotropy and mean diffusivity, calculated using participant DTI data, for the regions of interest defined by the JHU EVE-Type III atlas. We bootstrap 10 iterations at 19 levels of total sample size, 10 levels of sample size imbalance between sites, and 6 levels of mean age difference between sites to quantify (i) ßAGE, the linear regression coefficient of the relationship between FA and age; (ii) Î³/f*, the ComBat-estimated site-shift; and (iii) Î´/f*, the ComBat-estimated site-scaling. We characterize the reliability of ComBat by evaluating the root mean squared error in these three metrics and examine if there is a correlation between the reliability of ComBat and a violation of assumptions. Results: ComBat remains well behaved for ßAGE when N>162 and when the mean age difference is less than 4 years. The assumptions of the ComBat model regarding the normality of residual distributions are not violated as the model becomes unstable. Conclusion: Prior to harmonization of DTI data with ComBat, the input cohort should be examined for size and covariate distributions of each site. Direct assessment of residual distributions is less informative on stability than bootstrap analysis. We caution use ComBat of in situations that do not conform to the above thresholds.

SpecReFlow: an algorithm for specular reflection restoration using flow-guided video completion.

Purpose: Specular reflections (SRs) are highlight artifacts commonly found in endoscopy videos that can severely disrupt a surgeon's observation and judgment. Despite numerous attempts to restore SR, existing methods are inefficient and time consuming and can lead to false clinical interpretations. Therefore, we propose the first complete deep-learning solution, SpecReFlow, to detect and restore SR regions from endoscopy video with spatial and temporal coherence. Approach: SpecReFlow consists of three stages: (1) an image preprocessing stage to enhance contrast, (2) a detection stage to indicate where the SR region is present, and (3) a restoration stage in which we replace SR pixels with an accurate underlying tissue structure. Our restoration approach uses optical flow to seamlessly propagate color and structure from other frames of the endoscopy video. Results: Comprehensive quantitative and qualitative tests for each stage reveal that our SpecReFlow solution performs better than previous detection and restoration methods. Our detection stage achieves a Dice score of 82.8% and a sensitivity of 94.6%, and our restoration stage successfully incorporates temporal information with spatial information for more accurate restorations than existing techniques. Conclusions: SpecReFlow is a first-of-its-kind solution that combines temporal and spatial information for effective detection and restoration of SR regions, surpassing previous methods relying on single-frame spatial information. Future work will look to optimizing SpecReFlow for real-time applications. SpecReFlow is a software-only solution for restoring image content lost due to SR, making it readily deployable in existing clinical settings to improve endoscopy video quality for accurate diagnosis and treatment.

DeepN4: Learning N4ITK Bias Field Correction for T1-weighted Images.

T1-weighted (T1w) MRI has low frequency intensity artifacts due to magnetic field inhomogeneities. Removal of these biases in T1w MRI images is a critical preprocessing step to ensure spatially consistent image interpretation. N4ITK bias field correction, the current state-of-the-art, is implemented in such a way that makes it difficult to port between different pipelines and workflows, thus making it hard to reimplement and reproduce results across local, cloud, and edge platforms. Moreover, N4ITK is opaque to optimization before and after its application, meaning that methodological development must work around the inhomogeneity correction step. Given the importance of bias fields correction in structural preprocessing and flexible implementation, we pursue a deep learning approximation / reinterpretation of the N4ITK bias fields correction to create a method which is portable, flexible, and fully differentiable. In this paper, we trained a deep learning network "DeepN4" on eight independent cohorts from 72 different scanners and age ranges with N4ITK-corrected T1w MRI and bias field for supervision in log space. We found that we can closely approximate N4ITK bias fields correction with naïve networks. We evaluate the peak signal to noise ratio (PSNR) in test dataset against the N4ITK corrected images. The median PSNR of corrected images between N4ITK and DeepN4 was 47.96 dB. In addition, we assess the DeepN4 model on eight additional external datasets and show the generalizability of the approach. This study establishes that incompatible N4ITK preprocessing steps can be closely approximated by naïve deep neural networks, facilitating more flexibility. All code and models are released at https://github.com/MASILab/DeepN4 .

MidRISH: Unbiased harmonization of rotationally invariant harmonics of the diffusion signal.

Data harmonization is necessary for removing confounding effects in multi-site diffusion image analysis. One such harmonization method, LinearRISH, scales rotationally invariant spherical harmonic (RISH) features from one site ("target") to the second ("reference") to reduce confounding scanner effects. However, reference and target site designations are not arbitrary and resultant diffusion metrics (fractional anisotropy, mean diffusivity) are biased by this choice. In this work we propose MidRISH: rather than scaling reference RISH features to target RISH features, we project both sites to a mid-space. We validate MidRISH with the following experiments: harmonizing scanner differences from 37 matched patients free of cognitive impairment, and harmonizing acquisition and study differences on 117 matched patients free of cognitive impairment. We find that MidRISH reduces bias of reference selection while preserving harmonization efficacy of LinearRISH. Users should be cautious when performing LinearRISH harmonization. To select a reference site is to choose diffusion metric effect-size. Our proposed method eliminates the bias-inducing site selection step.

Evaluation of Mean Shift, ComBat, and CycleGAN for Harmonizing Brain Connectivity Matrices Across Sites.

Connectivity matrices derived from diffusion MRI (dMRI) provide an interpretable and generalizable way of understanding the human brain connectome. However, dMRI suffers from inter-site and between-scanner variation, which impedes analysis across datasets to improve robustness and reproducibility of results. To evaluate different harmonization approaches on connectivity matrices, we compared graph measures derived from these matrices before and after applying three harmonization techniques: mean shift, ComBat, and CycleGAN. The sample comprises 168 age-matched, sex-matched normal subjects from two studies: the Vanderbilt Memory and Aging Project (VMAP) and the Biomarkers of Cognitive Decline Among Normal Individuals (BIOCARD). First, we plotted the graph measures and used coefficient of variation (CoV) and the Mann-Whitney U test to evaluate different methods' effectiveness in removing site effects on the matrices and the derived graph measures. ComBat effectively eliminated site effects for global efficiency and modularity and outperformed the other two methods. However, all methods exhibited poor performance when harmonizing average betweenness centrality. Second, we tested whether our harmonization methods preserved correlations between age and graph measures. All methods except for CycleGAN in one direction improved correlations between age and global efficiency and between age and modularity from insignificant to significant with p-values less than 0.05.

Predicting Age from White Matter Diffusivity with Residual Learning.

Imaging findings inconsistent with those expected at specific chronological age ranges may serve as early indicators of neurological disorders and increased mortality risk. Estimation of chronological age, and deviations from expected results, from structural magnetic resonance imaging (MRI) data has become an important proxy task for developing biomarkers that are sensitive to such deviations. Complementary to structural analysis, diffusion tensor imaging (DTI) has proven effective in identifying age-related microstructural changes within the brain white matter, thereby presenting itself as a promising additional modality for brain age prediction. Although early studies have sought to harness DTI's advantages for age estimation, there is no evidence that the success of this prediction is owed to the unique microstructural and diffusivity features that DTI provides, rather than the macrostructural features that are also available in DTI data. Therefore, we seek to develop white-matter-specific age estimation to capture deviations from normal white matter aging. Specifically, we deliberately disregard the macrostructural information when predicting age from DTI scalar images, using two distinct methods. The first method relies on extracting only microstructural features from regions of interest (ROIs). The second applies 3D residual neural networks (ResNets) to learn features directly from the images, which are non-linearly registered and warped to a template to minimize macrostructural variations. When tested on unseen data, the first method yields mean absolute error (MAE) of 6.11 ± 0.19 years for cognitively normal participants and MAE of 6.62 ± 0.30 years for cognitively impaired participants, while the second method achieves MAE of 4.69 ± 0.23 years for cognitively normal participants and MAE of 4.96 ± 0.28 years for cognitively impaired participants. We find that the ResNet model captures subtler, non-macrostructural features for brain age prediction.

Characterizing patterns of DTI variance in aging brains.

Background: As large analyses merge data across sites, a deeper understanding of variance in statistical assessment across the sources of data becomes critical for valid analyses. Diffusion tensor imaging (DTI) exhibits spatially varying and correlated noise, so care must be taken with distributional assumptions. Purpose: We characterize the role of physiology, subject compliance, and the interaction of subject with the scanner in the understanding of DTI variability, as modeled in spatial variance of derived metrics in homogeneous regions. Methods: We analyze DTI data from 1035 subjects in the Baltimore Longitudinal Study of Aging (BLSA), with ages ranging from 22.4 to 103 years old. For each subject, up to 12 longitudinal sessions were conducted. We assess variance of DTI scalars within regions of interest (ROIs) defined by four segmentation methods and investigate the relationships between the variance and covariates, including baseline age, time from the baseline (referred to as "interval"), motion, sex, and whether it is the first scan or the second scan in the session. Results: Covariate effects are heterogeneous and bilaterally symmetric across ROIs. Inter-session interval is positively related (p ≪ 0.001) to FA variance in the cuneus and occipital gyrus, but negatively (p ≪ 0.001) in the caudate nucleus. Males show significantly (p ≪ 0.001) higher FA variance in the right putamen, thalamus, body of the corpus callosum, and cingulate gyrus. In 62 out of 176 ROIs defined by the Eve type-1 atlas, an increase in motion is associated (p < 0.05) with a decrease in FA variance. Head motion increases during the rescan of DTI (Δµ = 0.045 millimeters per volume). Conclusions: The effects of each covariate on DTI variance, and their relationships across ROIs are complex. Ultimately, we encourage researchers to include estimates of variance when sharing data and consider models of heteroscedasticity in analysis. This work provides a foundation for study planning to account for regional variations in metric variance.

NeuroGraph: Benchmarks for Graph Machine Learning in Brain Connectomics.

Machine learning provides a valuable tool for analyzing high-dimensional functional neuroimaging data, and is proving effective in predicting various neurological conditions, psychiatric disorders, and cognitive patterns. In functional magnetic resonance imaging (MRI) research, interactions between brain regions are commonly modeled using graph-based representations. The potency of graph machine learning methods has been established across myriad domains, marking a transformative step in data interpretation and predictive modeling. Yet, despite their promise, the transposition of these techniques to the neuroimaging domain has been challenging due to the expansive number of potential preprocessing pipelines and the large parameter search space for graph-based dataset construction. In this paper, we introduce NeuroGraph, a collection of graph-based neuroimaging datasets, and demonstrated its utility for predicting multiple categories of behavioral and cognitive traits. We delve deeply into the dataset generation search space by crafting 35 datasets that encompass static and dynamic brain connectivity, running in excess of 15 baseline methods for benchmarking. Additionally, we provide generic frameworks for learning on both static and dynamic graphs. Our extensive experiments lead to several key observations. Notably, using correlation vectors as node features, incorporating larger number of regions of interest, and employing sparser graphs lead to improved performance. To foster further advancements in graph-based data driven neuroimaging analysis, we offer a comprehensive open-source Python package that includes the benchmark datasets, baseline implementations, model training, and standard evaluation.

DeepN4: Learning N4ITK Bias Field Correction for T1-weighted Images.

T1-weighted (T1w) MRI has low frequency intensity artifacts due to magnetic field inhomogeneities. Removal of these biases in T1w MRI images is a critical preprocessing step to ensure spatially consistent image interpretation. N4ITK bias field correction, the current state-of-the-art, is implemented in such a way that makes it difficult to port between different pipelines and workflows, thus making it hard to reimplement and reproduce results across local, cloud, and edge platforms. Moreover, N4ITK is opaque to optimization before and after its application, meaning that methodological development must work around the inhomogeneity correction step. Given the importance of bias fields correction in structural preprocessing and flexible implementation, we pursue a deep learning approximation / reinterpretation of the N4ITK bias fields correction to create a method which is portable, flexible, and fully differentiable. In this paper, we trained a deep learning network "DeepN4" on eight independent cohorts from 72 different scanners and age ranges with N4ITK-corrected T1w MRI and bias field for supervision in log space. We found that we can closely approximate N4ITK bias fields correction with naïve networks. We evaluate the peak signal to noise ratio (PSNR) in test dataset against the N4ITK corrected images. The median PSNR of corrected images between N4ITK and DeepN4 was 47.96 dB. In addition, we assess the DeepN4 model on eight additional external datasets and show the generalizability of the approach. This study establishes that incompatible N4ITK preprocessing steps can be closely approximated by naïve deep neural networks, facilitating more flexibility. All code and models are released at https://github.com/MASILab/DeepN4.

White matter tract microstructure, macrostructure, and associated cortical gray matter morphology across the lifespan.

Characterizing how, when and where the human brain changes across the lifespan is fundamental to our understanding of developmental processes of childhood and adolescence, degenerative processes of aging, and divergence from normal patterns in disease and disorders. We aimed to provide detailed descriptions of white matter pathways across the lifespan by thoroughly characterizing white matter microstructure, white matter macrostructure, and morphology of the cortex associated with white matter pathways. We analyzed 4 large, high-quality, publicly-available datasets comprising 2789 total imaging sessions, and participants ranging from 0 to 100 years old, using advanced tractography and diffusion modeling. We first find that all microstructural, macrostructural, and cortical features of white matter bundles show unique lifespan trajectories, with rates and timing of development and degradation that vary across pathways - describing differences between types of pathways and locations in the brain, and developmental milestones of maturation of each feature. Second, we show cross-sectional relationships between different features that may help elucidate biological changes occurring during different stages of the lifespan. Third, we show unique trajectories of age-associations across features. Finally, we find that age associations during development are strongly related to those during aging. Overall, this study reports normative data for several features of white matter pathways of the human brain that will be useful for studying normal and abnormal white matter development and degeneration.

MidRISH: Unbiased harmonization of rotationally invariant harmonics of the diffusion signal.

Objective: Data harmonization is necessary for removing confounding effects in multi-site diffusion image analysis. One such harmonization method, LinearRISH, scales rotationally invariant spherical harmonic (RISH) features from one site ("target") to the second ("reference") to reduce confounding scanner effects. However, reference and target site designations are not arbitrary and resultant diffusion metrics (fractional anisotropy, mean diffusivity) are biased by this choice. In this work we propose MidRISH: rather than scaling reference RISH features to target RISH features, we project both sites to a mid-space. Methods: We validate MidRISH with the following experiments: harmonizing scanner differences from 37 matched patients free of cognitive impairment, and harmonizing acquisition and study differences on 117 matched patients free of cognitive impairment. Conclusion: MidRISH reduces bias of reference selection while preserving harmonization efficacy of LinearRISH. Significance: Users should be cautious when performing LinearRISH harmonization. To select a reference site is to choose diffusion metric effect-size. Our proposed method eliminates the bias-inducing site selection step.

Implementation considerations for deep learning with diffusion MRI streamline tractography.

One area of medical imaging that has recently experienced innovative deep learning advances is diffusion MRI (dMRI) streamline tractography with recurrent neural networks (RNNs). Unlike traditional imaging studies which utilize voxel-based learning, these studies model dMRI features at points in continuous space off the voxel grid in order to propagate streamlines, or virtual estimates of axons. However, implementing such models is non-trivial, and an open-source implementation is not yet widely available. Here, we describe a series of considerations for implementing tractography with RNNs and demonstrate they allow one to approximate a deterministic streamline propagator with comparable performance to existing algorithms. We release this trained model and the associated implementations leveraging popular deep learning libraries. We hope the availability of these resources will lower the barrier of entry into this field, spurring further innovation.

NeSVoR: Implicit Neural Representation for Slice-to-Volume Reconstruction in MRI.

Reconstructing 3D MR volumes from multiple motion-corrupted stacks of 2D slices has shown promise in imaging of moving subjects, e. g., fetal MRI. However, existing slice-to-volume reconstruction methods are time-consuming, especially when a high-resolution volume is desired. Moreover, they are still vulnerable to severe subject motion and when image artifacts are present in acquired slices. In this work, we present NeSVoR, a resolution-agnostic slice-to-volume reconstruction method, which models the underlying volume as a continuous function of spatial coordinates with implicit neural representation. To improve robustness to subject motion and other image artifacts, we adopt a continuous and comprehensive slice acquisition model that takes into account rigid inter-slice motion, point spread function, and bias fields. NeSVoR also estimates pixel-wise and slice-wise variances of image noise and enables removal of outliers during reconstruction and visualization of uncertainty. Extensive experiments are performed on both simulated and in vivo data to evaluate the proposed method. Results show that NeSVoR achieves state-of-the-art reconstruction quality while providing two to ten-fold acceleration in reconstruction times over the state-of-the-art algorithms.

Convolutional-recurrent neural networks approximate diffusion tractography from T1-weighted MRI and associated anatomical context.

Diffusion MRI (dMRI) streamline tractography is the gold-standard for in vivo estimation of white matter (WM) pathways in the brain. However, the high angular resolution dMRI acquisitions capable of fitting the microstructural models needed for tractography are often time-consuming and not routinely collected clinically, restricting the scope of tractography analyses. To address this limitation, we build on recent advances in deep learning which have demonstrated that streamline propagation can be learned from dMRI directly without traditional model fitting. Specifically, we propose learning the streamline propagator from T1w MRI to facilitate arbitrary tractography analyses when dMRI is unavailable. To do so, we present a novel convolutional-recurrent neural network (CoRNN) trained in a teacher-student framework that leverages T1w MRI, associated anatomical context, and streamline memory from data acquired for the Human Connectome Project. We characterize our approach under two common tractography paradigms, WM bundle analysis and structural connectomics, and find approximately a 5-15% difference between measures computed from streamlines generated with our approach and those generated using traditional dMRI tractography. When placed in the literature, these results suggest that the accuracy of WM measures computed from T1w MRI with our method is on the level of scan-rescan dMRI variability and raise an important question: is tractography truly a microstructural phenomenon, or has dMRI merely facilitated its discovery and implementation?

Efficacy of second CAR-T (CART2) infusion limited by poor CART expansion and antigen modulation.

Chimeric antigen receptor T-cells (CART) are active in relapsed/refractory (r/r) B-cell acute lymphoblastic leukemia (B-ALL), but relapse remains a substantial challenge. Reinfusion with the same CART product (CART2) in patients with suboptimal response or antigen positive relapse following first infusion (CART1) represents a potential treatment strategy, though early experiences suggest limited efficacy of CART2 with CD19 targeting. We report on our experience with CART2 across a host of novel CAR T-cell trials. This was a retrospective review of children and young adults with B-ALL who received reinfusion with an anti-CD19, anti-CD22, or anti-CD19/22 CART construct on one of 3 CAR T-cells trials at the National Cancer Institute (NCT01593696, NCT02315612, NCT0344839) between July 2012 and January 2021. All patients received lymphodepletion (LD) pre-CART (standard LD: 75 mg/m2 fludarabine, 900 mg/m2 cyclophosphamide; or intensified LD: 120 mg/m2 fludarabine, 1200 mg/m2 cyclophosphamide). Primary objectives were to describe response to and toxicity of CART2. Indication for CART2, impact of LD intensity, and CAR T-cell expansion and leukemia antigen expression between CART infusions was additionally evaluated. Eighteen patients proceeded to CART2 due to persistent (n=7) or relapsed antigen positive disease (n=11) following CART1. Seven of 18 (38.9%) demonstrated objective response (responders) to CART2: 5 achieved a minimal residual disease (MRD) negative CR, 1 had persistent MRD level disease, and 1 showed a partial remission, the latter with eradication of antigen positive disease and emergence of antigen negative B-ALL. Responders included four patients who had not achieved a CR with CART1. Limited cytokine release syndrome was seen following CART2. Peripheral blood CART1 expansion was higher than CART2 expansion (p=0.03). Emergence of antigen negative/dim B-ALL in 6 (33.3%) patients following CART2 contributed to lack of CR. Five of seven (71.4%) responders received intensified LD pre-CART2, which corresponded with higher CART2 expansion than in those receiving standard LD (p=0.029). Diminished CAR T-cell expansion and antigen downregulation/loss impeded robust responses to CART2. A subset of patients, however, may derive benefit from CART2 despite suboptimal response to CART1. Intensified LD may be one strategy to augment CART2 responses, though further study of factors associated with CART2 response, including serial monitoring of antigen expression, is warranted.

SVoRT: Iterative Transformer for Slice-to-Volume Registration in Fetal Brain MRI.

Volumetric reconstruction of fetal brains from multiple stacks of MR slices, acquired in the presence of almost unpredictable and often severe subject motion, is a challenging task that is highly sensitive to the initialization of slice-to-volume transformations. We propose a novel slice-to-volume registration method using Transformers trained on synthetically transformed data, which model multiple stacks of MR slices as a sequence. With the attention mechanism, our model automatically detects the relevance between slices and predicts the transformation of one slice using information from other slices. We also estimate the underlying 3D volume to assist slice-to-volume registration and update the volume and transformations alternately to improve accuracy. Results on synthetic data show that our method achieves lower registration error and better reconstruction quality compared with existing state-of-the-art methods. Experiments with real-world MRI data are also performed to demonstrate the ability of the proposed model to improve the quality of 3D reconstruction under severe fetal motion.

COVID-19's U.S. Temperature Response Profile.

We estimate the U.S. temperature response profile (TRP) for COVID-19 and show it is highly sensitive to temperature variation. Replacing the erratic daily death counts U.S. states initially reported with counts based on death certificate date, we build a week-ahead statistical forecasting model that explains most of their daily variation (R2 = 0.97) and isolates COVID-19's TRP (p < 0.001). These counts, normalized at 31 °C (U.S. mid-summer average), scale up to 160% at 5 °C in the static case where the infection pool is held constant. Positive case counts are substantially more temperature sensitive. When temperatures are declining, dynamic feedback through a growing infection pool can substantially amplify these temperature effects. Our estimated TRP can be incorporated into COVID-related planning exercises and used as an input to SEIR models employed for longer run forecasting. For the former, we show how our TRP is predictive of the realized pattern of growth rates in per capita positive cases across states five months after the end of our sample period. For the latter, we show the variation in herd immunity levels implied by temperature-driven, time-varying R0 series for the Alpha and Delta variants of COVID-19 for several representative states. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1007/s10640-021-00603-8.

Multimodal Representation Learning via Maximization of Local Mutual Information.

We propose and demonstrate a representation learning approach by maximizing the mutual information between local features of images and text. The goal of this approach is to learn useful image representations by taking advantage of the rich information contained in the free text that describes the findings in the image. Our method trains image and text encoders by encouraging the resulting representations to exhibit high local mutual information. We make use of recent advances in mutual information estimation with neural network discriminators. We argue that the sum of local mutual information is typically a lower bound on the global mutual information. Our experimental results in the downstream image classification tasks demonstrate the advantages of using local features for image-text representation learning. Our code is available at: https://github.com/RayRuizhiLiao/mutual_info_img_txt.