Release the Krakencoder: A unified brain connectome translation and fusion tool.

Brain connectivity can be estimated in many ways, depending on modality and processing strategy. Here we present the Krakencoder, a joint connectome mapping tool that simultaneously, bidirectionally translates between structural (SC) and functional connectivity (FC), and across different atlases and processing choices via a common latent representation. These mappings demonstrate unprecedented accuracy and individual-level identifiability; the mapping between SC and FC has identifiability 42-54% higher than existing models. The Krakencoder combines all connectome flavors via a shared low-dimensional latent space. This "fusion" representation i) better reflects familial relatedness, ii) preserves age- and sex-relevant information and iii) enhances cognition-relevant information. The Krakencoder can be applied without retraining to new, out-of-age-distribution data while still preserving inter-individual differences in the connectome predictions and familial relationships in the latent representations. The Krakencoder is a significant leap forward in capturing the relationship between multi-modal brain connectomes in an individualized, behaviorally- and demographically-relevant way.

Remote Associations Between Tau and Cortical Amyloid-ß Are Stage-Dependent.

Background: Histopathologic studies of Alzheimer's disease (AD) suggest that extracellular amyloid-ß (Aß) plaques promote the spread of neurofibrillary tau tangles. However, these two proteinopathies initiate in spatially distinct brain regions, so how they interact during AD progression is unclear. Objective: In this study, we utilized Aß and tau positron emission tomography (PET) scans from 572 older subjects (476 healthy controls (HC), 14 with mild cognitive impairment (MCI), 82 with mild AD), at varying stages of the disease, to investigate to what degree tau is associated with cortical Aß deposition. Methods: Using multiple linear regression models and a pseudo-longitudinal ordering technique, we investigated remote tau-Aß associations in four pathologic phases of AD progression based on tau spread: 1) no-tau, 2) pre-acceleration, 3) acceleration, and 4) post-acceleration. Results: No significant tau-Aß association was detected in the no-tau phase. In the pre-acceleration phase, the earliest stage of tau deposition, associations emerged between regional tau in medial temporal lobe (MTL) (i.e., entorhinal cortex, parahippocampal gyrus) and cortical Aß in lateral temporal lobe regions. The strongest tau-Aß associations were found in the acceleration phase, in which tau in MTL regions was strongly associated with cortical Aß (i.e., temporal and frontal lobes regions). Strikingly, in the post-acceleration phase, including 96% of symptomatic subjects, tau-Aß associations were no longer significant. Conclusions: The results indicate that associations between tau and Aß are stage-dependent, which could have important implications for understanding the interplay between these two proteinopathies during the progressive stages of AD.

Learning across diverse biomedical data modalities and cohorts: Challenges and opportunities for innovation.

In healthcare, machine learning (ML) shows significant potential to augment patient care, improve population health, and streamline healthcare workflows. Realizing its full potential is, however, often hampered by concerns about data privacy, diversity in data sources, and suboptimal utilization of different data modalities. This review studies the utility of cross-cohort cross-category (C4) integration in such contexts: the process of combining information from diverse datasets distributed across distinct, secure sites. We argue that C4 approaches could pave the way for ML models that are both holistic and widely applicable. This paper provides a comprehensive overview of C4 in health care, including its present stage, potential opportunities, and associated challenges.

Test Retest Reproducibility of Organ Volume Measurements in ADPKD Using 3D Multimodality Deep Learning.

RATIONALE AND OBJECTIVES: Following autosomal dominant polycystic kidney disease (ADPKD) progression by measuring organ volumes requires low measurement variability. The objective of this study is to reduce organ volume measurement variability on MRI of ADPKD patients by utilizing all pulse sequences to obtain multiple measurements which allows outlier analysis to find errors and averaging to reduce variability. MATERIALS AND METHODS: In order to make measurements on multiple pulse sequences practical, a 3D multi-modality multi-class segmentation model based on nnU-net was trained/validated using T1, T2, SSFP, DWI and CT from 413 subjects. Reproducibility was assessed with test-re-test methodology on ADPKD subjects (n = 19) scanned twice within a 3-week interval correcting outliers and averaging the measurements across all sequences. Absolute percent differences in organ volumes were compared to paired students t-test. RESULTS: Dice similarlity coefficient > 97%, Jaccard Index > 0.94, mean surface distance < 1 mm and mean Hausdorff Distance < 2 cm for all three organs and all five sequences were found on internal (n = 25), external (n = 37) and test-re-test reproducibility assessment (38 scans in 19 subjects). When averaging volumes measured from five MRI sequences, the model automatically segmented kidneys with test-re-test reproducibility (percent absolute difference between exam 1 and exam 2) of 1.3% which was better than all five expert observers. It reliably stratified ADPKD into Mayo Imaging Classification (area under the curve=100%) compared to radiologist. CONCLUSION: 3D deep learning measures organ volumes on five MRI sequences leveraging the power of outlier analysis and averaging to achieve 1.3% total kidney test-re-test reproducibility.

Human brain responses are modulated when exposed to optimized natural images or synthetically generated images.

Understanding how human brains interpret and process information is important. Here, we investigated the selectivity and inter-individual differences in human brain responses to images via functional MRI. In our first experiment, we found that images predicted to achieve maximal activations using a group level encoding model evoke higher responses than images predicted to achieve average activations, and the activation gain is positively associated with the encoding model accuracy. Furthermore, anterior temporal lobe face area (aTLfaces) and fusiform body area 1 had higher activation in response to maximal synthetic images compared to maximal natural images. In our second experiment, we found that synthetic images derived using a personalized encoding model elicited higher responses compared to synthetic images from group-level or other subjects' encoding models. The finding of aTLfaces favoring synthetic images than natural images was also replicated. Our results indicate the possibility of using data-driven and generative approaches to modulate macro-scale brain region responses and probe inter-individual differences in and functional specialization of the human visual system.

A robust and interpretable deep learning framework for multi-modal registration via keypoints.

We present KeyMorph, a deep learning-based image registration framework that relies on automatically detecting corresponding keypoints. State-of-the-art deep learning methods for registration often are not robust to large misalignments, are not interpretable, and do not incorporate the symmetries of the problem. In addition, most models produce only a single prediction at test-time. Our core insight which addresses these shortcomings is that corresponding keypoints between images can be used to obtain the optimal transformation via a differentiable closed-form expression. We use this observation to drive the end-to-end learning of keypoints tailored for the registration task, and without knowledge of ground-truth keypoints. This framework not only leads to substantially more robust registration but also yields better interpretability, since the keypoints reveal which parts of the image are driving the final alignment. Moreover, KeyMorph can be designed to be equivariant under image translations and/or symmetric with respect to the input image ordering. Finally, we show how multiple deformation fields can be computed efficiently and in closed-form at test time corresponding to different transformation variants. We demonstrate the proposed framework in solving 3D affine and spline-based registration of multi-modal brain MRI scans. In particular, we show registration accuracy that surpasses current state-of-the-art methods, especially in the context of large displacements. Our code is available at https://github.com/alanqrwang/keymorph.

Deep learning analysis of blood flow sounds to detect arteriovenous fistula stenosis.

For hemodialysis patients, arteriovenous fistula (AVF) patency determines whether adequate hemofiltration can be achieved, and directly influences clinical outcomes. Here, we report the development and performance of a deep learning model for automated AVF stenosis screening based on the sound of AVF blood flow using supervised learning with data validated by ultrasound. We demonstrate the importance of contextualizing the sound with location metadata as the characteristics of the blood flow sound varies significantly along the AVF. We found the best model to be a vision transformer trained on spectrogram images. Our model can screen for stenosis at a performance level comparable to that of a nephrologist performing a physical exam, but with the advantage of being automated and scalable. In a high-volume, resource-limited clinical setting, automated AVF stenosis screening can help ensure patient safety via early detection of at-risk vascular access, streamline the dialysis workflow, and serve as a patient-facing tool to allow for at-home, self-screening.

Modulating human brain responses via optimal natural image selection and synthetic image generation.

One of the main goals of neuroscience is to understand how biological brains interpret and process incoming environmental information. Building computational encoding models that map images to neural responses is one way to pursue this goal. Moreover, generating or selecting visual stimuli designed to achieve specific patterns of responses allows exploration and control of neuronal firing rates or regional brain activity responses. Here, we investigated the brain's regional activation selectivity and inter-individual differences in human brain responses to various sets of natural and synthetic (generated) images via two functional MRI (fMRI) studies. For our first fMRI study, we used a pre-trained group-level neural model for selecting or synthesizing images that are predicted to maximally activate targeted brain regions. We then presented these images to subjects while collecting their fMRI data. Our results show that optimized images indeed evoke larger magnitude responses than other images predicted to achieve average levels of activation.Furthermore, the activation gain is positively associated with the encoding model accuracy. While most regions' activations in response to maximal natural images and maximal synthetic images were not different, two regions, namely anterior temporal lobe faces (aTLfaces) and fusiform body area 1 (FBA1), had significantly higher activation in response to maximal synthetic images compared to maximal natural images. On the other hand, three regions; medial temporal lobe face area (mTLfaces), ventral word form area 1 (VWFA1) and ventral word form area 2 (VWFA2), had higher activation in response to maximal natural images compared to maximal synthetic images. In our second fMRI experiment, we focused on probing inter-individual differences in face regions' responses and found that individual-specific synthetic (and not natural) images derived using a personalized encoding model elicited significantly higher responses compared to synthetic images derived from the group-level or other subjects' encoding models. Finally, we replicated the finding showing synthetic images elicited larger activation responses in the aTLfaces region compared to natural image responses in that region. Here, for the first time, we leverage our data-driven and generative modeling framework NeuroGen to probe inter-individual differences in and functional specialization of the human visual system. Our results indicate that NeuroGen can be used to modulate macro-scale brain regions in specific individuals using synthetically generated visual stimuli.

Optic Nerve Diameter on Non-Contrast Computed Tomography and Intracranial Hypertension in Patients With Acute Brain Injury: A Validation Study.

Intracranial hypertension is a feared complication of acute brain injury that can cause ischemic stroke, herniation, and death. Identifying those at risk is difficult, and the physical examination is often confounded. Given the widespread availability and use of computed tomography (CT) in patients with acute brain injury, prior work has attempted to use optic nerve diameter measurements to identify those at risk of intracranial hypertension. We aimed to validate the use of optic nerve diameter measurements on CT as a screening tool for intracranial hypertension in a large cohort of brain-injured patients. We performed a retrospective observational cohort study in a single tertiary referral Neuroscience Intensive Care Unit. We identified patients with documented intracranial pressure (ICP) measures as part of their routine clinical care who had non-contrast CT head scans collected within 24 h, and then measured the optic nerve diameters and explored the relationship and test characteristics of these measures to identify those at risk of intracranial hypertension. In a cohort of 314 patients, optic nerve diameter on CT was linearly but weakly associated with ICP. When used to identify those with intracranial hypertension (> 20 mm Hg), the area under the receiver operator curve (AUROC) was 0.68. Using a previously proposed threshold of 0.6 cm, the sensitivity was 81%, specificity 43%, positive likelihood ratio 1.4, and negative likelihood ratio 0.45. CT-derived optic nerve diameter using a threshold of 0.6 cm is sensitive but not specific for intracranial hypertension, and the overall correlation is weak.

Hyper-convolutions via implicit kernels for medical image analysis.

The convolutional neural network (CNN) is one of the most commonly used architectures for computer vision tasks. The key building block of a CNN is the convolutional kernel that aggregates information from the pixel neighborhood and shares weights across all pixels. A standard CNN's capacity, and thus its performance, is directly related to the number of learnable kernel weights, which is determined by the number of channels and the kernel size (support). In this paper, we present the hyper-convolution, a novel building block that implicitly encodes the convolutional kernel using spatial coordinates. Unlike a regular convolutional kernel, whose weights are independently learned, hyper-convolution kernel weights are correlated through an encoder that maps spatial coordinates to their corresponding values. Hyper-convolutions decouple kernel size from the total number of learnable parameters, enabling a more flexible architecture design. We demonstrate in our experiments that replacing regular convolutions with hyper-convolutions can improve performance with less parameters, and increase robustness against noise. We provide our code here: https://github.com/tym002/Hyper-Convolution.

LARO: Learned acquisition and reconstruction optimization to accelerate quantitative susceptibility mapping.

Quantitative susceptibility mapping (QSM) involves acquisition and reconstruction of a series of images at multi-echo time points to estimate tissue field, which prolongs scan time and requires specific reconstruction technique. In this paper, we present our new framework, called Learned Acquisition and Reconstruction Optimization (LARO), which aims to accelerate the multi-echo gradient echo (mGRE) pulse sequence for QSM. Our approach involves optimizing a Cartesian multi-echo k-space sampling pattern with a deep reconstruction network. Next, this optimized sampling pattern was implemented in an mGRE sequence using Cartesian fan-beam k-space segmenting and ordering for prospective scans. Furthermore, we propose to insert a recurrent temporal feature fusion module into the reconstruction network to capture signal redundancies along echo time. Our ablation studies show that both the optimized sampling pattern and proposed reconstruction strategy help improve the quality of the multi-echo image reconstructions. Generalization experiments show that LARO is robust on the test data with new pathologies and different sequence parameters. Our code is available at https://github.com/Jinwei1209/LARO-QSM.git.

Pulse Sequence Dependence of a Simple and Interpretable Deep Learning Method for Detection of Clinically Significant Prostate Cancer Using Multiparametric MRI.

RATIONALE AND OBJECTIVES: Multiparametric magnetic resonance imaging (mpMRI) is increasingly used for risk stratification and localization of prostate cancer (PCa). Thanks to the great success of deep learning models in computer vision, the potential application for early detection of PCa using mpMRI is imminent. MATERIALS AND METHODS: Deep learning analysis of the PROSTATEx dataset. RESULTS: In this study, we show a simple convolutional neural network (CNN) with mpMRI can achieve high performance for detection of clinically significant PCa (csPCa), depending on the pulse sequences used. The mpMRI model with T2-ADC-DWI achieved 0.90 AUC score in the held-out test set, not significantly better than the model using Ktrans instead of DWI (AUC 0.89). Interestingly, the model incorporating T2-ADC- Ktrans better estimates grade. We also describe a saliency "heat" map. Our results show that csPCa detection models with mpMRI may be leveraged to guide clinical management strategies. CONCLUSION: Convolutional neural networks incorporating multiple pulse sequences show high performance for detection of clinically-significant prostate cancer, and the model including dynamic contrast-enhanced information correlates best with grade.

Machine learning can aid in prediction of IDH mutation from H&E-stained histology slides in infiltrating gliomas.

While Machine Learning (ML) models have been increasingly applied to a range of histopathology tasks, there has been little emphasis on characterizing these models and contrasting them with human experts. We present a detailed empirical analysis comparing expert neuropathologists and ML models at predicting IDH mutation status in H&E-stained histology slides of infiltrating gliomas, both independently and synergistically. We find that errors made by neuropathologists and ML models trained using the TCGA dataset are distinct, representing modest agreement between predictions (human-vs.-human κ = 0.656; human-vs.-ML model κ = 0.598). While no ML model surpassed human performance on an independent institutional test dataset (human AUC = 0.901, max ML AUC = 0.881), a hybrid model aggregating human and ML predictions demonstrates predictive performance comparable to the consensus of two expert neuropathologists (hybrid classifier AUC = 0.921 vs. two-neuropathologist consensus AUC = 0.920). We also show that models trained at different levels of magnification exhibit different types of errors, supporting the value of aggregation across spatial scales in the ML approach. Finally, we present a detailed interpretation of our multi-scale ML ensemble model which reveals that predictions are driven by human-identifiable features at the patch-level.

Machine learning based multi-modal prediction of future decline toward Alzheimer's disease: An empirical study.

Alzheimer's disease (AD) is a neurodegenerative condition that progresses over decades. Early detection of individuals at high risk of future progression toward AD is likely to be of critical significance for the successful treatment and/or prevention of this devastating disease. In this paper, we present an empirical study to characterize how predictable an individual subjects' future AD trajectory is, several years in advance, based on rich multi-modal data, and using modern deep learning methods. Crucially, the machine learning strategy we propose can handle different future time horizons and can be trained with heterogeneous data that exhibit missingness and non-uniform follow-up visit times. Our experiments demonstrate that our strategy yields predictions that are more accurate than a model trained on a single time horizon (e.g. 3 years), which is common practice in prior literature. We also provide a comparison between linear and nonlinear models, verifying the well-established insight that the latter can offer a boost in performance. Our results also confirm that predicting future decline for cognitively normal (CN) individuals is more challenging than for individuals with mild cognitive impairment (MCI). Intriguingly, however, we discover that prediction accuracy decreases with increasing time horizon for CN subjects, but the trend is in the opposite direction for MCI subjects. Additionally, we quantify the contribution of different data types in prediction, which yields novel insights into the utility of different biomarkers. We find that molecular biomarkers are not as helpful for CN individuals as they are for MCI individuals, whereas magnetic resonance imaging biomarkers (hippocampus volume, specifically) offer a significant boost in prediction accuracy for CN individuals. Finally, we show how our model's prediction reveals the evolution of individual-level progression risk over a five-year time horizon. Our code is available at https://github.com/batuhankmkaraman/mlbasedad.

A transformer-Based neural language model that synthesizes brain activation maps from free-form text queries.

Neuroimaging studies are often limited by the number of subjects and cognitive processes that can be feasibly interrogated. However, a rapidly growing number of neuroscientific studies have collectively accumulated an extensive wealth of results. Digesting this growing literature and obtaining novel insights remains to be a major challenge, since existing meta-analytic tools are constrained to keyword queries. In this paper, we present Text2Brain, an easy to use tool for synthesizing brain activation maps from open-ended text queries. Text2Brain was built on a transformer-based neural network language model and a coordinate-based meta-analysis of neuroimaging studies. Text2Brain combines a transformer-based text encoder and a 3D image generator, and was trained on variable-length text snippets and their corresponding activation maps sampled from 13,000 published studies. In our experiments, we demonstrate that Text2Brain can synthesize meaningful neural activation patterns from various free-form textual descriptions. Text2Brain is available at https://braininterpreter.com as a web-based tool for efficiently searching through the vast neuroimaging literature and generating new hypotheses.

NeuroGen: Activation optimized image synthesis for discovery neuroscience.

Functional MRI (fMRI) is a powerful technique that has allowed us to characterize visual cortex responses to stimuli, yet such experiments are by nature constructed based on a priori hypotheses, limited to the set of images presented to the individual while they are in the scanner, are subject to noise in the observed brain responses, and may vary widely across individuals. In this work, we propose a novel computational strategy, which we call NeuroGen, to overcome these limitations and develop a powerful tool for human vision neuroscience discovery. NeuroGen combines an fMRI-trained neural encoding model of human vision with a deep generative network to synthesize images predicted to achieve a target pattern of macro-scale brain activation. We demonstrate that the reduction of noise that the encoding model provides, coupled with the generative network's ability to produce images of high fidelity, results in a robust discovery architecture for visual neuroscience. By using only a small number of synthetic images created by NeuroGen, we demonstrate that we can detect and amplify differences in regional and individual human brain response patterns to visual stimuli. We then verify that these discoveries are reflected in the several thousand observed image responses measured with fMRI. We further demonstrate that NeuroGen can create synthetic images predicted to achieve regional response patterns not achievable by the best-matching natural images. The NeuroGen framework extends the utility of brain encoding models and opens up a new avenue for exploring, and possibly precisely controlling, the human visual system.

Predicting individual task contrasts from resting-state functional connectivity using a surface-based convolutional network.

Task-based and resting-state represent the two most common experimental paradigms of functional neuroimaging. While resting-state offers a flexible and scalable approach for characterizing brain function, task-based techniques provide superior localization. In this paper, we build on recent deep learning methods to create a model that predicts task-based contrast maps from resting-state fMRI scans. Specifically, we propose BrainSurfCNN, a surface-based fully-convolutional neural network model that works with a representation of the brain's cortical sheet. BrainSurfCNN achieves exceptional predictive accuracy on independent test data from the Human Connectome Project, which is on par with the repeat reliability of the measured subject-level contrast maps. Conversely, our analyses reveal that a previously published benchmark is no better than group-average contrast maps. Finally, we demonstrate that BrainSurfCNN can generalize remarkably well to novel domains with limited training data.

Cortical response to naturalistic stimuli is largely predictable with deep neural networks.

Naturalistic stimuli, such as movies, activate a substantial portion of the human brain, invoking a response shared across individuals. Encoding models that predict neural responses to arbitrary stimuli can be very useful for studying brain function. However, existing models focus on limited aspects of naturalistic stimuli, ignoring the dynamic interactions of modalities in this inherently context-rich paradigm. Using movie-watching data from the Human Connectome Project, we build group-level models of neural activity that incorporate several inductive biases about neural information processing, including hierarchical processing, temporal assimilation, and auditory-visual interactions. We demonstrate how incorporating these biases leads to remarkable prediction performance across large areas of the cortex, beyond the sensory-specific cortices into multisensory sites and frontal cortex. Furthermore, we illustrate that encoding models learn high-level concepts that generalize to task-bound paradigms. Together, our findings underscore the potential of encoding models as powerful tools for studying brain function in ecologically valid conditions.

Magnetic Resonance Imaging Radiomics-Based Machine Learning Prediction of Clinically Significant Prostate Cancer in Equivocal PI-RADS 3 Lesions.

BACKGROUND: While Prostate Imaging Reporting and Data System (PI-RADS) 4 and 5 lesions typically warrant prostate biopsy and PI-RADS 1 and 2 lesions may be safely observed, PI-RADS 3 lesions are equivocal. PURPOSE: To construct and cross-validate a machine learning model based on radiomics features from T2 -weighted imaging (T2 WI) of PI-RADS 3 lesions to identify clinically significant prostate cancer (csPCa), that is, pathological Grade Group ≥ 2. STUDY TYPE: Single-center retrospective study. POPULATION: A total of 240 patients were included (training cohort, n = 188, age range 43-82 years; test cohort, n = 52, age range 41-79 years). Eligibility criteria were 1) magnetic resonance imaging (MRI)-targeted biopsy between 2015 and 2020; 2) PI-RADS 3 index lesion identified on multiparametric MRI; (3) biopsy performed within 1 year of MRI. The percentages of csPCa lesions were 10.6% and 15.4% in the training and test cohorts, respectively. FIELD STRENGTH/SEQUENCE: A 3 T; T2 WI turbo-spin echo, diffusion-weighted spin-echo echo planar imaging, dynamic contrast-enhanced MRI with time-resolved T1-weighted imaging. ASSESSMENT: Multislice volumes-of-interest (VOIs) were drawn in the PI-RADS 3 index lesions on T2 WI. A total of 107 radiomics features (first-order histogram and second-order texture) were extracted from the segmented lesions. STATISTICAL TESTS: A random forest classifier using the radiomics features as input was trained and validated for prediction of csPCa. The performance of the machine learning classifier, prostate specific antigen (PSA) density, and prostate volume for csPCa prediction was evaluated using receiver operating characteristic (ROC) analysis. RESULTS: The trained random forest classifier constructed from the T2 WI radiomics features good and statistically significant area-under-the-curves (AUCs) of 0.76 (P = 0.022) for prediction of csPCa in the test set. Prostate volume and PSA density showed moderate and nonsignificant performance (AUC 0.62, P = 0.275 and 0.61, P = 0.348, respectively) for csPCa prediction in the test set. CONCLUSION: The machine learning classifier based on T2 WI radiomic features demonstrated good performance for prediction of csPCa in PI-RADS 3 lesions. EVIDENCE LEVEL: 4 TECHNICAL EFFICACY: 2.