massNet: integrated processing and classification of spatially resolved mass spectrometry data using deep learning for rapid tumor delineation.

MOTIVATION: Mass spectrometry imaging (MSI) provides rich biochemical information in a label-free manner and therefore holds promise to substantially impact current practice in disease diagnosis. However, the complex nature of MSI data poses computational challenges in its analysis. The complexity of the data arises from its large size, high-dimensionality and spectral nonlinearity. Preprocessing, including peak picking, has been used to reduce raw data complexity; however, peak picking is sensitive to parameter selection that, perhaps prematurely, shapes the downstream analysis for tissue classification and ensuing biological interpretation. RESULTS: We propose a deep learning model, massNet, that provides the desired qualities of scalability, nonlinearity and speed in MSI data analysis. This deep learning model was used, without prior preprocessing and peak picking, to classify MSI data from a mouse brain harboring a patient-derived tumor. The massNet architecture established automatically learning of predictive features, and automated methods were incorporated to identify peaks with potential for tumor delineation. The model's performance was assessed using cross-validation, and the results demonstrate higher accuracy and a substantial gain in speed compared to the established classical machine learning method, support vector machine. AVAILABILITY AND IMPLEMENTATION: https://github.com/wabdelmoula/massNet. The data underlying this article are available in the NIH Common Fund's National Metabolomics Data Repository (NMDR) Metabolomics Workbench under project id (PR001292) with http://dx.doi.org/10.21228/M8Q70T. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

Group-wise parcellation of the cortex through multi-scale spectral clustering.

The delineation of functionally and structurally distinct regions as well as their connectivity can provide key knowledge towards understanding the brain's behaviour and function. Cytoarchitecture has long been the gold standard for such parcellation tasks, but has poor scalability and cannot be mapped in vivo. Functional and diffusion magnetic resonance imaging allow in vivo mapping of brain's connectivity and the parcellation of the brain based on local connectivity information. Several methods have been developed for single subject connectivity driven parcellation, but very few have tackled the task of group-wise parcellation, which is essential for uncovering group specific behaviours. In this paper, we propose a group-wise connectivity-driven parcellation method based on spectral clustering that captures local connectivity information at multiple scales and directly enforces correspondences between subjects. The method is applied to diffusion Magnetic Resonance Imaging driven parcellation on two independent groups of 50 subjects from the Human Connectome Project. Promising quantitative and qualitative results in terms of information loss, modality comparisons, group consistency and inter-group similarities demonstrate the potential of the method.

Transperineal in-bore 3-T MR imaging-guided prostate biopsy: a prospective clinical observational study.

PURPOSE: To determine the detection rate, clinical relevance, Gleason grade, and location of prostate cancer ( PCa prostate cancer ) diagnosed with and the safety of an in-bore transperineal 3-T magnetic resonance (MR) imaging-guided prostate biopsy in a clinically heterogeneous patient population. MATERIALS AND METHODS: This prospective retrospectively analyzed study was HIPAA compliant and institutional review board approved, and informed consent was obtained. Eighty-seven men (mean age, 66.2 years ± 6.9) underwent multiparametric endorectal prostate MR imaging at 3 T and transperineal MR imaging-guided biopsy. Three subgroups of patients with at least one lesion suspicious for cancer were included: men with no prior PCa prostate cancer diagnosis, men with PCa prostate cancer who were undergoing active surveillance, and men with treated PCa prostate cancer and suspected recurrence. Exclusion criteria were prior prostatectomy and/or contraindication to 3-T MR imaging. The transperineal MR imaging-guided biopsy was performed in a 70-cm wide-bore 3-T device. Overall patient biopsy outcomes, cancer detection rates, Gleason grade, and location for each subgroup were evaluated and statistically compared by using χ(2) and one-way analysis of variance followed by Tukey honestly significant difference post hoc comparisons. RESULTS: Ninety biopsy procedures were performed with no serious adverse events, with a mean of 3.7 targets sampled per gland. Cancer was detected in 51 (56.7%) men: 48.1% (25 of 52) with no prior PCa prostate cancer , 61.5% (eight of 13) under active surveillance, and 72.0% (18 of 25) in whom recurrence was suspected. Gleason pattern 4 or higher was diagnosed in 78.1% (25 of 32) in the no prior PCa prostate cancer and active surveillance groups. Gleason scores were not assigned in the suspected recurrence group. MR targets located in the anterior prostate had the highest cancer yield (40 of 64, 62.5%) compared with those for the other parts of the prostate (P < .001). CONCLUSION: In-bore 3-T transperineal MR imaging-guided biopsy, with a mean of 3.7 targets per gland, allowed detection of many clinically relevant cancers, many of which were located anteriorly.

Magnetic resonance imaging of ionic currents in solution: the effect of magnetohydrodynamic flow.

PURPOSE: Reliably detecting MRI signals in the brain that are more tightly coupled to neural activity than blood-oxygen-level-dependent fMRI signals could not only prove valuable for basic scientific research but could also enhance clinical applications such as epilepsy presurgical mapping. This endeavor will likely benefit from an improved understanding of the behavior of ionic currents, the mediators of neural activity, in the presence of the strong magnetic fields that are typical of modern-day MRI scanners. THEORY: Of the various mechanisms that have been proposed to explain the behavior of ionic volume currents in a magnetic field, only one-magnetohydrodynamic flow-predicts a slow evolution of signals, on the order of a minute for normal saline in a typical MRI scanner. METHODS: This prediction was tested by scanning a volume-current phantom containing normal saline with gradient-echo-planar imaging at 3 T. RESULTS: Greater signal changes were observed in the phase of the images than in the magnitude, with the changes evolving on the order of a minute. CONCLUSION: These results provide experimental support for the MHD flow hypothesis. Furthermore, MHD-driven cerebrospinal fluid flow could provide a novel fMRI contrast mechanism.

Automatic iceball segmentation with adapted shape priors for MRI-guided cryoablation.

PURPOSE: To develop and evaluate an automatic segmentation method that extracts the 3D configuration of the ablation zone, the iceball, from images acquired during the freezing phase of MRI-guided cryoablation. MATERIALS AND METHODS: Intraprocedural images at 63 timepoints from 13 kidney tumor cryoablation procedures were examined retrospectively. The images were obtained using a 3 Tesla wide-bore MRI scanner and axial HASTE sequence. Initialized with semiautomatically localized cryoprobes, the iceball was segmented automatically at each timepoint using the graph cut (GC) technique with adapted shape priors. RESULTS: The average Dice Similarity Coefficients (DSC), compared with manual segmentations, were 0.88, 0.92, 0.92, 0.93, and 0.93 at 3, 6, 9, 12, and 15 min timepoints, respectively, and the average DSC of the total 63 segmentations was 0.92 ± 0.03. The proposed method improved the accuracy significantly compared with the approach without shape prior adaptation (P = 0.026). The number of probes involved in the procedure had no apparent influence on the segmentation results using our technique. The average computation time was 20 s, which was compatible with an intraprocedural setting. CONCLUSION: Our automatic iceball segmentation method demonstrated high accuracy and robustness for practical use in monitoring the progress of MRI-guided cryoablation.

ReMIND: The Brain Resection Multimodal Imaging Database.

The standard of care for brain tumors is maximal safe surgical resection. Neuronavigation augments the surgeon's ability to achieve this but loses validity as surgery progresses due to brain shift. Moreover, gliomas are often indistinguishable from surrounding healthy brain tissue. Intraoperative magnetic resonance imaging (iMRI) and ultrasound (iUS) help visualize the tumor and brain shift. iUS is faster and easier to incorporate into surgical workflows but offers a lower contrast between tumorous and healthy tissues than iMRI. With the success of data-hungry Artificial Intelligence algorithms in medical image analysis, the benefits of sharing well-curated data cannot be overstated. To this end, we provide the largest publicly available MRI and iUS database of surgically treated brain tumors, including gliomas (n=92), metastases (n=11), and others (n=11). This collection contains 369 preoperative MRI series, 320 3D iUS series, 301 iMRI series, and 356 segmentations collected from 114 consecutive patients at a single institution. This database is expected to help brain shift and image analysis research and neurosurgical training in interpreting iUS and iMRI.

ReMIND: The Brain Resection Multimodal Imaging Database.

The standard of care for brain tumors is maximal safe surgical resection. Neuronavigation augments the surgeon's ability to achieve this but loses validity as surgery progresses due to brain shift. Moreover, gliomas are often indistinguishable from surrounding healthy brain tissue. Intraoperative magnetic resonance imaging (iMRI) and ultrasound (iUS) help visualize the tumor and brain shift. iUS is faster and easier to incorporate into surgical workflows but offers a lower contrast between tumorous and healthy tissues than iMRI. With the success of data-hungry Artificial Intelligence algorithms in medical image analysis, the benefits of sharing well-curated data cannot be overstated. To this end, we provide the largest publicly available MRI and iUS database of surgically treated brain tumors, including gliomas (n = 92), metastases (n = 11), and others (n = 11). This collection contains 369 preoperative MRI series, 320 3D iUS series, 301 iMRI series, and 356 segmentations collected from 114 consecutive patients at a single institution. This database is expected to help brain shift and image analysis research and neurosurgical training in interpreting iUS and iMRI.

Extended Broca's area in the functional connectome of language in adults: combined cortical and subcortical single-subject analysis using fMRI and DTI tractography.

Traditional models of the human language circuitry encompass three cortical areas, Broca's, Geschwind's and Wernicke's, and their connectivity through white matter fascicles. The neural connectivity deep to these cortical areas remains poorly understood, as does the macroscopic functional organization of the cortico-subcortical language circuitry. In an effort to expand current knowledge, we combined functional MRI (fMRI) and diffusion tensor imaging to explore subject-specific structural and functional macroscopic connectivity, focusing on Broca's area. Fascicles were studied using diffusion tensor imaging fiber tracking seeded from volumes placed manually within the white matter. White matter fascicles and fMRI-derived clusters (antonym-generation task) of positive and negative blood-oxygen-level-dependent (BOLD) signal were co-registered with 3-D renderings of the brain in 12 healthy subjects. Fascicles connecting BOLD-derived clusters were analyzed within specific cortical areas: Broca's, with the pars triangularis, the pars opercularis, and the pars orbitaris; Geschwind's and Wernicke's; the premotor cortex, the dorsal supplementary motor area, the middle temporal gyrus, the dorsal prefrontal cortex and the frontopolar region. We found a functional connectome divisible into three systems-anterior, superior and inferior-around the insula, more complex than previously thought, particularly with respect to a new extended Broca's area. The extended Broca's area involves two new fascicles: the operculo-premotor fascicle comprised of well-organized U-shaped fibers that connect the pars opercularis with the premotor region; and (2) the triangulo-orbitaris system comprised of intermingled U-shaped fibers that connect the pars triangularis with the pars orbitaris. The findings enhance our understanding of language function.

Learning Expected Appearances for Intraoperative Registration during Neurosurgery.

We present a novel method for intraoperative patient-to-image registration by learning Expected Appearances. Our method uses preoperative imaging to synthesize patient-specific expected views through a surgical microscope for a predicted range of transformations. Our method estimates the camera pose by minimizing the dissimilarity between the intraoperative 2D view through the optical microscope and the synthesized expected texture. In contrast to conventional methods, our approach transfers the processing tasks to the preoperative stage, reducing thereby the impact of low-resolution, distorted, and noisy intraoperative images, that often degrade the registration accuracy. We applied our method in the context of neuronavigation during brain surgery. We evaluated our approach on synthetic data and on retrospective data from 6 clinical cases. Our method outperformed state-of-the-art methods and achieved accuracies that met current clinical standards.

Unified Brain MR-Ultrasound Synthesis using Multi-Modal Hierarchical Representations.

We introduce MHVAE, a deep hierarchical variational autoencoder (VAE) that synthesizes missing images from various modalities. Extending multi-modal VAEs with a hierarchical latent structure, we introduce a probabilistic formulation for fusing multi-modal images in a common latent representation while having the flexibility to handle incomplete image sets as input. Moreover, adversarial learning is employed to generate sharper images. Extensive experiments are performed on the challenging problem of joint intra-operative ultrasound (iUS) and Magnetic Resonance (MR) synthesis. Our model outperformed multi-modal VAEs, conditional GANs, and the current state-of-the-art unified method (ResViT) for synthesizing missing images, demonstrating the advantage of using a hierarchical latent representation and a principled probabilistic fusion operation. Our code is publicly available.

Deep Learning for Detection and Localization of B-Lines in Lung Ultrasound.

Lung ultrasound (LUS) is an important imaging modality used by emergency physicians to assess pulmonary congestion at the patient bedside. B-line artifacts in LUS videos are key findings associated with pulmonary congestion. Not only can the interpretation of LUS be challenging for novice operators, but visual quantification of B-lines remains subject to observer variability. In this work, we investigate the strengths and weaknesses of multiple deep learning approaches for automated B-line detection and localization in LUS videos. We curate and publish, BEDLUS, a new ultrasound dataset comprising 1,419 videos from 113 patients with a total of 15,755 expert-annotated B-lines. Based on this dataset, we present a benchmark of established deep learning methods applied to the task of B-line detection. To pave the way for interpretable quantification of B-lines, we propose a novel "single-point" approach to B-line localization using only the point of origin. Our results show that (a) the area under the receiver operating characteristic curve ranges from 0.864 to 0.955 for the benchmarked detection methods, (b) within this range, the best performance is achieved by models that leverage multiple successive frames as input, and (c) the proposed single-point approach for B-line localization reaches an F 1-score of 0.65, performing on par with the inter-observer agreement. The dataset and developed methods can facilitate further biomedical research on automated interpretation of lung ultrasound with the potential to expand the clinical utility.

fMRI-DTI modeling via landmark distance atlases for prediction and detection of fiber tracts.

The overall goal of this research is the design of statistical atlas models that can be created from normal subjects, but may generalize to be applicable to abnormal brains. We present a new style of joint modeling of fMRI, DTI, and structural MRI. Motivated by the fact that a white matter tract and related cortical areas are likely to displace together in the presence of a mass lesion (brain tumor), in this work we propose a rotation and translation invariant model that represents the spatial relationship between fiber tracts and anatomic and functional landmarks. This landmark distance model provides a new basis for representation of fiber tracts and can be used for detection and prediction of fiber tracts based on landmarks. Our results indicate that the measured model is consistent across normal subjects, and thus suitable for atlas building. Our experiments demonstrate that the model is robust to displacement and missing data, and can be successfully applied to a small group of patients with mass lesions.

Image registration for targeted MRI-guided transperineal prostate biopsy.

PURPOSE: To develop and evaluate image registration methodology for automated re-identification of tumor-suspicious foci from preprocedural MR exams during MR-guided transperineal prostate core biopsy. MATERIALS AND METHODS: A hierarchical approach for automated registration between planning and intra-procedural T2-weighted prostate MRI was developed and evaluated on the images acquired during 10 consecutive MR-guided biopsies. Registration accuracy was quantified at image-based landmarks and by evaluating spatial overlap for the manually segmented prostate and sub-structures. Registration reliability was evaluated by simulating initial mis-registration and analyzing the convergence behavior. Registration precision was characterized at the planned biopsy targets. RESULTS: The total computation time was compatible with a clinical setting, being at most 2 min. Deformable registration led to a significant improvement in spatial overlap of the prostate and peripheral zone contours compared with both rigid and affine registration. Average in-slice landmark registration error was 1.3 ± 0.5 mm. Experiments simulating initial mis-registration resulted in an estimated average capture range of 6 mm and an average in-slice registration precision of ±0.3 mm. CONCLUSION: Our registration approach requires minimum user interaction and is compatible with the time constraints of our interventional clinical workflow. The initial evaluation shows acceptable accuracy, reliability and consistency of the method.

A Bayesian nonrigid registration method to enhance intraoperative target definition in image-guided prostate procedures through uncertainty characterization.

PURPOSE: This study introduces a probabilistic nonrigid registration method for use in image-guided prostate brachytherapy. Intraoperative imaging for prostate procedures, usually transrectal ultrasound (TRUS), is typically inferior to diagnostic-quality imaging of the pelvis such as endorectal magnetic resonance imaging (MRI). MR images contain superior detail of the prostate boundaries and provide substructure features not otherwise visible. Previous efforts to register diagnostic prostate images with the intraoperative coordinate system have been deterministic and did not offer a measure of the registration uncertainty. The authors developed a Bayesian registration method to estimate the posterior distribution on deformations and provide a case-specific measure of the associated registration uncertainty. METHODS: The authors adapted a biomechanical-based probabilistic nonrigid method to register diagnostic to intraoperative images by aligning a physician's segmentations of the prostate in the two images. The posterior distribution was characterized with a Markov Chain Monte Carlo method; the maximum a posteriori deformation and the associated uncertainty were estimated from the collection of deformation samples drawn from the posterior distribution. The authors validated the registration method using a dataset created from ten patients with MRI-guided prostate biopsies who had both diagnostic and intraprocedural 3 Tesla MRI scans. The accuracy and precision of the estimated posterior distribution on deformations were evaluated from two predictive distance distributions: between the deformed central zone-peripheral zone (CZ-PZ) interface and the physician-labeled interface, and based on physician-defined landmarks. Geometric margins on the registration of the prostate's peripheral zone were determined from the posterior predictive distance to the CZ-PZ interface separately for the base, mid-gland, and apical regions of the prostate. RESULTS: The authors observed variation in the shape and volume of the segmented prostate in diagnostic and intraprocedural images. The probabilistic method allowed us to convey registration results in terms of posterior distributions, with the dispersion providing a patient-specific estimate of the registration uncertainty. The median of the predictive distance distribution between the deformed prostate boundary and the segmented boundary was ≤3 mm (95th percentiles within ±4 mm) for all ten patients. The accuracy and precision of the internal deformation was evaluated by comparing the posterior predictive distance distribution for the CZ-PZ interface for each patient, with the median distance ranging from -0.6 to 2.4 mm. Posterior predictive distances between naturally occurring landmarks showed registration errors of ≤5 mm in any direction. The uncertainty was not a global measure, but instead was local and varied throughout the registration region. Registration uncertainties were largest in the apical region of the prostate. CONCLUSIONS: Using a Bayesian nonrigid registration method, the authors determined the posterior distribution on deformations between diagnostic and intraprocedural MR images and quantified the uncertainty in the registration results. The feasibility of this approach was tested and results were positive. The probabilistic framework allows us to evaluate both patient-specific and location-specific estimates of the uncertainty in the registration result. Although the framework was tested on MR-guided procedures, the preliminary results suggest that it may be applied to TRUS-guided procedures as well, where the addition of diagnostic MR information may have a larger impact on target definition and clinical guidance.

Deep Diffusion MRI Registration (DDMReg): A Deep Learning Method for Diffusion MRI Registration.

In this paper, we present a deep learning method, DDMReg, for accurate registration between diffusion MRI (dMRI) datasets. In dMRI registration, the goal is to spatially align brain anatomical structures while ensuring that local fiber orientations remain consistent with the underlying white matter fiber tract anatomy. DDMReg is a novel method that uses joint whole-brain and tract-specific information for dMRI registration. Based on the successful VoxelMorph framework for image registration, we propose a novel registration architecture that leverages not only whole brain information but also tract-specific fiber orientation information. DDMReg is an unsupervised method for deformable registration between pairs of dMRI datasets: it does not require nonlinearly pre-registered training data or the corresponding deformation fields as ground truth. We perform comparisons with four state-of-the-art registration methods on multiple independently acquired datasets from different populations (including teenagers, young and elderly adults) and different imaging protocols and scanners. We evaluate the registration performance by assessing the ability to align anatomically corresponding brain structures and ensure fiber spatial agreement between different subjects after registration. Experimental results show that DDMReg obtains significantly improved registration performance compared to the state-of-the-art methods. Importantly, we demonstrate successful generalization of DDMReg to dMRI data from different populations with varying ages and acquired using different acquisition protocols and different scanners.

SuperWarp: Supervised Learning and Warping on U-Net for Invariant Subvoxel-Precise Registration.

In recent years, learning-based image registration methods have gradually moved away from direct supervision with target warps to instead use self-supervision, with excellent results in several registration benchmarks. These approaches utilize a loss function that penalizes the intensity differences between the fixed and moving images, along with a suitable regularizer on the deformation. However, since images typically have large untextured regions, merely maximizing similarity between the two images is not sufficient to recover the true deformation. This problem is exacerbated by texture in other regions, which introduces severe non-convexity into the landscape of the training objective and ultimately leads to overfitting. In this paper, we argue that the relative failure of supervised registration approaches can in part be blamed on the use of regular U-Nets, which are jointly tasked with feature extraction, feature matching and deformation estimation. Here, we introduce a simple but crucial modification to the U-Net that disentangles feature extraction and matching from deformation prediction, allowing the U-Net to warp the features, across levels, as the deformation field is evolved. With this modification, direct supervision using target warps begins to outperform self-supervision approaches that require segmentations, presenting new directions for registration when images do not have segmentations. We hope that our findings in this preliminary workshop paper will re-ignite research interest in supervised image registration techniques. Our code is publicly available from http://github.com/balbasty/superwarp.

Double-Uncertainty Guided Spatial and Temporal Consistency Regularization Weighting for Learning-based Abdominal Registration.

In order to tackle the difficulty associated with the ill-posed nature of the image registration problem, regularization is often used to constrain the solution space. For most learning-based registration approaches, the regularization usually has a fixed weight and only constrains the spatial transformation. Such convention has two limitations: (i) Besides the laborious grid search for the optimal fixed weight, the regularization strength of a specific image pair should be associated with the content of the images, thus the "one value fits all" training scheme is not ideal; (ii) Only spatially regularizing the transformation may neglect some informative clues related to the ill-posedness. In this study, we propose a mean-teacher based registration framework, which incorporates an additional temporal consistency regularization term by encouraging the teacher model's prediction to be consistent with that of the student model. More importantly, instead of searching for a fixed weight, the teacher enables automatically adjusting the weights of the spatial regularization and the temporal consistency regularization by taking advantage of the transformation uncertainty and appearance uncertainty. Extensive experiments on the challenging abdominal CT-MRI registration show that our training strategy can promisingly advance the original learning-based method in terms of efficient hyperparameter tuning and a better tradeoff between accuracy and smoothness.

PROSTATE CANCER DIAGNOSIS WITH SPARSE BIOPSY DATA AND IN PRESENCE OF LOCATION UNCERTAINTY.

Prostate cancer is the second most prevalent cancer in men worldwide. Deep neural networks have been successfully applied for prostate cancer diagnosis in magnetic resonance images (MRI). Pathology results from biopsy procedures are often used as ground truth to train such systems. There are several sources of noise in creating ground truth from biopsy data including sampling and registration errors. We propose: 1) A fully convolutional neural network (FCN) to produce cancer probability maps across the whole prostate gland in MRI; 2) A Gaussian weighted loss function to train the FCN with sparse biopsy locations; 3) A probabilistic framework to model biopsy location uncertainty and adjust cancer probability given the deep model predictions. We assess the proposed method on 325 biopsy locations from 203 patients. We observe that the proposed loss improves the area under the receiver operating characteristic curve and the biopsy location adjustment improves the sensitivity of the models.

Image registration: Maximum likelihood, minimum entropy and deep learning.

In this work, we propose a theoretical framework based on maximum profile likelihood for pairwise and groupwise registration. By an asymptotic analysis, we demonstrate that maximum profile likelihood registration minimizes an upper bound on the joint entropy of the distribution that generates the joint image data. Further, we derive the congealing method for groupwise registration by optimizing the profile likelihood in closed form, and using coordinate ascent, or iterative model refinement. We also describe a method for feature based registration in the same framework and demonstrate it on groupwise tractographic registration. In the second part of the article, we propose an approach to deep metric registration that implements maximum likelihood registration using deep discriminative classifiers. We show further that this approach can be used for maximum profile likelihood registration to discharge the need for well-registered training data, using iterative model refinement. We demonstrate that the method succeeds on a challenging registration problem where the standard mutual information approach does not perform well.