Affine image registration of arterial spin labeling MRI using deep learning networks.

Convolutional neural networks (CNN) have demonstrated good accuracy and speed in spatially registering high signal-to-noise ratio (SNR) structural magnetic resonance imaging (sMRI) images. However, some functional magnetic resonance imaging (fMRI) images, e.g., those acquired from arterial spin labeling (ASL) perfusion fMRI, are of intrinsically low SNR and therefore the quality of registering ASL images using CNN is not clear. In this work, we aimed to explore the feasibility of a CNN-based affine registration network (ARN) for registration of low-SNR three-dimensional ASL perfusion image time series and compare its performance with that from the state-of-the-art statistical parametric mapping (SPM) algorithm. The six affine parameters were learned from the ARN using both simulated motion and real acquisitions from ASL perfusion fMRI data and the registered images were generated by applying the transformation derived from the affine parameters. The speed and registration accuracy were compared between ARN and SPM. Several independent datasets, including meditation study (10 subjects × 2), bipolar disorder study (26 controls, 19 bipolar disorder subjects), and aging study (27 young subjects, 33 older subjects), were used to validate the generality of the trained ARN model. The ARN method achieves superior image affine registration accuracy (total translation/total rotation errors of ARN vs. SPM: 1.17 mm/1.23° vs. 6.09 mm/12.90° for simulated images and reduced MSE/L1/DSSIM/Total errors of 18.07% / 19.02% / 0.04% / 29.59% for real ASL test images) and 4.4 times (ARN vs. SPM: 0.50 s vs. 2.21 s) faster speed compared to SPM. The trained ARN can be generalized to align ASL perfusion image time series acquired with different scanners, and from different image resolutions, and from healthy or diseased populations. The results demonstrated that our ARN markedly outperforms the iteration-based SPM both for simulated motion and real acquisitions in terms of registration accuracy, speed, and generalization.

Cascading Affine and B-spline Registration Method for Large Deformation Registration of Lung X-rays.

Accurate registration of lung X-rays is an important task in medical image analysis. However, the conventional methods usually cost a lot in running time, and the existing deep learning methods are hard to deal with the large deformation caused by respiratory and cardiac motion. In this paper, we attempt to use deep learning methods to deal with large deformation and enable it to achieve the accuracy of conventional methods. We proposed the cascading affine and B-spline network (CABN), which consists of convolutional cross-stitch affine block (CCAB) and B-splines U-net-like block (BUB) for large lung motion. CCAB makes use of the convolutional cross-stitch model to learn global features among images. And BUB adopts the idea of cubic B-splines which is suitable for large deformation. We separately demonstrated CCAB, BUB, and CABN on two chest X-ray datasets. The experimental results indicate that our methods are highly competitive both in accuracy and runtime when compared to both other deep learning methods and iterative conventional approaches. Moreover, CCAB also can be used for the preprocessing of non-rigid registration methods, replacing affine in conventional methods.

Multi-object Model-Based Multi-atlas Segmentation Constrained Grid Cut for Automatic Segmentation of Lumbar Vertebrae from CT Images.

In this chapter, we present a multi-object model-based multi-atlas segmentation constrained grid cut method for automatic segmentation of lumbar vertebrae from a given lumbar spinal CT image. More specifically, our automatic lumbar vertebrae segmentation method consists of two steps: affine atlas-target registration-based label fusion and bone-sheetness assisted multi-label grid cut which has the inherent advantage of automatic separation of the five lumbar vertebrae from each other. We evaluate our method on 21 clinical lumbar spinal CT images with the associated manual segmentation and conduct a leave-one-out study. Our method achieved an average Dice coefficient of 93.9 ± 1.0% and an average symmetric surface distance of 0.41 ± 0.08 mm.

Characterization of the Physiological Displacement of the Aortic Arch Using Non-Rigid Registration and MR Imaging.

OBJECTIVES: The aim of this work was to study physiological aortic arch three-dimensional displacement using non-rigid registration methods and magnetic resonance imaging (MRI). MATERIALS AND METHODS: Ten healthy volunteers underwent thoracic MRI. Prospective cardiac gating was performed with a 3D turbo field echo sequence to obtain end-systolic and end-diastolic MR images. The rigid and elastic behavior between these two cardiac phases was detected and compared using either an affine or an elastic registration method. To assess reproducibility, a second MRI acquisition was performed 14 days later. RESULTS: Affine registration between the end-systolic and end-diastolic MR images showed significant global translations of the aortic arch and the supra-aortic vessels in the x, y, and z directions (2.02 ± 1.6, -0.71 ± 1.1, and -1.21 ± 1.4 mm, respectively). Corresponding elastic registration indicated significant local displacement with a vector magnitude of 5.1 ± 0.89 mm for the brachiocephalic artery (BCA), of 4.26 ± 0.83 mm for the left common carotid artery (LCCA), and of 4.8 ± 0.86 mm for the left subclavian artery (LSCA). There was a difference in displacement between the supra-aortic trunks of the order of 2 mm. Vector displacement was not statistically different between the repeated acquisitions. CONCLUSIONS: The present results showed important deformations in the ostia of supra-aortic vessels during the cardiac cycle. It seems that aortic arch motions should be taken into account when designing and manufacturing fenestrated endografts. The elastic registration method provides more precise results, but is more complex and time-consuming than other methods.

The facial expression of schizophrenic patients applied with infrared thermal facial image sequence.

BACKGROUND: Schizophrenia is a neurological disease characterized by alterations to patients' cognitive functions and emotional expressions. Relevant studies often use magnetic resonance imaging (MRI) of the brain to explore structural differences and responsiveness within brain regions. However, as this technique is expensive and commonly induces claustrophobia, it is frequently refused by patients. Thus, this study used non-contact infrared thermal facial images (ITFIs) to analyze facial temperature changes evoked by different emotions in moderately and markedly ill schizophrenia patients. METHODS: Schizophrenia is an emotion-related disorder, and images eliciting different types of emotions were selected from the international affective picture system (IAPS) and presented to subjects during ITFI collection. ITFIs were aligned using affine registration, and the changes induced by small irregular head movements were corrected. The average temperatures from the forehead, nose, mouth, left cheek, and right cheek were calculated, and continuous temperature changes were used as features. After performing dimensionality reduction and noise removal using the component analysis method, multivariate analysis of variance and the Support Vector Machine (SVM) classification algorithm were used to identify moderately and markedly ill schizophrenia patients. RESULTS: Analysis of five facial areas indicated significant temperature changes in the forehead and nose upon exposure to various emotional stimuli and in the right cheek upon evocation of high valence low arousal (HVLA) stimuli. The most significant P-value (lower than 0.001) was obtained in the forehead area upon evocation of disgust. Finally, when the features of forehead temperature changes in response to low valence high arousal (LVHA) were reduced to 9 using dimensionality reduction and noise removal, the identification rate was as high as 94.3%. CONCLUSIONS: Our results show that features obtained in the forehead, nose, and right cheek significantly differed between moderately and markedly ill schizophrenia patients. We then chose the features that most effectively distinguish between moderately and markedly ill schizophrenia patients using the SVM. These results demonstrate that the ITFI analysis protocol proposed in this study can effectively provide reference information regarding the phase of the disease in patients with schizophrenia.

Anatomy-aware and acquisition-agnostic joint registration with SynthMorph.

Affine image registration is a cornerstone of medical-image analysis. While classical algorithms can achieve excellent accuracy, they solve a time-consuming optimization for every image pair. Deep-learning (DL) methods learn a function that maps an image pair to an output transform. Evaluating the function is fast, but capturing large transforms can be challenging, and networks tend to struggle if a test-image characteristic shifts from the training domain, such as the resolution. Most affine methods are agnostic to the anatomy the user wishes to align, meaning the registration will be inaccurate if algorithms consider all structures in the image. We address these shortcomings with SynthMorph, a fast, symmetric, diffeomorphic, and easy-to-use DL tool for joint affine-deformable registration of any brain image without preprocessing. First, we leverage a strategy that trains networks with widely varying images synthesized from label maps, yielding robust performance across acquisition specifics unseen at training. Second, we optimize the spatial overlap of select anatomical labels. This enables networks to distinguish anatomy of interest from irrelevant structures, removing the need for preprocessing that excludes content which would impinge on anatomy-specific registration. Third, we combine the affine model with a deformable hypernetwork that lets users choose the optimal deformation-field regularity for their specific data, at registration time, in a fraction of the time required by classical methods. This framework is applicable to learning anatomy-aware, acquisition-agnostic registration of any anatomy with any architecture, as long as label maps are available for training. We analyze how competing architectures learn affine transforms and compare state-of-the-art registration tools across an extremely diverse set of neuroimaging data, aiming to truly capture the behavior of methods in the real world. SynthMorph demonstrates high accuracy and is available at https://w3id.org/synthmorph, as a single complete end-to-end solution for registration of brain magnetic resonance imaging (MRI) data.

A 3D Sparse Autoencoder for Fully Automated Quality Control of Affine Registrations in Big Data Brain MRI Studies.

This paper presents a fully automated pipeline using a sparse convolutional autoencoder for quality control (QC) of affine registrations in large-scale T1-weighted (T1w) and T2-weighted (T2w) magnetic resonance imaging (MRI) studies. Here, a customized 3D convolutional encoder-decoder (autoencoder) framework is proposed and the network is trained in a fully unsupervised manner. For cross-validating the proposed model, we used 1000 correctly aligned MRI images of the human connectome project young adult (HCP-YA) dataset. We proposed that the quality of the registration is proportional to the reconstruction error of the autoencoder. Further, to make this method applicable to unseen datasets, we have proposed dataset-specific optimal threshold calculation (using the reconstruction error) from ROC analysis that requires a subset of the correctly aligned and artificially generated misalignments specific to that dataset. The calculated optimum threshold is used for testing the quality of remaining affine registrations from the corresponding datasets. The proposed framework was tested on four unseen datasets from autism brain imaging data exchange (ABIDE I, 215 subjects), information eXtraction from images (IXI, 577 subjects), Open Access Series of Imaging Studies (OASIS4, 646 subjects), and "Food and Brain" study (77 subjects). The framework has achieved excellent performance for T1w and T2w affine registrations with an accuracy of 100% for HCP-YA. Further, we evaluated the generality of the model on four unseen datasets and obtained accuracies of 81.81% for ABIDE I (only T1w), 93.45% (T1w) and 81.75% (T2w) for OASIS4, and 92.59% for "Food and Brain" study (only T1w) and in the range 88-97% for IXI (for both T1w and T2w and stratified concerning scanner vendor and magnetic field strengths). Moreover, the real failures from "Food and Brain" and OASIS4 datasets were detected with sensitivities of 100% and 80% for T1w and T2w, respectively. In addition, AUCs of > 0.88 in all scenarios were obtained during threshold calculation on the four test sets.

Affine medical image registration with fusion feature mapping in local and global.

Objective. Medical image affine registration is a crucial basis before using deformable registration. On the one hand, the traditional affine registration methods based on step-by-step optimization are very time-consuming, so these methods are not compatible with most real-time medical applications. On the other hand, convolutional neural networks are limited in modeling long-range spatial relationships of the features due to inductive biases, such as weight sharing and locality. This is not conducive to affine registration tasks. Therefore, the evolution of real-time and high-accuracy affine medical image registration algorithms is necessary for registration applications.Approach. In this paper, we propose a deep learning-based coarse-to-fine global and local feature fusion architecture for fast affine registration, and we use an unsupervised approach for end-to-end training. We use multiscale convolutional kernels as our elemental convolutional blocks to enhance feature extraction. Then, to learn the long-range spatial relationships of the features, we propose a new affine registration framework with weighted global positional attention that fuses global feature mapping and local feature mapping. Moreover, the fusion regressor is designed to generate the affine parameters.Main results. The additive fusion method can be adaptive to global mapping and local mapping, which improves affine registration accuracy without the center of mass initialization. In addition, the max pooling layer and the multiscale convolutional kernel coding module increase the ability of the model in affine registration.Significance. We validate the effectiveness of our method on the OASIS dataset with 414 3D MRI brain maps. Comprehensive results demonstrate that our method achieves state-of-the-art affine registration accuracy and very efficient runtimes.

A Feature-based Affine Registration Method for Capturing Background Lung Tissue Deformation for Ground Glass Nodule Tracking.

Lung nodule tracking assessment relies on cross-sectional measurements of the largest lesion profile depicted in initial and follow-up computed tomography (CT) images. However, apparent changes in nodule size assessed via simple image-based measurements may also be compromised by the effect of the background lung tissue deformation on the GGN between the initial and follow-up images, leading to erroneous conclusions about nodule changes due to disease. To compensate for the lung deformation and enable consistent nodule tracking, here we propose a feature-based affine registration method and study its performance vis-a-vis several other registration methods. We implement and test each registration method using both a lung- and a lesion-centered region of interest on ten patient CT datasets featuring twelve nodules, including both benign and malignant GGO lesions containing pure GGNs, part-solid, or solid nodules. We evaluate each registration method according to the target registration error (TRE) computed across 30 - 50 homologous fiducial landmarks surrounding the lesions and selected by expert radiologists in both the initial and follow-up patient CT images. Our results show that the proposed feature-based affine lesion-centered registration yielded a 1.1 ± 1.2 mm TRE, while a Symmetric Normalization deformable registration yielded a 1.2 ± 1.2 mm TRE, and a least-square fit registration of the 30-50 validation fiducial landmark set yielded a 1.5 ± 1.2 mm TRE. Although the deformable registration yielded a slightly higher registration accuracy than the feature-based affine registration, it is significantly more computationally efficient, eliminates the need for ambiguous segmentation of GGNs featuring ill-defined borders, and reduces the susceptibility of artificial deformations introduced by the deformable registration, which may lead to increased similarity between the registered initial and follow-up images, over-compensating for the background lung tissue deformation, and, in turn, compromising the true disease-induced nodule change assessment. We also assessed the registration qualitatively, by visual inspection of the subtraction images, and conducted a pilot pre-clinical study that showed the proposed feature-based lesion-centered affine registration effectively compensates for the background lung tissue deformation between the initial and follow-up images and also serves as a reliable baseline registration method prior to assessing lung nodule changes due to disease.

Construction of a Multi-Phase Contrast Computed Tomography Kidney Atlas.

The Human BioMolecular Atlas Program (HuBMAP) seeks to create a molecular atlas at the cellular level of the human body to spur interdisciplinary innovations across spatial and temporal scales. While the preponderance of effort is allocated towards cellular and molecular scale mapping, differentiating and contextualizing findings within tissues, organs and systems are essential for the HuBMAP efforts. The kidney is an initial organ target of HuBMAP, and constructing a framework (or atlas) for integrating information across scales is needed for visualizing and integrating information. However, there is no abdominal atlas currently available in the public domain. Substantial variation in healthy kidneys exists with sex, body size, and imaging protocols. With the integration of clinical archives for secondary research use, we are able to build atlases based on a diverse population and clinically relevant protocols. In this study, we created a computed tomography (CT) phase-specific atlas for the abdomen, which is optimized for the kidney organ. A two-stage registration pipeline was used by registering extracted abdominal volume of interest from body part regression, to a high-resolution CT. Affine and non-rigid registration were performed to all scans hierarchically. To generate and evaluate the atlas, multiphase CT scans of 500 control subjects (age: 15 - 50, 250 males, 250 females) are registered to the atlas target through the complete pipeline. The abdominal body and kidney registration are shown to be stable with the variance map computed from the result average template. Both left and right kidneys are substantially localized in the high-resolution target space, which successfully demonstrated the sharp details of its anatomical characteristics across each phase. We illustrated the applicability of the atlas template for integrating across normal kidney variation from 64 cm3 to 302 cm3.

Edge-Aware Pyramidal Deformable Network for Unsupervised Registration of Brain MR Images.

Deformable image registration is of essential important for clinical diagnosis, treatment planning, and surgical navigation. However, most existing registration solutions require separate rigid alignment before deformable registration, and may not well handle the large deformation circumstances. We propose a novel edge-aware pyramidal deformable network (referred as EPReg) for unsupervised volumetric registration. Specifically, we propose to fully exploit the useful complementary information from the multi-level feature pyramids to predict multi-scale displacement fields. Such coarse-to-fine estimation facilitates the progressive refinement of the predicted registration field, which enables our network to handle large deformations between volumetric data. In addition, we integrate edge information with the original images as dual-inputs, which enhances the texture structures of image content, to impel the proposed network pay extra attention to the edge-aware information for structure alignment. The efficacy of our EPReg was extensively evaluated on three public brain MRI datasets including Mindboggle101, LPBA40, and IXI30. Experiments demonstrate our EPReg consistently outperformed several cutting-edge methods with respect to the metrics of Dice index (DSC), Hausdorff distance (HD), and average symmetric surface distance (ASSD). The proposed EPReg is a general solution for the problem of deformable volumetric registration.

A PRM approach for early prediction of breast cancer response to chemotherapy based on registered MR images.

PURPOSE: This study aims to provide and optimize a performing algorithm for predicting the breast cancer response rate to the first round of chemotherapy using Magnetic Resonance Imaging (MRI). This provides an early recognition of breast tumor reaction to chemotherapy by using the Parametric Response Map (PRM) method. METHODS: PRM may predict the breast cancer response to chemotherapy by analyzing voxel-by-voxel temporal intra-tumor changes during one round of chemotherapy. Indeed, the tumor recognizes intra-tumor changes concerning its vascularity, which is an important criterion in the present study. This method is mainly based on spatial image affine registration between the breast tumor MRI volumes, acquired before and after the first cycle of chemotherapy, and region growing segmentation of the tumor volume. To evaluate our method, we used a retrospective study of 40 patients provided by a collaborating institute. RESULTS: PRM allows a color map to be created with the percentages of positive, negative and stable breast tumor response during the first round of chemotherapy, identifying each region with its response rate. We assessed the accuracy of the proposed method using technical and medical validation methods. The technical validation was based on landmarks-based registration and fully manual segmentation. The medical evaluation was based on the accuracy calculation of the standard reference of anatomic pathology. The p-values and the Area Under the Curve (AUC) of the Receiver Operating Characteristics were calculated to evaluate the proposed PRM method. CONCLUSION: We performed and evaluated the proposed PRM method to study and analyze the behavior of a tumor during the first round of chemotherapy, based on the intra-tumor changes of MR breast tumor images. The AUC obtained for the PRM method is considered as relevant in the early prediction of breast tumor response.

Evaluation of Medical Image Registration Techniques Based on Nature and Domain of the Transformation.

A lot of research has been done during the past 20 years in the area of medical image registration for obtaining detailed, important, and complementary information from two or more images and aligning them into a single, more informative image. Nature of the transformation and domain of the transformation are two important medical image registration techniques that deal with characters of objects (motions) in images. This article presents a detailed survey of the registration techniques that belong to both categories with detailed elaboration on their features, issues, and challenges. An investigation estimating similarity and dissimilarity measures and performance evaluation is the main objective of this work. This article also provides reference knowledge in a compact form for researchers and clinicians looking for the proper registration technique for a particular application.

A procedure to estimate the origins and the insertions of the knee ligaments from computed tomography images.

The estimation of the origin and insertion of the four knee ligaments is crucial for individualised dynamic modelling of the knee. Commonly this information is obtained ex vivo or from high resolution MRI, which is not always available. Aim of this work is to devise a method to estimate the origins and insertions from computed tomography (CT) images. A reference registration atlas was created using a set of 16 bone landmarks visible in CT and eight origins and insertions estimated from MRI and in vitro data available in the literature for three knees. This atlas can be registered to the set of bone landmarks palpated on any given CT using an affine transformation. The resulting orientation and translation matrices and scaling factors can be used to find also the ligament origin and insertions. This procedure was validated on seven pathological knees for which both CT and MRI of the knee region were available, using a proprietary software tool (NMSBuilder, SCS srl, Italy). To assess the procedure reproducibility and repeatability, four different operators performed the landmarks palpation on all seven patients. The average difference between the values predicted by registration on the CT scan and those estimated on the MRI was 2.1±1.2 mm for the femur and 2.7±1.0 mm for the tibia, respectively. The procedure is highly repeatable, with no significant differences observed within or between the operators (p>0.1) and allows to estimate origins and insertions of the knee ligaments from a CT scan with the same level of accuracy obtainable with MRI.

Bidirectional elastic image registration using B-spline affine transformation.

A registration scheme termed as B-spline affine transformation (BSAT) is presented in this study to elastically align two images. We define an affine transformation instead of the traditional translation at each control point. Mathematically, BSAT is a generalized form of the affine transformation and the traditional B-spline transformation (BST). In order to improve the performance of the iterative closest point (ICP) method in registering two homologous shapes but with large deformation, a bidirectional instead of the traditional unidirectional objective/cost function is proposed. In implementation, the objective function is formulated as a sparse linear equation problem, and a sub-division strategy is used to achieve a reasonable efficiency in registration. The performance of the developed scheme was assessed using both two-dimensional (2D) synthesized dataset and three-dimensional (3D) volumetric computed tomography (CT) data. Our experiments showed that the proposed B-spline affine model could obtain reasonable registration accuracy.

Hyperspectral Imaging for Cancer Surgical Margin Delineation: Registration of Hyperspectral and Histological Images.

The determination of tumor margins during surgical resection remains a challenging task. A complete removal of malignant tissue and conservation of healthy tissue is important for the preservation of organ function, patient satisfaction, and quality of life. Visual inspection and palpation is not sufficient for discriminating between malignant and normal tissue types. Hyperspectral imaging (HSI) technology has the potential to noninvasively delineate surgical tumor margin and can be used as an intra-operative visual aid tool. Since histological images provide the ground truth of cancer margins, it is necessary to warp the cancer regions in ex vivo histological images back to in vivo hyperspectral images in order to validate the tumor margins detected by HSI and to optimize the imaging parameters. In this paper, principal component analysis (PCA) is utilized to extract the principle component bands of the HSI images, which is then used to register HSI images with the corresponding histological image. Affine registration is chosen to model the global transformation. A B-spline free form deformation (FFD) method is used to model the local non-rigid deformation. Registration experiment was performed on animal hyperspectral and histological images. Experimental results from animals demonstrated the feasibility of the hyperspectral imaging method for cancer margin detection.

Large Deformation Diffeomorphic Metric Mapping Registration of Reconstructed 3D Histological Section Images and in vivo MR Images.

Our current understanding of neuroanatomical abnormalities in neuropsychiatric diseases is based largely on magnetic resonance imaging (MRI) and post mortem histological analyses of the brain. Further advances in elucidating altered brain structure in these human conditions might emerge from combining MRI and histological methods. We propose a multistage method for registering 3D volumes reconstructed from histological sections to corresponding in vivo MRI volumes from the same subjects: (1) manual segmentation of white matter (WM), gray matter (GM) and cerebrospinal fluid (CSF) compartments in histological sections, (2) alignment of consecutive histological sections using 2D rigid transformation to construct a 3D histological image volume from the aligned sections, (3) registration of reconstructed 3D histological volumes to the corresponding 3D MRI volumes using 3D affine transformation, (4) intensity normalization of images via histogram matching, and (5) registration of the volumes via intensity based large deformation diffeomorphic metric (LDDMM) image matching algorithm. Here we demonstrate the utility of our method in the transfer of cytoarchitectonic information from histological sections to identify regions of interest in MRI scans of nine adult macaque brains for morphometric analyses. LDDMM improved the accuracy of the registration via decreased distances between GM/CSF surfaces after LDDMM (0.39 +/- 0.13 mm) compared to distances after affine registration (0.76 +/- 0.41 mm). Similarly, WM/GM distances decreased to 0.28 +/- 0.16 mm after LDDMM compared to 0.54 +/- 0.39 mm after affine registration. The multistage registration method may find broad application for mapping histologically based information, for example, receptor distributions, gene expression, onto MRI volumes.