Unsupervised Domain Adaptation for Medical Image Segmentation Using Transformer With Meta Attention.

Image segmentation is essential to medical image analysis as it provides the labeled regions of interest for the subsequent diagnosis and treatment. However, fully-supervised segmentation methods require high-quality annotations produced by experts, which is laborious and expensive. In addition, when performing segmentation on another unlabeled image modality, the segmentation performance will be adversely affected due to the domain shift. Unsupervised domain adaptation (UDA) is an effective way to tackle these problems, but the performance of the existing methods is still desired to improve. Also, despite the effectiveness of recent Transformer-based methods in medical image segmentation, the adaptability of Transformers is rarely investigated. In this paper, we present a novel UDA framework using a Transformer for building a cross-modality segmentation method with the advantages of learning long-range dependencies and transferring attentive information. To fully utilize the attention learned by the Transformer in UDA, we propose Meta Attention (MA) and use it to perform a fully attention-based alignment scheme, which can learn the hierarchical consistencies of attention and transfer more discriminative information between two modalities. We have conducted extensive experiments on cross-modality segmentation using three datasets, including a whole heart segmentation dataset (MMWHS), an abdominal organ segmentation dataset, and a brain tumor segmentation dataset. The promising results show that our method can significantly improve performance compared with the state-of-the-art UDA methods.

Retinal Vessel Segmentation by A Transformer-U-Net Hybrid Model with Dual-Path Decoder.

This paper introduces an effective and efficient framework for retinal vessel segmentation. First, we design a Transformer-CNN hybrid model in which a Transformer module is inserted inside the U-Net to capture long-range interactions. Second, we design a dual-path decoder in the U-Net framework, which contains two decoding paths for multi-task outputs. Specifically, we train the extra decoder to predict vessel skeletons as an auxiliary task which helps the model learn balanced features. The proposed framework, named as TSNet, not only achieves good performances in a fully supervised learning manner but also enables a rough skeleton annotation process. The annotators only need to roughly delineate vessel skeletons instead of giving precise pixel-wise vessel annotations. To learn with rough skeleton annotations plus a few precise vessel annotations, we propose a skeleton semi-supervised learning scheme. We adopt a mean teacher model to produce pseudo vessel annotations and conduct annotation correction for roughly labeled skeletons annotations. This learning scheme can achieve promising performance with fewer annotation efforts. We have evaluated TSNet through extensive experiments on five benchmarking datasets. Experimental results show that TSNet yields state-of-the-art performances on retinal vessel segmentation and provides an efficient training scheme in practice.

A Dual Coordinate System Vertebra Landmark Detection Network with Sparse-to-Dense Vertebral Line Interpolation.

Precise surveillance and assessment of spinal disorders are important for improving health care and patient survival rates. The assessment of spinal disorders, such as scoliosis assessment, depends heavily on precise vertebra landmark localization. However, existing methods usually search for only a handful of keypoints in a high-resolution image. In this paper, we propose the S2D-VLI VLDet network, a unified end-to-end vertebra landmark detection network for the assessment of scoliosis. The proposed network considers the spatially relevant information both from inside and between vertebrae. The new vertebral line interpolation method converts the training labels from sparse to dense, which can improve the network learning process and method performance. In addition, through the combined use of the Cartesian and polar coordinate systems in our method, the symmetric mean absolute percentage error (SMAPE) in scoliosis assessment can be reduced substantially. Specifically, as shown in the experiments, the SMAPE value decreases from 9.82 to 8.28. The experimental results indicate that our proposed approach is beneficial for estimating the Cobb angle and identifying landmarks in X-ray scans with low contrast.

Learn2Reg: Comprehensive Multi-Task Medical Image Registration Challenge, Dataset and Evaluation in the Era of Deep Learning.

Image registration is a fundamental medical image analysis task, and a wide variety of approaches have been proposed. However, only a few studies have comprehensively compared medical image registration approaches on a wide range of clinically relevant tasks. This limits the development of registration methods, the adoption of research advances into practice, and a fair benchmark across competing approaches. The Learn2Reg challenge addresses these limitations by providing a multi-task medical image registration data set for comprehensive characterisation of deformable registration algorithms. A continuous evaluation will be possible at https://learn2reg.grand-challenge.org. Learn2Reg covers a wide range of anatomies (brain, abdomen, and thorax), modalities (ultrasound, CT, MR), availability of annotations, as well as intra- and inter-patient registration evaluation. We established an easily accessible framework for training and validation of 3D registration methods, which enabled the compilation of results of over 65 individual method submissions from more than 20 unique teams. We used a complementary set of metrics, including robustness, accuracy, plausibility, and runtime, enabling unique insight into the current state-of-the-art of medical image registration. This paper describes datasets, tasks, evaluation methods and results of the challenge, as well as results of further analysis of transferability to new datasets, the importance of label supervision, and resulting bias. While no single approach worked best across all tasks, many methodological aspects could be identified that push the performance of medical image registration to new state-of-the-art performance. Furthermore, we demystified the common belief that conventional registration methods have to be much slower than deep-learning-based methods.

Disease prediction with edge-variational graph convolutional networks.

The need for computational models that can incorporate imaging data with non-imaging data while investigating inter-subject associations arises in the task of population-based disease analysis. Although off-the-shelf deep convolutional neural networks have empowered representation learning from imaging data, incorporating data of different modalities complementarily in a unified model to improve the disease diagnostic quality is still challenging. In this work, we propose a generalizable graph-convolutional framework for population-based disease prediction on multi-modal medical data. Unlike previous methods constructing a static affinity population graph in a hand-crafting manner, the proposed framework can automatically learn to build a population graph with variational edges, which we show can be optimized jointly with spectral graph convolutional networks. In addition, to estimate the predictive uncertainty related to the constructed graph, we propose Monte-Carlo edge dropout uncertainty estimation. Experimental results on four multi-modal datasets demonstrate that the proposed method can substantially improve the predictive accuracy for Autism Spectrum Disorder, Alzheimer's disease, and ocular diseases. A sufficient ablation study with in-depth discussion is conducted to evaluate the effectiveness of each component and the choice of algorithmic details of the proposed method. The results indicate the potential and extendability of the proposed framework in leveraging multi-modal data for population-based disease prediction.

Relax and focus on brain tumor segmentation.

In this paper, we present a Deep Convolutional Neural Networks (CNNs) for fully automatic brain tumor segmentation for both high- and low-grade gliomas in MRI images. Unlike normal tissues or organs that usually have a fixed location or shape, brain tumors with different grades have shown great variation in terms of the location, size, structure, and morphological appearance. Moreover, the severe data imbalance exists not only between the brain tumor and non-tumor tissues, but also among the different sub-regions inside brain tumor (e.g., enhancing tumor, necrotic, edema, and non-enhancing tumor). Therefore, we introduce a hybrid model to address the challenges in the multi-modality multi-class brain tumor segmentation task. First, we propose the dynamic focal Dice loss function that is able to focus more on the smaller tumor sub-regions with more complex structures during training, and the learning capacity of the model is dynamically distributed to each class independently based on its training performance in different training stages. Besides, to better recognize the overall structure of the brain tumor and the morphological relationship among different tumor sub-regions, we relax the boundary constraints for the inner tumor regions in coarse-to-fine fashion. Additionally, a symmetric attention branch is proposed to highlight the possible location of the brain tumor from the asymmetric features caused by growth and expansion of the abnormal tissues in the brain. Generally, to balance the learning capacity of the model between spatial details and high-level morphological features, the proposed model relaxes the constraints of the inner boundary and complex details and enforces more attention on the tumor shape, location, and the harder classes of the tumor sub-regions. The proposed model is validated on the publicly available brain tumor dataset from real patients, BRATS 2019. The experimental results reveal that our model improves the overall segmentation performance in comparison with the state-of-the-art methods, with major progress on the recognition of the tumor shape, the structural relationship of tumor sub-regions, and the segmentation of more challenging tumor sub-regions, e.g., the tumor core, and enhancing tumor.

MedQ: Lossless ultra-low-bit neural network quantization for medical image segmentation.

Implementing deep convolutional neural networks (CNNs) with boolean arithmetic is ideal for eliminating the notoriously high computational expense of deep learning models. However, although lossless model compression via weight-only quantization has been achieved in previous works, it is still an open problem about how to reduce the computation precision of CNNs without losing performance, especially for medical image segmentation tasks where data dimension is high and annotation is scarce. This paper presents a novel CNN quantization framework that can squeeze a deep model (both parameters and activation) to extremely low bitwidth, e.g., 1∼2 bits, while maintaining its high performance. In the new method, we first design a strong baseline quantizer with an optimizable quantization range. Then, to relieve the back-propagation difficulty caused by the discontinuous quantization function, we design a radical residual connection scheme that allows gradients to flow through every quantized layer freely. Moreover, a tanh-based derivative function is used to further boost gradient flow and a distributional loss is employed to regularize the model output. Extensive experiments and ablation studies are conducted on two well-established public 3D segmentation datasets, i.e., BRATS2020 and LiTS. Experimental results evidence that our framework not only outperforms state-of-the-art quantization approaches significantly, but also achieves lossless performance on both datasets with ternary (2-bit) quantization.

A novel framework for segmentation of deep brain structures based on Markov dependence tree.

The aim of this work is to develop a new framework for multi-object segmentation of deep brain structures (caudate nucleus, putamen and thalamus) in medical brain images. Deep brain segmentation is difficult and challenging because the structures of interest are of relatively small size and have significant shape variations. The structure boundaries may be blurry or even missing, and the surrounding background is full of irrelevant edges. To tackle these problems, we propose a template-based framework to fuse the information of edge features, region statistics and inter-structure constraints for detecting and locating all target brain structures such that initialization by hand is unnecessary. The multi-object template is organized in the form of a hierarchical Markov dependence tree (MDT), and multiple objects are efficiently matched to a target image by a top-to-down optimization strategy. The final segmentation is obtained through refinement by a B-spline based non-rigid registration between the exemplar image and the target image. Our approach needs only one example as training data. We have validated the proposed method on a publicly available T1-weighted magnetic resonance image database with expert-segmented brain structures. In the experiments, the proposed approach has obtained encouraging results with 0.80 Dice score for the caudate nuclei, 0.81 Dice score for the putamina and 0.84 Dice score for the thalami on average.

Efficient implementation for spherical flux computation and its application to vascular segmentation.

Spherical flux is the flux inside a spherical region, and it is very useful in the analysis of tubular structures in magnetic resonance angiography and computed tomographic angiography. The conventional approach is to estimate the spherical flux in the spatial domain. Its running time depends on the sphere radius quadratically, which leads to very slow spherical flux computation when the sphere size is large. This paper proposes a more efficient implementation for spherical flux computation in the Fourier domain. Our implementation is based on the reformulation of the spherical flux calculation using the divergence theorem, spherical step function, and the convolution operation. With this reformulation, most of the calculations are performed in the Fourier domain. We show how to select the frequency subband so that the computation accuracy can be maintained. It is experimentally demonstrated that, using the synthetic and clinical phase contrast magnetic resonance angiographic volumes, our implementation is more computationally efficient than the conventional spatial implementation. The accuracies of our implementation and that of the conventional spatial implementation are comparable. Finally, the proposed implementation can definitely benefit the computation of the multiscale spherical flux with a set of radii because, unlike the conventional spatial implementation, the time complexity of the proposed implementation does not depend on the sphere radius.

Dominant local binary patterns for texture classification.

This paper proposes a novel approach to extract image features for texture classification. The proposed features are robust to image rotation, less sensitive to histogram equalization and noise. It comprises of two sets of features: dominant local binary patterns (DLBP) in a texture image and the supplementary features extracted by using the circularly symmetric Gabor filter responses. The dominant local binary pattern method makes use of the most frequently occurred patterns to capture descriptive textural information, while the Gabor-based features aim at supplying additional global textural information to the DLBP features. Through experiments, the proposed approach has been intensively evaluated by applying a large number of classification tests to histogram-equalized, randomly rotated and noise corrupted images in Outex, Brodatz, Meastex, and CUReT texture image databases. Our method has also been compared with six published texture features in the experiments. It is experimentally demonstrated that the proposed method achieves the highest classification accuracy in various texture databases and image conditions.

3D Randomized Connection Network with Graph-based Label Inference.

In this paper, a novel 3D deep learning network is proposed for brain MR image segmentation with randomized connection, which can decrease the dependency between layers and increase the network capacity. The convolutional LSTM and 3D convolution are employed as network units to capture the long-term and short-term 3D properties respectively. To assemble these two kinds of spatial-temporal information and refine the deep learning outcomes, we further introduce an efficient graph-based node selection and label inference method. Experiments have been carried out on two publicly available databases and results demonstrate that the proposed method can obtain competitive performances as compared with other state-of-the-art methods.

Weighted local variance-based edge detection and its application to vascular segmentation in magnetic resonance angiography.

Accurate detection of vessel boundaries is particularly important for a precise extraction of vasculatures in magnetic resonance angiography (MRA). In this paper, we propose the use of weighted local variance (WLV)-based edge detection scheme for vessel boundary detection in MRA. The proposed method is robust against changes of intensity contrast of edges and capable of giving high detection responses on low contrast edges. These robustness and capabilities are essential for detecting the boundaries of vessels in low contrast regions of images, which can contain intensity inhomogeneity, such as bias field, interferences induced from other tissues, or fluctuation of the speed related vessel intensity. The performance of the WLV-based edge detection scheme is studied and shown to be able to return strong and consistent detection responses on low contrast edges in the experiments. The proposed edge detection scheme can be embedded naturally in the active contour models for vascular segmentation. The WLV-based vascular segmentation method is tested using MRA image volumes. It is experimentally shown that the WLV-based edge detection approach can achieve high-quality segmentation of vasculatures in MRA images.

Probabilistic vessel axis tracing and its application to vessel segmentation with stream surfaces and minimum cost paths.

We propose a novel framework to segment vessels on their cross-sections. It starts with a probabilistic vessel axis tracing in a gray-scale three-dimensional angiogram, followed by vessel boundary delineation on cross-sections derived from the extracted axis. It promotes a more intuitive delineation of vessel boundaries which are mostly round on the cross-sections. The prior probability density function of the axis tracer's formulation permits seamless integration of user guidance to produce continuous traces through regions that contain furcations, diseased portions, kissing vessels (vessels in close proximity to each other) and thin vessels. The contour that outlines the vessel boundary in a 3-D space is determined as the minimum cost path on a weighted directed acyclic graph derived from each cross-section. The user can place anchor points to force the contour to pass through. The contours obtained are tiled to approximate the vessel boundary surface. Since we use stream surfaces generated w.r.t. the traced axis as cross-sections, non-intersecting adjacent cross-sections are guaranteed. Therefore, the tiling can be achieved by joining vertices of adjacent contours. The vessel boundary surface is then deformed under constrained movements on the cross-sections and is voxelized to produce the final vascular segmentation. Experimental results on synthetic and clinical data have shown that the vessel axes extracted by our tracer are continuous and less jittered as compared with the other two trace-based algorithms. Furthermore, the segmentation algorithm with cross-sections are robust to noise and can delineate vessel boundaries that have level of variability similar to those obtained manually.

A segmentation model using compound Markov random fields based on a boundary model.

Markov random field (MRF) theory has been widely applied to the challenging problem of image segmentation. In this paper, we propose a new nontexture segmentation model using compound MRFs, in which the original label MRF is coupled with a new boundary MRF to help improve the segmentation performance. The boundary model is relatively general and does not need prior training on boundary patterns. Unlike some existing related work, the proposed method offers a more compact interaction between label and boundary MRFs. Furthermore, our boundary model systematically takes into account all the possible scenarios of a single edge existing in a 3 x 3 neighborhood and, thus, incorporates sophisticated prior information about the relation between label and boundary. It is experimentally shown that the proposed model can segment objects with complex boundaries and at the same time is able to work under noise corruption. The new method has been applied to medical image segmentation. Experiments on synthetic images and real clinical datasets show that the proposed model is able to produce more accurate segmentation results and satisfactorily keep the delicate boundary. It is also less sensitive to noise in both high and low signal-to-noise ratio regions than some of the existing models in common use.

A loose-packing approach for coil embolization of giant intracranial aneurysm.

Endoluminal occlusion of giant intracranial aneurysms with coil embolization is a viable endovascular treatment option alternative to surgical clipping. However, due to the relatively large aneurysm size, the use of embolization coils for giant aneurysms could be great. A loose-packing embolization strategy in which the fundus of the aneurysm is loosely packed while the aneurysm base is tightly packed is presented. Such a coiling strategy is best suited to giant aneurysms of elongated configuration and narrow neck as illustrated in the present case. While the use of the loose-packing approach is recommended for elongated aneurysms with a narrow neck, its use is not to be generalized for aneurysms of other configurations.

Feature Sensitive Label Fusion With Random Walker for Atlas-Based Image Segmentation.

In this paper, a novel label fusion method is proposed for brain magnetic resonance image segmentation. This label fusion method is formulated on a graph, which embraces both label priors from atlases and anatomical priors from target image. To represent a pixel in a comprehensive way, three kinds of feature vectors are generated, including intensity, gradient, and structural signature. To select candidate atlas nodes for fusion, rather than exact searching, randomized k-d tree with spatial constraint is introduced as an efficient approximation for high-dimensional feature matching. Feature sensitive label prior (FSLP), which takes both the consistency and variety of different features into consideration, is proposed to gather atlas priors. As FSLP is a non-convex problem, one heuristic approach is further designed to solve it efficiently. Moreover, based on the anatomical knowledge, parts of the target pixels are also employed as the graph seeds to assist the label fusion process, and an iterative strategy is utilized to gradually update the label map. The comprehensive experiments carried out on two publicly available databases give results to demonstrate that the proposed method can obtain better segmentation quality.

Augmented vessels for quantitative analysis of vascular abnormalities and endovascular treatment planning.

Endovascular treatment plays an important role in the minimally invasive treatment of patients with vascular diseases, a major cause of morbidity and mortality worldwide. Given a segmentation of an angiography, quantitative analysis of abnormal structures can aid radiologists in choosing appropriate treatments and apparatuses. However, effective quantitation is only attainable if the abnormalities are identified from the vasculature. To achieve this, a novel method is developed, which works on the simpler shape of normal vessels to identify different vascular abnormalities (viz. stenotic atherosclerotic plaque, and saccular and fusiform aneurysmal lumens) in an indirect fashion, instead of directly manipulating the complex-shaped abnormalities. The proposed method has been tested on three synthetic and 17 clinical data sets. Comparisons with two related works are also conducted. Experimental results show that our method can produce satisfactory identification of the abnormalities and approximations of the ideal post-treatment vessel lumens. In addition, it can help increase the repeatability of the measurement of clinical parameters significantly.

Statistical cerebrovascular segmentation in three-dimensional rotational angiography based on maximum intensity projections.

Segmentation of three-dimensional rotational angiography (3D-RA) can provide quantitative 3D morphological information of vasculature. The expectation maximization-(EM-) based segmentation techniques have been widely used in the medical image processing community, because of the implementation simplicity, and computational efficiency of the approach. In a brain 3D-RA, vascular regions usually occupy a very small proportion (around 1%) inside an entire image volume. This severe imbalance between the intensity distributions of vessels and background can lead to inaccurate statistical modeling in the EM-based segmentation methods, and thus adversely affect the segmentation quality for 3D-RA. In this paper we present a new method for the extraction of vasculature in 3D-RA images. The new method is fully automatic and computationally efficient. As compared with the original 3D-RA volume, there is a larger proportion (around 20%) of vessels in its corresponding maximum intensity projection (MIP) image. The proposed method exploits this property to increase the accuracy of statistical modeling with the EM algorithm. The algorithm takes an iterative approach to compiling the 3D vascular segmentation progressively with the segmentation of MIP images along the three principal axes, and use a winner-takes-all strategy to combine the results obtained along individual axes. Experimental results on 12 3D-RA clinical datasets indicate that the segmentations obtained by the new method exhibit a high degree of agreement to the ground truth segmentations and are comparable to those produced by the manual optimal global thresholding method.

Bayesian image segmentation using local iso-intensity structural orientation.

Image segmentation is a fundamental problem in early computer vision. In segmentation of flat shaded, nontextured objects in real-world images, objects are usually assumed to be piecewise homogeneous. This assumption, however, is not always valid with images such as medical images. As a result, any techniques based on this assumption may produce less-than-satisfactory image segmentation. In this work, we relax the piecewise homogeneous assumption. By assuming that the intensity nonuniformity is smooth in the imaged objects, a novel algorithm that exploits the coherence in the intensity profile to segment objects is proposed. The algorithm uses a novel smoothness prior to improve the quality of image segmentation. The formulation of the prior is based on the coherence of the local structural orientation in the image. The segmentation process is performed in a Bayesian framework. Local structural orientation estimation is obtained with an orientation tensor. Comparisons between the conventional Hessian matrix and the orientation tensor have been conducted. The experimental results on the synthetic images and the real-world images have indicated that our novel segmentation algorithm produces better segmentations than both the global thresholding with the maximum likelihood estimation and the algorithm with the multilevel logistic MRF model.

Vascular segmentation of phase contrast magnetic resonance angiograms based on statistical mixture modeling and local phase coherence.

In this paper, we present an approach to segmenting the brain vasculature in phase contrast magnetic resonance angiography (PC-MRA). According to our prior work, we can describe the overall probability density function of a PC-MRA speed image as either a Maxwell-uniform (MU) or Maxwell-Gaussian-uniform (MGU) mixture model. An automatic mechanism based on Kullback-Leibler divergence is proposed for selecting between the MGU and MU models given a speed image volume. A coherence measure, namely local phase coherence (LPC), which incorporates information about the spatial relationships between neighboring flow vectors, is defined and shown to be more robust to noise than previously described coherence measures. A statistical measure from the speed images and the LPC measure from the phase images are combined in a probabilistic framework, based on the maximum a posteriori method and Markov random fields, to estimate the posterior probabilities of vessel and background for classification. It is shown that segmentation based on both measures gives a more accurate segmentation than using either speed or flow coherence information alone. The proposed method is tested on synthetic, flow phantom and clinical datasets. The results show that the method can segment normal vessels and vascular regions with relatively low flow rate and low signal-to-noise ratio, e.g., aneurysms and veins.