A transfer learning approach to few-shot segmentation of novel white matter tracts.

Convolutional neural networks (CNNs) have achieved state-of-the-art performance for white matter (WM) tract segmentation based on diffusion magnetic resonance imaging (dMRI). The training of the CNN-based segmentation model generally requires a large number of manual delineations of WM tracts, which can be expensive and time-consuming. Although it is possible to carefully curate abundant training data for a set of WM tracts of interest, there can also be novel WM tracts-i.e., WM tracts that are not included in the existing annotated WM tracts-that are specific to a new scientific problem, and it is desired that the novel WM tracts can be segmented without repeating the laborious collection of a large number of manual delineations for these tracts. One possible solution to the problem is to transfer the knowledge learned for segmenting existing WM tracts to the segmentation of novel WM tracts with a fine-tuning strategy, where a CNN pretrained for segmenting existing WM tracts is fine-tuned with only a few annotated scans to segment the novel WM tracts. However, in classic fine-tuning, the information in the last task-specific layer for segmenting existing WM tracts is completely discarded. In this work, based on the pretraining and fine-tuning framework, we propose an improved transfer learning approach to the segmentation of novel WM tracts in the few-shot setting, where all knowledge in the pretrained model is incorporated into the fine-tuning procedure. Specifically, from the weights of the pretrained task-specific layer for segmenting existing WM tracts, we derive a better initialization of the last task-specific layer for the target model that segments novel WM tracts. In addition, to allow further improvement of the initialization of the last layer and thus the segmentation performance in the few-shot setting, we develop a simple yet effective data augmentation strategy that generates synthetic annotated images with tract-aware image mixing. To validate the proposed method, we performed experiments on brain dMRI scans from public and private datasets under various experimental settings, and the results indicate that our method improves the performance of few-shot segmentation of novel WM tracts.

Volumetric segmentation of white matter tracts with label embedding.

Convolutional neural networks have achieved state-of-the-art performance for white matter (WM) tract segmentation based on diffusion magnetic resonance imaging (dMRI). However, the segmentation can still be difficult for challenging WM tracts with thin bodies or complicated shapes; the segmentation is even more problematic in challenging scenarios with reduced data quality or domain shift between training and test data, which can be easily encountered in clinical settings. In this work, we seek to improve the segmentation of WM tracts, especially for challenging WM tracts in challenging scenarios. In particular, our method is based on volumetric WM tract segmentation, where voxels are directly labeled without performing tractography. To improve the segmentation, we exploit the characteristics of WM tracts that different tracts can cross or overlap and revise the network design accordingly. Specifically, because multiple tracts can co-exist in a voxel, we hypothesize that the different tract labels can be correlated. The tract labels at a single voxel are concatenated as a label vector, the length of which is the number of tract labels. Due to the tract correlation, this label vector can be projected into a lower-dimensional space-referred to as the embedded space-for each voxel, which allows the segmentation network to solve a simpler problem. By predicting the coordinate in the embedded space for the tracts at each voxel and subsequently mapping the coordinate to the label vector with a reconstruction module, the segmentation result can be achieved. To facilitate the learning of the embedded space, an auxiliary label reconstruction loss is integrated with the segmentation accuracy loss during network training, and network training and inference are end-to-end. Our method was validated on two dMRI datasets under various settings. The results show that the proposed method improves the accuracy of WM tract segmentation, and the improvement is more prominent for challenging tracts in challenging scenarios.