Learning U-Net Based Multi-Scale Features in Encoding-Decoding for MR Image Brain Tissue Segmentation.

Accurate brain tissue segmentation of MRI is vital to diagnosis aiding, treatment planning, and neurologic condition monitoring. As an excellent convolutional neural network (CNN), U-Net is widely used in MR image segmentation as it usually generates high-precision features. However, the performance of U-Net is considerably restricted due to the variable shapes of the segmented targets in MRI and the information loss of down-sampling and up-sampling operations. Therefore, we propose a novel network by introducing spatial and channel dimensions-based multi-scale feature information extractors into its encoding-decoding framework, which is helpful in extracting rich multi-scale features while highlighting the details of higher-level features in the encoding part, and recovering the corresponding localization to a higher resolution layer in the decoding part. Concretely, we propose two information extractors, multi-branch pooling, called MP, in the encoding part, and multi-branch dense prediction, called MDP, in the decoding part, to extract multi-scale features. Additionally, we designed a new multi-branch output structure with MDP in the decoding part to form more accurate edge-preserving predicting maps by integrating the dense adjacent prediction features at different scales. Finally, the proposed method is tested on datasets MRbrainS13, IBSR18, and ISeg2017. We find that the proposed network performs higher accuracy in segmenting MRI brain tissues and it is better than the leading method of 2018 at the segmentation of GM and CSF. Therefore, it can be a useful tool for diagnostic applications, such as brain MRI segmentation and diagnosing.

Multi-model Ensemble Learning Architecture Based on 3D CNN for Lung Nodule Malignancy Suspiciousness Classification.

Classification of benign and malignant in lung nodules using chest CT images is a key step in the diagnosis of early-stage lung cancer, as well as an effective way to improve the patients' survival rate. However, due to the diversity of lung nodules and the visual similarity of lung nodules to their surrounding tissues, it is difficult to construct a robust classification model with conventional deep learning-based diagnostic methods. To address this problem, we propose a multi-model ensemble learning architecture based on 3D convolutional neural network (MMEL-3DCNN). This approach incorporates three key ideas: (1) Constructed multi-model network architecture can be well adapted to the heterogeneity of lung nodules. (2) The input that concatenated of the intensity image corresponding to the nodule mask, the original image, and the enhanced image corresponding to which can help training model to extract advanced feature with more discriminative capacity. (3) Select the corresponding model to different nodule size dynamically for prediction, which can improve the generalization ability of the model effectively. In addition, ensemble learning is applied in this paper to further improve the robustness of the nodule classification model. The proposed method has been experimentally verified on the public dataset, LIDC-IDRI. The experimental results show that the proposed MMEL-3DCNN architecture can obtain satisfactory classification results.

Bone Marrow Cells Detection: A Technique for the Microscopic Image Analysis.

In the detection of myeloproliferative, the number of cells in each type of bone marrow cells (BMC) is an important parameter for the evaluation. In this study, we propose a new counting method, which consists of three modules including localization, segmentation and classification. The localization of BMC is achieved from a color transformation enhanced BMC sample image and stepwise averaging method. In the nucleus segmentation, both stepwise averaging method and Otsu's method are applied to obtain a weighted threshold for segmenting the patch into nucleus and non-nucleus. In the cytoplasm segmentation, a color weakening transformation, an improved region growing method and the K-Means algorithm are employed. The connected cells with BMC will be separated by the marker-controlled watershed algorithm. The features will be extracted for the classification after the segmentation. In this study, the BMC are classified using the support vector machine into five classes; namely, neutrophilic split granulocyte, neutrophilic stab granulocyte, metarubricyte, mature lymphocytes and the outlier (all other cells not listed). Experimental results show that the proposed method achieves superior segmentation and classification performance with an average segmentation accuracy of 91.76% and an average recall rate of 87.49%. The comparison shows that the proposed segmentation and classification methods outperform the existing methods.

Automatic Labeling of MR Brain Images Through the Hashing Retrieval Based Atlas Forest.

The multi-atlas method is one of the efficient and common automatic labeling method, which uses the prior information provided by expert-labeled images to guide the labeling of the target. However, most multi-atlas-based methods depend on the registration that may not give the correct information during the label propagation. To address the issue, we designed a new automatic labeling method through the hashing retrieval based atlas forest. The proposed method propagates labels without registration to reduce the errors, and constructs a target-oriented learning model to integrate information among the atlases. This method innovates a coarse classification strategy to preprocess the dataset, which retains the integrity of dataset and reduces computing time. Furthermore, the method considers each voxel in the atlas as a sample and encodes these samples with hashing for the fast sample retrieval. In the stage of labeling, the method selects suitable samples through hashing learning and trains atlas forests by integrating the information from the dataset. Then, the trained model is used to predict the labels of the target. Experimental results on two datasets illustrated that the proposed method is promising in the automatic labeling of MR brain images.

Reliability-Aware Cooperative Node Sleeping and Clustering in Duty-Cycled Sensors Networks.

Duty-cycled sensor networks provide a new perspective for improvement of energy efficiency and reliability assurance of multi-hop cooperative sensor networks. In this paper, we consider the energy-efficient cooperative node sleeping and clustering problems in cooperative sensor networks where clusters of relay nodes jointly transmit sensory data to the next hop. Our key idea for guaranteeing reliability is to exploit the on-demand number of cooperative nodes, facilitating the prediction of personalized end-to-end (ETE) reliability. Namely, a novel reliability-aware cooperative routing (RCR) scheme is proposed to select k-cooperative nodes at every hop (RCR-selection). After selecting k cooperative nodes at every hop, all of the non-cooperative nodes will go into sleep status. In order to solve the cooperative node clustering problem, we propose the RCR-based optimal relay assignment and cooperative data delivery (RCR-delivery) scheme to provide a low-communication-overhead data transmission and an optimal duty cycle for a given number of cooperative nodes when the network is dynamic, which enables part of cooperative nodes to switch into idle status for further energy saving. Through the extensive OPNET-based simulations, we show that the proposed scheme significantly outperforms the existing geographic routing schemes and beaconless geographic routings in wireless sensor networks with a highly dynamic wireless channel and controls energy consumption, while ETE reliability is effectively guaranteed.

An Augmented Reality Endoscope System for Ureter Position Detection.

Iatrogenic injury of ureter in the clinical operation may cause the serious complication and kidney damage. To avoid such a medical accident, it is necessary to provide the ureter position information to the doctor. For the detection of ureter position, an ureter position detection and display system with the augmented ris proposed to detect the ureter that is covered by human tissue. There are two key issues which should be considered in this new system. One is how to detect the covered ureter that cannot be captured by the electronic endoscope and the other is how to display the ureter position that provides stable and high-quality images. Simultaneously, any delayed processing of the system should disturb the surgery. The aided hardware detection method and target detection algorithms are proposed in this system. To mark the ureter position, a surface-lighting plastic optical fiber (POF) with the encoded light-emitting diode (LED) light is used to indicate the ureter position. The monochrome channel filtering algorithm (MCFA) is proposed to locate the ureter region more precisely. The ureter position is extracted using the proposed automatic region growing algorithm (ARGA) that utilizes the statistical information of the monochrome channel for the selection of growing seed point. In addition, according to the pulse signal of encoded light, the recognition of bright and dark frames based on the aided hardware (BDAH) is proposed to expedite the processing speed. Experimental results demonstrate that the proposed endoscope system can identify 92.04% ureter region in average.

A Novel Endoscope System for Position Detection and Depth Estimation of the Ureter.

Iatrogenic injury of ureter occurs occasionally in the clinical laparoscopic surgery. The ureter injury may cause the serious complications and kidney damage. To avoid such an injury, it is necessary to detect the ureter position in real-time. Currently, the endoscope cannot perform this type of function in detecting the ureter position in real-time. In order to have the real-time display of ureter position during the surgical operation, we propose a novel endoscope system which consists of a modified endoscope light and a new lumiontron tube with the LED light. The endoscope light is modified to detect the position of ureter by using our proposed dim target detection algorithm (DTDA). To make this new system functioning, two algorithmic approaches are proposed for the display of ureter position. The horizontal position of ureter is detected by the center line extraction method and the depth of ureter is estimated by the depth estimation method. Experimental results demonstrate that the proposed endoscope system can extract the position and depth information of ureter and exhibit superior performance in terms of accuracy and stabilization.

Automatic segmentation of pulmonary fissures in computed tomography images using 3D surface features.

Pulmonary interlobar fissures are important anatomic structures in human lungs and are useful in locating and classifying lung abnormalities. Automatic segmentation of fissures is a difficult task because of their low contrast and large variability. We developed a fully automatic training-free approach for fissure segmentation based on the local bending degree (LBD) and the maximum bending index (MBI). The LBD is determined by the angle between the eigenvectors of two Hessian matrices for a pair of adjacent voxels. It is used to construct a constraint to extract the candidate surfaces in three-dimensional (3D) space. The MBI is a measure to discriminate cylindrical surfaces from planar surfaces in 3D space. Our approach for segmenting fissures consists of five steps, including lung segmentation, plane-like structure enhancement, surface extraction with LBD, initial fissure identification with MBI, and fissure extension based on local plane fitting. When applying our approach to 15 chest computed tomography (CT) scans, the mean values of the positive predictive value, the sensitivity, the root-mean square (RMS) distance, and the maximal RMS are 91 %, 88 %, 1.01 ± 0.99 mm, and 11.56 mm, respectively, which suggests that our algorithm can efficiently segment fissures in chest CT scans.

Combining external-latent attention for medical image segmentation.

The attention mechanism comes as a new entry point for improving the performance of medical image segmentation. How to reasonably assign weights is a key element of the attention mechanism, and the current popular schemes include the global squeezing and the non-local information interactions using self-attention (SA) operation. However, these approaches over-focus on external features and lack the exploitation of latent features. The global squeezing approach crudely represents the richness of contextual information by the global mean or maximum value, while non-local information interactions focus on the similarity of external features between different regions. Both ignore the fact that the contextual information is presented more in terms of the latent features like the frequency change within the data. To tackle above problems and make proper use of attention mechanisms in medical image segmentation, we propose an external-latent attention collaborative guided image segmentation network, named TransGuider. This network consists of three key components: 1) a latent attention module that uses an improved entropy quantification method to accurately explore and locate the distribution of latent contextual information. 2) an external self-attention module using sparse representation, which can preserve external global contextual information while reducing computational overhead by selecting representative feature description map for SA operation. 3) a multi-attention collaborative module to guide the network to continuously focus on the region of interest, refining the segmentation mask. Our experimental results on several benchmark medical image segmentation datasets show that TransGuider outperforms the state-of-the-art methods, and extensive ablation experiments demonstrate the effectiveness of the proposed components. Our code will be available at https://github.com/chasingone/TransGuider.

An advanced nomogram model using deep learning radiomics and clinical data for predicting occult lymph node metastasis in lung adenocarcinoma.

PURPOSE: To evaluate the effectiveness of deep learning radiomics nomogram in distinguishing the occult lymph node metastasis (OLNM) status in clinical stage IA lung adenocarcinoma. METHODS: A cohort of 473 cases of lung adenocarcinomas from two hospitals was included, with 404 cases allocated to the training cohort and 69 cases to the testing cohort. Clinical characteristics and semantic features were collected, and radiomics features were extracted from the computed tomography (CT) images. Additionally, deep transfer learning (DTL) features were generated using RseNet50. Predictive models were developed using the logistic regression (LR) machine learning algorithm. Moreover, gene analysis was conducted on RNA sequencing data from 14 patients to explore the underlying biological basis of deep learning radiomics scores. RESULT: The training and testing cohorts achieved AUC values of 0.826 and 0.775 for the clinical model, 0.865 and 0.801 for the radiomics model, 0.927 and 0.885 for the DTL-radiomics model, and 0.928 and 0.898 for the nomogram model. The nomogram model demonstrated superiority over the clinical model. The decision curve analysis (DCA) revealed a net benefit in predicting OLNM for all models. The investigation into the biological basis of deep learning radiomics scores identified an association between high scores and pathways related to tumor proliferation and immune cell infiltration in the microenvironment. CONCLUSIONS: The nomogram model, incorporating clinical-semantic features, radiomics, and DTL features, exhibited promising performance in predicting OLNM. It has the potential to provide valuable information for non-invasive lymph node staging and individualized therapeutic approaches.

Development and Validation of a Deep Learning Radiomics Model to Predict High-Risk Pathologic Pulmonary Nodules Using Preoperative Computed Tomography.

RATIONALE AND OBJECTIVES: To accurately identify the high-risk pathological factors of pulmonary nodules, our study constructed a model combined with clinical features, radiomics features, and deep transfer learning features to predict high-risk pathological pulmonary nodules. MATERIALS AND METHODS: The study cohort consisted of 469 cases of lung adenocarcinoma patients, divided into a training cohort (n = 400) and an external validation cohort (n = 69). We obtained computed tomography (CT) semantic features and clinical characteristics, as well as extracted radiomics and deep transfer learning (DTL) features from the CT images. Selected features were used for constructing prediction models using the logistic regression (LR) algorithm. The performance of the models was evaluated through metrics including the area under the receiver operating characteristic curve (AUC), sensitivity, specificity, calibration curve, and decision curve analysis. RESULTS: The clinical model achieved an AUC of 0.774 (95% CI: 0.728-0.821) in the training cohort and 0.762 (95% confidence interval [CI]: 0.650-0.873) in the external validation cohort. The radiomics model demonstrated an AUC of 0.847 (95% CI: 0.810-0.884) in the training cohort and 0.800 (95% CI: 0.693-0.907) in the external validation cohort. The radiomics-DTL (Rad-DTL) model showed an AUC of 0.871 (95% CI: 0.838-0.905) in the training cohort and 0.806 (95% CI: 0.698-0.914) in the external validation cohort. The proposed combined model yielded AUC values of 0.872 and 0.814 in the training and external validation cohorts, respectively. The combined model demonstrated superiority over both the clinical model and the Rad-DTL model. There were no statistically significant differences observed in the comparison between the combined model incorporating clinical features and the Rad-DTL model. Decision curve analysis (DCA) indicated that the models provided a net benefit in predicting high-risk pathologic pulmonary nodules. CONCLUSION: Rad-DTL signature is a potential biomarker for predicting high-risk pathologic pulmonary nodules using preoperative CT, determining the appropriate surgical strategy, and guiding the extent of resection.

CT-based quantification of intratumoral heterogeneity for predicting pathologic complete response to neoadjuvant immunochemotherapy in non-small cell lung cancer.

Objectives: To investigate the prediction of pathologic complete response (pCR) in patients with non-small cell lung cancer (NSCLC) undergoing neoadjuvant immunochemotherapy (NAIC) using quantification of intratumoral heterogeneity from pre-treatment CT image. Methods: This retrospective study included 178 patients with NSCLC who underwent NAIC at 4 different centers. The training set comprised 108 patients from center A, while the external validation set consisted of 70 patients from center B, center C, and center D. The traditional radiomics model was contrasted using radiomics features. The radiomics features of each pixel within the tumor region of interest (ROI) were extracted. The optimal division of tumor subregions was determined using the K-means unsupervised clustering method. The internal tumor heterogeneity habitat model was developed using the habitats features from each tumor sub-region. The LR algorithm was employed in this study to construct a machine learning prediction model. The diagnostic performance of the model was evaluated using criteria such as area under the receiver operating characteristic curve (AUC), accuracy, specificity, sensitivity, positive predictive value (PPV), and negative predictive value (NPV). Results: In the training cohort, the traditional radiomics model achieved an AUC of 0.778 [95% confidence interval (CI): 0.688-0.868], while the tumor internal heterogeneity habitat model achieved an AUC of 0.861 (95% CI: 0.789-0.932). The tumor internal heterogeneity habitat model exhibits a higher AUC value. It demonstrates an accuracy of 0.815, surpassing the accuracy of 0.685 achieved by traditional radiomics models. In the external validation cohort, the AUC values of the two models were 0.723 (CI: 0.591-0.855) and 0.781 (95% CI: 0.673-0.889), respectively. The habitat model continues to exhibit higher AUC values. In terms of accuracy evaluation, the tumor heterogeneity habitat model outperforms the traditional radiomics model, achieving a score of 0.743 compared to 0.686. Conclusion: The quantitative analysis of intratumoral heterogeneity using CT to predict pCR in NSCLC patients undergoing NAIC holds the potential to inform clinical decision-making for resectable NSCLC patients, prevent overtreatment, and enable personalized and precise cancer management.

Magnetically driven capsules with multimodal response and multifunctionality for biomedical applications.

Untethered capsules hold clinical potential for the diagnosis and treatment of gastrointestinal diseases. Although considerable progress has been achieved recently in this field, the constraints imposed by the narrow spatial structure of the capsule and complex gastrointestinal tract environment cause many open-ended problems, such as poor active motion and limited medical functions. In this work, we describe the development of small-scale magnetically driven capsules with a distinct magnetic soft valve made of dual-layer ferromagnetic soft composite films. A core technological advancement achieved is the flexible opening and closing of the magnetic soft valve by using the competitive interactions between magnetic gradient force and magnetic torque, laying the foundation for the functional integration of both drug release and sampling. Meanwhile, we propose a magnetic actuation strategy based on multi-frequency response control and demonstrate that it can achieve effective decoupled regulation of the capsule's global motion and local responses. Finally, through a comprehensive approach encompassing ideal models, animal ex vivo models, and in vivo assessment, we demonstrate the versatility of the developed magnetic capsules and their multiple potential applications in the biomedical field, such as targeted drug delivery and sampling, selective dual-drug release, and light/thermal-assisted therapy.

Wireless capsule endoscopy video reduction based on camera motion estimation.

Wireless capsule endoscopy (WCE) is a novel technology aiming for investigating the diseases and abnormalities in small intestine. The major drawback of WCE examination is that it takes a long time to examine the whole WCE video. In this paper, we present a new reduction scheme for WCE video to reduce the examination time. To achieve this task, a WCE video motion model is proposed. Under this motion model, the WCE imaging motion is estimated in two stages (the coarse level and the fine level). In the coarse level, the WCE camera motion is estimated with a combination of Bee Algorithm and Mutual Information. In the fine level, the local gastrointestinal tract motion is estimated with SIFT flow. Based on the result of WCE imaging motion estimation, the reduction scheme preserves key images in WCE video with scene changes. From experimental results, we notice that the proposed motion model is suitable for the motion estimation in successive WCE images. Through the comparison with APRS and FCM-NMF scheme, our scheme can produce an acceptable reduction sequence for browsing and examination.

FSA-Net: Rethinking the attention mechanisms in medical image segmentation from releasing global suppressed information.

Attention mechanism-based medical image segmentation methods have developed rapidly recently. For the attention mechanisms, it is crucial to accurately capture the distribution weights of the effective features contained in the data. To accomplish this task, most attention mechanisms prefer using the global squeezing approach. However, it will lead to a problem of over-focusing on the global most salient effective features of the region of interest, while suppressing the secondary salient ones. Making partial fine-grained features are abandoned directly. To address this issue, we propose to use a multiple-local perception method to aggregate global effective features, and design a fine-grained medical image segmentation network, named FSA-Net. This network consists of two key components: 1) the novel Separable Attention Mechanisms which replace global squeezing with local squeezing to release the suppressed secondary salient effective features. 2) a Multi-Attention Aggregator (MAA) which can fuse multi-level attention to efficiently aggregate task-relevant semantic information. We conduct extensive experimental evaluations on five publicly available medical image segmentation datasets: MoNuSeg, COVID-19-CT100, GlaS, CVC-ClinicDB, ISIC2018, and DRIVE datasets. Experimental results show that FSA-Net outperforms state-of-the-art methods in medical image segmentation.

A 3D Cross-Modality Feature Interaction Network With Volumetric Feature Alignment for Brain Tumor and Tissue Segmentation.

Accurate volumetric segmentation of brain tumors and tissues is beneficial for quantitative brain analysis and brain disease identification in multi-modal Magnetic Resonance (MR) images. Nevertheless, due to the complex relationship between modalities, 3D Fully Convolutional Networks (3D FCNs) using simple multi-modal fusion strategies hardly learn the complex and nonlinear complementary information between modalities. Meanwhile, the indiscriminative feature aggregation between low-level and high-level features easily causes volumetric feature misalignment in 3D FCNs. On the other hand, the 3D convolution operations of 3D FCNs are excellent at modeling local relations but typically inefficient at capturing global relations between distant regions in volumetric images. To tackle these issues, we propose an Aligned Cross-Modality Interaction Network (ACMINet) for segmenting the regions of brain tumors and tissues from MR images. In this network, the cross-modality feature interaction module is first designed to adaptively and efficiently fuse and refine multi-modal features. Secondly, the volumetric feature alignment module is developed for dynamically aligning low-level and high-level features by the learnable volumetric feature deformation field. Thirdly, we propose the volumetric dual interaction graph reasoning module for graph-based global context modeling in spatial and channel dimensions. Our proposed method is applied to brain glioma, vestibular schwannoma, and brain tissue segmentation tasks, and we performed extensive experiments on BraTS2018, BraTS2020, Vestibular Schwannoma, and iSeg-2017 datasets. Experimental results show that ACMINet achieves state-of-the-art segmentation performance on all four benchmark datasets and obtains the highest DSC score of hard-segmented enhanced tumor region on the validation leaderboard of the BraTS2020 challenge.

Automatic 3D CT liver segmentation based on fast global minimization of probabilistic active contour.

PURPOSE: Automatic liver segmentation from computed tomography (CT) images is an essential preprocessing step for computer-aided diagnosis of liver diseases. However, due to the large differences in liver shapes, low-contrast to adjacent tissues, and existence of tumors or other abnormalities, liver segmentation has been very challenging. This study presents an accurate and fast liver segmentation method based on a novel probabilistic active contour (PAC) model and its fast global minimization scheme (3D-FGMPAC), which is explainable as compared with deep learning methods. METHODS: The proposed method first constructs a slice-indexed-histogram to localize the volume of interest (VOI) and estimate the probability that a voxel belongs to the liver according its intensity. The probabilistic image would be used to initialize the 3D PAC model. Secondly, a new contour indicator function, which is a component of the model, is produced by combining the gradient-based edge detection and Hessian-matrix-based surface detection. Then, a fast numerical scheme derived for the 3D PAC model is performed to evolve the initial probabilistic image into the global minimizer of the model, which is a smoothed probabilistic image showing a distinctly highlighted liver. Next, a simple region-growing strategy is applied to extract the whole liver mask from the smoothed probabilistic image. Finally, a B-spline surface is constructed to fit the patch of the rib cage to prevent possible leakage into adjacent intercostal tissues. RESULTS: The proposed method is evaluated on two public datasets. The average Dice score, volume overlap error, volume difference, symmetric surface distance and volume processing time are 0.96, 7.35%, 0.02%, 1.17 mm and 19.8 s for the Sliver07 dataset, and 0.95, 8.89%, - 0.02 % $-0.02\%$ , 1.45 mm and 23.08 s for the 3Dircadb dataset, respectively. CONCLUSIONS: The proposed fully-automatic approach can effectively segment the liver from low-contrast and complex backgrounds. The quantitative and qualitative results demonstrate that the proposed segmentation method outperforms state-of-the-art traditional automatic liver segmentation algorithms and achieves very competitive performance compared with recent deep leaning-based methods.

A bidirectional registration neural network for cardiac motion tracking using cine MRI images.

Using cine magnetic resonance imaging (cine MRI) images to track cardiac motion helps users to analyze the myocardial strain, and is of great importance in clinical applications. At present, most of the automatic deep learning-based motion tracking methods compare two images without considering temporal information between MRI frames, which easily leads to the lack of consistency of the generated motion fields. Even though a small number of works take into account the temporal factor, they are usually computationally intensive or have limitations on image length. To solve this problem, we propose a bidirectional convolution neural network for motion tracking of cardiac cine MRI images. This network leverages convolutional blocks to extract spatial features from three-dimensional (3D) image registration pairs, and models the temporal relations through a bidirectional recurrent neural network to obtain the Lagrange motion field between the reference image and other images. Compared with previous pairwise registration methods, the proposed method can automatically learn spatiotemporal information from multiple images with fewer parameters. We evaluated our model on three public cardiac cine MRI datasets. The experimental results demonstrated that the proposed method can significantly improve the motion tracking accuracy. The average Dice coefficient between estimated segmentation and manual segmentation has reached almost 0.85 on the widely used Automatic Cardiac Diagnostic Challenge (ACDC) dataset.

A discontinuity-preserving regularization for deep learning-based cardiac image registration.

Objective. Sliding motion may occur between organs in anatomical regions due to respiratory motion and heart beating. This issue is often neglected in previous studies, resulting in poor image registration performance. A new approach is proposed to handle discontinuity at the boundary and improve registration accuracy.Approach. The proposed discontinuity-preserving regularization (DPR) term can maintain local discontinuities. It leverages the segmentation mask to find organ boundaries and then relaxes the displacement field constraints in these boundary regions. A weakly supervised method using mask dissimilarity loss (MDL) is also proposed. It employs a simple formula to calculate the similarity between the fixed image mask and the deformed moving image mask. These two strategies are added to the loss function during network training to guide the model better to update parameters. Furthermore, during inference time, no segmentation mask information is needed.Main results. Adding the proposed DPR term increases the Dice coefficients by 0.005, 0.009, and 0.081 for three existing registration neural networks CRNet, VoxelMorph, and ViT-V-Net, respectively. It also shows significant improvements in other metrics, including Hausdorff Distance and Average Surface Distance. All quantitative indicator results with MDL have been slightly improved within 1%. After applying these two regularization terms, the generated displacement field is more reasonable at the boundary, and the deformed moving image is closer to the fixed image.Significance. This study demonstrates that the proposed regularization terms can effectively handle discontinuities at the boundaries of organs and improve the accuracy of deep learning-based cardiac image registration methods. Besides, they are generic to be extended to other networks.

A modality-collaborative convolution and transformer hybrid network for unpaired multi-modal medical image segmentation with limited annotations.

BACKGROUND: Multi-modal learning is widely adopted to learn the latent complementary information between different modalities in multi-modal medical image segmentation tasks. Nevertheless, the traditional multi-modal learning methods require spatially well-aligned and paired multi-modal images for supervised training, which cannot leverage unpaired multi-modal images with spatial misalignment and modality discrepancy. For training accurate multi-modal segmentation networks using easily accessible and low-cost unpaired multi-modal images in clinical practice, unpaired multi-modal learning has received comprehensive attention recently. PURPOSE: Existing unpaired multi-modal learning methods usually focus on the intensity distribution gap but ignore the scale variation problem between different modalities. Besides, within existing methods, shared convolutional kernels are frequently employed to capture common patterns in all modalities, but they are typically inefficient at learning global contextual information. On the other hand, existing methods highly rely on a large number of labeled unpaired multi-modal scans for training, which ignores the practical scenario when labeled data is limited. To solve the above problems, we propose a modality-collaborative convolution and transformer hybrid network (MCTHNet) using semi-supervised learning for unpaired multi-modal segmentation with limited annotations, which not only collaboratively learns modality-specific and modality-invariant representations, but also could automatically leverage extensive unlabeled scans for improving performance. METHODS: We make three main contributions to the proposed method. First, to alleviate the intensity distribution gap and scale variation problems across modalities, we develop a modality-specific scale-aware convolution (MSSC) module that can adaptively adjust the receptive field sizes and feature normalization parameters according to the input. Secondly, we propose a modality-invariant vision transformer (MIViT) module as the shared bottleneck layer for all modalities, which implicitly incorporates convolution-like local operations with the global processing of transformers for learning generalizable modality-invariant representations. Third, we design a multi-modal cross pseudo supervision (MCPS) method for semi-supervised learning, which enforces the consistency between the pseudo segmentation maps generated by two perturbed networks to acquire abundant annotation information from unlabeled unpaired multi-modal scans. RESULTS: Extensive experiments are performed on two unpaired CT and MR segmentation datasets, including a cardiac substructure dataset derived from the MMWHS-2017 dataset and an abdominal multi-organ dataset consisting of the BTCV and CHAOS datasets. Experiment results show that our proposed method significantly outperforms other existing state-of-the-art methods under various labeling ratios, and achieves a comparable segmentation performance close to single-modal methods with fully labeled data by only leveraging a small portion of labeled data. Specifically, when the labeling ratio is 25%, our proposed method achieves overall mean DSC values of 78.56% and 76.18% in cardiac and abdominal segmentation, respectively, which significantly improves the average DSC value of two tasks by 12.84% compared to single-modal U-Net models. CONCLUSIONS: Our proposed method is beneficial for reducing the annotation burden of unpaired multi-modal medical images in clinical applications.