EMOST: A dual-branch hybrid network for medical image fusion via efficient model module and sparse transformer.

ABSTRACT

Multimodal medical image fusion fuses images with different modalities and provides more comprehensive and integrated diagnostic information. However, current multimodal image fusion methods cannot effectively model non-local contextual feature relationships, and due to direct aggregation of the extracted features, they introduce unnecessary implicit noise into the fused images. To solve the above problems, this paper proposes a novel dual-branch hybrid fusion network called EMOST for medical image fusion that combines a convolutional neural network (CNN) and a transformer. First, to extract more comprehensive feature information, an effective feature extraction module is proposed, which consists of an efficient dense block (EDB), an attention module (AM), a multiscale convolution block (MCB), and three sparse transformer blocks (STB). Meanwhile, a lightweight efficient model (EMO) is used in the feature extraction module to exploit the efficiency of the CNN with the dynamic modeling capability of the transformer. Additionally, the STB is incorporated to adaptively maintain the most useful self-attention values and remove as much redundant noise as possible by developing the top-k selection operator. Moreover, a novel feature fusion rule is designed to efficiently integrate the features. Experiments are conducted on four types of multimodal medical images. The proposed method shows higher performance than the art-of-the-state methods in terms of quantitative and qualitative evaluations. The code of the proposed method is available at https//github.com/XUTauto/EMOST.