Multicolor image classification using the multimodal information bottleneck network (MMIB-Net) for detecting diabetic retinopathy.

Multicolor (MC) imaging is an imaging modality that records confocal scanning laser ophthalmoscope (cSLO) fundus images, which can be used for the diabetic retinopathy (DR) detection. By utilizing this imaging technique, multiple modal images can be obtained in a single case. Additional symptomatic features can be obtained if these images are considered during the diagnosis of DR. However, few studies have been carried out to classify MC Images using deep learning methods, let alone using multi modal features for analysis. In this work, we propose a novel model which uses the multimodal information bottleneck network (MMIB-Net) to classify the MC Images for the detection of DR. Our model can extract the features of multiple modalities simultaneously while finding concise feature representations of each modality using the information bottleneck theory. MC Images classification can be achieved by picking up the combined representations and features of all modalities. In our experiments, it is shown that the proposed method can achieve an accurate classification of MC Images. Comparative experiments also demonstrate that the use of multimodality and information bottleneck improves the performance of MC Images classification. To the best of our knowledge, this is the first report of DR identification utilizing the multimodal information bottleneck convolutional neural network in MC Images.

Multiview multimodal network for breast cancer diagnosis in contrast-enhanced spectral mammography images.

PURPOSE: CESM (contrast-enhanced spectral mammography) is an efficient tool for detecting breast cancer because of its image characteristics. However, among most deep learning-based methods for breast cancer classification, few models can integrate both its multiview and multimodal features. To effectively utilize the image features of CESM and thus help physicians to improve the accuracy of diagnosis, we propose a multiview multimodal network (MVMM-Net). METHODS: The experiment is carried out to evaluate the in-house CESM images dataset taken from 95 patients aged 21-74 years with 760 images. The framework consists of three main stages: the input of the model, image feature extraction, and image classification. The first stage is to preprocess the CESM to utilize its multiview and multimodal features effectively. In the feature extraction stage, a deep learning-based network is used to extract CESM images features. The last stage is to integrate different features for classification using the MVMM-Net model. RESULTS: According to the experimental results, the proposed method based on the Res2Net50 framework achieves an accuracy of 96.591%, sensitivity of 96.396%, specificity of 96.350%, precision of 96.833%, F1_score of 0.966, and AUC of 0.966 on the test set. Comparative experiments illustrate that the classification performance of the model can be improved by using multiview multimodal features. CONCLUSION: We proposed a deep learning classification model that combines multiple features of CESM. The results of the experiment indicate that our method is more precise than the state-of-the-art methods and produces accurate results for the classification of CESM images.