Convolutional Network With Twofold Feature Augmentation for Diabetic Retinopathy Recognition From Multi-Modal Images.

OBJECTIVE: With the scenario of limited labeled dataset, this paper introduces a deep learning-based approach that leverages Diabetic Retinopathy (DR) severity recognition performance using fundus images combined with wide-field swept-source optical coherence tomography angiography (SS-OCTA). METHODS: The proposed architecture comprises a backbone convolutional network associated with a Twofold Feature Augmentation mechanism, namely TFA-Net. The former includes multiple convolution blocks extracting representational features at various scales. The latter is constructed in a two-stage manner, i.e., the utilization of weight-sharing convolution kernels and the deployment of a Reverse Cross-Attention (RCA) stream. RESULTS: The proposed model achieves a Quadratic Weighted Kappa rate of 90.2% on the small-sized internal KHUMC dataset. The robustness of the RCA stream is also evaluated by the single-modal Messidor dataset, of which the obtained mean Accuracy (94.8%) and Area Under Receiver Operating Characteristic (99.4%) outperform those of the state-of-the-arts significantly. CONCLUSION: Utilizing a network strongly regularized at feature space to learn the amalgamation of different modalities is of proven effectiveness. Thanks to the widespread availability of multi-modal retinal imaging for each diabetes patient nowadays, such approach can reduce the heavy reliance on large quantity of labeled visual data. SIGNIFICANCE: Our TFA-Net is able to coordinate hybrid information of fundus photos and wide-field SS-OCTA for exhaustively exploiting DR-oriented biomarkers. Moreover, the embedded feature-wise augmentation scheme can enrich generalization ability efficiently despite learning from small-scale labeled data.

Bimodal learning via trilogy of skip-connection deep networks for diabetic retinopathy risk progression identification.

BACKGROUND: Diabetic Retinopathy (DR) is considered a pathology of retinal vascular complications, which stays in the top causes of vision impairment and blindness. Therefore, precisely inspecting its progression enables the ophthalmologists to set up appropriate next-visit schedule and cost-effective treatment plans. In the literature, existing work only makes use of numerical attributes in Electronic Medical Records (EMR) for acquiring such kind of DR-oriented knowledge through conventional machine learning techniques, which require an exhaustive job of engineering most impactful risk factors. OBJECTIVE: In this paper, an approach of deep bimodal learning is introduced to leverage the performance of DR risk progression identification. METHODS: In particular, we further involve valuable clinical information of fundus photography in addition to the aforementioned systemic attributes. Accordingly, a Trilogy of Skip-connection Deep Networks, namely Tri-SDN, is proposed to exhaustively exploit underlying relationships between the baseline and follow-up information of the fundus images and EMR-based attributes. Besides that, we adopt Skip-Connection Blocks as basis components of the Tri-SDN for making the end-to-end flow of signals more efficient during feedforward and backpropagation processes. RESULTS: Through a 10-fold cross validation strategy on a private dataset of 96 diabetic mellitus patients, the proposed method attains superior performance over the conventional EMR-modality learning approach in terms of Accuracy (90.6%), Sensitivity (96.5%), Precision (88.7%), Specificity (82.1%), and Area Under Receiver Operating Characteristics (88.8%). CONCLUSIONS: The experimental results show that the proposed Tri-SDN can combine features of different modalities (i.e., fundus images and EMR-based numerical risk factors) smoothly and effectively during training and testing processes, respectively. As a consequence, with impressive performance of DR risk progression recognition, the proposed approach is able to help the ophthalmologists properly decide follow-up schedule and subsequent treatment plans.