Improving Tumor Classification by Reusing Self-predicted Segmentation of Medical Images as Guiding Knowledge.

Differential diagnosis of tumors is important for computer-aided diagnosis. In computer-aided diagnosis systems, expert knowledge of lesion segmentation masks is limited as it is only used during preprocessing or as supervision to guide feature extraction. To improve the utilization of lesion segmentation masks, this study proposes a simple and effective multitask learning network that improves medical image classification using self-predicted segmentation as guiding knowledge; we call this network RS 2-net. In RS 2-net, the predicted segmentation probability map obtained from the initial segmentation inference is added to the original image to form a new input, which is then reinput to the network for the final classification inference. We validated the proposed RS 2-net using three datasets: the pNENs-Grade dataset, which tested the prediction of pancreatic neuroendocrine neoplasm grading, and the HCC-MVI dataset, which tested the prediction of microvascular invasion of hepatocellular carcinoma, and ISIC 2017 public skin lesion dataset. The experimental results indicate that the proposed strategy of reusing self-predicted segmentation is effective, and RS 2-net outperforms other popular networks and existing state-of-the-art studies. Interpretive analytics based on feature visualization demonstrates that the improved classification performance of our reuse strategy is due to the semantic information that can be acquired in advance in a shallow network.

CT-based radiomics signature of visceral adipose tissue for prediction of disease progression in patients with Crohn's disease: A multicentre cohort study.

Background: Visceral adipose tissue (VAT) is involved in the pathogenesis of Crohn's disease (CD). However, data describing its effects on CD progression remain scarce. We developed and validated a VAT-radiomics model (RM) using computed tomography (CT) images to predict disease progression in patients with CD and compared it with a subcutaneous adipose tissue (SAT)-RM. Methods: This retrospective study included 256 patients with CD (training, n = 156; test, n = 100) who underwent baseline CT examinations from June 19, 2015 to June 14, 2020 at three tertiary referral centres (The First Affiliated Hospital of Sun Yat-Sen University, The First Affiliated Hospital of Shantou University Medical College, and The First People's Hospital of Foshan City) in China. Disease progression referred to the development of penetrating or stricturing diseases or the requirement for CD-related surgeries during follow-up. A total of 1130 radiomics features were extracted from VAT on CT in the training cohort, and a machine-learning-based VAT-RM was developed to predict disease progression using selected reproducible features and validated in an external test cohort. Using the same modeling methodology, a SAT-RM was developed and compared with the VAT-RM. Findings: The VAT-RM exhibited satisfactory performance for predicting disease progression in total test cohort (the area under the ROC curve [AUC] = 0.850, 95% confidence Interval [CI] 0.764-0.913, P < 0.001) and in test cohorts 1 (AUC = 0.820, 95% CI 0.687-0.914, P < 0.001) and 2 (AUC = 0.871, 95% CI 0.744-0.949, P < 0.001). No significant differences in AUC were observed between test cohorts 1 and 2 (P = 0.673), suggesting considerable efficacy and robustness of the VAT-RM. In the total test cohort, the AUC of the VAT-RM for predicting disease progression was higher than that of SAT-RM (AUC = 0.786, 95% CI 0.692-0.861, P < 0.001). On multivariate Cox regression analysis, the VAT-RM (hazard ratio [HR] = 9.285, P = 0.005) was the most important independent predictor, followed by the SAT-RM (HR = 3.280, P = 0.060). Decision curve analysis further confirmed the better net benefit of the VAT-RM than the SAT-RM. Moreover, the SAT-RM failed to significantly improve predictive efficacy after it was added to the VAT-RM (integrated discrimination improvement = 0.031, P = 0.102). Interpretation: Our results suggest that VAT is an important determinant of disease progression in patients with CD. Our VAT-RM allows the accurate identification of high-risk patients prone to disease progression and offers notable advantages over SAT-RM. Funding: This study was supported by the National Natural Science Foundation of China, Guangdong Basic and Applied Basic Research Foundation, Shenzhen-Hong Kong Institute of Brain Science-Shenzhen Fundamental Research Institutions, Nature Science Foundation of Shenzhen, and Young S&T Talent Training Program of Guangdong Provincial Association for S&T. Translation: For the Chinese translation of the abstract see Supplementary Materials section.

Automatic quantification of retinal photoreceptor integrity to predict persistent disease activity in neovascular age-related macular degeneration using deep learning.

Purpose: Using deep learning (DL)-based technique, we identify risk factors and create a prediction model for refractory neovascular age-related macular degeneration (nAMD) characterized by persistent disease activity (PDA) in spectral domain optical coherence tomography (SD-OCT) images. Materials and methods: A total of 671 typical B-scans were collected from 186 eyes of 186 patients with nAMD. Spectral domain optical coherence tomography images were analyzed using a classification convolutional neural network (CNN) and a fully convolutional network (FCN) algorithm to extract six features involved in nAMD, including ellipsoid zone (EZ), external limiting membrane (ELM), intraretinal fluid (IRF), subretinal fluid (SRF), pigment epithelium detachment (PED), and subretinal hyperreflective material (SHRM). Random forest models were probed to predict 1-year disease activity (stable, PDA, and cured) based on the quantitative features computed from automated segmentation and evaluated with cross-validation. Results: The algorithm to segment six SD-OCT features achieved the mean accuracy of 0.930 (95% CI: 0.916-0.943), dice coefficients of 0.873 (95% CI: 0.847-0.899), a sensitivity of 0.873 (95% CI: 0.844-0.910), and a specificity of 0.922 (95% CI: 0.905-0.940). The six-metric model including EZ and ELM achieved the optimal performance to predict 1-year disease activity, with an area under the receiver operating characteristic (ROC) curve (AUC) of 0.980, the accuracy of 0.930, the sensitivity of 0.920, and the specificity of 0.962. The integrity of EZ and ELM significantly improved the performance of the six-metric model than that of the four-metric model. Conclusion: The prediction model reveals the potential to predict PDA in nAMD eyes. The integrity of EZ and ELM constituted the strongest predictive factor for PDA in nAMD eyes in real-world clinical practice. The results of this study are a significant step toward image-guided prediction of long-term disease activity in the management of nAMD and highlight the importance of the automatic identification of photoreceptor layers.

Intestinal fibrosis classification in patients with Crohn's disease using CT enterography-based deep learning: comparisons with radiomics and radiologists.

OBJECTIVES: Accurate evaluation of bowel fibrosis in patients with Crohn's disease (CD) remains challenging. Computed tomography enterography (CTE)-based radiomics enables the assessment of bowel fibrosis; however, it has some deficiencies. We aimed to develop and validate a CTE-based deep learning model (DLM) for characterizing bowel fibrosis more efficiently. METHODS: We enrolled 312 bowel segments of 235 CD patients (median age, 33 years old) from three hospitals in this retrospective study. A training cohort and test cohort 1 were recruited from center 1, while test cohort 2 from centers 2 and 3. All patients performed CTE within 3 months before surgery. The histological fibrosis was semi-quantitatively assessed. A DLM was constructed in the training cohort based on a 3D deep convolutional neural network with 10-fold cross-validation, and external independent validation was conducted on the test cohorts. The radiomics model (RM) was developed with 4 selected radiomics features extracted from CTE images by using logistic regression. The evaluation of CTE images was performed by two radiologists. DeLong's test and a non-inferiority test were used to compare the models' performance. RESULTS: DLM distinguished none-mild from moderate-severe bowel fibrosis with an area under the receiver operator characteristic curve (AUC) of 0.828 in the training cohort and 0.811, 0.808, and 0.839 in the total test cohort, test cohorts 1 and 2, respectively. In the total test cohort, DLM achieved better performance than two radiologists (*1 AUC = 0.579, *2 AUC = 0.646; both p < 0.05) and was not inferior to RM (AUC = 0.813, p < 0.05). The total processing time for DLM was much shorter than that of RM (p < 0.001). CONCLUSION: DLM is better than radiologists in diagnosing intestinal fibrosis on CTE in patients with CD and not inferior to RM; furthermore, it is more time-saving compared to RM. KEY POINTS: • Question Could computed tomography enterography (CTE)-based deep learning model (DLM) accurately distinguish intestinal fibrosis severity in patients with Crohn's disease (CD)? • Findings In this cross-sectional study that included 235 patients with CD, DLM achieved better performance than that of two radiologists' interpretation and was not inferior to RM with significant differences and much shorter processing time. • Meaning This DLM may accurately distinguish the degree of intestinal fibrosis in patients with CD and guide gastroenterologists to formulate individualized treatment strategies for those with bowel strictures.

Domain Adaptation Meets Zero-Shot Learning: An Annotation-Efficient Approach to Multi-Modality Medical Image Segmentation.

Due to the lack of properly annotated medical data, exploring the generalization capability of the deep model is becoming a public concern. Zero-shot learning (ZSL) has emerged in recent years to equip the deep model with the ability to recognize unseen classes. However, existing studies mainly focus on natural images, which utilize linguistic models to extract auxiliary information for ZSL. It is impractical to apply the natural image ZSL solutions directly to medical images, since the medical terminology is very domain-specific, and it is not easy to acquire linguistic models for the medical terminology. In this work, we propose a new paradigm of ZSL specifically for medical images utilizing cross-modality information. We make three main contributions with the proposed paradigm. First, we extract the prior knowledge about the segmentation targets, called relation prototypes, from the prior model and then propose a cross-modality adaptation module to inherit the prototypes to the zero-shot model. Second, we propose a relation prototype awareness module to make the zero-shot model aware of information contained in the prototypes. Last but not least, we develop an inheritance attention module to recalibrate the relation prototypes to enhance the inheritance process. The proposed framework is evaluated on two public cross-modality datasets including a cardiac dataset and an abdominal dataset. Extensive experiments show that the proposed framework significantly outperforms the state of the arts.

Accurate and Feasible Deep Learning Based Semi-Automatic Segmentation in CT for Radiomics Analysis in Pancreatic Neuroendocrine Neoplasms.

Current clinical practice or radiomics studies of pancreatic neuroendocrine neoplasms (pNENs) require manual delineation of the lesions in computed tomography (CT) images, which is time-consuming and subjective. We used a semi-automatic deep learning (DL) method for segmentation of pNENs and verified its feasibility in radiomics analysis. This retrospective study included two datasets: Dataset 1, contrast-enhanced CT images (CECT) of 80 and 18 patients respectively collected from two centers; and Dataset 2, CECT of 56 and 16 patients respectively from two centers. A DL-based semi-automatic segmentation model was developed and validated with Dataset 1 and Dataset 2, and the segmentation results were used for radiomics analysis from which the performance was compared against that based on manual segmentation. The mean Dice similarity coefficient of the trained segmentation model was 81.8% and 74.8% for external validation with Dataset 1 and Dataset 2 respectively. Four classifiers frequently used in radiomics studies were trained and tested with leave-one-out cross-validation strategy. For pathological grading prediction with Dataset 1, the area under the receiver operating characteristic curve (AUC) with semi-automatic segmentation was up to 0.76 and 0.87 respectively for internal and external validation. For recurrence study with Dataset 2, the AUC with semi-automatic segmentation was up to 0.78. All these AUCs were not statistically significant from the corresponding results based on manual segmentation. Our study showed that DL-based semi-automatic segmentation is accurate and feasible for the radiomics analysis in pNENs.

Development and Validation of a Novel Computed-Tomography Enterography Radiomic Approach for Characterization of Intestinal Fibrosis in Crohn's Disease.

BACKGROUND & AIMS: No reliable method for evaluating intestinal fibrosis in Crohn's disease (CD) exists; therefore, we developed a computed-tomography enterography (CTE)-based radiomic model (RM) for characterizing intestinal fibrosis in CD. METHODS: This retrospective multicenter study included 167 CD patients with 212 bowel lesions (training, 98 lesions; test, 114 lesions) who underwent preoperative CTE and bowel resection at 1 of the 3 tertiary referral centers from January 2014 through June 2020. Bowel fibrosis was histologically classified as none-mild or moderate-severe. In the training cohort, 1454 radiomic features were extracted from venous-phase CTE and a machine learning-based RM was developed based on the reproducible features using logistic regression. The RM was validated in an independent external test cohort recruited from 3 centers. The diagnostic performance of RM was compared with 2 radiologists' visual interpretation of CTE using receiver operating characteristic (ROC) curve analysis. RESULTS: In the training cohort, the area under the ROC curve (AUC) of RM for distinguishing moderate-severe from none-mild intestinal fibrosis was 0.888 (95% confidence interval [CI], 0.818-0.957). In the test cohort, the RM showed robust performance across 3 centers with an AUC of 0.816 (95% CI, 0.706-0.926), 0.724 (95% CI, 0.526-0.923), and 0.750 (95% CI, 0.560-0.940), respectively. Moreover, the RM was more accurate than visual interpretations by either radiologist (radiologist 1, AUC = 0.554; radiologist 2, AUC = 0.598; both, P < .001) in the test cohort. Decision curve analysis showed that the RM provided a better net benefit to predicting intestinal fibrosis than the radiologists. CONCLUSIONS: A CTE-based RM allows for accurate characterization of intestinal fibrosis in CD.

Deep Semantic Segmentation Feature-Based Radiomics for the Classification Tasks in Medical Image Analysis.

Recently, an emerging trend in medical image classification is to combine radiomics framework with deep learning classification network in an integrated system. Although this combination is efficient in some tasks, the deep learning-based classification network is often difficult to capture an effective representation of lesion regions, and prone to face the challenge of overfitting, leading to unreliable features and inaccurate results, especially when the sizes of the lesions are small or the training dataset is small. In addition, these combinations mostly lack an effective feature selection mechanism, which makes it difficult to obtain the optimal feature selection. In this paper, we introduce a novel and effective deep semantic segmentation feature-based radiomics (DSFR) framework to overcome the above-mentioned challenges, which consists of two modules: the deep semantic feature extraction module and the feature selection module. Specifically, the extraction module is utilized to extract hierarchical semantic features of the lesions from a trained segmentation network. The feature selection module aims to select the most representative features by using a novel feature similarity adaptation algorithm. Experiments are extensively conducted to evaluate our method in two clinical tasks: the pathological grading prediction in pancreatic neuroendocrine neoplasms (pNENs), and the prediction of thrombolytic therapy efficacy in deep venous thrombosis (DVT). Experimental results on both tasks demonstrate that the proposed method consistently outperforms the state-of-the-art approaches by a large margin.

AGE challenge: Angle Closure Glaucoma Evaluation in Anterior Segment Optical Coherence Tomography.

Angle closure glaucoma (ACG) is a more aggressive disease than open-angle glaucoma, where the abnormal anatomical structures of the anterior chamber angle (ACA) may cause an elevated intraocular pressure and gradually lead to glaucomatous optic neuropathy and eventually to visual impairment and blindness. Anterior Segment Optical Coherence Tomography (AS-OCT) imaging provides a fast and contactless way to discriminate angle closure from open angle. Although many medical image analysis algorithms have been developed for glaucoma diagnosis, only a few studies have focused on AS-OCT imaging. In particular, there is no public AS-OCT dataset available for evaluating the existing methods in a uniform way, which limits progress in the development of automated techniques for angle closure detection and assessment. To address this, we organized the Angle closure Glaucoma Evaluation challenge (AGE), held in conjunction with MICCAI 2019. The AGE challenge consisted of two tasks: scleral spur localization and angle closure classification. For this challenge, we released a large dataset of 4800 annotated AS-OCT images from 199 patients, and also proposed an evaluation framework to benchmark and compare different models. During the AGE challenge, over 200 teams registered online, and more than 1100 results were submitted for online evaluation. Finally, eight teams participated in the onsite challenge. In this paper, we summarize these eight onsite challenge methods and analyze their corresponding results for the two tasks. We further discuss limitations and future directions. In the AGE challenge, the top-performing approach had an average Euclidean Distance of 10 pixels (10 µm) in scleral spur localization, while in the task of angle closure classification, all the algorithms achieved satisfactory performances, with two best obtaining an accuracy rate of 100%. These artificial intelligence techniques have the potential to promote new developments in AS-OCT image analysis and image-based angle closure glaucoma assessment in particular.

Uncertainty-aware domain alignment for anatomical structure segmentation.

Automatic and accurate segmentation of anatomical structures on medical images is crucial for detecting various potential diseases. However, the segmentation performance of established deep neural networks may degenerate on different modalities or devices owing to the significant difference across the domains, a problem known as domain shift. In this work, we propose an uncertainty-aware domain alignment framework to address the domain shift problem in the cross-domain Unsupervised Domain Adaptation (UDA) task. Specifically, we design an Uncertainty Estimation and Segmentation Module (UESM) to obtain the uncertainty map estimation. Then, a novel Uncertainty-aware Cross Entropy (UCE) loss is proposed to leverage the uncertainty information to boost the segmentation performance on highly uncertain regions. To further improve the performance in the UDA task, an Uncertainty-aware Self-Training (UST) strategy is developed to choose the optimal target samples by uncertainty guidance. In addition, the Uncertainty Feature Recalibration Module (UFRM) is applied to enforce the framework to minimize the cross-domain discrepancy. The proposed framework is evaluated on a private cross-device Optical Coherence Tomography (OCT) dataset and a public cross-modality cardiac dataset released by MMWHS 2017. Extensive experiments indicate that the proposed UESM is both efficient and effective for the uncertainty estimation in the UDA task, achieving state-of-the-art performance on both cross-modality and cross-device datasets.

Fully Automated Segmentation of Lower Extremity Deep Vein Thrombosis Using Convolutional Neural Network.

OBJECTIVE: Deep vein thrombosis (DVT) is a disease caused by abnormal blood clots in deep veins. Accurate segmentation of DVT is important to facilitate the diagnosis and treatment. In the current study, we proposed a fully automatic method of DVT delineation based on deep learning (DL) and contrast enhanced magnetic resonance imaging (CE-MRI) images. METHODS: 58 patients (25 males; 28~96 years old) with newly diagnosed lower extremity DVT were recruited. CE-MRI was acquired on a 1.5 T system. The ground truth (GT) of DVT lesions was manually contoured. A DL network with an encoder-decoder architecture was designed for DVT segmentation. 8-Fold cross-validation strategy was applied for training and testing. Dice similarity coefficient (DSC) was adopted to evaluate the network's performance. RESULTS: It took about 1.5s for our CNN model to perform the segmentation task in a slice of MRI image. The mean DSC of 58 patients was 0.74± 0.17 and the median DSC was 0.79. Compared with other DL models, our CNN model achieved better performance in DVT segmentation (0.74± 0.17 versus 0.66±0.15, 0.55±0.20, and 0.57±0.22). CONCLUSION: Our proposed DL method was effective and fast for fully automatic segmentation of lower extremity DVT.