Automatic classification of heart failure based on Cine-CMR images.

PURPOSE: Heart failure (HF) is a serious and complex syndrome with a high mortality rate. In clinical diagnosis, the correct classification of HF is helpful. In our previous work, we proposed a self-supervised learning framework of HF classification (SSLHF) on cine cardiac magnetic resonance images (Cine-CMR). However, this method lacks the integration of three dimensions of spatial information and temporal information. Thus, this study aims at proposing an automatic 4D HF classification algorithm. METHODS: To construct a 4D classification model, we proposed an extensional framework called 4D-SSLHF. It mainly consists of self-supervised image restoration and HF classification. The image restoration proxy task utilizes three image transformation methods to enhance the exploration of spatial and temporal information in the Cine-CMR. In the classification task, we proposed a Siamese Conv-LSTM network by combining the Siamese network and bi-directional Conv-LSTM to integrate the features of the four dimensions simultaneously. RESULTS: Experimental results on 184 patients from Shanghai Chest Hospital achieved an AUC of 0.8794 and an ACC of 0.8402 in the five-fold cross-validation. Compared with our previous work, the improvements in AUC and ACC were 2.89 % and 1.94 %, respectively. CONCLUSIONS: In this study, we proposed a novel self-supervised learning framework named 4D-SSLHF for HF classification based on Cine-CMR. The proposed 4D-SSLHF can mine 3D spatial information and temporal information in Cine-CMR images well and accurately classify different categories of HF. The good classification results show our method's potential to assist physicians in choosing personalized treatment.

Dosiomics and radiomics improve the prediction of post-radiotherapy neutrophil-lymphocyte ratio in locally advanced non-small cell lung cancer.

PURPOSE: To develop and validate a dosiomics and radiomics model based on three-dimensional (3D) dose distribution map and computed tomography (CT) images for the prediction of the post-radiotherapy (post-RT) neutrophil-to-lymphocyte ratio (NLR). METHODS: This work retrospectively collected 242 locally advanced non-small cell lung cancer (LA-NSCLC) patients who were treated with definitive radiotherapy from 2012 to 2016. The NLR collected one month after the completion of RT was defined as the primary outcome. Clinical characteristics and two-dimensional dosimetric factors calculated from the dose-volume histogram (DVH) were included. A total of 4165 dosiomics and radiomics features were extracted from the 3D dose maps and CT images within five different anatomical regions of interest (ROIs), respectively. Then, a three-step feature selection method was proposed to progressively filter features from coarse to fine: (i) model-based ranking according to individual feature's performance, (ii) maximum relevance and minimum redundancy (mRMR), (iii) select from model based on feature importance calculated with an ensemble of several decision trees. The selected feature subsets were utilized to develop the prediction model with GBDT. All patients were divided into a development set and an independent testing set (2:1). Five-fold cross-validation was applied to the development set for both feature selection and model training procedure. Finally, a fusion model combining dosiomics, radiomics and clinical features was constructed to further improve the prediction results. The area under receiver operating characteristic curve (ROC) were used to evaluate the model performance. RESULTS: The clinical-based and DVH-based models showed limited predictive power with AUCs of 0.632 (95% CI: 0.490-0.773) and 0.634 (95% CI: 0.497-0.771), respectively, in the independent testing set. The 9 feature-based dosiomics and 3 feature-based radiomics models showed improved AUCs of 0.738 (95% CI: 0.628-0.849) and 0.689 (95% CI: 0.566-0.813), respectively. The dosiomics & radiomics & clinical fusion model further improved the model's generalization ability with an AUC of 0.765 (95% CI: 0.656-0.874). CONCLUSIONS: Dosiomics and radiomics can benefit the prediction of post-RT NLR of LA-NSCLC patients. This can provide a reference for evaluating radiotherapy-related inflammation.

CoADS: Cross attention based dual-space graph network for survival prediction of lung cancer using whole slide images.

BACKGROUND AND OBJECTIVE: Accurate overall survival (OS) prediction for lung cancer patients is of great significance, which can help classify patients into different risk groups to benefit from personalized treatment. Histopathology slides are considered the gold standard for cancer diagnosis and prognosis, and many algorithms have been proposed to predict the OS risk. Most methods rely on selecting key patches or morphological phenotypes from whole slide images (WSIs). However, OS prediction using the existing methods exhibits limited accuracy and remains challenging. METHODS: In this paper, we propose a novel cross-attention-based dual-space graph convolutional neural network model (CoADS). To facilitate the improvement of survival prediction, we fully take into account the heterogeneity of tumor sections from different perspectives. CoADS utilizes the information from both physical and latent spaces. With the guidance of cross-attention, both the spatial proximity in physical space and the feature similarity in latent space between different patches from WSIs are integrated effectively. RESULTS: We evaluated our approach on two large lung cancer datasets of 1044 patients. The extensive experimental results demonstrated that the proposed model outperforms state-of-the-art methods with the highest concordance index. CONCLUSIONS: The qualitative and quantitative results show that the proposed method is more powerful for identifying the pathology features associated with prognosis. Furthermore, the proposed framework can be extended to other pathological images for predicting OS or other prognosis indicators, and thus delivering individualized treatment.

Hierarchical multimodal fusion framework based on noisy label learning and attention mechanism for cancer classification with pathology and genomic features.

Classification of subtype and grade is imperative in the clinical diagnosis and prognosis of cancer. Many deep learning-based studies related to cancer classification are based on pathology and genomics. However, most of them are late fusion-based and require full supervision in pathology image analysis. To address these problems, we present an integrated framework for cancer classification with pathology and genomics data. This framework consists of two major parts, a weakly supervised model for extracting patch features from whole slide images (WSIs), and a hierarchical multimodal fusion model. The weakly supervised model can make full use of WSI labels, and mitigate the effects of label noises by the self-training strategy. The generic multimodal fusion model is capable of capturing deep interaction information through multi-level attention mechanisms and controlling the expressiveness of each modal representation. We validate our approach on glioma and lung cancer datasets from The Cancer Genome Atlas (TCGA). The results demonstrate that the proposed method achieves superior performance compared to state-of-the-art methods, with the competitive AUC of 0.872 and 0.977 on these two datasets respectively. This paper establishes insight on how to build deep networks on multimodal biomedical data and proposes a more general framework for pathology image analysis without pixel-level annotation.

PUNDIT: Pulmonary nodule detection with image category transformation.

BACKGROUND: Convolutional neural networks (CNNs) have achieved great success in pulmonary nodules detection, which plays an important role in lung cancer screening. PURPOSE: In this paper, we proposed a novel strategy for pulmonary nodule detection by learning it from a harder task, which was to transform nodule images into normal images. We named this strategy as pulmonary nodule detection with image category transformation (PUNDIT). METHODS: There were two steps for nodules detection, nodule candidate detection and false positive (FP) reduction. In nodule candidate detection step, a segmentation-based framework was built for detection. We designed an image category transformation (ICT) task to translate nodule images into pixel-to-pixel normal images and share the information of detection and transformation tasks by multitask learning. As for references of transformation tasks, we proposed background consistency losses into standard cycle-consistent adversarial networks, which can solve the problem of background uncontrolled changing. A three-dimensional network was used in FP reduction step. RESULTS: PUNDIT was evaluated in two datasets, cancer screening dataset (CSD) with 1186 nodules for cross-validation and (CTD) with 3668 nodules for external test. Results were mainly evaluated by competition performance metric (CPM), the average sensitivity at seven predefined FP rates. The CPM was improved from 0.906 to 0.931 in CSD, and from 0.835 to 0.848 in CTD. CONCLUSIONS: Experimental results showed that PUNDIT can improve the performance of pulmonary nodules detection effectively.

A Self-supervised Learning Based Framework for Automatic Heart Failure Classification on Cine Cardiac Magnetic Resonance Image.

Heart failure (HF) is a serious syndrome, with high rates of mortality. Accurate classification of HF according to the left ventricular ejection faction (EF) plays an important role in the clinical treatment. Compared to echocardiography, cine cardiac magnetic resonance images (Cine-CMR) can estimate more accurate EF, whereas rare studies focus on the application of Cine-CMR. In this paper, a self-supervised learning framework for HF classification called SSLHF was proposed to automatically classify the HF patients into HF patients with preserved EF and HF patients with reduced EF based on Cine-CMR. In order to enable the classification network better learn the spatial and temporal information contained in the Cine-CMR, the SSLHF consists of two stages: self-supervised image restoration and HF classification. In the first stage, an image restoration proxy task was designed to help a U-Net like network mine the HF information in the spatial and temporal dimensions. In the second stage, a HF classification network whose weights were initialized by the encoder part of the U-Net like network was trained to complete the HF classification. Benefitting from the proxy task, the SSLHF achieved an AUC of 0.8505 and an ACC of 0.8208 in the 5-fold cross-validation.

Lung cancer subtype classification using histopathological images based on weakly supervised multi-instance learning.

Objective.Subtype classification plays a guiding role in the clinical diagnosis and treatment of non-small-cell lung cancer (NSCLC). However, due to the gigapixel of whole slide images (WSIs) and the absence of definitive morphological features, most automatic subtype classification methods for NSCLC require manually delineating the regions of interest (ROIs) on WSIs.Approach.In this paper, a weakly supervised framework is proposed for accurate subtype classification while freeing pathologists from pixel-level annotation. With respect to the characteristics of histopathological images, we design a two-stage structure with ROI localization and subtype classification. We first develop a method called multi-resolution expectation-maximization convolutional neural network (MR-EM-CNN) to locate ROIs for subsequent subtype classification. The EM algorithm is introduced to select the discriminative image patches for training a patch-wise network, with only WSI-wise labels available. A multi-resolution mechanism is designed for fine localization, similar to the coarse-to-fine process of manual pathological analysis. In the second stage, we build a novel hierarchical attention multi-scale network (HMS) for subtype classification. HMS can capture multi-scale features flexibly driven by the attention module and implement hierarchical features interaction.Results.Experimental results on the 1002-patient Cancer Genome Atlas dataset achieved an AUC of 0.9602 in the ROI localization and an AUC of 0.9671 for subtype classification.Significance.The proposed method shows superiority compared with other algorithms in the subtype classification of NSCLC. The proposed framework can also be extended to other classification tasks with WSIs.

Lymph Node Parameters Predict Adjuvant Chemoradiotherapy Efficacy and Disease-Free Survival in Pathologic N2 Non-Small Cell Lung Cancer.

Pathologic N2 non-small cell lung cancer (NSCLC) is prominently intrinsically heterogeneous. We aimed to identify homogeneous prognostic subgroups and evaluate the role of different adjuvant treatments. We retrospectively collected patients with resected pathologic T1-3N2M0 NSCLC from the Shanghai Chest Hospital as the primary cohort and randomly allocated them (3:1) to the training set and the validation set 1. We had patients from the Fudan University Shanghai Cancer Center as an external validation cohort (validation set 2) with the same inclusion and exclusion criteria. Variables significantly related to disease-free survival (DFS) were used to build an adaptive Elastic-Net Cox regression model. Nomogram was used to visualize the model. The discriminative and calibration abilities of the model were assessed by time-dependent area under the receiver operating characteristic curves (AUCs) and calibration curves. The primary cohort consisted of 1,312 patients. Tumor size, histology, grade, skip N2, involved N2 stations, lymph node ratio (LNR), and adjuvant treatment pattern were identified as significant variables associated with DFS and integrated into the adaptive Elastic-Net Cox regression model. A nomogram was developed to predict DFS. The model showed good discrimination (the median AUC in the validation set 1: 0.66, range 0.62 to 0.71; validation set 2: 0.66, range 0.61 to 0.73). We developed and validated a nomogram that contains multiple variables describing lymph node status (skip N2, involved N2 stations, and LNR) to predict the DFS of patients with resected pathologic N2 NSCLC. Through this model, we could identify a subtype of NSCLC with a more malignant clinical biological behavior and found that this subtype remained at high risk of disease recurrence after adjuvant chemoradiotherapy.

Detailed Analysis and Radiomic Prediction of First Progression Sites of First-Line Targeted Therapy for EGFR-Mutant Lung Adenocarcinoma Patients With Systemic Metastasis.

BACKGROUND: We aimed to analyze the first progression sites of first-line tyrosine kinase inhibitor (TKI) treatment for EGFR-mutant lung adenocarcinoma patients with systemic metastasis to recognize the potential candidates who might benefit from radiotherapy and establish a radiomic-based model to predict the first progression sites. MATERIALS AND METHODS: We retrospectively collected the clinical information and pre-treatment chest CT images of patients in Shanghai Chest Hospital from 2013 to 2017. All patients were diagnosed with stage IV EGFR-mutant lung adenocarcinoma and received TKI as first-line treatment. The first progression sites and survival were analyzed. The pre-treatment chest non-contrast CT images were utilized to establish a radiomic-based model to predict the first progression sites. RESULTS: We totally collected 233 patients with systemic metastasis, among whom, there were 84 (36.1%) and 149 (63.9%) patients developing first progression in original lesions (OP) and new lesions (NP), respectively. The PFS and OS of patients with OP were longer than those with NP (PFS 11 months vs. 8 months, p = 0.03, OS 50 months vs. 35 months, p = 0.046). For 67.9% of the patients with OF, disease progressed within five sites (oligoprogression). The radiomic-based model could predict the progression sites with an AUC value of 0.736, a specificity of 0.60, and a sensitivity of 0.750 in the independent validation set. CONCLUSION: Among patients with systemic metastasis, there were 36.1% of patients developing OP at first progression who had a better prognosis than those developing NP. Patients with OP may be potential candidates who might benefit from radiotherapy. Radiomics is a useful method to distinguish patients developing OP and could provide some indications for radiotherapy.

Predicting Tyrosine Kinase Inhibitor Treatment Response in Stage IV Lung Adenocarcinoma Patients With EGFR Mutation Using Model-Based Deep Transfer Learning.

BACKGROUND: For stage IV patients harboring EGFR mutations, there is a differential response to the first-line TKI treatment. We constructed three-dimensional convolutional neural networks (CNN) with deep transfer learning to stratify patients into subgroups with different response and progression risks. MATERIALS AND METHODS: From 2013 to 2017, 339 patients with EGFR mutation receiving first-line TKI treatment were included. Progression-free survival (PFS) time and progression patterns were confirmed by routine follow-up and restaging examinations. Patients were divided into two subgroups according to the median PFS (<=9 months, > 9 months). We developed a PFS prediction model and a progression pattern classification model using transfer learning from a pre-trained EGFR mutation classification 3D CNN. Clinical features were fused with the 3D CNN to build the final hybrid prediction model. The performance was quantified using area under receiver operating characteristic curve (AUC), and model performance was compared by AUCs with Delong test. RESULTS: The PFS prediction CNN showed an AUC of 0.744 (95% CI, 0.645-0.843) in the independent validation set and the hybrid model of CNNs and clinical features showed an AUC of 0.771 (95% CI, 0.676-0.866), which are significantly better than clinical features-based model (AUC, 0.624, P<0.01). The progression pattern prediction model showed an AUC of 0.762(95% CI, 0.643-0.882) and the hybrid model with clinical features showed an AUC of 0.794 (95% CI, 0.681-0.908), which can provide compensate information for clinical features-based model (AUC, 0.710; 95% CI, 0.582-0.839). CONCLUSION: The CNN exhibits potential ability to stratify progression status in patients with EGFR mutation treated with first-line TKI, which might help make clinical decisions.

Predicting Malignancy and Invasiveness of Pulmonary Subsolid Nodules on CT Images Using Deep Learning.

BACKGROUND: To develop and validate a deep learning-based model on CT images for the malignancy and invasiveness prediction of pulmonary subsolid nodules (SSNs). MATERIALS AND METHODS: This study retrospectively collected patients with pulmonary SSNs treated by surgery in our hospital from 2012 to 2018. Postoperative pathology was used as the diagnostic reference standard. Three-dimensional convolutional neural network (3D CNN) models were constructed using preoperative CT images to predict the malignancy and invasiveness of SSNs. Then, an observer reader study conducted by two thoracic radiologists was used to compare with the CNN model. The diagnostic power of the models was evaluated with receiver operating characteristic curve (ROC) analysis. RESULTS: A total of 2,614 patients were finally included and randomly divided for training (60.9%), validation (19.1%), and testing (20%). For the benign and malignant classification, the best 3D CNN model achieved a satisfactory AUC of 0.913 (95% CI: 0.885-0.940), sensitivity of 86.1%, and specificity of 83.8% at the optimal decision point, which outperformed all observer readers' performance (AUC: 0.846±0.031). For pre-invasive and invasive classification of malignant SSNs, the 3D CNN also achieved satisfactory AUC of 0.908 (95% CI: 0.877-0.939), sensitivity of 87.4%, and specificity of 80.8%. CONCLUSION: The deep-learning model showed its potential to accurately identify the malignancy and invasiveness of SSNs and thus can help surgeons make treatment decisions.

Prognostic index for estimating the survival benefit of postoperative radiotherapy in pathologic N2 non-small cell lung cancer: A real-world validation study.

OBJECTIVES: This study aimed to evaluate the effect of postoperative radiotherapy (PORT) in patients with resected pathologic N2 (pN2) non-small cell lung cancer (NSCLC) with different locoregional recurrence (LRR) risks. MATERIALS AND METHODS: The primary cohort and validation cohort were retrieved from two independent medical centres. Data for all consecutive patients with completely resected pathologic stage T1-3N2M0 NSCLC were analysed. Patients without PORT in the primary cohort were identified as a training set. Significant prognostic factors for LRR were identified by the Fine-Gray model to develop a prognostic index (PI) in the training set. RESULTS: The primary cohort consisted of 357 patients who met the eligibility criteria (training set, 287 patients without PORT). The external validation cohort consisted of 1044 patients who met the eligibility criteria (validation set, 711 patients without PORT). Heavy cigarette smoking history, clinical N2 status (cN2), and the number of positive lymph nodes >4 were identified as independent risk factors. The PI was computed as follows: PI＝0.8*smoking history＋0.5*cN2＋0.7*the number of involved lymph nodes (reference level was assigned the value 1 and risk level the value 2). In the low-risk group (PI score< = 3), PORT showed a trend towards decreased LRR rates but not significantly improved overall survival (OS). In the high-risk group (PI score>3), PORT significantly reduced the risk of LRR and improved OS. CONCLUSIONS: We constructed and validated a PI to predict individually the effect of PORT in patients with completely resected pN2 NSCLC. Patients with a higher PI score can benefit from PORT in terms of OS.

A Weak Supervision-based Framework for Automatic Lung Cancer Classification on Whole Slide Image.

Classification of normal lung tissue, lung adenocarcinoma (LUAD) and lung squamous cell carcinoma (LUSC) by pathological images is significant for clinical diagnosis and treatment. Due to the large scale of pathological images and the absence of definitive morphological features between LUAD and LUSC, it is time-consuming, laborious and challenging for pathologists to analyze the microscopic histopathology slides by visual observation. In this paper, a pixel-level annotation-free framework was proposed to classify normal tissue, LUAD and LUSC slides. This framework can be divided into two stages: tumor classification and localization, and subtype classification. In the first stage, EM-CNN was utilized to distinguish tumor slides from normal tissue slides and locate the discriminative regions for subsequent analysis with only image-level labels provided. In the second stage, a multi-scale network was proposed to improve the accuracy of subtype classification. This method achieved an AUC of 0.9978 for tumor classification and an AUC of 0.9684 for subtype classification, showing its superiority in lung pathological image classification compared with other methods.

Predictive model of the first failure pattern in patients receiving definitive chemoradiotherapy for inoperable locally advanced non-small cell lung cancer (LA-NSCLC).

PURPOSE: To analyze patterns of failure in patients with LA-NSCLC who received definitive chemoradiotherapy (CRT) and to build a nomogram for predicting the failure patterns in this population of patients. MATERIALS AND METHODS: Clinicopathological data of patients with LA-NSCLC who received definitive chemoradiotherapy and follow-up between 2013 and 2016 in our hospital were collected. The endpoint was the first failure after definitive chemoradiotherapy. With using elastic net regression and 5-fold nested cross-validation, the optimal model with better generalization ability was selected. Based on the selected model and corresponding features, a nomogram prediction model was built. This model was also validated by ROC curves, calibration curve and decision curve analysis (DCA). RESULTS: With a median follow-up of 28 months, 100 patients experienced failure. There were 46 and 54 patients who experience local failure and distant failure, respectively. Predictive model including 9 factors (smoking, pathology, location, EGFR mutation, age, tumor diameter, clinical N stage, consolidation chemotherapy and radiation dose) was finally built with the best performance. The average area under the ROC curve (AUC) with 5-fold nested cross-validation was 0.719, which was better than any factors alone. The calibration curve revealed a satisfactory consistency between the predicted distant failure rates and the actual observations. DCA showed most of the threshold probabilities in this model were with good net benefits. CONCLUSION: Clinicopathological factors could collaboratively predict failure patterns in patients with LA-NSCLC who are receiving definitive chemoradiotherapy. A nomogram was built and validated based on these factors, showing a potential predictive value in clinical practice.

Detection of epithelial growth factor receptor (EGFR) mutations on CT images of patients with lung adenocarcinoma using radiomics and/or multi-level residual convolutionary neural networks.

BACKGROUND: We aim to analyze the ability to detect epithelial growth factor receptor (EGFR) mutations on chest CT images of patients with lung adenocarcinoma using radiomics and/or multi-level residual convolutionary neural networks (MCNNs). METHODS: We retrospectively collected 1,010 consecutive patients in Shanghai Chest Hospital from 2013 to 2017, among which 510 patients were EGFR-mutated and 500 patients were wild-type. The patients were randomly divided into a training set (810 patients) and a validation set (200 patients) according to a balanced distribution of clinical features. The CT images and the corresponding EGFR status measured by Amplification Refractory Mutation System (ARMS) method of the patients in the training set were utilized to construct both a radiomics-based model (MRadiomics) and MCNNs-based model (MMCNNs). The MRadiomics and MMCNNs were combined to build the ModelRadiomics+MCNNs (MRadiomics+MCNNs). Clinical data of gender and smoking history constructed the clinical features-based model (MClinical). MClinical was then added into MRadiomics, MMCNNs, and MRadiomics+MCNNs to establish the ModelRadiomics+Clinical (MRadiomics+Clinical), the ModelMCNNs+Clinical (MMCNNs+Clinical) and the ModelRadiomics+MCNNs+Clinical (MRadiomics+MCNNs+Clinical). All the seven models were tested in the validation set to ascertain whether they were competent to detect EGFR mutations. The detection efficiency of each model was also compared in terms of area under the curve (AUC), sensitivity and specificity. RESULTS: The AUC of the MRadiomics, MMCNNs and MRadiomics+MCNNs to predict EGFR mutations was 0.740, 0.810 and 0.811 respectively. The performance of MMCNNs was better than that of MRadiomics (P=0.0225). The addition of clinical features did not improve the AUC of the MRadiomics (P=0.623), the MMCNNs (P=0.114) and the MRadiomics+MCNNs (P=0.058). The MRadiomics+MCNNs+Clinical demonstrated the highest AUC value of 0.834. The MMCNNs did not demonstrate any inferiority when compared with the MRadiomics+MCNNs (P=0.742) and the MRadiomics+MCNNs+Clinical (P=0.056). CONCLUSIONS: Both of the MRadiomics and the MCNNs could predict EGFR mutations on CT images of patients with lung adenocarcinoma. The MMCNNs outperformed the MRadiomics in the detection of EGFR mutations. The combination of these two models, even added with clinical features, is not significantly more efficient than MMCNNs alone.