BAF-Net: bidirectional attention-aware fluid pyramid feature integrated multimodal fusion network for diagnosis and prognosis.

Objective. To go beyond the deficiencies of the three conventional multimodal fusion strategies (i.e. input-, feature- and output-level fusion), we propose a bidirectional attention-aware fluid pyramid feature integrated fusion network (BAF-Net) with cross-modal interactions for multimodal medical image diagnosis and prognosis.Approach. BAF-Net is composed of two identical branches to preserve the unimodal features and one bidirectional attention-aware distillation stream to progressively assimilate cross-modal complements and to learn supplementary features in both bottom-up and top-down processes. Fluid pyramid connections were adopted to integrate the hierarchical features at different levels of the network, and channel-wise attention modules were exploited to mitigate cross-modal cross-level incompatibility. Furthermore, depth-wise separable convolution was introduced to fuse the cross-modal cross-level features to alleviate the increase in parameters to a great extent. The generalization abilities of BAF-Net were evaluated in terms of two clinical tasks: (1) an in-house PET-CT dataset with 174 patients for differentiation between lung cancer and pulmonary tuberculosis. (2) A public multicenter PET-CT head and neck cancer dataset with 800 patients from nine centers for overall survival prediction.Main results. On the LC-PTB dataset, improved performance was found in BAF-Net (AUC = 0.7342) compared with input-level fusion model (AUC = 0.6825;p< 0.05), feature-level fusion model (AUC = 0.6968;p= 0.0547), output-level fusion model (AUC = 0.7011;p< 0.05). On the H&N cancer dataset, BAF-Net (C-index = 0.7241) outperformed the input-, feature-, and output-level fusion model, with 2.95%, 3.77%, and 1.52% increments of C-index (p= 0.3336, 0.0479 and 0.2911, respectively). The ablation experiments demonstrated the effectiveness of all the designed modules regarding all the evaluated metrics in both datasets.Significance. Extensive experiments on two datasets demonstrated better performance and robustness of BAF-Net than three conventional fusion strategies and PET or CT unimodal network in terms of diagnosis and prognosis.

Optimal batch determination for improved harmonization and prognostication of multi-center PET/CT radiomics feature in head and neck cancer.

Objective. To determine the optimal approach for identifying and mitigating batch effects in PET/CT radiomics features, and further improve the prognosis of patients with head and neck cancer (HNC), this study investigated the performance of three batch harmonization methods.Approach. Unsupervised harmonization identified the batch labels by K-means clustering. Supervised harmonization regarding the image acquisition factors (center, manufacturer, scanner, filter kernel) as known/given batch labels, and Combat harmonization was then implemented separately and sequentially based on the batch labels, i.e. harmonizing features among batches determined by each factor individually or harmonizing features among batches determined by multiple factors successively. Extensive experiments were conducted to predict overall survival (OS) on public PET/CT datasets that contain 800 patients from 9 centers.Main results. In the external validation cohort, results show that compared to original models without harmonization, Combat harmonization would be beneficial in OS prediction with C-index of 0.687-0.740 versus 0.684-0.767. Supervised harmonization slightly outperformed unsupervised harmonization in all models (C-index: 0.692-0.767 versus 0.684-0.750). Separate harmonization outperformed sequential harmonization in CT_m+clinic and CT_cm+clinic models with C-index of 0.752 and 0.722, respectively, while sequential harmonization involved clinical features in PET_rs+clinic model further improving the performance and achieving the highest C-index of 0.767.Significance. Optimal batch determination especially sequential harmonization for Combat holds the potential to improve the prognostic power of radiomics model in multi-center HNC dataset with PET/CT imaging.

Deep learning PET/CT-based radiomics integrates clinical data: A feasibility study to distinguish between tuberculosis nodules and lung cancer.

BACKGROUND: Radiomic diagnosis models generally consider only a single dimension of information, leading to limitations in their diagnostic accuracy and reliability. The integration of multiple dimensions of information into the deep learning model have the potential to improve its diagnostic capabilities. The purpose of study was to evaluate the performance of deep learning model in distinguishing tuberculosis (TB) nodules and lung cancer (LC) based on deep learning features, radiomic features, and clinical information. METHODS: Positron emission tomography (PET) and computed tomography (CT) image data from 97 patients with LC and 77 patients with TB nodules were collected. One hundred radiomic features were extracted from both PET and CT imaging using the pyradiomics platform, and 2048 deep learning features were obtained through a residual neural network approach. Four models included traditional machine learning model with radiomic features as input (traditional radiomics), a deep learning model with separate input of image features (deep convolutional neural networks [DCNN]), a deep learning model with two inputs of radiomic features and deep learning features (radiomics-DCNN) and a deep learning model with inputs of radiomic features and deep learning features and clinical information (integrated model). The models were evaluated using area under the curve (AUC), sensitivity, accuracy, specificity, and F1-score metrics. RESULTS: The results of the classification of TB nodules and LC showed that the integrated model achieved an AUC of 0.84 (0.82-0.88), sensitivity of 0.85 (0.80-0.88), and specificity of 0.84 (0.83-0.87), performing better than the other models. CONCLUSION: The integrated model was found to be the best classification model in the diagnosis of TB nodules and solid LC.

Deep learning-based harmonization of CT reconstruction kernels towards improved clinical task performance.

OBJECTIVES: To develop a deep learning-based harmonization framework, assessing whether it can improve performance of radiomics models given different kernels in different clinical tasks and additionally generalize to mitigate the effects of new/unobserved kernels on radiomics features. METHODS: Patient data with 2 reconstruction kernels and phantom data with 22 reconstruction kernels were included. Eighty-five patients were studied for lymph node metastasis (LNM) prediction, and 164 patients for differential diagnosis between lung cancer (LC) and pulmonary tuberculosis (TB). Two convolutional neural network (CNN) models were developed to convert images (i) from B70f to B30f (CNNa) and (ii) from B30f to B70f (CNNb). Model performance between the two kernels was evaluated using AUC and compared with other well-known harmonization methods. Patient-normalized feature difference (PNFD) was used to identify the incompatible kernels (i.e., kernel with median PNFD > 1) with baseline (B30f/B70f), and measure the ability of the CNN models to convert the non-comparable kernels. RESULTS: For LC versus pulmonary TB diagnosis, AUCs of CNNa vs. others were 0.85 vs. 0.54-0.74 (p = 0.0001-0.0003), and for CNNb vs. others: 0.87 vs. 0.54-0.86 (p = 0.0001-0.55). For LNM prediction, AUCs of CNNa vs. others were 0.68 vs. 0.56-0.61 (p = 0.10-0.39), and for CNNb vs. others: 0.78 vs. 0.70-0.73 (p = 0.07-0.40). After CNN harmonization, 17 of 20 (85%) of investigated unknown kernels produced comparable radiomics feature values relative to baseline (median PNFD from 1.10-2.31 to 0.23-1.13). CONCLUSION: The CNN harmonization effectively improved performance of radiomics models between reconstruction kernels in different clinical tasks, and reduced feature differences between unknown kernels vs. baseline. KEY POINTS: • The soft (B30f) and sharp (B70f) kernels strongly affect radiomics reproducibility and generalizability. • The convolutional neural network (CNN) harmonization methods performed better than location-scale (ComBat and centering-scaling) and matrix factorization harmonization methods (based on singular value decomposition (SVD) and independent component analysis (ICA)) in both clinical tasks. • The CNN harmonization methods improve feature reproducibility not only between specific kernels (B30f and B70f) from the same scanner, but also between unobserved kernels from different scanners of different vendors.

Imbalanced Data Correction Based PET/CT Radiomics Model for Predicting Lymph Node Metastasis in Clinical Stage T1 Lung Adenocarcinoma.

OBJECTIVES: To develop and validate the imbalanced data correction based PET/CT radiomics model for predicting lymph node metastasis (LNM) in clinical stage T1 lung adenocarcinoma (LUAD). METHODS: A total of 183 patients (148/35 non-metastasis/LNM) with pathologically confirmed LUAD were retrospectively included. The cohorts were divided into training vs. validation cohort in a ratio of 7:3. A total of 487 radiomics features were extracted from PET and CT components separately for radiomics model construction. Four clinical features and seven PET/CT radiological features were extracted for traditional model construction. To balance the distribution of majority (non-metastasis) class and minority (LNM) class, the imbalance-adjustment strategies using ten data re-sampling methods were adopted. Three multivariate models (denoted as Traditional, Radiomics, and Combined) were constructed using multivariable logistic regression analysis, where the combined model incorporated all of the significant clinical, radiological, and radiomics features. One hundred times repeated Monte Carlo cross-validation was used to assess the application order of feature selection and imbalance-adjustment strategies in the machine learning pipeline. Prediction performance of each model was evaluated using the area under the receiver operating characteristic curve (AUC) and Geometric mean score (G-mean). RESULTS: A total of 2 clinical parameters, 2 radiological features, 3 PET, and 5 CT radiomics features were significantly associated with LNM. The combined model with Edited Nearest Neighbors (ENN) re-sampling methods showed strong prediction performance than traditional model or radiomics model with the AUC of 0.94 (95%CI = 0.86-0.97) vs. 0.89 (95%CI = 0.79-0.93), 0.92 (95%CI = 0.85-0.97), and G-mean of 0.88 vs. 0.82, 0.80 in the training cohort, and the AUC of 0.75 (95%CI = 0.57-0.91) vs. 0.68 (95%CI = 0.36-0.83), 0.71 (95%CI = 0.48-0.83) and G-mean of 0.76 vs. 0.64, 0.51 in the validation cohort. The combination of performing feature selection before data re-sampling obtains a better result than the reverse combination (AUC 0.76 ± 0.06 vs. 0.70 ± 0.07, p<0.001). CONCLUSIONS: The combined model (consisting of age, histological type, C/T ratio, MATV, and radiomics signature) integrated with ENN re-sampling methods had strong lymph node metastasis prediction performance for imbalance cohorts in clinical stage T1 LUAD. Radiomics signatures extracted from PET/CT images could provide complementary prediction information compared with traditional model.

Identification of Stage IIIC/IV EGFR-Mutated Non-Small Cell Lung Cancer Populations Sensitive to Targeted Therapy Based on a PET/CT Radiomics Risk Model.

OBJECTIVES: This project aimed to construct an individualized PET/CT prognostic biomarker to accurately quantify the progression risk of patients with stage IIIC-IV epidermal growth factor receptor (EGFR)-mutated Non-small cell lung cancer (NSCLC) after first-line first and second generation EGFR- tyrosine kinase inhibitor (TKI) drug therapy and identify the first and second generation EGFR-TKI treatment-sensitive population. METHODS: A total of 250 patients with stage IIIC-IV EGFR-mutated NSCLC underwent first-line first and second generation EGFR-TKI drug therapy were included from two institutions (140 patients in training cohort; 60 patients in internal validation cohort, and 50 patients in external validation cohort). 1037 3D radiomics features were extracted to quantify the phenotypic characteristics of the tumor region in PET and CT images, respectively. A four-step feature selection method was performed to enable derivation of stable and effective signature in the training cohort. According to the median value of radiomics signature score (Rad-score), patients were divided into low- and high-risk groups. The progression-free survival (PFS) behaviors of the two subgroups were compared by Kaplan-Meier survival analysis. RESULTS: Our results shown that higher Rad-scores were significantly associated with worse PFS in the training (p < 0.0001), internal validation (p = 0.0153), and external validation (p = 0.0006) cohorts. Rad-score can effectively identify patients with a high risk of rapid progression. The Kaplan-Meier survival curves of the three cohorts present significant differences in PFS between the stratified slow and rapid progression subgroups. CONCLUSION: The PET/CT-derived Rad-score can realize the precise quantitative stratification of progression risk after first-line first and second generation EGFR-TKI drug therapy for NSCLC and identify EGFR-mutated NSCLC populations sensitive to targeted therapy, which might help to provide precise treatment options for NSCLC.

Integration of PET/CT Radiomics and Semantic Features for Differentiation between Active Pulmonary Tuberculosis and Lung Cancer.

PURPOSE: We aim to accurately differentiate between active pulmonary tuberculosis (TB) and lung cancer (LC) based on radiomics and semantic features as extracted from pre-treatment positron emission tomography/X-ray computed tomography (PET/CT) images. PROCEDURES: A total of 174 patients (77/97 pulmonary TB/LC as confirmed by pathology) were retrospectively selected, with 122 in the training cohort and 52 in the validation cohort. Four hundred eighty-seven radiomics features were initially extracted to quantify phenotypic characteristics of the lesion region in both PET and CT images. Eleven semantic features were additionally defined by two experienced nuclear medicine physicians. Feature selection was performed in 5 steps to enable derivation of robust and effective signatures. Multivariable logistic regression analysis was subsequently used to develop a radiomics nomogram. The calibration, discrimination, and clinical usefulness of the nomogram were evaluated in both the training and independent validation cohorts. RESULTS: The individualized radiomics nomogram, which combined PET/CT radiomics signature with semantic features, demonstrated good calibration and significantly improved the diagnostic performance with respect to the semantic model alone or PET/CT signature alone in training cohort (AUC 0.97 vs. 0.94 or 0.91, p = 0.0392 or 0.0056), whereas did not significantly improve the performance in validation cohort (AUC 0.93 vs. 0.89 or 0.91, p = 0.3098 or 0.3323). CONCLUSION: The radiomics nomogram showed potential for individualized differential diagnosis between solid active pulmonary TB and solid LC, although the improvement of performance was not significant relative to semantic model.

Evaluation of the diagnostic value of joint PET myocardial perfusion and metabolic imaging for vascular stenosis in patients with obstructive coronary artery disease.

BACKGROUND: To investigate the diagnostic value of joint PET myocardial perfusion and metabolic imaging for vascular stenosis in patients with suspected obstructive coronary artery disease (CAD). METHODS: Eighty-eight patients (53 and 35 applied for training and validation, respectively) with suspected obstructive CAD were referred to 13N-NH3 PET/CT myocardial perfusion imaging (MPI) and 18F-FDG PET/CT myocardial metabolic imaging (MMI) with available coronary angiography for analysis. One semi-quantitative indicator summed rest score (SRS) and five quantitative indicators, namely, perfusion defect extent (EXT), total perfusion deficit (TPD), myocardial blood flow (MBF), scar degree (SCR), and metabolism-perfusion mismatch (MIS), were extracted from the PET rest MPI and MMI scans. Different combinations of indicators and seven machine learning methods were used to construct diagnostic models. Diagnostic performance was evaluated using the sum of four metrics (noted as sumScore), namely, area under the receiver operating characteristic curve (AUC), accuracy, sensitivity, and specificity. RESULTS: In univariate analysis, MIS outperformed other individual indicators in terms of sumScore (2.816-3.042 vs 2.138-2.908). In multivariate analysis, support vector machine (SVM) consisting of three indicators (MBF, SCR, and MIS) achieved the best performance (AUC 0.856, accuracy 0.810, sensitivity 0.838, specificity 0.757, and sumScore 3.261). This model consistently achieved significantly higher AUC compared with the SRS method for four specific subgroups (0.897, 0.839, 0.875, and 0.949 vs 0.775, 0.606, 0.713, and 0.744; P = 0.041, 0.005, 0.034 0.003, respectively). CONCLUSIONS: The joint evaluation of PET rest MPI and MMI could improve the diagnostic performance for obstructive CAD. The multivariate model (MBF, SCR, and MIS) combined with SVM outperformed other methods.

Machine Learning Methods for Optimal Radiomics-Based Differentiation Between Recurrence and Inflammation: Application to Nasopharyngeal Carcinoma Post-therapy PET/CT Images.

PURPOSE: To identify optimal machine learning methods for radiomics-based differentiation of local recurrence versus inflammation from post-treatment nasopharyngeal positron emission tomography/X-ray computed tomography (PET/CT) images. PROCEDURES: Seventy-six nasopharyngeal carcinoma (NPC) patients were enrolled (41/35 local recurrence/inflammation as confirmed by pathology). Four hundred eighty-seven radiomics features were extracted from PET images for each patient. The diagnostic performance was investigated for 42 cross-combinations derived from 6 feature selection methods and 7 classifiers. Of the original cohort, 70 % was applied for feature selection and classifier development, and the remaining 30 % used as an independent validation set. The diagnostic performance was evaluated using area under the ROC curve (AUC), test error, sensitivity, and specificity. Furthermore, the performance of the radiomics signatures against routine features was statistically compared using DeLong's method. RESULTS: The cross-combination fisher score (FSCR) + k-nearest neighborhood (KNN), FSCR + support vector machines with radial basis function kernel (RBF-SVM), FSCR + random forest (RF), and minimum redundancy maximum relevance (MRMR) + RBF-SVM outperformed others in terms of accuracy (AUC 0.883, 0.867, 0.892, 0.883; sensitivity 0.833, 0.864, 0.831, 0.750; specificity 1, 1, 0.873, 1) and reliability (test error 0.091, 0.136, 0.150, 0.136). Compared with conventional metrics, the radiomics signatures showed higher AUC values (0.867-0.892 vs. 0.817), though the differences were not statistically significant (p = 0.462-0.560). CONCLUSION: This study identified the most accurate and reliable machine learning methods, which could enhance the application of radiomics methods in the precision of diagnosis of NPC.

Subregional Radiomics Analysis of PET/CT Imaging with Intratumor Partitioning: Application to Prognosis for Nasopharyngeal Carcinoma.

PURPOSE: This work aims to identify intratumoral habitats with distinct heterogeneity based on 2-deoxy-2-[18F]fluro-D-glucose positron emission tomography (PET)/X-ray computed tomography (CT) imaging, and to develop a subregional radiomics approach to predict progression-free survival (PFS) in patients with nasopharyngeal carcinoma (NPC). PROCEDURES: In total, 128 NPC patients (85 vs. 43 for primary vs. validation cohorts) who underwent pre-treatment PET/CT scan were enrolled retrospectively. Each tumor was partitioned into several phenotypically consistent subregions based on individual- and population-level clustering. For each subregion, 202 radiomics features were extracted to construct imaging biomarker for prognosis via Cox's proportional hazard model combined with forward stepwise feature selection. Relevance of imaging biomarkers and clinicopathological factors were assessed by multivariate Cox regression analysis and Spearman's correlation analysis. To investigate whether imaging biomarkers could provide complementary prognosis information beyond existing predictors, a scoring system was further developed for risk stratification and compared with AJCC staging system. RESULTS: Three subregions (denoted as S1, S2, and S3) were discovered with distinct PET/CT imaging characteristics in the two cohorts. The prognostic performance of imaging biomarker S3 outperformed the whole tumor (C-index, 0.69 vs. 0.58; log-rank test, p < 0.001 vs. p = 0.552). Imaging biomarker S3 and AJCC stage were identified as independent predictors (p = 0.011 and 0.042, respectively) after adjusting for clinicopathological factors. The scoring system outperformed the traditional AJCC staging system (log-rank test, p < 0.0001 vs. p = 0.0002 in primary cohort and p = 0.0021 vs. p = 0.0277 in validation cohort, respectively). CONCLUSIONS: Subregional radiomics analysis of PET/CT imaging has the potential to predict PFS in patients with NPC, which also provides complementary prognostic information for traditional predictors.

[Palm vein recognition based on end-to-end convolutional neural network].

OBJECTIVE: We propose a novel palm-vein recognition model based on the end-to-end convolutional neural network. In this model, the convolutional layer and the pooling layer were alternately connected to extract the image features, and the categorical attribute was estimated simultaneously via the neural network classifier. The classification error was minimized via the mini-batch stochastic gradient descent algorithm with momentum to optimize the feature descriptor along with the direction of the gradient descent. Four strategies including data augmentation, batch normalization, dropout, and L2 parameter regularization were applied in the model to reduce the generalization error. The experimental results showed that for classifying 500 subjects form PolyU database and a self-established database, this model achieved identification rates of 99.90% and 98.05%, respectively, with an identification time for a single sample less than 9 ms. The proposed approach, as compared with the traditional method, could improve the accuracy of palm vein recognition in clincal applications and provides a new approach to palm vein recognition.