Age-related Muscle Fat Infiltration in Lung Screening Participants: Impact of Smoking Cessation.

Rationale: Skeletal muscle fat infiltration progresses with aging and is worsened among individuals with a history of cigarette smoking. Many negative impacts of smoking on muscles are likely reversible with smoking cessation. Objectives: To determine if the progression of skeletal muscle fat infiltration with aging is altered by smoking cessation among lung cancer screening participants. Methods: This was a secondary analysis based on the National Lung Screening Trial. Skeletal muscle attenuation in Hounsfield unit (HU) was derived from the baseline and follow-up low-dose CT scans using a previously validated artificial intelligence algorithm. Lower attenuation indicates greater fatty infiltration. Linear mixed-effects models were constructed to evaluate the associations between smoking status and the muscle attenuation trajectory. Measurements and Main Results: Of 19,019 included participants (age: 61 years, 5 [SD]; 11,290 males), 8,971 (47.2%) were actively smoking cigarettes. Accounting for body mass index, pack-years, percent emphysema, and other confounding factors, actively smoking predicted a lower attenuation in both males (ß0 =-0.88 HU, P<.001) and females (ß0 =-0.69 HU, P<.001), and an accelerated muscle attenuation decline-rate in males (ß1=-0.08 HU/y, P<.05). Age-stratified analyses indicated that the accelerated muscle attenuation decline associated with smoking likely occurred at younger age, especially in females. Conclusions: Among lung cancer screening participants, active cigarette smoking was associated with greater skeletal muscle fat infiltration in both males and females, and accelerated muscle adipose accumulation rate in males. These findings support the important role of smoking cessation in preserving muscle health.

UNesT: Local spatial representation learning with hierarchical transformer for efficient medical segmentation.

Transformer-based models, capable of learning better global dependencies, have recently demonstrated exceptional representation learning capabilities in computer vision and medical image analysis. Transformer reformats the image into separate patches and realizes global communication via the self-attention mechanism. However, positional information between patches is hard to preserve in such 1D sequences, and loss of it can lead to sub-optimal performance when dealing with large amounts of heterogeneous tissues of various sizes in 3D medical image segmentation. Additionally, current methods are not robust and efficient for heavy-duty medical segmentation tasks such as predicting a large number of tissue classes or modeling globally inter-connected tissue structures. To address such challenges and inspired by the nested hierarchical structures in vision transformer, we proposed a novel 3D medical image segmentation method (UNesT), employing a simplified and faster-converging transformer encoder design that achieves local communication among spatially adjacent patch sequences by aggregating them hierarchically. We extensively validate our method on multiple challenging datasets, consisting of multiple modalities, anatomies, and a wide range of tissue classes, including 133 structures in the brain, 14 organs in the abdomen, 4 hierarchical components in the kidneys, inter-connected kidney tumors and brain tumors. We show that UNesT consistently achieves state-of-the-art performance and evaluate its generalizability and data efficiency. Particularly, the model achieves whole brain segmentation task complete ROI with 133 tissue classes in a single network, outperforming prior state-of-the-art method SLANT27 ensembled with 27 networks. Our model performance increases the mean DSC score of the publicly available Colin and CANDI dataset from 0.7264 to 0.7444 and from 0.6968 to 0.7025, respectively. Code, pre-trained models, and use case pipeline are available at: https://github.com/MASILab/UNesT.

Stratification of Lung Cancer Risk with Thoracic Imaging Phenotypes.

In lung cancer screening, estimation of future lung cancer risk is usually guided by demographics and smoking status. The role of constitutional profiles of human body, a.k.a. body habitus, is increasingly understood to be important, but has not been integrated into risk models. Chest low dose computed tomography (LDCT) is the standard imaging study in lung cancer screening, with the capability to discriminate differences in body composition and organ arrangement in the thorax. We hypothesize that the primary phenotypes identified using lung screening chest LDCT can form a representation of body habitus and add predictive power for lung cancer risk stratification. In this pilot study, we evaluated the feasibility of body habitus image-based phenotyping on a large lung screening LDCT dataset. A thoracic imaging manifold was estimated based on an intensity-based pairwise (dis)similarity metric for pairs of spatial normalized chest LDCT images. We applied the hierarchical clustering method on this manifold to identify the primary phenotypes. Body habitus features of each identified phenotype were evaluated and associated with future lung cancer risk using time-to-event analysis. We evaluated the method on the baseline LDCT scans of 1,200 male subjects sampled from National Lung Screening Trial. Five primary phenotypes were identified, which were associated with highly distinguishable clinical and body habitus features. Time-to-event analysis against future lung cancer incidences showed two of the five identified phenotypes were associated with elevated future lung cancer risks (HR=1.61, 95% CI = [1.08, 2.38], p=0.019; HR=1.67, 95% CI = [0.98, 2.86], p=0.057). These results indicated that it is feasible to capture the body habitus by image-base phenotyping using lung screening LDCT and the learned body habitus representation can potentially add value for future lung cancer risk stratification.

AI Body Composition in Lung Cancer Screening: Added Value Beyond Lung Cancer Detection.

Background An artificial intelligence (AI) algorithm has been developed for fully automated body composition assessment of lung cancer screening noncontrast low-dose CT of the chest (LDCT) scans, but the utility of these measurements in disease risk prediction models has not been assessed. Purpose To evaluate the added value of CT-based AI-derived body composition measurements in risk prediction of lung cancer incidence, lung cancer death, cardiovascular disease (CVD) death, and all-cause mortality in the National Lung Screening Trial (NLST). Materials and Methods In this secondary analysis of the NLST, body composition measurements, including area and attenuation attributes of skeletal muscle and subcutaneous adipose tissue, were derived from baseline LDCT examinations by using a previously developed AI algorithm. The added value of these measurements was assessed with sex- and cause-specific Cox proportional hazards models with and without the AI-derived body composition measurements for predicting lung cancer incidence, lung cancer death, CVD death, and all-cause mortality. Models were adjusted for confounding variables including age; body mass index; quantitative emphysema; coronary artery calcification; history of diabetes, heart disease, hypertension, and stroke; and other PLCOM2012 lung cancer risk factors. Goodness-of-fit improvements were assessed with the likelihood ratio test. Results Among 20 768 included participants (median age, 61 years [IQR, 57-65 years]; 12 317 men), 865 were diagnosed with lung cancer and 4180 died during follow-up. Including the AI-derived body composition measurements improved risk prediction for lung cancer death (male participants: χ2 = 23.09, P < .001; female participants: χ2 = 15.04, P = .002), CVD death (males: χ2 = 69.94, P < .001; females: χ2 = 16.60, P < .001), and all-cause mortality (males: χ2 = 248.13, P < .001; females: χ2 = 94.54, P < .001), but not for lung cancer incidence (male participants: χ2 = 2.53, P = .11; female participants: χ2 = 1.73, P = .19). Conclusion The body composition measurements automatically derived from baseline low-dose CT examinations added predictive value for lung cancer death, CVD death, and all-cause death, but not for lung cancer incidence in the NLST. Clinical trial registration no. NCT00047385 © RSNA, 2023 Supplemental material is available for this article. See also the editorial by Fintelmann in this issue.

Predicting Crohn's disease severity in the colon using mixed cell nucleus density from pseudo labels.

Crohn's disease (CD) is a debilitating inflammatory bowel disease with no known cure. Computational analysis of hematoxylin and eosin (H&E) stained colon biopsy whole slide images (WSIs) from CD patients provides the opportunity to discover unknown and complex relationships between tissue cellular features and disease severity. While there have been works using cell nuclei-derived features for predicting slide-level traits, this has not been performed on CD H&E WSIs for classifying normal tissue from CD patients vs active CD and assessing slide label-predictive performance while using both separate and combined information from pseudo-segmentation labels of nuclei from neutrophils, eosinophils, epithelial cells, lymphocytes, plasma cells, and connective cells. We used 413 WSIs of CD patient biopsies and calculated normalized histograms of nucleus density for the six cell classes for each WSI. We used a support vector machine to classify the truncated singular value decomposition representations of the normalized histograms as normal or active CD with four-fold cross-validation in rounds where nucleus types were first compared individually, the best was selected, and further types were added each round. We found that neutrophils were the most predictive individual nucleus type, with an AUC of 0.92 ± 0.0003 on the withheld test set. Adding information improved cross-validation performance for the first two rounds and on the withheld test set for the first three rounds, though performance metrics did not increase substantially beyond when neutrophils were used alone.

Deep multimodal fusion of image and non-image data in disease diagnosis and prognosis: a review.

The rapid development of diagnostic technologies in healthcare is leading to higher requirements for physicians to handle and integrate the heterogeneous, yet complementary data that are produced during routine practice. For instance, the personalized diagnosis and treatment planning for a single cancer patient relies on various images (e.g. radiology, pathology and camera images) and non-image data (e.g. clinical data and genomic data). However, such decision-making procedures can be subjective, qualitative, and have large inter-subject variabilities. With the recent advances in multimodal deep learning technologies, an increasingly large number of efforts have been devoted to a key question: how do we extract and aggregate multimodal information to ultimately provide more objective, quantitative computer-aided clinical decision making? This paper reviews the recent studies on dealing with such a question. Briefly, this review will include the (a) overview of current multimodal learning workflows, (b) summarization of multimodal fusion methods, (c) discussion of the performance, (d) applications in disease diagnosis and prognosis, and (e) challenges and future directions.

Characterizing browser-based medical imaging AI with serverless edge computing: towards addressing clinical data security constraints.

Artificial intelligence (AI) has been widely introduced to various medical imaging applications ranging from disease visualization to medical decision support. However, data privacy has become an essential concern in clinical practice of deploying the deep learning algorithms through cloud computing. The sensitivity of patient health information (PHI) commonly limits network transfer, installation of bespoke desktop software, and access to computing resources. Serverless edge-computing shed light on privacy preserved model distribution maintaining both high flexibility (as cloud computing) and security (as local deployment). In this paper, we propose a browser-based, cross-platform, and privacy preserved medical imaging AI deployment system working on consumer-level hardware via serverless edge-computing. Briefly we implement this system by deploying a 3D medical image segmentation model for computed tomography (CT) based lung cancer screening. We further curate tradeoffs in model complexity and data size by characterizing the speed, memory usage, and limitations across various operating systems and browsers. Our implementation achieves a deployment with (1) a 3D convolutional neural network (CNN) on CT volumes (256×256×256 resolution), (2) an average runtime of 80 seconds across Firefox v.102.0.1/Chrome v.103.0.5060.114/Microsoft Edge v.103.0.1264.44 and 210 seconds on Safari v.14.1.1, and (3) an average memory usage of 1.5 GB on Microsoft Windows laptops, Linux workstation, and Apple Mac laptops. In conclusion, this work presents a privacy-preserved solution for medical imaging AI applications that minimizes the risk of PHI exposure. We characterize the tools, architectures, and parameters of our framework to facilitate the translation of modern deep learning methods into routine clinical care.

PI3K Inhibition Restores and Amplifies Response to Ruxolitinib in Patients with Myelofibrosis.

PURPOSE: Treatment options are limited beyond JAK inhibitors for patients with primary myelofibrosis (MF) or secondary MF. Preclinical studies have revealed that PI3Kδ inhibition cooperates with ruxolitinib, a JAK1/2 inhibitor, to reduce proliferation and induce apoptosis of JAK2V617F-mutant cell lines. PATIENTS AND METHODS: In a phase I dose-escalation and -expansion study, we evaluated the safety and efficacy of a selective PI3Kδ inhibitor, umbralisib, in combination with ruxolitinib in patients with MF who had a suboptimal response or lost response to ruxolitinib. Enrolled subjects were required to be on a stable dose of ruxolitinib for ≥8 weeks and continue that MTD at study enrollment. The recommended dose of umbralisib in combination with ruxolitinib was determined using a modified 3+3 dose-escalation design. Safety, pharmacokinetics, and efficacy outcomes were evaluated, and spleen size was measured with a novel automated digital atlas. RESULTS: Thirty-seven patients with MF (median age, 67 years) with prior exposure to ruxolitinib were enrolled. A total of 2 patients treated with 800 mg umbralisib experienced reversible grade 3 asymptomatic pancreatic enzyme elevation, but no dose-limiting toxicities were seen at lower umbralisib doses. Two patients (5%) achieved a durable complete response, and 12 patients (32%) met the International Working Group-Myeloproliferative Neoplasms Research and Treatment response criteria of clinical improvement. With a median follow-up of 50.3 months for censored patients, overall survival was greater than 70% after 3 years of follow-up. CONCLUSIONS: Adding umbralisib to ruxolitinib in patients was well tolerated and may resensitize patients with MF to ruxolitinib without unacceptable rates of adverse events seen with earlier generation PI3Kδ inhibitors. Randomized trials testing umbralisib in the treatment of MF should be pursued.

Recent Advances in Melanoma Diagnosis and Prognosis Using Machine Learning Methods.

PURPOSE OF REVIEW: The purpose was to summarize the current role and state of artificial intelligence and machine learning in the diagnosis and management of melanoma. RECENT FINDINGS: Deep learning algorithms can identify melanoma from clinical, dermoscopic, and whole slide pathology images with increasing accuracy. Efforts to provide more granular annotation to datasets and to identify new predictors are ongoing. There have been many incremental advances in both melanoma diagnostics and prognostic tools using artificial intelligence and machine learning. Higher quality input data will further improve these models' capabilities.

The association between hepatic steatosis and incident cardiovascular disease, cancer, and all-cause mortality in a US multicohort study.

BACKGROUND AND AIMS: NAFLD strongly associates with cardiovascular disease (CVD) risk factors; however, the association between NAFLD and incident CVD, CVD-related mortality, incident cancer, and all-cause mortality is unclear. APPROACH AND RESULTS: We included 10,040 participants from the Framingham Heart Study, the Coronary Artery Risk Development in Young Adults Study, and the Multi-ethnic Study of Atherosclerosis to assess the longitudinal association between liver fat (defined on CT) and incident CVD, CVD-related mortality, incident cancer, and all-cause mortality. We performed multivariable-adjusted Cox regression models including age, sex, diabetes, systolic blood pressure, alcohol use, smoking, HDL, triglycerides, and body mass index at baseline or time-varying covariates. The average age was 51.3±3.3 years and 50.6% were women. Hepatic steatosis was associated with all-cause mortality after 12.7 years of mean follow-up when adjusting for baseline CVD risk factors, including body mass index (HR: 1.21, 1.04-1.40); however, the results were attenuated when utilizing time-varying covariates. The association between hepatic steatosis and incident CVD was not statistically significant after we accounted for body mass index in models considering baseline covariates or time-varying covariates. We observed no association between hepatic steatosis and CVD-related mortality or incident cancer. CONCLUSIONS: In this large, multicohort study of participants with CT-defined hepatic steatosis, accounting for change in CVD risk factors over time attenuated associations between liver fat and overall mortality or incident CVD. Our work highlights the need to consider concurrent cardiometabolic disease when determining associations between NAFLD and CVD and mortality outcomes.

An Accelerated Pipeline for Multi-label Renal Pathology Image Segmentation at the Whole Slide Image Level.

Deep-learning techniques have been used widely to alleviate the labour-intensive and time-consuming manual annotation required for pixel-level tissue characterization. Our previous study introduced an efficient single dynamic network - Omni-Seg - that achieved multi-class multi-scale pathological segmentation with less computational complexity. However, the patch-wise segmentation paradigm still applies to Omni-Seg, and the pipeline is time-consuming when providing segmentation for Whole Slide Images (WSIs). In this paper, we propose an enhanced version of the Omni-Seg pipeline in order to reduce the repetitive computing processes and utilize a GPU to accelerate the model's prediction for both better model performance and faster speed. Our proposed method's innovative contribution is two-fold: (1) a Docker is released for an end-to-end slide-wise multi-tissue segmentation for WSIs; and (2) the pipeline is deployed on a GPU to accelerate the prediction, achieving better segmentation quality in less time. The proposed accelerated implementation reduced the average processing time (at the testing stage) on a standard needle biopsy WSI from 2.3 hours to 22 minutes, using 35 WSIs from the Kidney Tissue Atlas (KPMP) Datasets. The source code and the Docker have been made publicly available at https://github.com/ddrrnn123/Omni-Seg.

An End-to-end Pipeline for 3D Slide-wise Multi-stain Renal Pathology Registration.

Tissue examination and quantification in a 3D context on serial section whole slide images (WSIs) were labor-intensive and time-consuming tasks. Our previous study proposed a novel registration-based method (Map3D) to automatically align WSIs to the same physical space, reducing the human efforts of screening serial sections from WSIs. However, the registration performance of our Map3D method was only evaluated on single-stain WSIs with large-scale kidney tissue samples. In this paper, we provide a Docker for an end-to-end 3D slide-wise registration pipeline on needle biopsy serial sections in a multi-stain paradigm. The contribution of this study is three-fold: (1) We release a containerized Docker for an end-to-end multi-stain WSI registration. (2) We prove that the Map3D pipeline is capable of sectional registration from multi-stain WSI. (3) We verify that the Map3D pipeline can also be applied to needle biopsy tissue samples. The source code and the Docker have been made publicly available at https://github.com/hrlblab/Map3D.

Feasibility of Universal Anomaly Detection without Knowing the Abnormality in Medical Images.

Many anomaly detection approaches, especially deep learning methods, have been recently developed to identify abnormal image morphology by only employing normal images during training. Unfortunately, many prior anomaly detection methods were optimized for a specific "known" abnormality (e.g., brain tumor, bone fraction, cell types). Moreover, even though only the normal images were used in the training process, the abnormal images were often employed during the validation process (e.g., epoch selection, hyper-parameter tuning), which might leak the supposed "unknown" abnormality unintentionally. In this study, we investigated these two essential aspects regarding universal anomaly detection in medical images by (1) comparing various anomaly detection methods across four medical datasets, (2) investigating the inevitable but often neglected issues on how to unbiasedly select the optimal anomaly detection model during the validation phase using only normal images, and (3) proposing a simple decision-level ensemble method to leverage the advantage of different kinds of anomaly detection without knowing the abnormality. The results of our experiments indicate that none of the evaluated methods consistently achieved the best performance across all datasets. Our proposed method enhanced the robustness of performance in general (average AUC 0.956).

Reducing uncertainty in cancer risk estimation for patients with indeterminate pulmonary nodules using an integrated deep learning model.

OBJECTIVE: Patients with indeterminate pulmonary nodules (IPN) with an intermediate to a high probability of lung cancer generally undergo invasive diagnostic procedures. Chest computed tomography image and clinical data have been in estimating the pretest probability of lung cancer. In this study, we apply a deep learning network to integrate multi-modal data from CT images and clinical data (including blood-based biomarkers) to improve lung cancer diagnosis. Our goal is to reduce uncertainty and to avoid morbidity, mortality, over- and undertreatment of patients with IPNs. METHOD: We use a retrospective study design with cross-validation and external-validation from four different sites. We introduce a deep learning framework with a two-path structure to learn from CT images and clinical data. The proposed model can learn and predict with single modality if the multi-modal data is not complete. We use 1284 patients in the learning cohort for model development. Three external sites (with 155, 136 and 96 patients, respectively) provided patient data for external validation. We compare our model to widely applied clinical prediction models (Mayo and Brock models) and image-only methods (e.g., Liao et al. model). RESULTS: Our co-learning model improves upon the performance of clinical-factor-only (Mayo and Brock models) and image-only (Liao et al.) models in both cross-validation of learning cohort (e.g. , AUC: 0.787 (ours) vs. 0.707-0.719 (baselines), results reported in validation fold and external-validation using three datasets from University of Pittsburgh Medical Center (e.g., 0.918 (ours) vs. 0.828-0.886 (baselines)), Detection of Early Cancer Among Military Personnel (e.g., 0.712 (ours) vs. 0.576-0.709 (baselines)), and University of Colorado Denver (e.g., 0.847 (ours) vs. 0.679-0.746 (baselines)). In addition, our model achieves better re-classification performance (cNRI 0.04 to 0.20) in all cross- and external-validation sets compared to the Mayo model. CONCLUSIONS: Lung cancer risk estimation in patients with IPNs can benefit from the co-learning of CT image and clinical data. Learning from more subjects, even though those only have a single modality, can improve the prediction accuracy. An integrated deep learning model can achieve reasonable discrimination and re-classification performance.

Extending the value of routine lung screening CT with quantitative body composition assessment.

Certain body composition phenotypes, like sarcopenia, are well established as predictive markers for post-surgery complications and overall survival of lung cancer patients. However, their association with incidental lung cancer risk in the screening population is still unclear. We study the feasibility of body composition analysis using chest low dose computed tomography (LDCT). A two-stage fully automatic pipeline is developed to assess the cross-sectional area of body composition components including subcutaneous adipose tissue (SAT), muscle, visceral adipose tissue (VAT), and bone on T5, T8 and T10 vertebral levels. The pipeline is developed using 61 cases of the VerSe'20 dataset, 40 annotated cases of NLST, and 851 inhouse screening cases. On a test cohort consisting of 30 cases from the inhouse screening cohort (age 55 - 73, 50% female) and 42 cases of NLST (age 55 - 75, 59.5% female), the pipeline achieves a root mean square error (RMSE) of 7.25 mm (95% CI: [6.61, 7.85]) for the vertebral level identification and mean Dice similarity score (DSC) 0.99 ± 0.02, 0.96 ± 0.03, and 0.95 ± 0.04 for SAT, muscle, and VAT, respectively for body composition segmentation. The pipeline is generalized to the CT arm of the NLST dataset (25,205 subjects, 40.8% female, 1,056 lung cancer incidences). Time-to-event analysis for lung cancer incidence indicates inverse association between measured muscle cross-sectional area and incidental lung cancer risks (p < 0.001 female, p < 0.001 male). In conclusion, automatic body composition analysis using routine lung screening LDCT is feasible.

Multi-modal Learning with Missing Data for Cancer Diagnosis Using Histopathological and Genomic Data.

Multi-modal learning (e.g., integrating pathological images with genomic features) tends to improve the accuracy of cancer diagnosis and prognosis as compared to learning with a single modality. However, missing data is a common problem in clinical practice, i.e., not every patient has all modalities available. Most of the previous works directly discarded samples with missing modalities, which might lose information in these data and increase the likelihood of overfitting. In this work, we generalize the multi-modal learning in cancer diagnosis with the capacity of dealing with missing data using histological images and genomic data. Our integrated model can utilize all available data from patients with both complete and partial modalities. The experiments on the public TCGA-GBM and TCGA-LGG datasets show that the data with missing modalities can contribute to multi-modal learning, which improves the model performance in grade classification of glioma cancer.

Cancer Risk Estimation Combining Lung Screening CT with Clinical Data Elements.

PURPOSE: To develop a model to estimate lung cancer risk using lung cancer screening CT and clinical data elements (CDEs) without manual reading efforts. MATERIALS AND METHODS: Two screening cohorts were retrospectively studied: the National Lung Screening Trial (NLST; participants enrolled between August 2002 and April 2004) and the Vanderbilt Lung Screening Program (VLSP; participants enrolled between 2015 and 2018). Fivefold cross-validation using the NLST dataset was used for initial development and assessment of the co-learning model using whole CT scans and CDEs. The VLSP dataset was used for external testing of the developed model. Area under the receiver operating characteristic curve (AUC) and area under the precision-recall curve were used to measure the performance of the model. The developed model was compared with published risk-prediction models that used only CDEs or imaging data alone. The Brock model was also included for comparison by imputing missing values for patients without a dominant pulmonary nodule. RESULTS: A total of 23 505 patients from the NLST (mean age, 62 years ± 5 [standard deviation]; 13 838 men, 9667 women) and 147 patients from the VLSP (mean age, 65 years ± 5; 82 men, 65 women) were included. Using cross-validation on the NLST dataset, the AUC of the proposed co-learning model (AUC, 0.88) was higher than the published models predicted with CDEs only (AUC, 0.69; P < .05) and with images only (AUC, 0.86; P < .05). Additionally, using the external VLSP test dataset, the co-learning model had a higher performance than each of the published individual models (AUC, 0.91 [co-learning] vs 0.59 [CDE-only] and 0.88 [image-only]; P < .05 for both comparisons). CONCLUSION: The proposed co-learning predictive model combining chest CT images and CDEs had a higher performance for lung cancer risk prediction than models that contained only CDE or only image data; the proposed model also had a higher performance than the Brock model.Keywords: Computer-aided Diagnosis (CAD), CT, Lung, Thorax Supplemental material is available for this article. © RSNA, 2021.

Deep Multi-path Network Integrating Incomplete Biomarker and Chest CT Data for Evaluating Lung Cancer Risk.

Clinical data elements (CDEs) (e.g., age, smoking history), blood markers and chest computed tomography (CT) structural features have been regarded as effective means for assessing lung cancer risk. These independent variables can provide complementary information and we hypothesize that combining them will improve the prediction accuracy. In practice, not all patients have all these variables available. In this paper, we propose a new network design, termed as multi-path multi-modal missing network (M3Net), to integrate the multi-modal data (i.e., CDEs, biomarker and CT image) considering missing modality with multiple paths neural network. Each path learns discriminative features of one modality, and different modalities are fused in a second stage for an integrated prediction. The network can be trained end-to-end with both medical image features and CDEs/biomarkers, or make a prediction with single modality. We evaluate M3Net with datasets including three sites from the Consortium for Molecular and Cellular Characterization of Screen-Detected Lesions (MCL) project. Our method is cross validated within a cohort of 1291 subjects (383 subjects with complete CDEs/biomarkers and CT images), and externally validated with a cohort of 99 subjects (99 with complete CDEs/biomarkers and CT images). Both cross-validation and external-validation results show that combining multiple modality significantly improves the predicting performance of single modality. The results suggest that integrating subjects with missing either CDEs/biomarker or CT imaging features can contribute to the discriminatory power of our model (p < 0.05, bootstrap two-tailed test). In summary, the proposed M3Net framework provides an effective way to integrate image and non-image data in the context of missing information.

Development and Characterization of a Chest CT Atlas.

A major goal of lung cancer screening is to identify individuals with particular phenotypes that are associated with high risk of cancer. Identifying relevant phenotypes is complicated by the variation in body position and body composition. In the brain, standardized coordinate systems (e.g., atlases) have enabled separate consideration of local features from gross/global structure. To date, no analogous standard atlas has been presented to enable spatial mapping and harmonization in chest computational tomography (CT). In this paper, we propose a thoracic atlas built upon a large low dose CT (LDCT) database of lung cancer screening program. The study cohort includes 466 male and 387 female subjects with no screening detected malignancy (age 46-79 years, mean 64.9 years). To provide spatial mapping, we optimize a multi-stage inter-subject non-rigid registration pipeline for the entire thoracic space. Briefly, with 50 scans of 50 randomly selected female subjects as fine tuning dataset, we search for the optimal configuration of the non-rigid registration module in a range of adjustable parameters including: registration searching radius, degree of keypoint dispersion, regularization coefficient and similarity patch size, to minimize the registration failure rate approximated by the number of samples with low Dice similarity score (DSC) for lung and body segmentation. We evaluate the optimized pipeline on a separate cohort (100 scans of 50 female and 50 male subjects) relative to two baselines with alternative non-rigid registration module: the same software with default parameters and an alternative software. We achieve a significant improvement in terms of registration success rate based on manual QA. For the entire study cohort, the optimized pipeline achieves a registration success rate of 91.7%. The application validity of the developed atlas is evaluated in terms of discriminative capability for different anatomic phenotypes, including body mass index (BMI), chronic obstructive pulmonary disease (COPD), and coronary artery calcification (CAC).

Deep Learning for Fully Automated Prediction of Overall Survival in Patients with Oropharyngeal Cancer Using FDG-PET Imaging.

PURPOSE: Accurate prognostic stratification of patients with oropharyngeal squamous cell carcinoma (OPSCC) is crucial. We developed an objective and robust deep learning-based fully-automated tool called the DeepPET-OPSCC biomarker for predicting overall survival (OS) in OPSCC using [18F]fluorodeoxyglucose (FDG)-PET imaging. EXPERIMENTAL DESIGN: The DeepPET-OPSCC prediction model was built and tested internally on a discovery cohort (n = 268) by integrating five convolutional neural network models for volumetric segmentation and ten models for OS prognostication. Two external test cohorts were enrolled-the first based on the Cancer Imaging Archive (TCIA) database (n = 353) and the second being a clinical deployment cohort (n = 31)-to assess the DeepPET-OPSCC performance and goodness of fit. RESULTS: After adjustment for potential confounders, DeepPET-OPSCC was found to be an independent predictor of OS in both discovery and TCIA test cohorts [HR = 2.07; 95% confidence interval (CI), 1.31-3.28 and HR = 2.39; 95% CI, 1.38-4.16; both P = 0.002]. The tool also revealed good predictive performance, with a c-index of 0.707 (95% CI, 0.658-0.757) in the discovery cohort, 0.689 (95% CI, 0.621-0.757) in the TCIA test cohort, and 0.787 (95% CI, 0.675-0.899) in the clinical deployment test cohort; the average time taken was 2 minutes for calculation per exam. The integrated nomogram of DeepPET-OPSCC and clinical risk factors significantly outperformed the clinical model [AUC at 5 years: 0.801 (95% CI, 0.727-0.874) vs. 0.749 (95% CI, 0.649-0.842); P = 0.031] in the TCIA test cohort. CONCLUSIONS: DeepPET-OPSCC achieved an accurate OS prediction in patients with OPSCC and enabled an objective, unbiased, and rapid assessment for OPSCC prognostication.