Deep learning integrates histopathology and proteogenomics at a pan-cancer level.

We introduce a pioneering approach that integrates pathology imaging with transcriptomics and proteomics to identify predictive histology features associated with critical clinical outcomes in cancer. We utilize 2,755 H&E-stained histopathological slides from 657 patients across 6 cancer types from CPTAC. Our models effectively recapitulate distinctions readily made by human pathologists: tumor vs. normal (AUROC = 0.995) and tissue-of-origin (AUROC = 0.979). We further investigate predictive power on tasks not normally performed from H&E alone, including TP53 prediction and pathologic stage. Importantly, we describe predictive morphologies not previously utilized in a clinical setting. The incorporation of transcriptomics and proteomics identifies pathway-level signatures and cellular processes driving predictive histology features. Model generalizability and interpretability is confirmed using TCGA. We propose a classification system for these tasks, and suggest potential clinical applications for this integrated human and machine learning approach. A publicly available web-based platform implements these models.

Artificial intelligence and deep learning to map immune cell types in inflamed human tissue.

Biopsies of inflammatory tissue contain a complex network of interacting cells, orchestrating the immune or autoimmune response. While standard histological examination can identify relationships, it is clear that a great amount of data on each slide is not quantitated or categorized in standard microscopic examinations. To deal with the huge amount of data present in biopsy tissue in an unbiased and comprehensive way, we have developed a deep learning algorithm to identify immune cells in biopsies of inflammatory lesions. We focused on T follicular helper (Tfh) cell subsets and B cells in dermatomyositis biopsy images. We achieved strong performance on detection and classification of cells, including the rare Tfh cell subsets present in the tissue. This algorithm could be used to perform distance mapping between cell types in tissue, and could be easily adapted to other disease states.

Deep Learning and Pathomics Analyses Reveal Cell Nuclei as Important Features for Mutation Prediction of BRAF-Mutated Melanomas.

Image-based analysis as a method for mutation detection can be advantageous in settings when tumor tissue is limited or unavailable for direct testing. In this study, we utilize two distinct and complementary machine-learning methods of analyzing whole-slide images for predicting mutated BRAF. In the first method, whole-slide images of melanomas from 256 patients were used to train a deep convolutional neural network to develop a fully automated model that first selects for tumor-rich areas (area under the curve = 0.96) and then predicts for mutated BRAF (area under the curve = 0.71). Saliency mapping was performed and revealed that pixels corresponding to nuclei were the most relevant to network learning. In the second method, whole-slide images were analyzed using a pathomics pipeline that first annotates nuclei and then quantifies nuclear features, showing that mutated BRAF nuclei were significantly larger and rounder than BRAF‒wild-type nuclei. Finally, we developed a model that combines clinical information, deep learning, and pathomics that improves the predictive performance for mutated BRAF to an area under the curve of 0.89. Not only does this provide additional insights on how BRAF mutations affect tumor structural characteristics, but machine learning‒based analysis of whole-slide images also has the potential to be integrated into higher-order models for understanding tumor biology.

Predicting endometrial cancer subtypes and molecular features from histopathology images using multi-resolution deep learning models.

The determination of endometrial carcinoma histological subtypes, molecular subtypes, and mutation status is critical for the diagnostic process, and directly affects patients' prognosis and treatment. Sequencing, albeit slower and more expensive, can provide additional information on molecular subtypes and mutations that can be used to better select treatments. Here, we implement a customized multi-resolution deep convolutional neural network, Panoptes, that predicts not only the histological subtypes but also the molecular subtypes and 18 common gene mutations based on digitized H&E-stained pathological images. The model achieves high accuracy and generalizes well on independent datasets. Our results suggest that Panoptes, with further refinement, has the potential for clinical application to help pathologists determine molecular subtypes and mutations of endometrial carcinoma without sequencing.

State of the Art: Machine Learning Applications in Glioma Imaging.

OBJECTIVE: Machine learning has recently gained considerable attention because of promising results for a wide range of radiology applications. Here we review recent work using machine learning in brain tumor imaging, specifically segmentation and MRI radiomics of gliomas. CONCLUSION: We discuss available resources, state-of-the-art segmentation methods, and machine learning radiomics for glioma. We highlight the challenges of these techniques as well as the future potential in clinical diagnostics, prognostics, and decision making.

Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning.

Visual inspection of histopathology slides is one of the main methods used by pathologists to assess the stage, type and subtype of lung tumors. Adenocarcinoma (LUAD) and squamous cell carcinoma (LUSC) are the most prevalent subtypes of lung cancer, and their distinction requires visual inspection by an experienced pathologist. In this study, we trained a deep convolutional neural network (inception v3) on whole-slide images obtained from The Cancer Genome Atlas to accurately and automatically classify them into LUAD, LUSC or normal lung tissue. The performance of our method is comparable to that of pathologists, with an average area under the curve (AUC) of 0.97. Our model was validated on independent datasets of frozen tissues, formalin-fixed paraffin-embedded tissues and biopsies. Furthermore, we trained the network to predict the ten most commonly mutated genes in LUAD. We found that six of them-STK11, EGFR, FAT1, SETBP1, KRAS and TP53-can be predicted from pathology images, with AUCs from 0.733 to 0.856 as measured on a held-out population. These findings suggest that deep-learning models can assist pathologists in the detection of cancer subtype or gene mutations. Our approach can be applied to any cancer type, and the code is available at https://github.com/ncoudray/DeepPATH .