Weakly supervised annotation-free cancer detection and prediction of genotype in routine histopathology.

Deep learning is a powerful tool in computational pathology: it can be used for tumor detection and for predicting genetic alterations based on histopathology images alone. Conventionally, tumor detection and prediction of genetic alterations are two separate workflows. Newer methods have combined them, but require complex, manually engineered computational pipelines, restricting reproducibility and robustness. To address these issues, we present a new method for simultaneous tumor detection and prediction of genetic alterations: The Slide-Level Assessment Model (SLAM) uses a single off-the-shelf neural network to predict molecular alterations directly from routine pathology slides without any manual annotations, improving upon previous methods by automatically excluding normal and non-informative tissue regions. SLAM requires only standard programming libraries and is conceptually simpler than previous approaches. We have extensively validated SLAM for clinically relevant tasks using two large multicentric cohorts of colorectal cancer patients, Darmkrebs: Chancen der Verhütung durch Screening (DACHS) from Germany and Yorkshire Cancer Research Bowel Cancer Improvement Programme (YCR-BCIP) from the UK. We show that SLAM yields reliable slide-level classification of tumor presence with an area under the receiver operating curve (AUROC) of 0.980 (confidence interval 0.975, 0.984; n = 2,297 tumor and n = 1,281 normal slides). In addition, SLAM can detect microsatellite instability (MSI)/mismatch repair deficiency (dMMR) or microsatellite stability/mismatch repair proficiency with an AUROC of 0.909 (0.888, 0.929; n = 2,039 patients) and BRAF mutational status with an AUROC of 0.821 (0.786, 0.852; n = 2,075 patients). The improvement with respect to previous methods was validated in a large external testing cohort in which MSI/dMMR status was detected with an AUROC of 0.900 (0.864, 0.931; n = 805 patients). In addition, SLAM provides human-interpretable visualization maps, enabling the analysis of multiplexed network predictions by human experts. In summary, SLAM is a new simple and powerful method for computational pathology that could be applied to multiple disease contexts. © 2021 The Authors. The Journal of Pathology published by John Wiley & Sons, Ltd. on behalf of The Pathological Society of Great Britain and Ireland.

Deep learning identifies inflamed fat as a risk factor for lymph node metastasis in early colorectal cancer.

The spread of early-stage (T1 and T2) adenocarcinomas to locoregional lymph nodes is a key event in disease progression of colorectal cancer (CRC). The cellular mechanisms behind this event are not completely understood and existing predictive biomarkers are imperfect. Here, we used an end-to-end deep learning algorithm to identify risk factors for lymph node metastasis (LNM) status in digitized histopathology slides of the primary CRC and its surrounding tissue. In two large population-based cohorts, we show that this system can predict the presence of more than one LNM in pT2 CRC patients with an area under the receiver operating curve (AUROC) of 0.733 (0.67-0.758) and patients with any LNM with an AUROC of 0.711 (0.597-0.797). Similarly, in pT1 CRC patients, the presence of more than one LNM or any LNM was predictable with an AUROC of 0.733 (0.644-0.778) and 0.567 (0.542-0.597), respectively. Based on these findings, we used the deep learning system to guide human pathology experts towards highly predictive regions for LNM in the whole slide images. This hybrid human observer and deep learning approach identified inflamed adipose tissue as the highest predictive feature for LNM presence. Our study is a first proof of concept that artificial intelligence (AI) systems may be able to discover potentially new biological mechanisms in cancer progression. Our deep learning algorithm is publicly available and can be used for biomarker discovery in any disease setting. © 2021 The Pathological Society of Great Britain and Ireland. Published by John Wiley & Sons, Ltd.

The future of artificial intelligence in digital pathology - results of a survey across stakeholder groups.

AIMS: Artificial intelligence (AI) provides a powerful tool to extract information from digitised histopathology whole slide images. During the last 5 years, academic and commercial actors have developed new technical solutions for a diverse set of tasks, including tissue segmentation, cell detection, mutation prediction, prognostication and prediction of treatment response. In the light of limited overall resources, it is presently unclear for researchers, practitioners and policymakers which of these topics are stable enough for clinical use in the near future and which topics are still experimental, but worth investing time and effort into. METHODS AND RESULTS: To identify potentially promising applications of AI in pathology, we performed an anonymous online survey of 75 computational pathology domain experts from academia and industry. Participants enrolled in 2021 were queried about their subjective opinion on promising and appealing subfields of computational pathology with a focus upon solid tumours. The results of this survey indicate that the prediction of treatment response directly from routine pathology slides is regarded as the most promising future application. This item was ranked highest in the overall analysis and in subgroups by age and professional background. Furthermore, prediction of genetic alterations, gene expression and survival directly from routine pathology images scored consistently high throughout subgroups. CONCLUSIONS: Together, these data demonstrate a possible direction for the development of computational pathology systems in clinical, academic and industrial research in the near future.

Deep learning detects genetic alterations in cancer histology generated by adversarial networks.

Deep learning can detect microsatellite instability (MSI) from routine histology images in colorectal cancer (CRC). However, ethical and legal barriers impede sharing of images and genetic data, hampering development of new algorithms for detection of MSI and other biomarkers. We hypothesized that histology images synthesized by conditional generative adversarial networks (CGANs) retain information about genetic alterations. To test this, we developed a 'histology CGAN' which was trained on 256 patients (training cohort 1) and 1457 patients (training cohort 2). The CGAN synthesized 10 000 synthetic MSI and non-MSI images which contained a range of tissue types and were deemed realistic by trained observers in a blinded study. Subsequently, we trained a deep learning detector of MSI on real or synthetic images and evaluated the performance of MSI detection in a held-out set of 142 patients. When trained on real images from training cohort 1, this system achieved an area under the receiver operating curve (AUROC) of 0.742 [0.681, 0.854]. Training on the larger cohort 2 only marginally improved the AUROC to 0.757 [0.707, 0.869]. Training on purely synthetic data resulted in an AUROC of 0.743 [0.658, 0.801]. Training on both real and synthetic data further increased AUROC to 0.777 [0.715, 0.821]. We conclude that synthetic histology images retain information reflecting underlying genetic alterations in colorectal cancer. Using synthetic instead of real images to train deep learning systems yields non-inferior classifiers. This approach can be used to create large shareable data sets or to augment small data sets with rare molecular features. © 2021 The Authors. The Journal of Pathology published by John Wiley & Sons, Ltd. on behalf of The Pathological Society of Great Britain and Ireland.

Deep learning in cancer pathology: a new generation of clinical biomarkers.

Clinical workflows in oncology rely on predictive and prognostic molecular biomarkers. However, the growing number of these complex biomarkers tends to increase the cost and time for decision-making in routine daily oncology practice; furthermore, biomarkers often require tumour tissue on top of routine diagnostic material. Nevertheless, routinely available tumour tissue contains an abundance of clinically relevant information that is currently not fully exploited. Advances in deep learning (DL), an artificial intelligence (AI) technology, have enabled the extraction of previously hidden information directly from routine histology images of cancer, providing potentially clinically useful information. Here, we outline emerging concepts of how DL can extract biomarkers directly from histology images and summarise studies of basic and advanced image analysis for cancer histology. Basic image analysis tasks include detection, grading and subtyping of tumour tissue in histology images; they are aimed at automating pathology workflows and consequently do not immediately translate into clinical decisions. Exceeding such basic approaches, DL has also been used for advanced image analysis tasks, which have the potential of directly affecting clinical decision-making processes. These advanced approaches include inference of molecular features, prediction of survival and end-to-end prediction of therapy response. Predictions made by such DL systems could simplify and enrich clinical decision-making, but require rigorous external validation in clinical settings.

Clinical-Grade Detection of Microsatellite Instability in Colorectal Tumors by Deep Learning.

BACKGROUND & AIMS: Microsatellite instability (MSI) and mismatch-repair deficiency (dMMR) in colorectal tumors are used to select treatment for patients. Deep learning can detect MSI and dMMR in tumor samples on routine histology slides faster and less expensively than molecular assays. However, clinical application of this technology requires high performance and multisite validation, which have not yet been performed. METHODS: We collected H&E-stained slides and findings from molecular analyses for MSI and dMMR from 8836 colorectal tumors (of all stages) included in the MSIDETECT consortium study, from Germany, the Netherlands, the United Kingdom, and the United States. Specimens with dMMR were identified by immunohistochemistry analyses of tissue microarrays for loss of MLH1, MSH2, MSH6, and/or PMS2. Specimens with MSI were identified by genetic analyses. We trained a deep-learning detector to identify samples with MSI from these slides; performance was assessed by cross-validation (N = 6406 specimens) and validated in an external cohort (n = 771 specimens). Prespecified endpoints were area under the receiver operating characteristic (AUROC) curve and area under the precision-recall curve (AUPRC). RESULTS: The deep-learning detector identified specimens with dMMR or MSI with a mean AUROC curve of 0.92 (lower bound, 0.91; upper bound, 0.93) and an AUPRC of 0.63 (range, 0.59-0.65), or 67% specificity and 95% sensitivity, in the cross-validation development cohort. In the validation cohort, the classifier identified samples with dMMR with an AUROC of 0.95 (range, 0.92-0.96) without image preprocessing and an AUROC of 0.96 (range, 0.93-0.98) after color normalization. CONCLUSIONS: We developed a deep-learning system that detects colorectal cancer specimens with dMMR or MSI using H&E-stained slides; it detected tissues with dMMR with an AUROC of 0.96 in a large, international validation cohort. This system might be used for high-throughput, low-cost evaluation of colorectal tissue specimens.

Validation of the prognostic value of CD3 and CD8 cell densities analogous to the Immunoscore® by stage and location of colorectal cancer: an independent patient cohort study.

In addition to the traditional staging system in colorectal cancer (CRC), the Immunoscore® has been proposed to characterize the level of immune infiltration in tumor tissue and as a potential prognostic marker. The aim of this study was to examine and validate associations of an immune cell score analogous to the Immunoscore® with established molecular tumor markers and with CRC patient survival in a routine setting. Patients from a population-based cohort study with available CRC tumor tissue blocks were included in this analysis. CD3+ and CD8+ tumor infiltrating lymphocytes in the tumor center and invasive margin were determined in stained tumor tissue slides. Based on the T-cell density in each region, an  immune cell score closely analogous to the concept of the Immunoscore® was calculated and tumors categorized into IS-low, IS-intermediate, or IS-high. Logistic regression models were used to assess associations between clinicopathological characteristics with the immune cell score, and Cox proportional hazards models to analyze associations with cancer-specific, relapse-free, and overall survival. From 1,535 patients with CRC, 411 (27%) had IS-high tumors. Microsatellite instability (MSI-high) was strongly associated with higher immune cell score levels (p < 0.001). Stage I-III patients with IS-high had better CRC-specific and relapse-free survival compared to patients with IS-low (hazard ratio [HR] = 0.42 [0.27-0.66] and HR = 0.45 [0.31-0.67], respectively). Patients with microsatellite stable (MSS) tumors and IS-high had better survival (HRCSS  = 0.60 [0.42-0.88]) compared to MSS/IS-low patients. In this population-based cohort of CRC patients, the immune cell score was significantly associated with better patient survival. It was a similarly strong prognostic marker in patients with MSI-high tumors and in the larger group of patients with MSS tumors. Additionally, this study showed that it is possible to implement an analogous immune cell score approach and validate the Immunoscore® using open source software in an academic setting. Thus, the Immunoscore® could be useful to improve the traditional staging system in colon and rectal cancer used in clinical practice.

Benchmarking weakly-supervised deep learning pipelines for whole slide classification in computational pathology.

Artificial intelligence (AI) can extract visual information from histopathological slides and yield biological insight and clinical biomarkers. Whole slide images are cut into thousands of tiles and classification problems are often weakly-supervised: the ground truth is only known for the slide, not for every single tile. In classical weakly-supervised analysis pipelines, all tiles inherit the slide label while in multiple-instance learning (MIL), only bags of tiles inherit the label. However, it is still unclear how these widely used but markedly different approaches perform relative to each other. We implemented and systematically compared six methods in six clinically relevant end-to-end prediction tasks using data from N=2980 patients for training with rigorous external validation. We tested three classical weakly-supervised approaches with convolutional neural networks and vision transformers (ViT) and three MIL-based approaches with and without an additional attention module. Our results empirically demonstrate that histological tumor subtyping of renal cell carcinoma is an easy task in which all approaches achieve an area under the receiver operating curve (AUROC) of above 0.9. In contrast, we report significant performance differences for clinically relevant tasks of mutation prediction in colorectal, gastric, and bladder cancer. In these mutation prediction tasks, classical weakly-supervised workflows outperformed MIL-based weakly-supervised methods for mutation prediction, which is surprising given their simplicity. This shows that new end-to-end image analysis pipelines in computational pathology should be compared to classical weakly-supervised methods. Also, these findings motivate the development of new methods which combine the elegant assumptions of MIL with the empirically observed higher performance of classical weakly-supervised approaches. We make all source codes publicly available at https://github.com/KatherLab/HIA, allowing easy application of all methods to any similar task.

Predicting Mutational Status of Driver and Suppressor Genes Directly from Histopathology With Deep Learning: A Systematic Study Across 23 Solid Tumor Types.

In the last four years, advances in Deep Learning technology have enabled the inference of selected mutational alterations directly from routine histopathology slides. In particular, recent studies have shown that genetic changes in clinically relevant driver genes are reflected in the histological phenotype of solid tumors and can be inferred by analysing routine Haematoxylin and Eosin (H&E) stained tissue sections with Deep Learning. However, these studies mostly focused on selected individual genes in selected tumor types. In addition, genetic changes in solid tumors primarily act by changing signaling pathways that regulate cell behaviour. In this study, we hypothesized that Deep Learning networks can be trained to directly predict alterations of genes and pathways across a spectrum of solid tumors. We manually outlined tumor tissue in H&E-stained tissue sections from 7,829 patients with 23 different tumor types from The Cancer Genome Atlas. We then trained convolutional neural networks in an end-to-end way to detect alterations in the most clinically relevant pathways or genes, directly from histology images. Using this automatic approach, we found that alterations in 12 out of 14 clinically relevant pathways and numerous single gene alterations appear to be detectable in tissue sections, many of which have not been reported before. Interestingly, we show that the prediction performance for single gene alterations is better than that for pathway alterations. Collectively, these data demonstrate the predictability of genetic alterations directly from routine cancer histology images and show that individual genes leave a stronger morphological signature than genetic pathways.

Pan-cancer image-based detection of clinically actionable genetic alterations.

Molecular alterations in cancer can cause phenotypic changes in tumor cells and their micro-environment. Routine histopathology tissue slides - which are ubiquitously available - can reflect such morphological changes. Here, we show that deep learning can consistently infer a wide range of genetic mutations, molecular tumor subtypes, gene expression signatures and standard pathology biomarkers directly from routine histology. We developed, optimized, validated and publicly released a one-stop-shop workflow and applied it to tissue slides of more than 5000 patients across multiple solid tumors. Our findings show that a single deep learning algorithm can be trained to predict a wide range of molecular alterations from routine, paraffin-embedded histology slides stained with hematoxylin and eosin. These predictions generalize to other populations and are spatially resolved. Our method can be implemented on mobile hardware, potentially enabling point-of-care diagnostics for personalized cancer treatment. More generally, this approach could elucidate and quantify genotype-phenotype links in cancer.