Generative Deep Learning in Digital Pathology Workflows.

Many modern histopathology laboratories are in the process of digitizing their workflows. Digitization of tissue images has made it feasible to research the augmentation or automation of clinical reporting and diagnosis. The application of modern computer vision techniques, based on deep learning, promises systems that can identify pathologies in slide images with a high degree of accuracy. Generative modeling is an approach to machine learning and deep learning that can be used to transform and generate data. It can be applied to a broad range of tasks within digital pathology, including the removal of color and intensity artifacts, the adaption of images in one domain into those of another, and the generation of synthetic digital tissue samples. This review provides an introduction to the topic, considers these applications, and discusses future directions for generative models within histopathology.

Quantification of tumour budding, lymphatic vessel density and invasion through image analysis in colorectal cancer.

BACKGROUND: Tumour budding (TB), lymphatic vessel density (LVD) and lymphatic vessel invasion (LVI) have shown promise as prognostic factors in colorectal cancer (CRC) but reproducibility using conventional histopathology is challenging. We demonstrate image analysis methodology to quantify the histopathological features which could permit standardisation across institutes and aid risk stratification of Dukes B patients. METHODS: Multiplexed immunofluorescence of pan-cytokeratin, D2-40 and DAPI identified epithelium, lymphatic vessels and all nuclei respectively in tissue sections from 50 patients diagnosed with Dukes A (n = 13), Dukes B (n = 29) and Dukes C (n = 8) CRC. An image analysis algorithm was developed and performed, on digitised images of the CRC tissue sections, to quantify TB, LVD, and LVI at the invasive front. RESULTS: TB (HR =5.7; 95% CI, 2.38-13.8), LVD (HR =5.1; 95% CI, 2.04-12.99) and LVI (HR =9.9; 95% CI, 3.57-27.98) were successfully quantified through image analysis and all were shown to be significantly associated with poor survival, in univariate analyses. LVI (HR =6.08; 95% CI, 1.17-31.41) is an independent prognostic factor within the study and was correlated to both TB (Pearson r =0.71, p <0.0003) and LVD (Pearson r =0.69, p <0.0003). CONCLUSION: We demonstrate methodology through image analysis which can standardise the quantification of TB, LVD and LVI from a single tissue section while decreasing observer variability. We suggest this technology is capable of stratifying a high risk Dukes B CRC subpopulation and we show the three histopathological features to be of prognostic significance.

Use of High-Plex Data Reveals Novel Insights into the Tumour Microenvironment of Clear Cell Renal Cell Carcinoma.

Although immune checkpoint inhibitors (ICIs) have significantly improved the oncological outcomes, about one-third of patients affected by clear cell renal cell carcinoma (ccRCC) still experience recurrence. Current prognostic algorithms, such as the Leibovich score (LS), rely on morphological features manually assessed by pathologists and are therefore subject to bias. Moreover, these tools do not consider the heterogeneous molecular milieu present in the Tumour Microenvironment (TME), which may have prognostic value. We systematically developed a semi-automated method to investigate 62 markers and their combinations in 150 primary ccRCCs using Multiplex Immunofluorescence (mIF), NanoString GeoMx® Digital Spatial Profiling (DSP) and Artificial Intelligence (AI)-assisted image analysis in order to find novel prognostic signatures and investigate their spatial relationship. We found that coexpression of cancer stem cell (CSC) and epithelial-to-mesenchymal transition (EMT) markers such as OCT4 and ZEB1 are indicative of poor outcome. OCT4 and the immune markers CD8, CD34, and CD163 significantly stratified patients at intermediate LS. Furthermore, augmenting the LS with OCT4 and CD34 improved patient stratification by outcome. Our results support the hypothesis that combining molecular markers has prognostic value and can be integrated with morphological features to improve risk stratification and personalised therapy. To conclude, GeoMx® DSP and AI image analysis are complementary tools providing high multiplexing capability required to investigate the TME of ccRCC, while reducing observer bias.

Identification of plasma proteins associated with oesophageal cancer chemotherapeutic treatment outcomes using SWATH-MS.

Oesophageal adenocarcinoma (OAC) is an aggressive cancer with a five-year survival of <15%. Current chemotherapeutic strategies only benefit a minority (20-30%) of patients and there are no methods available to differentiate between responders and non-responders. We performed quantitative proteomics using Sequential Window Acquisition of all THeoretical fragment-ion spectra-Mass Spectrometry (SWATH-MS) on albumin/IgG-depleted and non-depleted plasma samples from 23 patients with locally advanced OAC prior to treatment. Individuals were grouped based on tumour regression (TRG) score (TRG1/2/3 vs TRG4/5) after chemotherapy, and differentially abundant proteins were compared. Protein depletion of highly abundant proteins led to the identification of around twice as many proteins. SWATH-MS revealed significant quantitative differences in the abundance of several proteins between the two groups. These included complement c1q subunit proteins, C1QA, C1QB and C1QC, which were of higher abundance in the low TRG group. Of those that were found to be of higher abundance in the high TRG group, glutathione S-transferase pi (GSTP1) exhibited the lowest p-value and highest classification accuracy and Cohen's kappa value. Concentrations of these proteins were further examined using ELISA-based assays. This study provides quantitative information relating to differences in the plasma proteome that underpin response to chemotherapeutic treatment in oesophageal cancers. SIGNIFICANCE: Oesophageal cancers, including oesophageal adenocarcinoma (OAC) and oesophageal gastric junction cancer (OGJ), are one of the leading causes of cancer mortality worldwide. Curative therapy consists of surgery, either alone or in combination with adjuvant or neoadjuvant chemotherapy or radiation, or combination chemoradiotherapy regimens. There are currently no clinico-pathological means of predicting which patients will benefit from chemotherapeutic treatments. There is therefore an urgent need to improve oesophageal cancer disease management and treatment strategies. This work compared proteomic differences in OAC patients who responded well to chemotherapy as compared to those who did not, using quantitative proteomics prior to treatment commencement. SWATH-MS analysis of plasma (with and without albumin/IgG-depletion) from OAC patients prior to chemotherapy was performed. This approach was adopted to determine whether depletion offered a significant improvement in peptide coverage. Resultant datasets demonstrated that depletion increased peptide coverage significantly. Additionally, there was good quantitative agreement between commonly observed peptides. Data analysis was performed by adopting both univariate as well as multivariate analysis strategies. Differentially abundant proteins were identified between treatment response groups based on tumour regression grade. Such proteins included complement C1q sub-components and GSTP1. This study provides a platform for further work, utilising larger sample sets across different treatment regimens for oesophageal cancer, that will aid the development of 'treatment response prediction assays' for stratification of OAC patients prior to chemotherapy.

Prognostic features of the tumour microenvironment in oesophageal adenocarcinoma.

Oesophageal adenocarcinoma (OAC) is a disease with an incredibly poor survival rate and a complex makeup. The growth and spread of OAC tumours are profoundly influenced by their surrounding microenvironment and the properties of the tumour itself. Constant crosstalk between the tumour and its microenvironment is key to the survival of the tumour and ultimately the death of the patient. The tumour microenvironment (TME) is composed of a complex milieu of cell types including cancer associated fibroblasts (CAFs) which make up the tumour stroma, endothelial cells which line blood and lymphatic vessels and infiltrating immune cell populations. These various cell types and the tumour constantly communicate through environmental cues including fluctuations in pH, hypoxia and the release of mitogens such as cytokines, chemokines and growth factors, many of which help promote malignant progression. Eventually clusters of tumour cells such as tumour buds break away and spread through the lymphatic system to nearby lymph nodes or enter the circulation forming secondary metastasis. Collectively, these factors need to be considered when assessing and treating patients clinically. This review aims to summarise the ways in which these various factors are currently assessed and how they relate to patient treatment and outcome at an individual level.

YAP Translocation Precedes Cytoskeletal Rearrangement in Podocyte Stress Response: A Podometric Investigation of Diabetic Nephropathy.

Podocyte loss plays a pivotal role in the pathogenesis of glomerular disease. However, the mechanisms underlying podocyte damage and loss remain poorly understood. Although detachment of viable cells has been documented in experimental Diabetic Nephropathy, correlations between reduced podocyte density and disease severity have not yet been established. YAP, a mechanosensing protein, has recently been shown to correlate with glomerular disease progression, however, the underlying mechanism has yet to be fully elucidated. In this study, we sought to document podocyte density in Diabetic Nephropathy using an amended podometric methodology, and to investigate the interplay between YAP and cytoskeletal integrity during podocyte injury. Podocyte density was quantified using TLE4 and GLEPP1 multiplexed immunofluorescence. Fourteen Diabetic Nephropathy cases were analyzed for both podocyte density and cytoplasmic translocation of YAP via automated image analysis. We demonstrate a significant decrease in podocyte density in Grade III/IV cases (124.5 per 106 µm3) relative to Grade I/II cases (226 per 106 µm3) (Student's t-test, p < 0.001), and further show that YAP translocation precedes cytoskeletal rearrangement following injury. Based on these findings we hypothesize that a significant decrease in podocyte density in late grade Diabetic Nephropathy may be explained by early cytoplasmic translocation of YAP.

Automated Detection and Classification of Desmoplastic Reaction at the Colorectal Tumour Front Using Deep Learning.

The categorisation of desmoplastic reaction (DR) present at the colorectal cancer (CRC) invasive front into mature, intermediate or immature type has been previously shown to have high prognostic significance. However, the lack of an objective and reproducible assessment methodology for the assessment of DR has been a major hurdle to its clinical translation. In this study, a deep learning algorithm was trained to automatically classify immature DR on haematoxylin and eosin digitised slides of stage II and III CRC cases (n = 41). When assessing the classifier's performance on a test set of patient samples (n = 40), a Dice score of 0.87 for the segmentation of myxoid stroma was reported. The classifier was then applied to the full cohort of 528 stage II and III CRC cases, which was then divided into a training (n = 396) and a test set (n = 132). Automatically classed DR was shown to have superior prognostic significance over the manually classed DR in both the training and test cohorts. The findings demonstrated that deep learning algorithms could be applied to assist pathologists in the detection and classification of DR in CRC in an objective, standardised and reproducible manner.

Assessment of Immunological Features in Muscle-Invasive Bladder Cancer Prognosis Using Ensemble Learning.

The clinical staging and prognosis of muscle-invasive bladder cancer (MIBC) routinely includes the assessment of patient tissue samples by a pathologist. Recent studies corroborate the importance of image analysis in identifying and quantifying immunological markers from tissue samples that can provide further insight into patient prognosis. In this paper, we apply multiplex immunofluorescence to MIBC tissue sections to capture whole-slide images and quantify potential prognostic markers related to lymphocytes, macrophages, tumour buds, and PD-L1. We propose a machine-learning-based approach for the prediction of 5 year prognosis with different combinations of image, clinical, and spatial features. An ensemble model comprising several functionally different models successfully stratifies MIBC patients into two risk groups with high statistical significance (p value < 1×10-5). Critical to improving MIBC survival rates, our method correctly classifies 71.4% of the patients who succumb to MIBC, which is significantly more than the 28.6% of the current clinical gold standard, the TNM staging system.

Corrigendum: Deep Learning for Whole Slide Image Analysis: An Overview.

[This corrects the article on p. 264 in vol. 6, PMID: 31824952.].

Prognostic significance of mesothelin expression in colorectal cancer disclosed by area-specific four-point tissue microarrays.

Mesothelin (MSLN) is a cell surface glycoprotein present in many cancer types. Its expression is generally associated with an unfavorable prognosis. This study examined the prognostic significance of MSLN expression in different areas of individual colorectal cancers (CRCs) using tissue microarrays (TMAs) by enrolling 314 patients with stage II (T3-T4, N0, M0) CRCs. Using formalin-fixed paraffin-embedded tissue blocks from patients, TMA blocks were constructed. Tissue core specimens were obtained from submucosal invasive front (Fr-sm), subserosal invasive front (Fr-ss), central area (Ce), and rolled edge (Ro) of each tumor. Using these four-point TMA sets, MSLN expression was immunohistochemically surveyed. The area-specific prognostic significance of MSLN expression was evaluated. A deep learning convolutional neural network algorithm was used for imaging analysis and evaluating our judgment's objectivity. MSLN staining ratio was positively correlated between the manual and machine-learning analyses (r = 0.71). The correlation coefficient between Ro and Ce, Ro and Fr-sm, and Ro and Fr-ss was r = 0.63, r = 0.54, and r = 0.61, respectively. Disease-specific survival curves for the MSLN-positive and MSLN-negative groups in Fr-sm, Fr-ss, and Ro were significantly different (five-year survival rates 88.1% and 95.5% (P = 0.024), 85.0 and 96.2% (P = 0.0087), 87.8 and 95.5% (P = 0.051), and 77.9 and 95.8% (P = 0.046) for Fr-sm, Fr-ss, Ce, and Ro, respectively). The analysis performed using area-specific four-point TMAs clearly demonstrated that MSLN expression in stage II CRC was relatively homogeneous within tumors. Additionally, high MSLN expression showed or tended to show unfavorable prognostic significance regardless of the tumor area.

Computerized Image Analysis of Tumor Cell Nuclear Morphology Can Improve Patient Selection for Clinical Trials in Localized Clear Cell Renal Cell Carcinoma.

BACKGROUND: Clinicopathological scores are used to predict the likelihood of recurrence-free survival for patients with clear cell renal cell carcinoma (ccRCC) after surgery. These are fallible, particularly in the middle range. This inevitably means that a significant proportion of ccRCC patients who will not develop recurrent disease enroll into clinical trials. As an exemplar of using digital pathology, we sought to improve the predictive power of "recurrence free" designation in localized ccRCC patients, by precise measurement of ccRCC nuclear morphological features using computational image analysis, thereby replacing manual nuclear grade assessment. MATERIALS AND METHODS: TNM 8 UICC pathological stage pT1-pT3 ccRCC cases were recruited in Scotland and in Singapore. A Leibovich score (LS) was calculated. Definiens Tissue studio® (Definiens GmbH, Munich) image analysis platform was used to measure tumor nuclear morphological features in digitized hematoxylin and eosin (H&E) images. RESULTS: Replacing human-defined nuclear grade with computer-defined mean perimeter generated a modified Leibovich algorithm, improved overall specificity 0.86 from 0.76 in the training cohort. The greatest increase in specificity was seen in LS 5 and 6, which went from 0 to 0.57 and 0.40, respectively. The modified Leibovich algorithm increased the specificity from 0.84 to 0.94 in the validation cohort. CONCLUSIONS: CcRCC nuclear mean perimeter, measured by computational image analysis, together with tumor stage and size, node status and necrosis improved the accuracy of predicting recurrence-free in the localized ccRCC patients. This finding was validated in an ethnically different Singaporean cohort, despite the different H and E staining protocol and scanner used. This may be a useful patient selection tool for recruitment to multicenter studies, preventing some patients from receiving unnecessary additional treatment while reducing the number of patients required to achieve adequate power within neoadjuvant and adjuvant clinical studies.

Spatial immune profiling of the colorectal tumor microenvironment predicts good outcome in stage II patients.

Cellular subpopulations within the colorectal tumor microenvironment (TME) include CD3+ and CD8+ lymphocytes, CD68+ and CD163+ macrophages, and tumor buds (TBs), all of which have known prognostic significance in stage II colorectal cancer. However, the prognostic relevance of their spatial interactions remains unknown. Here, by applying automated image analysis and machine learning approaches, we evaluate the prognostic significance of these cellular subpopulations and their spatial interactions. Resultant data, from a training cohort retrospectively collated from Edinburgh, UK hospitals (n = 113), were used to create a combinatorial prognostic model, which identified a subpopulation of patients who exhibit 100% survival over a 5-year follow-up period. The combinatorial model integrated lymphocytic infiltration, the number of lymphocytes within 50-µm proximity to TBs, and the CD68+/CD163+ macrophage ratio. This finding was confirmed on an independent validation cohort, which included patients treated in Japan and Scotland (n = 117). This work shows that by analyzing multiple cellular subpopulations from the complex TME, it is possible to identify patients for whom surgical resection alone may be curative.

Deep Learning for Whole Slide Image Analysis: An Overview.

The widespread adoption of whole slide imaging has increased the demand for effective and efficient gigapixel image analysis. Deep learning is at the forefront of computer vision, showcasing significant improvements over previous methodologies on visual understanding. However, whole slide images have billions of pixels and suffer from high morphological heterogeneity as well as from different types of artifacts. Collectively, these impede the conventional use of deep learning. For the clinical translation of deep learning solutions to become a reality, these challenges need to be addressed. In this paper, we review work on the interdisciplinary attempt of training deep neural networks using whole slide images, and highlight the different ideas underlying these methodologies.

Automated Analysis of Lymphocytic Infiltration, Tumor Budding, and Their Spatial Relationship Improves Prognostic Accuracy in Colorectal Cancer.

Both immune profiling and tumor budding significantly correlate with colorectal cancer patient outcome but are traditionally reported independently. This study evaluated the association and interaction between lymphocytic infiltration and tumor budding, coregistered on a single slide, in order to determine a more precise prognostic algorithm for patients with stage II colorectal cancer. Multiplexed immunofluorescence and automated image analysis were used for the quantification of CD3+CD8+ T cells, and tumor buds (TBs), across whole slide images of three independent cohorts (training cohort: n = 114, validation cohort 1: n = 56, validation cohort 2: n = 62). Machine learning algorithms were used for feature selection and prognostic risk model development. High numbers of TBs [HR = 5.899; 95% confidence interval (CI) 1.875-18.55], low CD3+ T-cell density (HR = 9.964; 95% CI, 3.156-31.46), and low mean number of CD3+CD8+ T cells within 50 µm of TBs (HR = 8.907; 95% CI, 2.834-28.0) were associated with reduced disease-specific survival. A prognostic signature, derived from integrating TBs, lymphocyte infiltration, and their spatial relationship, reported a more significant cohort stratification (HR = 18.75; 95% CI, 6.46-54.43), than TBs, Immunoscore, or pT stage. This was confirmed in two independent validation cohorts (HR = 12.27; 95% CI, 3.524-42.73; HR = 15.61; 95% CI, 4.692-51.91). The investigation of the spatial relationship between lymphocytes and TBs within the tumor microenvironment improves accuracy of prognosis of patients with stage II colorectal cancer through an automated image analysis and machine learning workflow.

Automated tumour budding quantification by machine learning augments TNM staging in muscle-invasive bladder cancer prognosis.

Tumour budding has been described as an independent prognostic feature in several tumour types. We report for the first time the relationship between tumour budding and survival evaluated in patients with muscle invasive bladder cancer. A machine learning-based methodology was applied to accurately quantify tumour buds across immunofluorescence labelled whole slide images from 100 muscle invasive bladder cancer patients. Furthermore, tumour budding was found to be correlated to TNM (p = 0.00089) and pT (p = 0.0078) staging. A novel classification and regression tree model was constructed to stratify all stage II, III, and IV patients into three new staging criteria based on disease specific survival. For the stratification of non-metastatic patients into high or low risk of disease specific death, our decision tree model reported that tumour budding was the most significant feature (HR = 2.59, p = 0.0091), and no clinical feature was utilised to categorise these patients. Our findings demonstrate that tumour budding, quantified using automated image analysis provides prognostic value for muscle invasive bladder cancer patients and a better model fit than TNM staging.

Novel Internationally Verified Method Reports Desmoplastic Reaction as the Most Significant Prognostic Feature For Disease-specific Survival in Stage II Colorectal Cancer.

Multiple histopathologic features have been reported as candidates for predicting aggressive stage II colorectal cancer (CRC). These include tumor budding (TB), poorly differentiated clusters (PDC), Crohn-like lymphoid reaction and desmoplastic reaction (DR) categorization. Although their individual prognostic significance has been established, their association with disease-specific survival (DSS) has not been compared in stage II CRC. This study aimed to evaluate and compare the prognostic value of the above features in a Japanese (n=283) and a Scottish (n=163) cohort, as well as to compare 2 different reporting methodologies: analyzing each feature from across every tissue slide from the whole tumor and a more efficient methodology reporting each feature from a single slide containing the deepest tumor invasion. In the Japanese cohort, there was an excellent agreement between the multi-slide and single-slide methodologies for TB, PDC, and DR (κ=0.798 to 0.898) and a good agreement when assessing Crohn-like lymphoid reaction (κ=0.616). TB (hazard ratio [HR]=1.773; P=0.016), PDC (HR=1.706; P=0.028), and DR (HR=2.982; P<0.001) based on the single-slide method were all significantly associated with DSS. DR was the only candidate feature reported to be a significant independent prognostic factor (HR=2.982; P<0.001) with both multi-slide and single-slide methods. The single-slide result was verified in the Scottish cohort, where multivariate Cox regression analysis reported that DR was the only significant independent feature (HR=1.778; P=0.002) associated with DSS. DR was shown to be the most significant of all the analyzed histopathologic features to predict disease-specific death in stage II CRC. We further show that analyzing the features from a single-slide containing the tumor's deepest invasion is an efficient and quicker method of evaluation.

A principled machine learning framework improves accuracy of stage II colorectal cancer prognosis.

Accurate prognosis is fundamental in planning an appropriate therapy for cancer patients. Consequent to the heterogeneity of the disease, intra- and inter-pathologist variability, and the inherent limitations of current pathological reporting systems, patient outcome varies considerably within similarly staged patient cohorts. This is particularly true when classifying stage II colorectal cancer patients using the current TNM guidelines. The aim of the present work is to address this problem through the use of machine learning. In particular, we introduce a data driven framework which makes use of a large number of diverse types of features, readily collected from immunofluorescence imagery. Its outstanding performance in predicting mortality in stage II patients (AUROC = 0:94), exceeds that of current clinical guidelines such as pT stage (AUROC = 0:65), and is demonstrated on a cohort of 173 colorectal cancer patients.

Next-Generation Pathology.

The field of pathology is rapidly transforming from a semiquantitative and empirical science toward a big data discipline. Large data sets from across multiple omics fields may now be extracted from a patient's tissue sample. Tissue is, however, complex, heterogeneous, and prone to artifact. A reductionist view of tissue and disease progression, which does not take this complexity into account, may lead to single biomarkers failing in clinical trials. The integration of standardized multi-omics big data and the retention of valuable information on spatial heterogeneity are imperative to model complex disease mechanisms. Mathematical modeling through systems pathology approaches is the ideal medium to distill the significant information from these large, multi-parametric, and hierarchical data sets. Systems pathology may also predict the dynamical response of disease progression or response to therapy regimens from a static tissue sample. Next-generation pathology will incorporate big data with systems medicine in order to personalize clinical practice for both prognostic and predictive patient care.

Novel histopathologic feature identified through image analysis augments stage II colorectal cancer clinical reporting.

A number of candidate histopathologic factors show promise in identifying stage II colorectal cancer (CRC) patients at a high risk of disease-specific death, however they can suffer from low reproducibility and none have replaced classical pathologic staging. We developed an image analysis algorithm which standardized the quantification of specific histopathologic features and exported a multi-parametric feature-set captured without bias. The image analysis algorithm was executed across a training set (n = 50) and the resultant big data was distilled through decision tree modelling to identify the most informative parameters to sub-categorize stage II CRC patients. The most significant, and novel, parameter identified was the 'sum area of poorly differentiated clusters' (AreaPDC). This feature was validated across a second cohort of stage II CRC patients (n = 134) (HR = 4; 95% CI, 1.5- 11). Finally, the AreaPDC was integrated with the significant features within the clinical pathology report, pT stage and differentiation, into a novel prognostic index (HR = 7.5; 95% CI, 3-18.5) which improved upon current clinical staging (HR = 4.26; 95% CI, 1.7- 10.3). The identification of poorly differentiated clusters as being highly significant in disease progression presents evidence to suggest that these features could be the source of novel targets to decrease the risk of disease specific death.

Human tissue in systems medicine.

Histopathology, the examination of an architecturally artefactual, two-dimensional and static image remains a potent tool allowing diagnosis and empirical expectation of prognosis. Considerable optimism exists that the advent of molecular genetic testing and other biomarker strategies will improve or even replace this ancient technology. A number of biomarkers already add considerable value for prediction of whether a treatment will work. In this short review we argue that a systems medicine approach to pathology will not seek to replace traditional pathology, but rather augment it. Systems approaches need to incorporate quantitative morphological, protein, mRNA and DNA data. A significant challenge for clinical implementation of systems pathology is how to optimize information available from tissue, which is frequently sub-optimal in quality and amount, and yet generate useful predictive models that work. The transition of histopathology to systems pathophysiology and the use of multiscale data sets usher in a new era in diagnosis, prognosis and prediction based on the analysis of human tissue.