Image-based consensus molecular subtyping in rectal cancer biopsies and response to neoadjuvant chemoradiotherapy.

The development of deep learning (DL) models to predict the consensus molecular subtypes (CMS) from histopathology images (imCMS) is a promising and cost-effective strategy to support patient stratification. Here, we investigate whether imCMS calls generated from whole slide histopathology images (WSIs) of rectal cancer (RC) pre-treatment biopsies are associated with pathological complete response (pCR) to neoadjuvant long course chemoradiotherapy (LCRT) with single agent fluoropyrimidine. DL models were trained to classify WSIs of colorectal cancers stained with hematoxylin and eosin into one of the four CMS classes using a multi-centric dataset of resection and biopsy specimens (n = 1057 WSIs) with paired transcriptional data. Classifiers were tested on a held out RC biopsy cohort (ARISTOTLE) and correlated with pCR to LCRT in an independent dataset merging two RC cohorts (ARISTOTLE, n = 114 and SALZBURG, n = 55 patients). DL models predicted CMS with high classification performance in multiple comparative analyses. In the independent cohorts (ARISTOTLE, SALZBURG), cases with WSIs classified as imCMS1 had a significantly higher likelihood of achieving pCR (OR = 2.69, 95% CI 1.01-7.17, p = 0.048). Conversely, imCMS4 was associated with lack of pCR (OR = 0.25, 95% CI 0.07-0.88, p = 0.031). Classification maps demonstrated pathologist-interpretable associations with high stromal content in imCMS4 cases, associated with poor outcome. No significant association was found in imCMS2 or imCMS3. imCMS classification of pre-treatment biopsies is a fast and inexpensive solution to identify patient groups that could benefit from neoadjuvant LCRT. The significant associations between imCMS1/imCMS4 with pCR suggest the existence of predictive morphological features that could enhance standard pathological assessment.

Continuous Indexing of Fibrosis (CIF): improving the assessment and classification of MPN patients.

The grading of fibrosis in myeloproliferative neoplasms (MPN) is an important component of disease classification, prognostication and monitoring. However, current fibrosis grading systems are only semi-quantitative and fail to fully capture sample heterogeneity. To improve the quantitation of reticulin fibrosis, we developed a machine learning approach using bone marrow trephine (BMT) samples (n = 107) from patients diagnosed with MPN or a reactive marrow. The resulting Continuous Indexing of Fibrosis (CIF) enhances the detection and monitoring of fibrosis within BMTs, and aids MPN subtyping. When combined with megakaryocyte feature analysis, CIF discriminates between the frequently challenging differential diagnosis of essential thrombocythemia (ET) and pre-fibrotic myelofibrosis with high predictive accuracy [area under the curve = 0.94]. CIF also shows promise in the identification of MPN patients at risk of disease progression; analysis of samples from 35 patients diagnosed with ET and enrolled in the Primary Thrombocythemia-1 trial identified features predictive of post-ET myelofibrosis (area under the curve = 0.77). In addition to these clinical applications, automated analysis of fibrosis has clear potential to further refine disease classification boundaries and inform future studies of the micro-environmental factors driving disease initiation and progression in MPN and other stem cell disorders.

A Graph Based Neural Network Approach to Immune Profiling of Multiplexed Tissue Samples.

Multiplexed immunofluorescence provides an un-precedented opportunity for studying specific cell-to-cell and cell microenvironment interactions. We employ graph neural networks to combine features obtained from tissue morphology with measurements of protein expression to profile the tumour microenvironment associated with different tumour stages. Our framework presents a new approach to analysing and processing these complex multi-dimensional datasets that overcomes some of the key challenges in analysing these data and opens up the opportunity to abstract biologically meaningful interactions.

Loss of COP9 signalosome genes at 2q37 is associated with IMiD resistance in multiple myeloma.

The acquisition of a multidrug refractory state is a major cause of mortality in myeloma. Myeloma drugs that target the cereblon (CRBN) protein include widely used immunomodulatory drugs (IMiDs), and newer CRBN E3 ligase modulator drugs (CELMoDs), in clinical trials. CRBN genetic disruption causes resistance and poor outcomes with IMiDs. Here, we investigate alternative genomic associations of IMiD resistance, using large whole-genome sequencing patient datasets (n = 522 cases) at newly diagnosed, lenalidomide (LEN)-refractory and lenalidomide-then-pomalidomide (LEN-then-POM)-refractory timepoints. Selecting gene targets reproducibly identified by published CRISPR/shRNA IMiD resistance screens, we found little evidence of genetic disruption by mutation associated with IMiD resistance. However, we identified a chromosome region, 2q37, containing COP9 signalosome members COPS7B and COPS8, copy loss of which significantly enriches between newly diagnosed (incidence 5.5%), LEN-refractory (10.0%), and LEN-then-POM-refractory states (16.4%), and may adversely affect outcomes when clonal fraction is high. In a separate dataset (50 patients) with sequential samples taken throughout treatment, we identified acquisition of 2q37 loss in 16% cases with IMiD exposure, but none in cases without IMiD exposure. The COP9 signalosome is essential for maintenance of the CUL4-DDB1-CRBN E3 ubiquitin ligase. This region may represent a novel marker of IMiD resistance with clinical utility.

Automated quality assessment of large digitised histology cohorts by artificial intelligence.

Research using whole slide images (WSIs) of histopathology slides has increased exponentially over recent years. Glass slides from retrospective cohorts, some with patient follow-up data are digitised for the development and validation of artificial intelligence (AI) tools. Such resources, therefore, become very important, with the need to ensure that their quality is of the standard necessary for downstream AI development. However, manual quality control of large cohorts of WSIs by visual assessment is unfeasible, and whilst quality control AI algorithms exist, these focus on bespoke aspects of image quality, e.g. focus, or use traditional machine-learning methods, which are unable to classify the range of potential image artefacts that should be considered. In this study, we have trained and validated a multi-task deep neural network to automate the process of quality control of a large retrospective cohort of prostate cases from which glass slides have been scanned several years after production, to determine both the usability of the images at the diagnostic level (considered in this study to be the minimal standard for research) and the common image artefacts present. Using a two-layer approach, quality overlays of WSIs were generated from a quality assessment (QA) undertaken at patch-level at [Formula: see text] magnification. From these quality overlays the slide-level quality scores were predicted and then compared to those generated by three specialist urological pathologists, with a Pearson correlation of 0.89 for overall 'usability' (at a diagnostic level), and 0.87 and 0.82 for focus and H&E staining quality scores respectively. To demonstrate its wider potential utility, we subsequently applied our QA pipeline to the TCGA prostate cancer cohort and to a colorectal cancer cohort, for comparison. Our model, designated as PathProfiler, indicates comparable predicted usability of images from the cohorts assessed (86-90% of WSIs predicted to be usable), and perhaps more significantly is able to predict WSIs that could benefit from an intervention such as re-scanning or re-staining for quality improvement. We have shown in this study that AI can be used to automate the process of quality control of large retrospective WSI cohorts to maximise their utility for research.

Learning Cellular Phenotypes through Supervision.

Image-based cell phenotyping is an important and open problem in computational pathology. The two principal challenges are: 1) making the cell cluster properties insensitive to experimental settings (like seed point and feature selection) and 2) ensuring that the phenotypes emerging are biologically relevant and support clinical reporting. To gauge robustness, we first compare the consistency of the phenotypes using self-supervised and supervised features. Through case classification, we analyse the relevance of the self-supervised and supervised feature sets with respect to the clinical diagnosis. In addition, we demonstrate how we can add model explainability through Shapley values to identify more disease relevant cellular phenotypes and measure their importance in context of the disease. Here, myeloproliferative neoplasms, a haematopoietic stem cell disorder, where one particular cell type is of diagnostic relevance is used as an exemplar. The experiments conducted on a set of bone marrow trephines demonstrate an improvement of 7.4 % in accuracy for case classification using cellular phenotypes derived from the supervised scenario.

Self-Organizing Maps for Cellular In Silico Staining and Cell Substate Classification.

Cellular composition and structural organization of cells in the tissue determine effective antitumor response and can predict patient outcome and therapy response. Here we present Seg-SOM, a method for dimensionality reduction of cell morphology in H&E-stained tissue images. Seg-SOM resolves cellular tissue heterogeneity and reveals complex tissue architecture. We leverage a self-organizing map (SOM) artificial neural network to group cells based on morphological features like shape and size. Seg-SOM allows for cell segmentation, systematic classification, and in silico cell labeling. We apply the Seg-SOM to a dataset of breast cancer progression images and find that clustering of SOM classes reveals groups of cells corresponding to fibroblasts, epithelial cells, and lymphocytes. We show that labeling the Lymphocyte SOM class on the breast tissue images accurately estimates lymphocytic infiltration. We further demonstrate how to use Seq-SOM in combination with non-negative matrix factorization to statistically describe the interaction of cell subtypes and use the interaction information as highly interpretable features for a histological classifier. Our work provides a framework for use of SOM in human pathology to resolve cellular composition of complex human tissues. We provide a python implementation and an easy-to-use docker deployment, enabling researchers to effortlessly featurize digitalized H&E-stained tissue.

Transcriptome and genome evolution during HER2-amplified breast neoplasia.

BACKGROUND: The acquisition of oncogenic drivers is a critical feature of cancer progression. For some carcinomas, it is clear that certain genetic drivers occur early in neoplasia and others late. Why these drivers are selected and how these changes alter the neoplasia's fitness is less understood. METHODS: Here we use spatially oriented genomic approaches to identify transcriptomic and genetic changes at the single-duct level within precursor neoplasia associated with invasive breast cancer. We study HER2 amplification in ductal carcinoma in situ (DCIS) as an event that can be both quantified and spatially located via fluorescence in situ hybridization (FISH) and immunohistochemistry on fixed paraffin-embedded tissue. RESULTS: By combining the HER2-FISH with the laser capture microdissection (LCM) Smart-3SEQ method, we found that HER2 amplification in DCIS alters the transcriptomic profiles and increases diversity of copy number variations (CNVs). Particularly, interferon signaling pathway is activated by HER2 amplification in DCIS, which may provide a prolonged interferon signaling activation in HER2-positive breast cancer. Multiple subclones of HER2-amplified DCIS with distinct CNV profiles are observed, suggesting that multiple events occurred for the acquisition of HER2 amplification. Notably, DCIS acquires key transcriptomic changes and CNV events prior to HER2 amplification, suggesting that pre-amplified DCIS may create a cellular state primed to gain HER2 amplification for growth advantage. CONCLUSION: By using genomic methods that are spatially oriented, this study identifies several features that appear to generate insights into neoplastic progression in precancer lesions at a single-duct level.

Associations of reproductive breast cancer risk factors with breast tissue composition.

BACKGROUND: We investigated the associations of reproductive factors with the percentage of epithelium, stroma, and fat tissue in benign breast biopsy samples. METHODS: This study included 983 cancer-free women with biopsy-confirmed benign breast disease (BBD) within the Nurses' Health Study and Nurses' Health Study II cohorts. The percentage of each tissue type (epithelium, stroma, and fat) was measured on whole-section images with a deep-learning technique. All tissue measures were log-transformed in all the analyses to improve normality. The data on reproductive variables and other breast cancer risk factors were obtained from biennial questionnaires. Generalized linear regression was used to examine the associations of reproductive factors with the percentage of tissue types, while adjusting for known breast cancer risk factors. RESULTS: As compared to parous women, nulliparous women had a smaller percentage of epithelium (ß = - 0.26, 95% confidence interval [CI] - 0.41, - 0.11) and fat (ß = - 0.34, 95% CI - 0.54, - 0.13) and a greater percentage of stroma (ß = 0.04, 95% CI 0.01, 0.08). Among parous women, the number of children was inversely associated with the percentage of stroma (ß per child = - 0.01, 95% CI - 0.02, - 0.00). The duration of breastfeeding of ≥ 24 months was associated with a reduced proportion of fat (ß = - 0.30, 95% CI - 0.54, - 0.06; p-trend = 0.04). In a separate analysis restricted to premenopausal women, older age at first birth was associated with a greater proportion of epithelium and a smaller proportion of stroma. CONCLUSIONS: Our findings suggest that being nulliparous as well as having a fewer number of children (both positively associated with breast cancer risk) is associated with a smaller proportion of epithelium and a greater proportion of stroma, potentially suggesting the importance of epithelial-stromal interactions. Future studies are warranted to confirm our findings and to elucidate the underlying biological mechanisms.

Artificial intelligence for advance requesting of immunohistochemistry in diagnostically uncertain prostate biopsies.

The use of immunohistochemistry in the reporting of prostate biopsies is an important adjunct when the diagnosis is not definite on haematoxylin and eosin (H&E) morphology alone. The process is however inherently inefficient with delays while waiting for pathologist review to make the request and duplicated effort reviewing a case more than once. In this study, we aimed to capture the workflow implications of immunohistochemistry requests and demonstrate a novel artificial intelligence tool to identify cases in which immunohistochemistry (IHC) is required and generate an automated request. We conducted audits of the workflow for prostate biopsies in order to understand the potential implications of automated immunohistochemistry requesting and collected prospective cases to train a deep neural network algorithm to detect tissue regions that presented ambiguous morphology on whole slide images. These ambiguous foci were selected on the basis of the pathologist requesting immunohistochemistry to aid diagnosis. A gradient boosted trees classifier was then used to make a slide-level prediction based on the outputs of the neural network prediction. The algorithm was trained on annotations of 219 immunohistochemistry-requested and 80 control images, and tested by threefold cross-validation. Validation was conducted on a separate validation dataset of 222 images. Non IHC-requested cases were diagnosed in 17.9 min on average, while IHC-requested cases took 33.4 min over multiple reporting sessions. We estimated 11 min could be saved on average per case by automated IHC requesting, by removing duplication of effort. The tool attained 99% accuracy and 0.99 Area Under the Curve (AUC) on the test data. In the validation, the average agreement with pathologists was 0.81, with a mean AUC of 0.80. We demonstrate the proof-of-principle that an AI tool making automated immunohistochemistry requests could create a significantly leaner workflow and result in pathologist time savings.

The Potential of Artificial Intelligence to Detect Lymphovascular Invasion in Testicular Cancer.

Testicular cancer is the most common cancer in men aged from 15 to 34 years. Lymphovascular invasion refers to the presence of tumours within endothelial-lined lymphatic or vascular channels, and has been shown to have prognostic significance in testicular germ cell tumours. In non-seminomatous tumours, lymphovascular invasion is the most powerful prognostic factor for stage 1 disease. For the pathologist, searching multiple slides for lymphovascular invasion can be highly time-consuming. The aim of this retrospective study was to develop and assess an artificial intelligence algorithm that can identify areas suspicious for lymphovascular invasion in histological digital whole slide images. Areas of possible lymphovascular invasion were annotated in a total of 184 whole slide images of haematoxylin and eosin (H&E) stained tissue from 19 patients with testicular germ cell tumours, including a mixture of seminoma and non-seminomatous cases. Following consensus review by specialist uropathologists, we trained a deep learning classifier for automatic segmentation of areas suspicious for lymphovascular invasion. The classifier identified 34 areas within a validation set of 118 whole slide images from 10 patients, each of which was reviewed by three expert pathologists to form a majority consensus. The precision was 0.68 for areas which were considered to be appropriate to flag, and 0.56 for areas considered to be definite lymphovascular invasion. An artificial intelligence tool which highlights areas of possible lymphovascular invasion to reporting pathologists, who then make a final judgement on its presence or absence, has been demonstrated as feasible in this proof-of-concept study. Further development is required before clinical deployment.

Early-Life and Adult Adiposity, Adult Height, and Benign Breast Tissue Composition.

BACKGROUND: Early-life and adult anthropometrics are associated with breast density and breast cancer risk. However, little is known about whether these factors also influence breast tissue composition beyond what is captured by breast density among women with benign breast disease (BBD). METHODS: This analysis included 788 controls from a nested case-control study of breast cancer within the Nurses' Health Study BBD subcohorts. Body fatness at ages 5 and 10 years was recalled using a 9-level pictogram. Weight at age 18, current weight, and height were reported via questionnaires. A deep-learning image analysis was used to quantify the percentages of epithelial, fibrous stromal, and adipose tissue areas within BBD slides. We performed linear mixed models to estimate beta coefficients (ß) and 95% confidence intervals (CI) for the relationships between anthropometrics and the log-transformed percentages of individual tissue type, adjusting for confounders. RESULTS: Childhood body fatness (level ≥ 4.5 vs. 1), BMI at age 18 (≥23 vs. <19 kg/m2), and current adult BMI (≥30 vs. <21 kg/m2) were associated with higher proportions of adipose tissue [ß (95% CI) = 0.34 (0.03, 0.65), 0.19 (-0.04-0.42), 0.40 (0.12, 0.68), respectively] and lower proportions of fibrous stromal tissue [-0.05 (-0.10, 0.002), -0.03 (-0.07, 0.003), -0.12 (-0.16, -0.07), respectively] during adulthood (all P trend < 0.04). BMI at age 18 was also inversely associated with epithelial tissue (P trend = 0.03). Adult height was not associated with any of the individual tissue types. CONCLUSIONS: Our data suggest that body fatness has long-term impacts on breast tissue composition. IMPACT: This study contributes to our understanding of the link between body fatness and breast cancer risk.See related commentary by Oskar et al., p. 590.

Image-based consensus molecular subtype (imCMS) classification of colorectal cancer using deep learning.

OBJECTIVE: Complex phenotypes captured on histological slides represent the biological processes at play in individual cancers, but the link to underlying molecular classification has not been clarified or systematised. In colorectal cancer (CRC), histological grading is a poor predictor of disease progression, and consensus molecular subtypes (CMSs) cannot be distinguished without gene expression profiling. We hypothesise that image analysis is a cost-effective tool to associate complex features of tissue organisation with molecular and outcome data and to resolve unclassifiable or heterogeneous cases. In this study, we present an image-based approach to predict CRC CMS from standard H&E sections using deep learning. DESIGN: Training and evaluation of a neural network were performed using a total of n=1206 tissue sections with comprehensive multi-omic data from three independent datasets (training on FOCUS trial, n=278 patients; test on rectal cancer biopsies, GRAMPIAN cohort, n=144 patients; and The Cancer Genome Atlas (TCGA), n=430 patients). Ground truth CMS calls were ascertained by matching random forest and single sample predictions from CMS classifier. RESULTS: Image-based CMS (imCMS) accurately classified slides in unseen datasets from TCGA (n=431 slides, AUC)=0.84) and rectal cancer biopsies (n=265 slides, AUC=0.85). imCMS spatially resolved intratumoural heterogeneity and provided secondary calls correlating with bioinformatic prediction from molecular data. imCMS classified samples previously unclassifiable by RNA expression profiling, reproduced the expected correlations with genomic and epigenetic alterations and showed similar prognostic associations as transcriptomic CMS. CONCLUSION: This study shows that a prediction of RNA expression classifiers can be made from H&E images, opening the door to simple, cheap and reliable biological stratification within routine workflows.

Artificial intelligence-based morphological fingerprinting of megakaryocytes: a new tool for assessing disease in MPN patients.

Accurate diagnosis and classification of myeloproliferative neoplasms (MPNs) requires integration of clinical, morphological, and genetic findings. Despite major advances in our understanding of the molecular and genetic basis of MPNs, the morphological assessment of bone marrow trephines (BMT) is critical in differentiating MPN subtypes and their reactive mimics. However, morphological assessment is heavily constrained by a reliance on subjective, qualitative, and poorly reproducible criteria. To improve the morphological assessment of MPNs, we have developed a machine learning approach for the automated identification, quantitative analysis, and abstract representation of megakaryocyte features using reactive/nonneoplastic BMT samples (n = 43) and those from patients with established diagnoses of essential thrombocythemia (n = 45), polycythemia vera (n = 18), or myelofibrosis (n = 25). We describe the application of an automated workflow for the identification and delineation of relevant histological features from routinely prepared BMTs. Subsequent analysis enabled the tissue diagnosis of MPN with a high predictive accuracy (area under the curve = 0.95) and revealed clear evidence of the potential to discriminate between important MPN subtypes. Our method of visually representing abstracted megakaryocyte features in the context of analyzed patient cohorts facilitates the interpretation and monitoring of samples in a manner that is beyond conventional approaches. The automated BMT phenotyping approach described here has significant potential as an adjunct to standard genetic and molecular testing in established or suspected MPN patients, either as part of the routine diagnostic pathway or in the assessment of disease progression/response to treatment.

Detailed Molecular and Immune Marker Profiling of Archival Prostate Cancer Samples Reveals an Inverse Association between TMPRSS2:ERG Fusion Status and Immune Cell Infiltration.

Prostate cancer is a significant global health issue, and limitations to current patient management pathways often result in overtreatment or undertreatment. New ways to stratify patients are urgently needed. We conducted a feasibility study of such novel assessments, looking for associations between genomic changes and lymphocyte infiltration. An innovative workflow using an in-house targeted sequencing panel, immune cell profiling using an image analysis pipeline, RNA sequencing, and exome sequencing in select cases was tested. Gene fusions were profiled by RNA sequencing in 27 of 27 cases, and a significantly higher tumor-infiltrating lymphocyte (TIL) count was noted in tumors without a TMPRSS2:ERG fusion compared with those with the fusion (P = 0.01). Although this finding was not replicated in a larger validation set (n = 436) of The Cancer Genome Atlas images, there was a trend in the same direction. Differential expression analysis of TIL-high and TIL-low tumors revealed the enrichment of both innate and adaptive immune response pathways. Mutations in mismatch repair genes (MLH1 and MSH6 mutations in 1 of 27 cases) were identified. We describe a potential immune escape mechanism in TMPRSS2:ERG fusion-positive tumors. Detailed profiling, as shown herein, can provide novel insights into tumor biology. Likely differences with findings with other cohorts are related to methods used to define region of interest, but this warrants further study in a larger cohort.

Author Correction: Why rankings of biomedical image analysis competitions should be interpreted with care.

In the original version of this Article the values in the rightmost column of Table 1 were inadvertently shifted relative to the other columns. This has now been corrected in the PDF and HTML versions of the Article.

Precision immunoprofiling by image analysis and artificial intelligence.

Clinical success of immunotherapy is driving the need for new prognostic and predictive assays to inform patient selection and stratification. This requirement can be met by a combination of computational pathology and artificial intelligence. Here, we critically assess computational approaches supporting the development of a standardized methodology in the assessment of immune-oncology biomarkers, such as PD-L1 and immune cell infiltrates. We examine immunoprofiling through spatial analysis of tumor-immune cell interactions and multiplexing technologies as a predictor of patient response to cancer treatment. Further, we discuss how integrated bioinformatics can enable the amalgamation of complex morphological phenotypes with the multiomics datasets that drive precision medicine. We provide an outline to machine learning (ML) and artificial intelligence tools and illustrate fields of application in immune-oncology, such as pattern-recognition in large and complex datasets and deep learning approaches for survival analysis. Synergies of surgical pathology and computational analyses are expected to improve patient stratification in immuno-oncology. We propose that future clinical demands will be best met by (1) dedicated research at the interface of pathology and bioinformatics, supported by professional societies, and (2) the integration of data sciences and digital image analysis in the professional education of pathologists.

Why rankings of biomedical image analysis competitions should be interpreted with care.

International challenges have become the standard for validation of biomedical image analysis methods. Given their scientific impact, it is surprising that a critical analysis of common practices related to the organization of challenges has not yet been performed. In this paper, we present a comprehensive analysis of biomedical image analysis challenges conducted up to now. We demonstrate the importance of challenges and show that the lack of quality control has critical consequences. First, reproducibility and interpretation of the results is often hampered as only a fraction of relevant information is typically provided. Second, the rank of an algorithm is generally not robust to a number of variables such as the test data used for validation, the ranking scheme applied and the observers that make the reference annotations. To overcome these problems, we recommend best practice guidelines and define open research questions to be addressed in the future.

Novel digital signatures of tissue phenotypes for predicting distant metastasis in colorectal cancer.

Distant metastasis is the major cause of death in colorectal cancer (CRC). Patients at high risk of developing distant metastasis could benefit from appropriate adjuvant and follow-up treatments if stratified accurately at an early stage of the disease. Studies have increasingly recognized the role of diverse cellular components within the tumor microenvironment in the development and progression of CRC tumors. In this paper, we show that automated analysis of digitized images from locally advanced colorectal cancer tissue slides can provide estimate of risk of distant metastasis on the basis of novel tissue phenotypic signatures of the tumor microenvironment. Specifically, we determine what cell types are found in the vicinity of other cell types, and in what numbers, rather than concentrating exclusively on the cancerous cells. We then extract novel tissue phenotypic signatures using statistical measurements about tissue composition. Such signatures can underpin clinical decisions about the advisability of various types of adjuvant therapy.