Artificial Intelligence-Based Mitosis Scoring in Breast Cancer: Clinical Application.

In recent years, artificial intelligence (AI) has demonstrated exceptional performance in mitosis identification and quantification. However, the implementation of AI in clinical practice needs to be evaluated against the existing methods. This study is aimed at assessing the optimal method of using AI-based mitotic figure scoring in breast cancer (BC). We utilized whole slide images from a large cohort of BC with extended follow-up comprising a discovery (n = 1715) and a validation (n = 859) set (Nottingham cohort). The Cancer Genome Atlas of breast invasive carcinoma (TCGA-BRCA) cohort (n = 757) was used as an external test set. Employing automated mitosis detection, the mitotic count was assessed using 3 different methods, the mitotic count per tumor area (MCT; calculated by dividing the number of mitotic figures by the total tumor area), the mitotic index (MI; defined as the average number of mitotic figures per 1000 malignant cells), and the mitotic activity index (MAI; defined as the number of mitotic figures in 3 mm2 area within the mitotic hotspot). These automated metrics were evaluated and compared based on their correlation with the well-established visual scoring method of the Nottingham grading system and Ki67 score, clinicopathologic parameters, and patient outcomes. AI-based mitotic scores derived from the 3 methods (MCT, MI, and MAI) were significantly correlated with the clinicopathologic characteristics and patient survival (P < .001). However, the mitotic counts and the derived cutoffs varied significantly between the 3 methods. Only MAI and MCT were positively correlated with the gold standard visual scoring method used in Nottingham grading system (r = 0.8 and r = 0.7, respectively) and Ki67 scores (r = 0.69 and r = 0.55, respectively), and MAI was the only independent predictor of survival (P < .05) in multivariate Cox regression analysis. For clinical applications, the optimum method of scoring mitosis using AI needs to be considered. MAI can provide reliable and reproducible results and can accurately quantify mitotic figures in BC.

Screening of normal endoscopic large bowel biopsies with interpretable graph learning: a retrospective study.

OBJECTIVE: To develop an interpretable artificial intelligence algorithm to rule out normal large bowel endoscopic biopsies, saving pathologist resources and helping with early diagnosis. DESIGN: A graph neural network was developed incorporating pathologist domain knowledge to classify 6591 whole-slides images (WSIs) of endoscopic large bowel biopsies from 3291 patients (approximately 54% female, 46% male) as normal or abnormal (non-neoplastic and neoplastic) using clinically driven interpretable features. One UK National Health Service (NHS) site was used for model training and internal validation. External validation was conducted on data from two other NHS sites and one Portuguese site. RESULTS: Model training and internal validation were performed on 5054 WSIs of 2080 patients resulting in an area under the curve-receiver operating characteristic (AUC-ROC) of 0.98 (SD=0.004) and AUC-precision-recall (PR) of 0.98 (SD=0.003). The performance of the model, named Interpretable Gland-Graphs using a Neural Aggregator (IGUANA), was consistent in testing over 1537 WSIs of 1211 patients from three independent external datasets with mean AUC-ROC=0.97 (SD=0.007) and AUC-PR=0.97 (SD=0.005). At a high sensitivity threshold of 99%, the proposed model can reduce the number of normal slides to be reviewed by a pathologist by approximately 55%. IGUANA also provides an explainable output highlighting potential abnormalities in a WSI in the form of a heatmap as well as numerical values associating the model prediction with various histological features. CONCLUSION: The model achieved consistently high accuracy showing its potential in optimising increasingly scarce pathologist resources. Explainable predictions can guide pathologists in their diagnostic decision-making and help boost their confidence in the algorithm, paving the way for its future clinical adoption.

Role of AI and digital pathology for colorectal immuno-oncology.

Immunotherapy deals with therapeutic interventions to arrest the progression of tumours using the immune system. These include checkpoint inhibitors, T-cell manipulation, cytokines, oncolytic viruses and tumour vaccines. In this paper, we present a survey of the latest developments on immunotherapy in colorectal cancer (CRC) and the role of artificial intelligence (AI) in this context. Among these, microsatellite instability (MSI) is perhaps the most popular IO biomarker globally. We first discuss the MSI status of tumours, its implications for patient management, and its relationship to immune response. In recent years, several aspiring studies have used AI to predict the MSI status of patients from digital whole-slide images (WSIs) of routine diagnostic slides. We present a survey of AI literature on the prediction of MSI and tumour mutation burden from digitised WSIs of haematoxylin and eosin-stained diagnostic slides. We discuss AI approaches in detail and elaborate their contributions, limitations and key takeaways to drive future research. We further expand this survey to other IO-related biomarkers like immune cell infiltrates and alternate data modalities like immunohistochemistry and gene expression. Finally, we underline possible future directions in immunotherapy for CRC and promise of AI to accelerate this exploration for patient benefits.

Evaluation of tumour infiltrating lymphocytes in luminal breast cancer using artificial intelligence.

BACKGROUND: Tumour infiltrating lymphocytes (TILs) are a prognostic parameter in triple-negative and human epidermal growth factor receptor 2 (HER2)-positive breast cancer (BC). However, their role in luminal (oestrogen receptor positive and HER2 negative (ER + /HER2-)) BC remains unclear. In this study, we used artificial intelligence (AI) to assess the prognostic significance of TILs in a large well-characterised cohort of luminal BC. METHODS: Supervised deep learning model analysis of Haematoxylin and Eosin (H&E)-stained whole slide images (WSI) was applied to a cohort of 2231 luminal early-stage BC patients with long-term follow-up. Stromal TILs (sTILs) and intratumoural TILs (tTILs) were quantified and their spatial distribution within tumour tissue, as well as the proportion of stroma involved by sTILs were assessed. The association of TILs with clinicopathological parameters and patient outcome was determined. RESULTS: A strong positive linear correlation was observed between sTILs and tTILs. High sTILs and tTILs counts, as well as their proximity to stromal and tumour cells (co-occurrence) were associated with poor clinical outcomes and unfavourable clinicopathological parameters including high tumour grade, lymph node metastasis, large tumour size, and young age. AI-based assessment of the proportion of stroma composed of sTILs (as assessed visually in routine practice) was not predictive of patient outcome. tTILs was an independent predictor of worse patient outcome in multivariate Cox Regression analysis. CONCLUSION: AI-based detection of TILs counts, and their spatial distribution provides prognostic value in luminal early-stage BC patients. The utilisation of AI algorithms could provide a comprehensive assessment of TILs as a morphological variable in WSIs beyond eyeballing assessment.

Deciphering the Morphology of Tumor-Stromal Features in Invasive Breast Cancer Using Artificial Intelligence.

Tumor-associated stroma in breast cancer (BC) is complex and exhibits a high degree of heterogeneity. To date, no standardized assessment method has been established. Artificial intelligence (AI) could provide an objective morphologic assessment of tumors and stroma, with the potential to identify new features not discernible by visual microscopy. In this study, we used AI to assess the clinical significance of (1) stroma-to-tumor ratio (S:TR) and (2) the spatial arrangement of stromal cells, tumor cell density, and tumor burden in BC. Whole-slide images of a large cohort (n = 1968) of well-characterized luminal BC cases were examined. Region and cell-level annotation was performed, and supervised deep learning models were applied for automated quantification of tumor and stromal features. S:TR was calculated in terms of surface area and cell count ratio, and the S:TR heterogeneity and spatial distribution were also assessed. Tumor cell density and tumor size were used to estimate tumor burden. Cases were divided into discovery (n = 1027) and test (n = 941) sets for validation of the findings. In the whole cohort, the stroma-to-tumor mean surface area ratio was 0.74, and stromal cell density heterogeneity score was high (0.7/1). BC with high S:TR showed features characteristic of good prognosis and longer patient survival in both the discovery and test sets. Heterogeneous spatial distribution of S:TR areas was predictive of worse outcome. Higher tumor burden was associated with aggressive tumor behavior and shorter survival and was an independent predictor of worse outcome (BC-specific survival; hazard ratio: 1.7, P = .03, 95% CI, 1.04-2.83 and distant metastasis-free survival; hazard ratio: 1.64, P = .04, 95% CI, 1.01-2.62) superior to absolute tumor size. The study concludes that AI provides a tool to assess major and subtle morphologic stromal features in BC with prognostic implications. Tumor burden is more prognostically informative than tumor size.

A Federated Learning Approach to Tumor Detection in Colon Histology Images.

Federated learning (FL), a relatively new area of research in medical image analysis, enables collaborative learning of a federated deep learning model without sharing the data of participating clients. In this paper, we propose FedDropoutAvg, a new federated learning approach for detection of tumor in images of colon tissue slides. The proposed method leverages the power of dropout, a commonly employed scheme to avoid overfitting in neural networks, in both client selection and federated averaging processes. We examine FedDropoutAvg against other FL benchmark algorithms for two different image classification tasks using a publicly available multi-site histopathology image dataset. We train and test the proposed model on a large dataset consisting of 1.2 million image tiles from 21 different sites. For testing the generalization of all models, we select held-out test sets from sites that were not used during training. We show that the proposed approach outperforms other FL methods and reduces the performance gap (to less than 3% in terms of AUC on independent test sites) between FL and a central deep learning model that requires all data to be shared for centralized training, demonstrating the potential of the proposed FedDropoutAvg model to be more generalizable than other state-of-the-art federated models. To the best of our knowledge, ours is the first study to effectively utilize the dropout strategy in a federated setting for tumor detection in histology images.

Better physical activity classification using smartphone acceleration sensor.

Obesity is becoming one of the serious problems for the health of worldwide population. Social interactions on mobile phones and computers via internet through social e-networks are one of the major causes of lack of physical activities. For the health specialist, it is important to track the record of physical activities of the obese or overweight patients to supervise weight loss control. In this study, acceleration sensor present in the smartphone is used to monitor the physical activity of the user. Physical activities including Walking, Jogging, Sitting, Standing, Walking upstairs and Walking downstairs are classified. Time domain features are extracted from the acceleration data recorded by smartphone during different physical activities. Time and space complexity of the whole framework is done by optimal feature subset selection and pruning of instances. Classification results of six physical activities are reported in this paper. Using simple time domain features, 99 % classification accuracy is achieved. Furthermore, attributes subset selection is used to remove the redundant features and to minimize the time complexity of the algorithm. A subset of 30 features produced more than 98 % classification accuracy for the six physical activities.

Social network analysis of cell networks improves deep learning for prediction of molecular pathways and key mutations in colorectal cancer.

Colorectal cancer (CRC) is a primary global health concern, and identifying the molecular pathways, genetic subtypes, and mutations associated with CRC is crucial for precision medicine. However, traditional measurement techniques such as gene sequencing are costly and time-consuming, while most deep learning methods proposed for this task lack interpretability. This study offers a new approach to enhance the state-of-the-art deep learning methods for molecular pathways and key mutation prediction by incorporating cell network information. We build cell graphs with nuclei as nodes and nuclei connections as edges of the network and leverage Social Network Analysis (SNA) measures to extract abstract, perceivable, and interpretable features that explicitly describe the cell network characteristics in an image. Our approach does not rely on precise nuclei segmentation or feature extraction, is computationally efficient, and is easily scalable. In this study, we utilize the TCGA-CRC-DX dataset, comprising 499 patients and 502 diagnostic slides from primary colorectal tumours, sourced from 36 distinct medical centres in the United States. By incorporating the SNA features alongside deep features in two multiple instance learning frameworks, we demonstrate improved performance for chromosomal instability (CIN), hypermutated tumour (HM), TP53 gene, BRAF gene, and Microsatellite instability (MSI) status prediction tasks (2.4%-4% and 7-8.8% improvement in AUROC and AUPRC on average). Additionally, our method achieves outstanding performance on MSI prediction in an external PAIP dataset (99% AUROC and 98% AUPRC), demonstrating its generalizability. Our findings highlight the discrimination power of SNA features and how they can be beneficial to deep learning models' performance and provide insights into the correlation of cell network profiles with molecular pathways and key mutations.

AI-based intra-tumor heterogeneity score of Ki67 expression as a prognostic marker for early-stage ER+/HER2- breast cancer.

Early-stage estrogen receptor positive and human epidermal growth factor receptor negative (ER+/HER2-) luminal breast cancer (BC) is quite heterogeneous and accounts for about 70% of all BCs. Ki67 is a proliferation marker that has a significant prognostic value in luminal BC despite the challenges in its assessment. There is increasing evidence that spatial colocalization, which measures the evenness of different types of cells, is clinically important in several types of cancer. However, reproducible quantification of intra-tumor spatial heterogeneity remains largely unexplored. We propose an automated pipeline for prognostication of luminal BC based on the analysis of spatial distribution of Ki67 expression in tumor cells using a large well-characterized cohort (n = 2,081). The proposed Ki67 colocalization (Ki67CL) score can stratify ER+/HER2- BC patients with high significance in terms of BC-specific survival (p < 0.00001) and distant metastasis-free survival (p = 0.0048). Ki67CL score is shown to be highly significant compared with the standard Ki67 index. In addition, we show that the proposed Ki67CL score can help identify luminal BC patients who can potentially benefit from adjuvant chemotherapy.

An aggregation of aggregation methods in computational pathology.

Image analysis and machine learning algorithms operating on multi-gigapixel whole-slide images (WSIs) often process a large number of tiles (sub-images) and require aggregating predictions from the tiles in order to predict WSI-level labels. In this paper, we present a review of existing literature on various types of aggregation methods with a view to help guide future research in the area of computational pathology (CPath). We propose a general CPath workflow with three pathways that consider multiple levels and types of data and the nature of computation to analyse WSIs for predictive modelling. We categorize aggregation methods according to the context and representation of the data, features of computational modules and CPath use cases. We compare and contrast different methods based on the principle of multiple instance learning, perhaps the most commonly used aggregation method, covering a wide range of CPath literature. To provide a fair comparison, we consider a specific WSI-level prediction task and compare various aggregation methods for that task. Finally, we conclude with a list of objectives and desirable attributes of aggregation methods in general, pros and cons of the various approaches, some recommendations and possible future directions.

AI-enabled routine H&E image based prognostic marker for early-stage luminal breast cancer.

Breast cancer (BC) grade is a well-established subjective prognostic indicator of tumour aggressiveness. Tumour heterogeneity and subjective assessment result in high degree of variability among observers in BC grading. Here we propose an objective Haematoxylin & Eosin (H&E) image-based prognostic marker for early-stage luminal/Her2-negative BReAst CancEr that we term as the BRACE marker. The proposed BRACE marker is derived from AI based assessment of heterogeneity in BC at a detailed level using the power of deep learning. The prognostic ability of the marker is validated in two well-annotated cohorts (Cohort-A/Nottingham: n = 2122 and Cohort-B/Coventry: n = 311) on early-stage luminal/HER2-negative BC patients treated with endocrine therapy and with long-term follow-up. The BRACE marker is able to stratify patients for both distant metastasis free survival (p = 0.001, C-index: 0.73) and BC specific survival (p < 0.0001, C-index: 0.84) showing comparable prediction accuracy to Nottingham Prognostic Index and Magee scores, which are both derived from manual histopathological assessment, to identify luminal BC patients that may be likely to benefit from adjuvant chemotherapy.

Development and validation of artificial intelligence-based prescreening of large-bowel biopsies taken in the UK and Portugal: a retrospective cohort study.

BACKGROUND: Histopathological examination is a crucial step in the diagnosis and treatment of many major diseases. Aiming to facilitate diagnostic decision making and improve the workload of pathologists, we developed an artificial intelligence (AI)-based prescreening tool that analyses whole-slide images (WSIs) of large-bowel biopsies to identify typical, non-neoplastic, and neoplastic biopsies. METHODS: This retrospective cohort study was conducted with an internal development cohort of slides acquired from a hospital in the UK and three external validation cohorts of WSIs acquired from two hospitals in the UK and one clinical laboratory in Portugal. To learn the differential histological patterns from digitised WSIs of large-bowel biopsy slides, our proposed weakly supervised deep-learning model (Colorectal AI Model for Abnormality Detection [CAIMAN]) used slide-level diagnostic labels and no detailed cell or region-level annotations. The method was developed with an internal development cohort of 5054 biopsy slides from 2080 patients that were labelled with corresponding diagnostic categories assigned by pathologists. The three external validation cohorts, with a total of 1536 slides, were used for independent validation of CAIMAN. Each WSI was classified into one of three classes (ie, typical, atypical non-neoplastic, and atypical neoplastic). Prediction scores of image tiles were aggregated into three prediction scores for the whole slide, one for its likelihood of being typical, one for its likelihood of being non-neoplastic, and one for its likelihood of being neoplastic. The assessment of the external validation cohorts was conducted by the trained and frozen CAIMAN model. To evaluate model performance, we calculated area under the convex hull of the receiver operating characteristic curve (AUROC), area under the precision-recall curve, and specificity compared with our previously published iterative draw and rank sampling (IDaRS) algorithm. We also generated heat maps and saliency maps to analyse and visualise the relationship between the WSI diagnostic labels and spatial features of the tissue microenvironment. The main outcome of this study was the ability of CAIMAN to accurately identify typical and atypical WSIs of colon biopsies, which could potentially facilitate automatic removing of typical biopsies from the diagnostic workload in clinics. FINDINGS: A randomly selected subset of all large bowel biopsies was obtained between Jan 1, 2012, and Dec 31, 2017. The AI training, validation, and assessments were done between Jan 1, 2021, and Sept 30, 2022. WSIs with diagnostic labels were collected between Jan 1 and Sept 30, 2022. Our analysis showed no statistically significant differences across prediction scores from CAIMAN for typical and atypical classes based on anatomical sites of the biopsy. At 0·99 sensitivity, CAIMAN (specificity 0·5592) was more accurate than an IDaRS-based weakly supervised WSI-classification pipeline (0·4629) in identifying typical and atypical biopsies on cross-validation in the internal development cohort (p<0·0001). At 0·99 sensitivity, CAIMAN was also more accurate than IDaRS for two external validation cohorts (p<0·0001), but not for a third external validation cohort (p=0·10). CAIMAN provided higher specificity than IDaRS at some high-sensitivity thresholds (0·7763 vs 0·6222 for 0·95 sensitivity, 0·7126 vs 0·5407 for 0·97 sensitivity, and 0·5615 vs 0·3970 for 0·99 sensitivity on one of the external validation cohorts) and showed high classification performance in distinguishing between neoplastic biopsies (AUROC 0·9928, 95% CI 0·9927-0·9929), inflammatory biopsies (0·9658, 0·9655-0·9661), and atypical biopsies (0·9789, 0·9786-0·9792). On the three external validation cohorts, CAIMAN had AUROC values of 0·9431 (95% CI 0·9165-0·9697), 0·9576 (0·9568-0·9584), and 0·9636 (0·9615-0·9657) for the detection of atypical biopsies. Saliency maps supported the representation of disease heterogeneity in model predictions and its association with relevant histological features. INTERPRETATION: CAIMAN, with its high sensitivity in detecting atypical large-bowel biopsies, might be a promising improvement in clinical workflow efficiency and diagnostic decision making in prescreening of typical colorectal biopsies. FUNDING: The Pathology Image Data Lake for Analytics, Knowledge and Education Centre of Excellence; the UK Government's Industrial Strategy Challenge Fund; and Innovate UK on behalf of UK Research and Innovation.

TIAToolbox as an end-to-end library for advanced tissue image analytics.

Background: Computational pathology has seen rapid growth in recent years, driven by advanced deep-learning algorithms. Due to the sheer size and complexity of multi-gigapixel whole-slide images, to the best of our knowledge, there is no open-source software library providing a generic end-to-end API for pathology image analysis using best practices. Most researchers have designed custom pipelines from the bottom up, restricting the development of advanced algorithms to specialist users. To help overcome this bottleneck, we present TIAToolbox, a Python toolbox designed to make computational pathology accessible to computational, biomedical, and clinical researchers. Methods: By creating modular and configurable components, we enable the implementation of computational pathology algorithms in a way that is easy to use, flexible and extensible. We consider common sub-tasks including reading whole slide image data, patch extraction, stain normalization and augmentation, model inference, and visualization. For each of these steps, we provide a user-friendly application programming interface for commonly used methods and models. Results: We demonstrate the use of the interface to construct a full computational pathology deep-learning pipeline. We show, with the help of examples, how state-of-the-art deep-learning algorithms can be reimplemented in a streamlined manner using our library with minimal effort. Conclusions: We provide a usable and adaptable library with efficient, cutting-edge, and unit-tested tools for data loading, pre-processing, model inference, post-processing, and visualization. This enables a range of users to easily build upon recent deep-learning developments in the computational pathology literature.

Semantic annotation for computational pathology: multidisciplinary experience and best practice recommendations.

Recent advances in whole-slide imaging (WSI) technology have led to the development of a myriad of computer vision and artificial intelligence-based diagnostic, prognostic, and predictive algorithms. Computational Pathology (CPath) offers an integrated solution to utilise information embedded in pathology WSIs beyond what can be obtained through visual assessment. For automated analysis of WSIs and validation of machine learning (ML) models, annotations at the slide, tissue, and cellular levels are required. The annotation of important visual constructs in pathology images is an important component of CPath projects. Improper annotations can result in algorithms that are hard to interpret and can potentially produce inaccurate and inconsistent results. Despite the crucial role of annotations in CPath projects, there are no well-defined guidelines or best practices on how annotations should be carried out. In this paper, we address this shortcoming by presenting the experience and best practices acquired during the execution of a large-scale annotation exercise involving a multidisciplinary team of pathologists, ML experts, and researchers as part of the Pathology image data Lake for Analytics, Knowledge and Education (PathLAKE) consortium. We present a real-world case study along with examples of different types of annotations, diagnostic algorithm, annotation data dictionary, and annotation constructs. The analyses reported in this work highlight best practice recommendations that can be used as annotation guidelines over the lifecycle of a CPath project.

Development and validation of a weakly supervised deep learning framework to predict the status of molecular pathways and key mutations in colorectal cancer from routine histology images: a retrospective study.

BACKGROUND: Determining the status of molecular pathways and key mutations in colorectal cancer is crucial for optimal therapeutic decision making. We therefore aimed to develop a novel deep learning pipeline to predict the status of key molecular pathways and mutations from whole-slide images of haematoxylin and eosin-stained colorectal cancer slides as an alternative to current tests. METHODS: In this retrospective study, we used 502 diagnostic slides of primary colorectal tumours from 499 patients in The Cancer Genome Atlas colon and rectal cancer (TCGA-CRC-DX) cohort and developed a weakly supervised deep learning framework involving three separate convolutional neural network models. Whole-slide images were divided into equally sized tiles and model 1 (ResNet18) extracted tumour tiles from non-tumour tiles. These tumour tiles were inputted into model 2 (adapted ResNet34), trained by iterative draw and rank sampling to calculate a prediction score for each tile that represented the likelihood of a tile belonging to the molecular labels of high mutation density (vs low mutation density), microsatellite instability (vs microsatellite stability), chromosomal instability (vs genomic stability), CpG island methylator phenotype (CIMP)-high (vs CIMP-low), BRAFmut (vs BRAFWT), TP53mut (vs TP53WT), and KRASWT (vs KRASmut). These scores were used to identify the top-ranked titles from each slide, and model 3 (HoVer-Net) segmented and classified the different types of cell nuclei in these tiles. We calculated the area under the convex hull of the receiver operating characteristic curve (AUROC) as a model performance measure and compared our results with those of previously published methods. FINDINGS: Our iterative draw and rank sampling method yielded mean AUROCs for the prediction of hypermutation (0·81 [SD 0·03] vs 0·71), microsatellite instability (0·86 [0·04] vs 0·74), chromosomal instability (0·83 [0·02] vs 0·73), BRAFmut (0·79 [0·01] vs 0·66), and TP53mut (0·73 [0·02] vs 0·64) in the TCGA-CRC-DX cohort that were higher than those from previously published methods, and an AUROC for KRASmut that was similar to previously reported methods (0·60 [SD 0·04] vs 0·60). Mean AUROC for predicting CIMP-high status was 0·79 (SD 0·05). We found high proportions of tumour-infiltrating lymphocytes and necrotic tumour cells to be associated with microsatellite instability, and high proportions of tumour-infiltrating lymphocytes and a low proportion of necrotic tumour cells to be associated with hypermutation. INTERPRETATION: After large-scale validation, our proposed algorithm for predicting clinically important mutations and molecular pathways, such as microsatellite instability, in colorectal cancer could be used to stratify patients for targeted therapies with potentially lower costs and quicker turnaround times than sequencing-based or immunohistochemistry-based approaches. FUNDING: The UK Medical Research Council.