Reproducible Reporting of the Collection and Evaluation of Annotations for Artificial Intelligence Models.

This work puts forth and demonstrates the utility of a reporting framework for collecting and evaluating annotations of medical images used for training and testing artificial intelligence (AI) models in assisting detection and diagnosis. AI has unique reporting requirements, as shown by the AI extensions to the Consolidated Standards of Reporting Trials (CONSORT) and Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) checklists and the proposed AI extensions to the Standards for Reporting Diagnostic Accuracy (STARD) and Transparent Reporting of a Multivariable Prediction model for Individual Prognosis or Diagnosis (TRIPOD) checklists. AI for detection and/or diagnostic image analysis requires complete, reproducible, and transparent reporting of the annotations and metadata used in training and testing data sets. In an earlier work by other researchers, an annotation workflow and quality checklist for computational pathology annotations were proposed. In this manuscript, we operationalize this workflow into an evaluable quality checklist that applies to any reader-interpreted medical images, and we demonstrate its use for an annotation effort in digital pathology. We refer to this quality framework as the Collection and Evaluation of Annotations for Reproducible Reporting of Artificial Intelligence (CLEARR-AI).

Training pathologists to assess stromal tumour-infiltrating lymphocytes in breast cancer synergises efforts in clinical care and scientific research.

A growing body of research supports stromal tumour-infiltrating lymphocyte (TIL) density in breast cancer to be a robust prognostic and predicive biomarker. The gold standard for stromal TIL density quantitation in breast cancer is pathologist visual assessment using haematoxylin and eosin-stained slides. Artificial intelligence/machine-learning algorithms are in development to automate the stromal TIL scoring process, and must be validated against a reference standard such as pathologist visual assessment. Visual TIL assessment may suffer from significant interobserver variability. To improve interobserver agreement, regulatory science experts at the US Food and Drug Administration partnered with academic pathologists internationally to create a freely available online continuing medical education (CME) course to train pathologists in assessing breast cancer stromal TILs using an interactive format with expert commentary. Here we describe and provide a user guide to this CME course, whose content was designed to improve pathologist accuracy in scoring breast cancer TILs. We also suggest subsequent steps to translate knowledge into clinical practice with proficiency testing.

Comparison of SP142 and 22C3 PD-L1 assays in a population-based cohort of triple-negative breast cancer patients in the context of their clinically established scoring algorithms.

BACKGROUND: Immunohistochemical (IHC) PD-L1 expression is commonly employed as predictive biomarker for checkpoint inhibitors in triple-negative breast cancer (TNBC). However, IHC evaluation methods are non-uniform and further studies are needed to optimize clinical utility. METHODS: We compared the concordance, prognostic value and gene expression between PD-L1 IHC expression by SP142 immune cell (IC) score and 22C3 combined positive score (CPS; companion IHC diagnostic assays for atezolizumab and pembrolizumab, respectively) in a population-based cohort of 232 early-stage TNBC patients. RESULTS: The expression rates of PD-L1 for SP142 IC ≥ 1%, 22C3 CPS ≥ 10, 22C3 CPS ≥ 1 and 22C3 IC ≥ 1% were 50.9%, 27.2%, 53.9% and 41.8%, respectively. The analytical concordance (kappa values) between SP142 IC+ and these three different 22C3 scorings were 73.7% (0.48, weak agreement), 81.5% (0.63) and 86.6% (0.73), respectively. The SP142 assay was better at identifying 22C3 positive tumors than the 22C3 assay was at detecting SP142 positive tumors. PD-L1 (CD274) gene expression (mRNA) showed a strong positive association with all two-categorical IHC scorings of the PD-L1 expression, irrespective of antibody and cut-off (Spearman Rho ranged from 0.59 to 0.62; all p-values < 0.001). PD-L1 IHC positivity and abundance of tumor infiltrating lymphocytes were of positive prognostic value in univariable regression analyses in patients treated with (neo)adjuvant chemotherapy, where it was strongest for 22C3 CPS ≥ 10 and distant relapse-free interval (HR = 0.18, p = 0.019). However, PD-L1 status was not independently prognostic when adjusting for abundance of tumor infiltrating lymphocytes in multivariable analyses. CONCLUSION: Our findings support that the SP142 and 22C3 IHC assays, with their respective clinically applied scoring algorithms, are not analytically equivalent where they identify partially non-overlapping subpopulations of TNBC patients and cannot be substituted with one another regarding PD-L1 detection. Trial registration The Swedish Cancerome Analysis Network - Breast (SCAN-B) study, retrospectively registered 2nd Dec 2014 at ClinicalTrials.gov; ID NCT02306096.

Molecular analyses of triple-negative breast cancer in the young and elderly.

BACKGROUND: Breast cancer in young adults has been implicated with a worse outcome. Analyses of genomic traits associated with age have been heterogenous, likely because of an incomplete accounting for underlying molecular subtypes. We aimed to resolve whether triple-negative breast cancer (TNBC) in younger versus older patients represent similar or different molecular diseases in the context of genetic and transcriptional subtypes and immune cell infiltration. PATIENTS AND METHODS: In total, 237 patients from a reported population-based south Swedish TNBC cohort profiled by RNA sequencing and whole-genome sequencing (WGS) were included. Patients were binned in 10-year intervals. Complimentary PD-L1 and CD20 immunohistochemistry and estimation of tumor-infiltrating lymphocytes (TILs) were performed. Cases were analyzed for differences in patient outcome, genomic, transcriptional, and immune landscape features versus age at diagnosis. Additionally, 560 public WGS breast cancer profiles were used for validation. RESULTS: Median age at diagnosis was 62 years (range 26-91). Age was not associated with invasive disease-free survival or overall survival after adjuvant chemotherapy. Among the BRCA1-deficient cases (82/237), 90% were diagnosed before the age of 70 and were predominantly of the basal-like subtype. In the full TNBC cohort, reported associations of patient age with changes in Ki67 expression, PIK3CA mutations, and a luminal androgen receptor subtype were confirmed. Within DNA repair deficiency or gene expression defined molecular subgroups, age-related alterations in, e.g., overall gene expression, immune cell marker gene expression, genetic mutational and rearrangement signatures, amount of copy number alterations, and tumor mutational burden did, however, not appear distinct. Similar non-significant associations for genetic alterations with age were obtained for other breast cancer subgroups in public WGS data. Consistent with age-related immunosenescence, TIL counts decreased linearly with patient age across different genetic TNBC subtypes. CONCLUSIONS: Age-related alterations in TNBC, as well as breast cancer in general, need to be viewed in the context of underlying genomic phenotypes. Based on this notion, age at diagnosis alone does not appear to provide an additional layer of biological complexity above that of proposed genetic and transcriptional phenotypes of TNBC. Consequently, treatment decisions should be less influenced by age and more driven by tumor biology.

Distinct mechanisms of resistance to fulvestrant treatment dictate level of ER independence and selective response to CDK inhibitors in metastatic breast cancer.

BACKGROUND: Resistance to endocrine treatment in metastatic breast cancer is a major clinical challenge. Clinical tools to predict both drug resistance and possible treatment combination approaches to overcome it are lacking. This unmet need is mainly due to the heterogeneity underlying both the mechanisms involved in resistance development and breast cancer itself. METHODS: To study the complexity of the mechanisms involved in the resistance to the selective estrogen receptor degrader (SERD) fulvestrant, we performed comprehensive biomarker analyses using several in vitro models that recapitulate the heterogeneity of developed resistance. We further corroborated our findings in tissue samples from patients treated with fulvestrant. RESULTS: We found that different in vitro models of fulvestrant resistance show variable stability in their phenotypes, which corresponded with distinct genomic alterations. Notably, the studied models presented adaptation at different cell cycle nodes to facilitate progression through the cell cycle and responded differently to CDK inhibitors. Cyclin E2 overexpression was identified as a biomarker of a persistent fulvestrant-resistant phenotype. Comparison of pre- and post-treatment paired tumor biopsies from patients treated with fulvestrant revealed an upregulation of cyclin E2 upon development of resistance. Moreover, overexpression of this cyclin was found to be a prognostic factor determining resistance to fulvestrant and shorter progression-free survival. CONCLUSIONS: These data highlight the complexity of estrogen receptor positive breast cancer and suggest that the development of diverse resistance mechanisms dictate levels of ER independence and potentially cross-resistance to CDK inhibitors.

The path to a better biomarker: application of a risk management framework for the implementation of PD-L1 and TILs as immuno-oncology biomarkers in breast cancer clinical trials and daily practice.

Immune checkpoint inhibitor therapies targeting PD-1/PD-L1 are now the standard of care in oncology across several hematologic and solid tumor types, including triple negative breast cancer (TNBC). Patients with metastatic or locally advanced TNBC with PD-L1 expression on immune cells occupying ≥1% of tumor area demonstrated survival benefit with the addition of atezolizumab to nab-paclitaxel. However, concerns regarding variability between immunohistochemical PD-L1 assay performance and inter-reader reproducibility have been raised. High tumor-infiltrating lymphocytes (TILs) have also been associated with response to PD-1/PD-L1 inhibitors in patients with breast cancer (BC). TILs can be easily assessed on hematoxylin and eosin-stained slides and have shown reliable inter-reader reproducibility. As an established prognostic factor in early stage TNBC, TILs are soon anticipated to be reported in daily practice in many pathology laboratories worldwide. Because TILs and PD-L1 are parts of an immunological spectrum in BC, we propose the systematic implementation of combined PD-L1 and TIL analyses as a more comprehensive immuno-oncological biomarker for patient selection for PD-1/PD-L1 inhibition-based therapy in patients with BC. Although practical and regulatory considerations differ by jurisdiction, the pathology community has the responsibility to patients to implement assays that lead to optimal patient selection. We propose herewith a risk-management framework that may help mitigate the risks of suboptimal patient selection for immuno-therapeutic approaches in clinical trials and daily practice based on combined TILs/PD-L1 assessment in BC. © 2020 Pathological Society of Great Britain and Ireland. Published by John Wiley & Sons, Ltd.

Refinement of breast cancer molecular classification by miRNA expression profiles.

BACKGROUND: Accurate classification of breast cancer using gene expression profiles has contributed to a better understanding of the biological mechanisms behind the disease and has paved the way for better prognostication and treatment prediction. RESULTS: We found that miRNA profiles largely recapitulate intrinsic subtypes. In the case of HER2-enriched tumors a small set of miRNAs including the HER2-encoded mir-4728 identifies the group with very high specificity. We also identified differential expression of the miR-99a/let-7c/miR-125b miRNA cluster as a marker for separation of the Luminal A and B subtypes. High expression of this miRNA cluster is linked to better overall survival among patients with Luminal A tumors. Correlation between the miRNA cluster and their precursor LINC00478 is highly significant suggesting that its expression could help improve the accuracy of present day's signatures. CONCLUSIONS: We show here that miRNA expression can be translated into mRNA profiles and that the inclusion of miRNA information facilitates the molecular diagnosis of specific subtypes, in particular the clinically relevant sub-classification of luminal tumors.

Agreement between molecular subtyping and surrogate subtype classification: a contemporary population-based study of ER-positive/HER2-negative primary breast cancer.

PURPOSE: Oestrogen receptor-positive (ER+) and human epidermal receptor 2-negative (HER2-) breast cancers are classified as Luminal A or B based on gene expression, but immunohistochemical markers are used for surrogate subtyping. The aims of this study were to examine the agreement between molecular subtyping (MS) and surrogate subtyping and to identify subgroups consisting mainly of Luminal A or B tumours. METHODS: The cohort consisted of 2063 patients diagnosed between 2013-2017, with primary ER+/HER2- breast cancer, analysed by RNA sequencing. Surrogate subtyping was performed according to three algorithms (St. Gallen 2013, Maisonneuve and our proposed Grade-based classification). Agreement (%) and kappa statistics (κ) were used as concordance measures and ROC analysis for luminal distinction. Ki67, progesterone receptor (PR) and histological grade (HG) were further investigated as surrogate markers. RESULTS: The agreement rates between the MS and St. Gallen 2013, Maisonneuve and Grade-based classifications were 62% (κ = 0.30), 66% (κ = 0.35) and 70% (κ = 0.41), respectively. PR did not contribute to distinguishing Luminal A from B tumours (auROC = 0.56). By classifying HG1-2 tumours as Luminal A-like and HG3 as Luminal B-like, agreement with MS was 80% (κ = 0.46). Moreover, by combining HG and Ki67 status, a large subgroup of patients (51% of the cohort) having > 90% Luminal A tumours could be identified. CONCLUSIONS: Agreement between MS and surrogate classifications was generally poor. However, a post hoc analysis showed that a combination of HG and Ki67 could identify patients very likely to have Luminal A tumours according to MS.

Analytical validation of a standardised scoring protocol for Ki67 immunohistochemistry on breast cancer excision whole sections: an international multicentre collaboration.

AIMS: The nuclear proliferation marker Ki67 assayed by immunohistochemistry has multiple potential uses in breast cancer, but an unacceptable level of interlaboratory variability has hampered its clinical utility. The International Ki67 in Breast Cancer Working Group has undertaken a systematic programme to determine whether Ki67 measurement can be analytically validated and standardised among laboratories. This study addresses whether acceptable scoring reproducibility can be achieved on excision whole sections. METHODS AND RESULTS: Adjacent sections from 30 primary ER+ breast cancers were centrally stained for Ki67 and sections were circulated among 23 pathologists in 12 countries. All pathologists scored Ki67 by two methods: (i) global: four fields of 100 tumour cells each were selected to reflect observed heterogeneity in nuclear staining; (ii) hot-spot: the field with highest apparent Ki67 index was selected and up to 500 cells scored. The intraclass correlation coefficient (ICC) for the global method [confidence interval (CI) = 0.87; 95% CI = 0.799-0.93] marginally met the prespecified success criterion (lower 95% CI ≥ 0.8), while the ICC for the hot-spot method (0.83; 95% CI = 0.74-0.90) did not. Visually, interobserver concordance in location of selected hot-spots varies between cases. The median times for scoring were 9 and 6 min for global and hot-spot methods, respectively. CONCLUSIONS: The global scoring method demonstrates adequate reproducibility to warrant next steps towards evaluation for technical and clinical validity in appropriate cohorts of cases. The time taken for scoring by either method is practical using counting software we are making publicly available. Establishment of external quality assessment schemes is likely to improve the reproducibility between laboratories further.

Histological grade provides significant prognostic information in addition to breast cancer subtypes defined according to St Gallen 2013.

BACKGROUND: The St Gallen surrogate definition of the intrinsic subtypes of breast cancer consist of five subgroups based on estrogen receptor (ER), progesterone receptor (PgR), human epidermal growth factor receptor type 2 (HER2), and Ki-67. PgR and Ki-67 are used for discriminating between the 'Luminal A-like' and 'Luminal B-like (HER2-negative)' subtypes. Histological grade (G) has prognostic value in breast cancer; however, its relationship to the St Gallen subtypes is not clear. Based on a previous pilot study, we hypothesized that G could be a primary discriminator for ER-positive/HER2-negative breast cancers that were G1 or G3, whereas Ki-67 and PgR could provide additional prognostic information specifically for patients with G2 tumors. To test this hypothesis, a larger patient cohort was examined. PATIENTS AND METHODS: Six hundred seventy-one patients (≥35 years of age, pT1-2, pN0-1) with ER-positive/HER2-negative breast cancer and complete data for PgR, Ki-67, G, lymph node status, tumor size, age, and distant disease-free survival (DDFS; median follow-up 9.2 years) were included. RESULTS: 'Luminal A-like' tumors were mostly G1 or G2 (90%) whereas 'Luminal B-like' tumors were mostly G2 or G3 (87%) and corresponded with good and poor DDFS, respectively. In 'Luminal B-like' tumors that were G1 (n = 23), no metastasis occurred, whereas 14 of 40 'Luminal A-like' tumors that were G3 metastasized. In the G2 subgroup, low PgR and high Ki-67 were associated with an increased risk of distant metastases, hazard ratio (HR) and 95% confidence interval (CI) 1.8 (0.95-3.4) and 1.5 (0.80-2.8), respectively. CONCLUSIONS: Patients with ER-positive/HER2-negative/G1 breast cancer have a good prognosis, similar to that of 'Luminal A-like', while those with ER-positive/HER2-negative/G3 breast cancer have a worse prognosis, similar to that of 'Luminal B-like', when assessed independently of PgR and Ki-67. Therapy decisions based on Ki-67 and PgR might thus be restricted to the subgroup G2.

Contralateral breast cancer can represent a metastatic spread of the first primary tumor: determination of clonal relationship between contralateral breast cancers using next-generation whole genome sequencing.

INTRODUCTION: By convention, a contralateral breast cancer (CBC) is treated as a new primary tumor, independent of the first cancer (BC1). Although there have been indications that the second tumor (BC2) sometimes may represent a metastatic spread of BC1, this has never been conclusively shown. We sought to apply next-generation sequencing to determine a "genetic barcode" for each tumor and reveal the clonal relationship of CBCs. METHODS: Ten CBC patients with detailed clinical information and available fresh frozen tumor tissue were studied. Using low-coverage whole genome DNA-sequencing data for each tumor, chromosomal rearrangements were enumerated and copy number profiles were generated. Comparisons between tumors provided an estimate of clonal relatedness for tumor pairs within individual patients. RESULTS: Between 15-256 rearrangements were detected in each tumor (median 87). For one patient, 76 % (68 out of 90) of the rearrangements were shared between BC1 and BC2, highly consistent with what has been seen for true primary-metastasis pairs (>50 %) and thus confirming a common clonal origin of the two tumors. For most of the remaining cases, BC1 and BC2 had similarly low overlap as unmatched randomized pairs of tumors from different individuals, suggesting the CBC to represent a new independent primary tumor. CONCLUSION: Using rearrangement fingerprinting, we show for the first time with certainty that a contralateral BC2 can represent a metastatic spread of BC1. Given the poor prognosis of a generalized disease compared to a new primary tumor, these women need to be identified at diagnosis of CBC for appropriate determination of treatment. Our approach generates a promising new method to assess clonal relationship between tumors. Additional studies are required to confirm the frequency of CBCs representing metastatic events.

Digital PCR quantification of ultrahigh ERBB2 copy number identifies poor breast cancer survival after trastuzumab.

HER2/ERBB2 evaluation is necessary for treatment decision-making in breast cancer (BC), however current methods have limitations and considerable variability exists. DNA copy number (CN) evaluation by droplet digital PCR (ddPCR) has complementary advantages for HER2/ERBB2 diagnostics. In this study, we developed a single-reaction multiplex ddPCR assay for determination of ERBB2 CN in reference to two control regions, CEP17 and a copy-number-stable region of chr. 2p13.1, validated CN estimations to clinical in situ hybridization (ISH) HER2 status, and investigated the association of ERBB2 CN with clinical outcomes. 909 primary BC tissues were evaluated and the area under the curve for concordance to HER2 status was 0.93 and 0.96 for ERBB2 CN using either CEP17 or 2p13.1 as reference, respectively. The accuracy of ddPCR ERBB2 CN was 93.7% and 94.1% in the training and validation groups, respectively. Positive and negative predictive value for the classic HER2 amplification and non-amplification groups was 97.2% and 94.8%, respectively. An identified biological "ultrahigh" ERBB2 ddPCR CN group had significantly worse survival within patients treated with adjuvant trastuzumab for both recurrence-free survival (hazard ratio, HR: 3.3; 95% CI 1.1-9.6; p = 0.031, multivariable Cox regression) and overall survival (HR: 3.6; 95% CI 1.1-12.6; p = 0.041). For validation using RNA-seq data as a surrogate, in a population-based SCAN-B cohort (NCT02306096) of 682 consecutive patients receiving adjuvant trastuzumab, the ultrahigh-ERBB2 mRNA group had significantly worse survival. Multiplex ddPCR is useful for ERBB2 CN estimation and ultrahigh ERBB2 may be a predictive factor for decreased long-term survival after trastuzumab treatment.

The Prognostic Role of Intratumoral Stromal Content in Lobular Breast Cancer.

Previous studies have shown that high intratumoral stromal content is associated with a worse prognosis in breast cancer, especially in the triple-negative subtype. However, contradictory results have been reported for estrogen-receptor-positive (ER+) breast cancer, indicating that the prognostic role of intratumoral stromal content may be subtype-dependent. In this study, we investigated the importance of intratumoral stromal content for breast cancer-specific mortality (BCM) in a well-defined subgroup (n = 182) of ER+/human-epidermal growth-factor-receptor-2 negative (HER2-) invasive lobular breast cancer (ILC). The intratumoral stromal content was assessed on hematoxylin-eosin-stained whole sections and graded into high stroma (>50%) or low stroma (≤50%). A total of 82 (45%) patients had high-stroma tumors, and 100 (55%) had low-stroma tumors. High-stroma tumors were associated with a lower Nottingham histological grade, low Ki67, and a luminal A-like subtype. After a 10-year follow-up, the patients with high-stroma tumors had a lower BCM (HR: 0.43, 95% CI: 0.21-0.89, p = 0.023) in univariable analysis. Essentially the same effect was found in both the multivariable analysis (10-year follow-up) and univariable analysis (25-year follow-up), but these findings were not strictly significant. In ER+/HER2- ILC, high intratumoral stromal content is an easily assessable histological indicator of a good prognosis.

How Reliable Are Gene Expression-Based and Immunohistochemical Biomarkers Assessed on a Core-Needle Biopsy? A Study of Paired Core-Needle Biopsies and Surgical Specimens in Early Breast Cancer.

In early breast cancer, a preoperative core-needle biopsy (CNB) is vital to confirm the malignancy of suspected lesions and for assessing the expression of treatment predictive and prognostic biomarkers in the tumor to choose the optimal treatments, emphasizing the importance of obtaining reliable results when biomarker status is assessed on a CNB specimen. This study aims to determine the concordance between biomarker status assessed as part of clinical workup on a CNB compared to a medically untreated surgical specimen. Paired CNB and surgical specimens from 259 patients that were part of the SCAN-B cohort were studied. The concordance between immunohistochemical (IHC) and gene expression (GEX) based biomarker status was investigated. Biomarkers of interest included estrogen receptor (ER; specifically, the alpha variant), progesterone receptor (PgR), Ki67, HER2, and tumor molecular subtype. In general, moderate to very good correlation in biomarker status between the paired CNB and surgical specimens was observed for both IHC assessment (83-99% agreement, kappa range 0.474-0.917) and GEX assessment (70-97% agreement, kappa range 0.552-0.800), respectively. However, using IHC, 52% of cases with low Ki67 status in the CNB shifted to high Ki67 status in the surgical specimen (McNemar's p = 0.011). Similarly, when using GEX, a significant shift from negative to positive ER (47%) and from low to high Ki67 (16%) was observed between the CNB and surgical specimen (McNemar's p = 0.027 and p = 0.002 respectively). When comparing biomarker status between different techniques (IHC vs. GEX) performed on either CNBs or surgical specimens, the agreement in ER, PgR, and HER2 status was generally over 80% in both CNBs and surgical specimens (kappa range 0.395-0.708), but Ki67 and tumor molecular subtype showed lower concordance levels between IHC and GEX (48-62% agreement, kappa range 0.152-0.398). These results suggest that both the techniques used for collecting tissue samples and analyzing biomarker status have the potential to affect the results of biomarker assessment, potentially also impacting treatment decisions and patient survival outcomes.

Development of Training Materials for Pathologists to Provide Machine Learning Validation Data of Tumor-Infiltrating Lymphocytes in Breast Cancer.

The High Throughput Truthing project aims to develop a dataset for validating artificial intelligence and machine learning models (AI/ML) fit for regulatory purposes. The context of this AI/ML validation dataset is the reporting of stromal tumor-infiltrating lymphocytes (sTILs) density evaluations in hematoxylin and eosin-stained invasive breast cancer biopsy specimens. After completing the pilot study, we found notable variability in the sTILs estimates as well as inconsistencies and gaps in the provided training to pathologists. Using the pilot study data and an expert panel, we created custom training materials to improve pathologist annotation quality for the pivotal study. We categorized regions of interest (ROIs) based on their mean sTILs density and selected ROIs with the highest and lowest sTILs variability. In a series of eight one-hour sessions, the expert panel reviewed each ROI and provided verbal density estimates and comments on features that confounded the sTILs evaluation. We aggregated and shaped the comments to identify pitfalls and instructions to improve our training materials. From these selected ROIs, we created a training set and proficiency test set to improve pathologist training with the goal to improve data collection for the pivotal study. We are not exploring AI/ML performance in this paper. Instead, we are creating materials that will train crowd-sourced pathologists to be the reference standard in a pivotal study to create an AI/ML model validation dataset. The issues discussed here are also important for clinicians to understand about the evaluation of sTILs in clinical practice and can provide insight to developers of AI/ML models.

Pilot study to evaluate tools to collect pathologist annotations for validating machine learning algorithms.

Purpose: Validation of artificial intelligence (AI) algorithms in digital pathology with a reference standard is necessary before widespread clinical use, but few examples focus on creating a reference standard based on pathologist annotations. This work assesses the results of a pilot study that collects density estimates of stromal tumor-infiltrating lymphocytes (sTILs) in breast cancer biopsy specimens. This work will inform the creation of a validation dataset for the evaluation of AI algorithms fit for a regulatory purpose. Approach: Collaborators and crowdsourced pathologists contributed glass slides, digital images, and annotations. Here, "annotations" refer to any marks, segmentations, measurements, or labels a pathologist adds to a report, image, region of interest (ROI), or biological feature. Pathologists estimated sTILs density in 640 ROIs from hematoxylin and eosin stained slides of 64 patients via two modalities: an optical light microscope and two digital image viewing platforms. Results: The pilot study generated 7373 sTILs density estimates from 29 pathologists. Analysis of annotations found the variability of density estimates per ROI increases with the mean; the root mean square differences were 4.46, 14.25, and 26.25 as the mean density ranged from 0% to 10%, 11% to 40%, and 41% to 100%, respectively. The pilot study informs three areas of improvement for future work: technical workflows, annotation platforms, and agreement analysis methods. Upgrades to the workflows and platforms will improve operability and increase annotation speed and consistency. Conclusions: Exploratory data analysis demonstrates the need to develop new statistical approaches for agreement. The pilot study dataset and analysis methods are publicly available to allow community feedback. The development and results of the validation dataset will be publicly available to serve as an instructive tool that can be replicated by developers and researchers.

Difficulties in diagnostics of lung tumours in biopsies: an interpathologist concordance study evaluating the international diagnostic guidelines.

AIMS: Accurate and reliable diagnosis is essential for lung cancer treatment. The study aim was to investigate interpathologist diagnostic concordance for pulmonary tumours according to WHO diagnostic criteria. METHODS: Fifty-two unselected lung and bronchial biopsies were diagnosed by a thoracic pathologist based on a broad spectrum of immunohistochemical (IHC) stainings, molecular data and clinical/radiological information. Slides stained with H&E, thyroid transcription factor-1 (TTF-1) clone SPT24 and p40 were scanned and provided digitally to 20 pathologists unaware of reference diagnoses. The pathologists independently diagnosed the cases and stated if further diagnostic markers were deemed necessary. RESULTS: In 31 (60%) of the cases, ≥80% of the pathologists agreed with each other and with the reference diagnosis. Lower agreement was seen in non-small cell neuroendocrine tumours and in squamous cell carcinoma with diffuse TTF-1 positivity. Agreement with the reference diagnosis ranged from 26 to 45 (50%-87%) for the individual pathologists. The pathologists requested additional IHC staining in 15-44 (29%-85%) of the 52 cases. In nearly half (17 of 36) of the malignant cases, one or more pathologist advocated for a different final diagnosis than the reference without need of additional IHC markers, potentially leading to different clinical treatment. CONCLUSIONS: Interpathologist diagnostic agreement is moderate for small unselected bronchial and lung biopsies based on a minimal panel of markers. Neuroendocrine morphology is sometimes missed and TTF-1 clone SPT24 should be interpreted with caution. Our results suggest an intensified education need for thoracic pathologists and a more generous use of diagnostic IHC markers.

RNA sequencing-based single sample predictors of molecular subtype and risk of recurrence for clinical assessment of early-stage breast cancer.

Multigene assays for molecular subtypes and biomarkers can aid management of early invasive breast cancer. Using RNA-sequencing we aimed to develop single-sample predictor (SSP) models for clinical markers, subtypes, and risk of recurrence (ROR). A cohort of 7743 patients was divided into training and test set. We trained SSPs for subtypes and ROR assigned by nearest-centroid (NC) methods and SSPs for biomarkers from histopathology. Classifications were compared with Prosigna in two external cohorts (ABiM, n = 100 and OSLO2-EMIT0, n = 103). Prognostic value was assessed using distant recurrence-free interval. Agreement between SSP and NC for PAM50 (five subtypes) was high (85%, Kappa = 0.78) for Subtype (four subtypes) very high (90%, Kappa = 0.84) and for ROR risk category high (84%, Kappa = 0.75, weighted Kappa = 0.90). Prognostic value was assessed as equivalent and clinically relevant. Agreement with histopathology was very high or high for receptor status, while moderate for Ki67 status and poor for Nottingham histological grade. SSP and Prosigna concordance was high for subtype (OSLO-EMIT0 83%, Kappa = 0.73 and ABiM 80%, Kappa = 0.72) and moderate and high for ROR risk category (68 and 84%, Kappa = 0.50 and 0.70, weighted Kappa = 0.70 and 0.78). Pooled concordance for emulated treatment recommendation dichotomized for chemotherapy was high (85%, Kappa = 0.66). Retrospective evaluation suggested that SSP application could change chemotherapy recommendations for up to 17% of postmenopausal ER+/HER2-/N0 patients with balanced escalation and de-escalation. Results suggest that NC and SSP models are interchangeable on a group-level and nearly so on a patient level and that SSP models can be derived to closely match clinical tests.

Variability in Breast Cancer Biomarker Assessment and the Effect on Oncological Treatment Decisions: A Nationwide 5-Year Population-Based Study.

We compared estrogen receptor (ER), progesterone receptor (PR), human epidermal growth-factor receptor 2 (HER2), Ki67, and grade scores among the pathology departments in Sweden. We investigated how ER and HER2 positivity rates affect the distribution of endocrine and HER2-targeted treatments among oncology departments. All breast cancer patients diagnosed between 2013 and 2018 in Sweden were identified in the National Quality Register for Breast Cancer. Cases with data on ER, PR, HER2, Ki67, grade, and treatment were selected (43,261 cases from 29 departments following the guidelines for biomarker testing). The ER positivity rates ranged from 84.2% to 97.6% with 6/29 labs out of the overall confidence intervals (CIs), while PR rates varied between 64.8% and 86.6% with 7/29 labs out of the CIs. HER2 positivity rates ranged from 9.4% to 16.3%, with 3/29 labs out of the overall CIs. Median Ki67 varied between 15% and 30%, where 19/29 labs showed significant intra-laboratory variability. The proportion of grade-II cases varied between 42.9% and 57.1%, and 13/29 labs were outside of the CI. Adjusting for patient characteristics, the proportion of endocrine and anti-HER2 treatments followed the rate of ER and HER2 positivity, illustrating the clinical effect of inter- and intra-laboratory variability. There was limited variability among departments in ER, PR, and HER2 testing. However, even a few outlier pathology labs affected endocrine and HER2-targeted treatment rates in a clinically relevant proportion, suggesting the need for improvement. High variability was found in grading and Ki67 assessment, illustrating the need for the adoption of new technologies in practice.