Reproducible Reporting of the Collection and Evaluation of Annotations for Artificial Intelligence Models.

This work puts forth and demonstrates the utility of a reporting framework for collecting and evaluating annotations of medical images used for training and testing artificial intelligence (AI) models in assisting detection and diagnosis. AI has unique reporting requirements, as shown by the AI extensions to the Consolidated Standards of Reporting Trials (CONSORT) and Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) checklists and the proposed AI extensions to the Standards for Reporting Diagnostic Accuracy (STARD) and Transparent Reporting of a Multivariable Prediction model for Individual Prognosis or Diagnosis (TRIPOD) checklists. AI for detection and/or diagnostic image analysis requires complete, reproducible, and transparent reporting of the annotations and metadata used in training and testing data sets. In an earlier work by other researchers, an annotation workflow and quality checklist for computational pathology annotations were proposed. In this manuscript, we operationalize this workflow into an evaluable quality checklist that applies to any reader-interpreted medical images, and we demonstrate its use for an annotation effort in digital pathology. We refer to this quality framework as the Collection and Evaluation of Annotations for Reproducible Reporting of Artificial Intelligence (CLEARR-AI).

Training pathologists to assess stromal tumour-infiltrating lymphocytes in breast cancer synergises efforts in clinical care and scientific research.

A growing body of research supports stromal tumour-infiltrating lymphocyte (TIL) density in breast cancer to be a robust prognostic and predicive biomarker. The gold standard for stromal TIL density quantitation in breast cancer is pathologist visual assessment using haematoxylin and eosin-stained slides. Artificial intelligence/machine-learning algorithms are in development to automate the stromal TIL scoring process, and must be validated against a reference standard such as pathologist visual assessment. Visual TIL assessment may suffer from significant interobserver variability. To improve interobserver agreement, regulatory science experts at the US Food and Drug Administration partnered with academic pathologists internationally to create a freely available online continuing medical education (CME) course to train pathologists in assessing breast cancer stromal TILs using an interactive format with expert commentary. Here we describe and provide a user guide to this CME course, whose content was designed to improve pathologist accuracy in scoring breast cancer TILs. We also suggest subsequent steps to translate knowledge into clinical practice with proficiency testing.

Interobserver Variation in the Assessment of Immunohistochemistry Expression Levels in HER2-Negative Breast Cancer: Can We Improve the Identification of Low Levels of HER2 Expression by Adjusting the Criteria? An International Interobserver Study.

The classification of human epidermal growth factor receptor 2 (HER2) expression is optimized to detect HER2-amplified breast cancer (BC). However, novel HER2-targeting agents are also effective for BCs with low levels of HER2. This raises the question whether the current guidelines for HER2 testing are sufficiently reproducible to identify HER2-low BC. The aim of this multicenter international study was to assess the interobserver agreement of specific HER2 immunohistochemistry scores in cases with negative HER2 results (0, 1+, or 2+/in situ hybridization negative) according to the current American Society of Clinical Oncology/College of American Pathologists (ASCO/CAP) guidelines. Furthermore, we evaluated whether the agreement improved by redefining immunohistochemistry (IHC) scoring criteria or by adding fluorescent in situ hybridization (FISH). We conducted a 2-round study of 105 nonamplified BCs. During the first assessment, 16 pathologists used the latest version of the ASCO/CAP guidelines. After a consensus meeting, the same pathologists scored the same digital slides using modified IHC scoring criteria based on the 2007 ASCO/CAP guidelines, and an extra "ultralow" category was added. Overall, the interobserver agreement was limited (4.7% of cases with 100% agreement) in the first round, but this was improved by clustering IHC categories. In the second round, the highest reproducibility was observed when comparing IHC 0 with the ultralow/1+/2+ grouped cluster (74.3% of cases with 100% agreement). The FISH results were not statistically different between HER2-0 and HER2-low cases, regardless of the IHC criteria used. In conclusion, our study suggests that the modified 2007 ASCO/CAP criteria were more reproducible in distinguishing HER2-0 from HER2-low cases than the 2018 ASCO/CAP criteria. However, the reproducibility was still moderate, which was not improved by adding FISH. This could lead to a suboptimal selection of patients eligible for novel HER2-targeting agents. If the threshold between HER2 IHC 0 and 1+ is to be clinically actionable, there is a need for clearer, more reproducible IHC definitions, training, and/or development of more accurate methods to detect this subtle difference in protein expression levels.

Clinical validity of circulating tumour cells in patients with metastatic breast cancer: a pooled analysis of individual patient data.

BACKGROUND: We aimed to assess the clinical validity of circulating tumour cell (CTC) quantification for prognostication of patients with metastatic breast cancer by undertaking a pooled analysis of individual patient data. METHODS: We contacted 51 European centres and asked them to provide reported and unreported anonymised data for individual patients with metastatic breast cancer who participated in studies between January, 2003, and July, 2012. Eligible studies had participants starting a new line of therapy, data for progression-free survival or overall survival, or both, and CTC quantification by the CellSearch method at baseline (before start of new treatment). We used Cox regression models, stratified by study, to establish the association between CTC count and progression-free survival and overall survival. We used the landmark method to assess the prognostic value of CTC and serum marker changes during treatment. We assessed the added value of CTCs or serum markers to prognostic clinicopathological models in a resampling procedure using likelihood ratio (LR) χ(2) statistics. FINDINGS: 17 centres provided data for 1944 eligible patients from 20 studies. 911 patients (46·9%) had a CTC count of 5 per 7·5 mL or higher at baseline, which was associated with decreased progression-free survival (hazard ratio [HR] 1·92, 95% CI 1·73-2·14, p<0·0001) and overall survival (HR 2·78, 95% CI 2·42-3·19, p<0·0001) compared with patients with a CTC count of less than 5 per 7·5 mL at baseline. Increased CTC counts 3-5 weeks after start of treatment, adjusted for CTC count at baseline, were associated with shortened progression-free survival (HR 1·85, 95% CI 1·48-2·32, p<0·0001) and overall survival (HR 2·26, 95% CI 1·68-3·03) as were increased CTC counts after 6-8 weeks (progression-free survival HR 2·20, 95% CI 1·66-2·90, p<0·0001; overall survival HR 2·91, 95% CI 2·01-4·23, p<0·0001). Survival prediction was significantly improved by addition of baseline CTC count to the clinicopathological models (progression-free survival LR 38·4, 95% CI 21·9-60·3, p<0·0001; overall survival LR 64·9, 95% CI 41·3-93·4, p<0·0001). This model was further improved by addition of CTC change at 3-5 weeks (progression-free survival LR 8·2, 95% CI 0·78-20·4, p=0·004; overall survival LR 11·5, 95% CI 2·6-25·1, p=0·0007) and at 6-8 weeks (progression-free survival LR 15·3, 95% CI 5·2-28·3; overall survival LR 14·6, 95% CI 4·0-30·6; both p<0·0001). Carcinoembryonic antigen and cancer antigen 15-3 concentrations at baseline and during therapy did not add significant information to the best baseline model. INTERPRETATION: These data confirm the independent prognostic effect of CTC count on progression-free survival and overall survival. CTC count also improves the prognostication of metastatic breast cancer when added to full clinicopathological predictive models, whereas serum tumour markers do not. FUNDING: Janssen Diagnostics, the Nuovo-Soldati foundation for cancer research.

Expression profiling of cancerous and normal breast tissues identifies microRNAs that are differentially expressed in serum from patients with (metastatic) breast cancer and healthy volunteers.

INTRODUCTION: MicroRNAs (miRNAs) are a group of small noncoding RNAs involved in the regulation of gene expression. As such, they regulate a large number of cellular pathways, and deregulation or altered expression of miRNAs is associated with tumorigenesis. In the current study, we evaluated the feasibility and clinical utility of circulating miRNAs as biomarkers for the detection and staging of breast cancer. METHODS: miRNAs were extracted from a set of 84 tissue samples from patients with breast cancer and eight normal tissue samples obtained after breast-reductive surgery. After reverse transcription and preamplification, 768 miRNAs were profiled by using the TaqMan low-density arrays. After data normalization, unsupervised hierarchical cluster analysis (UHCA) was used to investigate global differences in miRNA expression between cancerous and normal samples. With fold-change analysis, the most discriminating miRNAs between both tissue types were selected, and their expression was analyzed on serum samples from 20 healthy volunteers and 75 patients with breast cancer, including 16 patients with untreated metastatic breast cancer. miRNAs were extracted from 200 µl of serum, reverse transcribed, and analyzed in duplicate by using polymerase chain reaction (qRT-PCR). RESULTS: UHCA showed major differences in miRNA expression between tissue samples from patients with breast cancer and tissue samples from breast-reductive surgery (P < 0.0001). Generally, miRNA expression in cancerous samples tends to be repressed when compared with miRNA expression in healthy controls (P = 0.0685). The four most discriminating miRNAs by fold-change (miR-215, miR-299-5p, miR-411, and miR-452) were selected for further analysis on serum samples. All miRNAs at least tended to be differentially expressed between serum samples from patients with cancer and serum samples from healthy controls (miR-215, P = 0.094; miR-299-5P, P = 0.019; miR-411, P = 0.002; and miR-452, P = 0.092). For all these miRNAs, except for miR-452, the greatest difference in expression was observed between serum samples from healthy volunteers and serum samples from untreated patients with metastatic breast cancer. CONCLUSIONS: Our study provides a basis for the establishment of miRNAs as biomarkers for the detection and eventually staging of breast cancer through blood-borne testing. We identified and tested a set of putative biomarkers of breast cancer and demonstrated that altered levels of these miRNAs in serum from patients with breast cancer are particularly associated with the presence of metastatic disease.

Development of Training Materials for Pathologists to Provide Machine Learning Validation Data of Tumor-Infiltrating Lymphocytes in Breast Cancer.

The High Throughput Truthing project aims to develop a dataset for validating artificial intelligence and machine learning models (AI/ML) fit for regulatory purposes. The context of this AI/ML validation dataset is the reporting of stromal tumor-infiltrating lymphocytes (sTILs) density evaluations in hematoxylin and eosin-stained invasive breast cancer biopsy specimens. After completing the pilot study, we found notable variability in the sTILs estimates as well as inconsistencies and gaps in the provided training to pathologists. Using the pilot study data and an expert panel, we created custom training materials to improve pathologist annotation quality for the pivotal study. We categorized regions of interest (ROIs) based on their mean sTILs density and selected ROIs with the highest and lowest sTILs variability. In a series of eight one-hour sessions, the expert panel reviewed each ROI and provided verbal density estimates and comments on features that confounded the sTILs evaluation. We aggregated and shaped the comments to identify pitfalls and instructions to improve our training materials. From these selected ROIs, we created a training set and proficiency test set to improve pathologist training with the goal to improve data collection for the pivotal study. We are not exploring AI/ML performance in this paper. Instead, we are creating materials that will train crowd-sourced pathologists to be the reference standard in a pivotal study to create an AI/ML model validation dataset. The issues discussed here are also important for clinicians to understand about the evaluation of sTILs in clinical practice and can provide insight to developers of AI/ML models.

The clinical use of circulating tumor cells (CTCs) enumeration for staging of metastatic breast cancer (MBC): International expert consensus paper.

BACKGROUND: The heterogeneity of metastatic breast cancer (MBC) necessitates novel biomarkers allowing stratification of patients for treatment selection and drug development. We propose to use the prognostic utility of circulating tumor cells (CTCs) for stratification of patients with stage IV disease. METHODS: In a retrospective, pooled analysis of individual patient data from 18 cohorts, including 2436 MBC patients, a CTC threshold of 5 cells per 7.5 ml was used for stratification based on molecular subtypes, disease location, and prior treatments. Patients with ≥ 5 CTCs were classified as Stage IVaggressive, those with < 5 CTCs as Stage IVindolent. Survival was analyzed using Kaplan-Meier curves and the log rank test. RESULTS: For all patients, Stage IVindolent patients had longer median overall survival than those with Stage IVaggressive (36.3 months vs. 16.0 months, P < 0.0001) and similarly for de novo MBC patients (41.4 months Stage IVindolent vs. 18.7 months Stage IVaggressive, p < 0.0001). Moreover, patients with Stage IVindolent disease had significantly longer overall survival across all disease subtypes compared to the aggressive cohort: hormone receptor-positive (44 months vs. 17.3 months, P < 0.0001), HER2-positive (36.7 months vs. 20.4 months, P < 0.0001), and triple negative (23.8 months vs. 9.0 months, P < 0.0001). Similar results were obtained regardless of prior treatment or disease location. CONCLUSIONS: We confirm the identification of two subgroups of MBC, Stage IVindolent and Stage IVaggressive, independent of clinical and molecular variables. Thus, CTC count should be considered an important tool for staging of advanced disease and for disease stratification in prospective clinical trials.

Circulating tumour cells and lung microvascular tumour cell retention in patients with metastatic breast and cervical cancer.

We have shown that in up to half of the patients with metastatic breast cancer (MBC), higher numbers of circulating tumour cells (CTCs) are present in the central venous blood (CVB) compared to the peripheral venous blood (PVB), suggesting that the lungs might retain a substantial number of CTCs. Here we report the presence of tumour cell emboli (TCE) in the microvasculature of the lungs in three out of eight patients with MBC and one patient with metastatic cervical carcinoma who had markedly elevated numbers of CTCs in the blood. All these patients suffered from symptomatic dyspnoea not easily attributable to other causes. No TCE were observed in five patients with MBC and elevated CTC counts and three patients with MBC who had low CTC counts (<5/7.5 ml). To investigate whether CTCs derived from CVB or PVB exhibit different transcriptional characteristics that might explain selective CTC retention, paired CTC samples from CVB and PVB of 12 patients with advanced breast cancer were subjected to gene expression analysis of 105 genes. No significant differences in CTC gene expression were observed. Together, these data suggest that potentially clinically relevant CTC retention in the microvasculature of the lung can occur in a subset of patients with advanced metastatic breast and cervical cancer, which seems to be transcriptionally non-selectively.

Gene expression profiles of circulating tumor cells versus primary tumors in metastatic breast cancer.

Before using circulating tumor cells (CTCs) as liquid biopsy, insight into molecular discrepancies between CTCs and primary tumors is essential. We characterized CellSearch-enriched CTCs from 62 metastatic breast cancer (MBC) patients with ≥5 CTCs starting first-line systemic treatment. Expression levels of 35 tumor-associated, CTC-specific genes, including ESR1, coding for the estrogen receptor (ER), were measured by reverse transcription quantitative polymerase chain reaction and correlated to corresponding primary tumors. In 30 patients (48%), gene expression profiles of 35 genes were discrepant between CTCs and the primary tumor, but this had no prognostic consequences. In 15 patients (24%), the expression of ER was discrepant. Patients with ER-negative primary tumors and ER-positive CTCs had a longer median TTS compared to those with concordantly ER-negative CTCs (8.5 versus 2.1 months, P = 0.05). From seven patients, an axillary lymph node metastasis was available. In two patients, the CTC profiles better resembled the lymph node metastasis than the primary tumor. Our findings suggest that molecular discordances between CTCs and primary tumors frequently occur, but that this bears no prognostic consequences. Alterations in ER-status between primary tumors and CTCs might have prognostic implications.

Microarray-based oncogenic pathway profiling in advanced serous papillary ovarian carcinoma.

INTRODUCTION: The identification of specific targets for treatment of ovarian cancer patients remains a challenge. The objective of this study is the analysis of oncogenic pathways in ovarian cancer and their relation with clinical outcome. METHODOLOGY: A meta-analysis of 6 gene expression datasets was done for oncogenic pathway activation scores: AKT, ß-Catenin, BRCA, E2F1, EGFR, ER, HER2, INFα, INFγ, MYC, p53, p63, PI3K, PR, RAS, SRC, STAT3, TNFα, and TGFß and VEGF-A. Advanced serous papillary tumours from uniformly treated patients were selected (N = 464) to find differences independent from stage-, histology- and treatment biases. Survival and correlations with documented prognostic signatures (wound healing response signature WHR/genomic grade index GGI/invasiveness gene signature IGS) were analysed. RESULTS: The GGI, WHR, IGS score were unexpectedly increased in chemosensitive versus chemoresistant patients. PR and RAS activation score were associated with survival outcome (p = 0.002;p = 0.004). Increased activations of ß-Catenin (p = 0.0009), E2F1 (p = 0.005), PI3K (p = 0.003) and p63 (p = 0.05) were associated with more favourable clinical outcome and were consistently correlated with three prognostic gene signatures. CONCLUSIONS: Oncogenic pathway profiling of advanced serous ovarian tumours revealed that increased ß-Catenin, E2F1, p63, PI3K, PR and RAS-pathway activation scores were significantly associated with favourable clinical outcome. WHR, GGI and IGS scores were unexpectedly increased in chemosensitive tumours. Earlier studies have shown that WHR, GGI and IGS are strongly associated with proliferation and that high-proliferative ovarian tumours are more chemosensitive. These findings may indicate opposite confounding of prognostic versus predictive factors when studying biomarkers in epithelial ovarian cancer.