Pilot study to evaluate tools to collect pathologist annotations for validating machine learning algorithms.

Purpose: Validation of artificial intelligence (AI) algorithms in digital pathology with a reference standard is necessary before widespread clinical use, but few examples focus on creating a reference standard based on pathologist annotations. This work assesses the results of a pilot study that collects density estimates of stromal tumor-infiltrating lymphocytes (sTILs) in breast cancer biopsy specimens. This work will inform the creation of a validation dataset for the evaluation of AI algorithms fit for a regulatory purpose. Approach: Collaborators and crowdsourced pathologists contributed glass slides, digital images, and annotations. Here, "annotations" refer to any marks, segmentations, measurements, or labels a pathologist adds to a report, image, region of interest (ROI), or biological feature. Pathologists estimated sTILs density in 640 ROIs from hematoxylin and eosin stained slides of 64 patients via two modalities: an optical light microscope and two digital image viewing platforms. Results: The pilot study generated 7373 sTILs density estimates from 29 pathologists. Analysis of annotations found the variability of density estimates per ROI increases with the mean; the root mean square differences were 4.46, 14.25, and 26.25 as the mean density ranged from 0% to 10%, 11% to 40%, and 41% to 100%, respectively. The pilot study informs three areas of improvement for future work: technical workflows, annotation platforms, and agreement analysis methods. Upgrades to the workflows and platforms will improve operability and increase annotation speed and consistency. Conclusions: Exploratory data analysis demonstrates the need to develop new statistical approaches for agreement. The pilot study dataset and analysis methods are publicly available to allow community feedback. The development and results of the validation dataset will be publicly available to serve as an instructive tool that can be replicated by developers and researchers.

Development of Training Materials for Pathologists to Provide Machine Learning Validation Data of Tumor-Infiltrating Lymphocytes in Breast Cancer.

The High Throughput Truthing project aims to develop a dataset for validating artificial intelligence and machine learning models (AI/ML) fit for regulatory purposes. The context of this AI/ML validation dataset is the reporting of stromal tumor-infiltrating lymphocytes (sTILs) density evaluations in hematoxylin and eosin-stained invasive breast cancer biopsy specimens. After completing the pilot study, we found notable variability in the sTILs estimates as well as inconsistencies and gaps in the provided training to pathologists. Using the pilot study data and an expert panel, we created custom training materials to improve pathologist annotation quality for the pivotal study. We categorized regions of interest (ROIs) based on their mean sTILs density and selected ROIs with the highest and lowest sTILs variability. In a series of eight one-hour sessions, the expert panel reviewed each ROI and provided verbal density estimates and comments on features that confounded the sTILs evaluation. We aggregated and shaped the comments to identify pitfalls and instructions to improve our training materials. From these selected ROIs, we created a training set and proficiency test set to improve pathologist training with the goal to improve data collection for the pivotal study. We are not exploring AI/ML performance in this paper. Instead, we are creating materials that will train crowd-sourced pathologists to be the reference standard in a pivotal study to create an AI/ML model validation dataset. The issues discussed here are also important for clinicians to understand about the evaluation of sTILs in clinical practice and can provide insight to developers of AI/ML models.

A Pathologist-Annotated Dataset for Validating Artificial Intelligence: A Project Description and Pilot Study.

PURPOSE: Validating artificial intelligence algorithms for clinical use in medical images is a challenging endeavor due to a lack of standard reference data (ground truth). This topic typically occupies a small portion of the discussion in research papers since most of the efforts are focused on developing novel algorithms. In this work, we present a collaboration to create a validation dataset of pathologist annotations for algorithms that process whole slide images. We focus on data collection and evaluation of algorithm performance in the context of estimating the density of stromal tumor-infiltrating lymphocytes (sTILs) in breast cancer. METHODS: We digitized 64 glass slides of hematoxylin- and eosin-stained invasive ductal carcinoma core biopsies prepared at a single clinical site. A collaborating pathologist selected 10 regions of interest (ROIs) per slide for evaluation. We created training materials and workflows to crowdsource pathologist image annotations on two modes: an optical microscope and two digital platforms. The microscope platform allows the same ROIs to be evaluated in both modes. The workflows collect the ROI type, a decision on whether the ROI is appropriate for estimating the density of sTILs, and if appropriate, the sTIL density value for that ROI. RESULTS: In total, 19 pathologists made 1645 ROI evaluations during a data collection event and the following 2 weeks. The pilot study yielded an abundant number of cases with nominal sTIL infiltration. Furthermore, we found that the sTIL densities are correlated within a case, and there is notable pathologist variability. Consequently, we outline plans to improve our ROI and case sampling methods. We also outline statistical methods to account for ROI correlations within a case and pathologist variability when validating an algorithm. CONCLUSION: We have built workflows for efficient data collection and tested them in a pilot study. As we prepare for pivotal studies, we will investigate methods to use the dataset as an external validation tool for algorithms. We will also consider what it will take for the dataset to be fit for a regulatory purpose: study size, patient population, and pathologist training and qualifications. To this end, we will elicit feedback from the Food and Drug Administration via the Medical Device Development Tool program and from the broader digital pathology and AI community. Ultimately, we intend to share the dataset, statistical methods, and lessons learned.

Mucinous borderline ovarian tumors: points of general agreement and persistent controversies regarding nomenclature, diagnostic criteria, and behavior.

This report focuses on the borderline category of ovarian mucinous tumors and summarizes the points of general agreement and persistent controversies identified by experts in the field who participated in the Borderline Ovarian Tumor Workshop held in Bethesda, MD, in August 2003. Points of agreement and persistent controversies regarding nomenclature, diagnostic criteria, and behavior are addressed for the following ovarian mucinous tumor categories: mucinous borderline ovarian tumor (M-BOT; synonymously referred to as atypical proliferative mucinous tumor of ovary or mucinous ovarian tumor of low malignant potential), M-BOT with intraepithelial carcinoma, and M-BOT with microinvasion. The morphologic spectrum of M-BOTs with regard to distinction from mucinous cystadenoma and the confluent glandular/expansile type of invasive mucinous carcinoma is also addressed. Non-ovarian mucinous tumors, including the secondary ovarian mucinous tumors associated with pseudomyxoma peritonei and metastatic mucinous carcinomas with a deceptive pattern of invasion, are recognized as tumors that can simulate primary M-BOTs. Improved classification of these mucinous tumors has clarifed the behavior of true M-BOTs by excluding these simulators from the M-BOT category.

Histopathology, FIGO stage, and BRCA mutation status of ovarian cancers from the Gilda Radner Familial Ovarian Cancer Registry.

Studies of the histopathology of ovarian cancer arising in patients with germline mutations in BRCA1 or BRCA2 have shown inconsistent findings. We analyzed the large number of tumors from women enrolled in the Gilda Radner Familial Ovarian Cancer Registry for correlations between histopathology and BRCA mutation status. Histopathology slides and reports were reviewed for histology, grade, and stage for cancers of the ovary or peritoneum in 220 women from 126 Gilda Radner Familial Ovarian Cancer Registry families. At least one affected member of each family was analyzed for mutations in the BRCA1 and BRCA2 genes, and tumors from mutation-positive families were compared with those from mutation-negative families. Of 70 patients from 38 BRCA1-positive families, 69 had epithelial ovarian carcinoma and one had a dysgerminoma. Fifteen of 16 patients from nine BRCA2-positive families had epithelial ovarian cancer, and one had a primary peritoneal cancer. Of 134 patients from 79 BRCA-negative families, 118 had epithelial ovarian carcinoma, 11 had ovarian borderline tumors, three had nonepithelial tumors, and two had primary peritoneal carcinoma. There were fewer grade 1 (p < 0.001) and stage I (p = 0.005) cancers in patients from BRCA-positive families than in patients from BRCA-negative families. Neither mucinous nor borderline tumors were found in the BRCA-positive families. In conclusion, ovarian cancers arising in women from BRCA-positive families are more likely to be high-grade and have extraovarian spread than tumors arising in women from BRCA-negative families. Borderline and mucinous tumors do not appear to be part of the phenotype of families with germline mutations in the BRCA genes.

BRCA1/2 mutation status influences somatic genetic progression in inherited and sporadic epithelial ovarian cancer cases.

Metaphase comparative genomic hybridization was used to analyze the spectrum of genetic alterations in 141 epithelial ovarian cancers from BRCA1 and BRCA2 mutation carriers, individuals with familial non-BRCA1/2 epithelial ovarian cancer, and women with nonfamilial epithelial ovarian cancer. Multiple genetic alterations were identified in almost all tumors. The high frequency with which some alterations were identified suggests the location of genes that are commonly altered during ovarian tumor development. In multiple chromosome regions, there were significant differences in alteration frequency between the four tumor types suggesting that BRCA1/2 mutation status and a family history of ovarian cancer influences the somatic genetic pathway of ovarian cancer progression. These findings were supported by hierarchical cluster analysis, which identified genetic events that tend to occur together during tumorigenesis and several alterations that were specific to tumors of a particular type. In addition, some genetic alterations were strongly associated with differences in tumor differentiation and disease stage. Taken together, these data provide molecular genetic evidence to support previous findings from histopathological studies, which suggest that clinical features of ovarian and breast tumors differ with respect to BRCA1/2 mutation status and/or cancer family history.

Endometrial carcinoma cells are nonpermissive for CD44-erbB2 interactions.

The erbB2 receptor tyrosine kinase and the CD44 transmembrane glycoprotein interact with one another in numerous cell types. This interaction helps to maintain erbB2 activity that contributes to tumor progression. We investigated whether CD44 and erbB2 similarly interact in endometrial carcinomas in vitro and in situ. In contrast to other carcinomas, CD44 did not colocalize with erbB2 in any of the 51 cases of endometrial cancer analyzed. CD44 also did not coimmunoprecipitate or colocalize with erbB2 in two endometrial carcinoma cell lines. We propose that the lack of CD44-erbB2 interactions may reduce the contribution of erbB2 to endometrial carcinoma progression.