Use of Artificial Intelligence Chatbots in Interpretation of Pathology Reports.

Importance: Anatomic pathology reports are an essential part of health care, containing vital diagnostic and prognostic information. Currently, most patients have access to their test results online. However, the reports are complex and are generally incomprehensible to laypeople. Artificial intelligence chatbots could potentially simplify pathology reports. Objective: To evaluate the ability of large language model chatbots to accurately explain pathology reports to patients. Design, Setting, and Participants: This cross-sectional study used 1134 pathology reports from January 1, 2018, to May 31, 2023, from a multispecialty hospital in Brooklyn, New York. A new chat was started for each report, and both chatbots (Bard [Google Inc], hereinafter chatbot 1; GPT-4 [OpenAI], hereinafter chatbot 2) were asked in sequential prompts to explain the reports in simple terms and identify key information. Chatbot responses were generated between June 1 and August 31, 2023. The mean readability scores of the original and simplified reports were compared. Two reviewers independently screened and flagged reports with potential errors. Three pathologists reviewed the flagged reports and categorized them as medically correct, partially medically correct, or medically incorrect; they also recorded any instances of hallucinations. Main Outcomes and Measures: Outcomes included improved mean readability scores and a medically accurate interpretation. Results: For the 1134 reports included, the Flesch-Kincaid grade level decreased from a mean of 13.19 (95% CI, 12.98-13.41) to 8.17 (95% CI, 8.08-8.25; t = 45.29; P < .001) by chatbot 1 and 7.45 (95% CI, 7.35-7.54; t = 49.69; P < .001) by chatbot 2. The Flesch Reading Ease score was increased from a mean of 10.32 (95% CI, 8.69-11.96) to 61.32 (95% CI, 60.80-61.84; t = -63.19; P < .001) by chatbot 1 and 70.80 (95% CI, 70.32-71.28; t = -74.61; P < .001) by chatbot 2. Chatbot 1 interpreted 993 reports (87.57%) correctly, 102 (8.99%) partially correctly, and 39 (3.44%) incorrectly; chatbot 2 interpreted 1105 reports (97.44%) correctly, 24 (2.12%) partially correctly, and 5 (0.44%) incorrectly. Chatbot 1 had 32 instances of hallucinations (2.82%), while chatbot 2 had 3 (0.26%). Conclusions and Relevance: The findings of this cross-sectional study suggest that artificial intelligence chatbots were able to simplify pathology reports. However, some inaccuracies and hallucinations occurred. Simplified reports should be reviewed by clinicians before distribution to patients.

Understanding the financial aspects of digital pathology: A dynamic customizable return on investment calculator for informed decision-making.

Background: The adoption of digital pathology has transformed the field of pathology, however, the economic impact and cost analysis of implementing digital pathology solutions remain a critical consideration for institutions to justify. Digital pathology implementation requires a thorough evaluation of associated costs and should identify and optimize resource allocation to facilitate informed decision-making. A dynamic cost calculator to estimate the financial implications of deploying digital pathology systems was needed to estimate the financial effects on transitioning to a digital workflow. Methods: A systematic approach was used to comprehensively assess the various components involved in implementing and maintaining a digital pathology system. This consisted of: (1) identification of key cost categories associated with digital pathology implementation; (2) data collection and analysis of cost estimation; (3) cost categorization and quantification of direct and indirect costs associated with different use cases, allowing customization of each factor based on specific intended uses and market rates, industry standards, and regional variations; (4) opportunities for savings realized by digitization of glass slides and (5) integration of the cost calculator into a unified framework for a holistic view of the financial implications associated with digital pathology implementation. The online tool enables the user to test various scenarios specific to their institution and provides adjustable parameters to assure organization specific relatability. Results: The Digital Pathology Association has developed a web-based calculator as a companion tool to provide an exhaustive list of the necessary concepts needed when assessing the financial implications of transitioning to a digital pathology system. The dynamic return on investment (ROI) calculator successfully integrated relevant cost and cost-saving components associated with digital pathology implementation and maintenance. Considerations include factors such as digital pathology infrastructure, clinical operations, staffing, hardware and software, information technology, archive and retrieval, medical-legal, and potential reimbursements. The ROI calculator developed for digital pathology workflows offers a comprehensive, customizable tool for institutions to assess their anticipated upfront and ongoing annual costs as they start or expand their digital pathology journey. It also offers cost-savings analysis based on specific user case volume, institutional geographic considerations, and actual costs. In addition, the calculator also serves as a tool to estimate number of required whole slide scanners, scanner throughput, and data storage (TB). This tool is intended to estimate the potential costs and cost savings resulting from the transition to digital pathology for business plan justifications and return on investment calculations. Conclusions: The digital pathology online cost calculator provides a comprehensive and reliable means of estimating the financial implications associated with implementing and maintaining a digital pathology system. By considering various cost factors and allowing customization based on institution-specific variables, the calculator empowers pathology laboratories, healthcare institutions, and administrators to make informed decisions and optimize resource allocation when adopting or expanding digital pathology technologies. The ROI calculator will enable healthcare institutions to assess the financial feasibility and potential return on investment on adopting digital pathology, facilitating informed decision-making and resource allocation.

The 1000 Mitoses Project: A Consensus-Based International Collaborative Study on Mitotic Figures Classification.

Introduction. The identification of mitotic figures is essential for the diagnosis, grading, and classification of various different tumors. Despite its importance, there is a paucity of literature reporting the consistency in interpreting mitotic figures among pathologists. This study leverages publicly accessible datasets and social media to recruit an international group of pathologists to score an image database of more than 1000 mitotic figures collectively. Materials and Methods. Pathologists were instructed to randomly select a digital slide from The Cancer Genome Atlas (TCGA) datasets and annotate 10-20 mitotic figures within a 2 mm2 area. The first 1010 submitted mitotic figures were used to create an image dataset, with each figure transformed into an individual tile at 40x magnification. The dataset was redistributed to all pathologists to review and determine whether each tile constituted a mitotic figure. Results. Overall pathologists had a median agreement rate of 80.2% (range 42.0%-95.7%). Individual mitotic figure tiles had a median agreement rate of 87.1% and a fair inter-rater agreement across all tiles (kappa = 0.284). Mitotic figures in prometaphase had lower percentage agreement rates compared to other phases of mitosis. Conclusion. This dataset stands as the largest international consensus study for mitotic figures to date and can be utilized as a training set for future studies. The agreement range reflects a spectrum of criteria that pathologists use to decide what constitutes a mitotic figure, which may have potential implications in tumor diagnostics and clinical management.

Training pathologists to assess stromal tumour-infiltrating lymphocytes in breast cancer synergises efforts in clinical care and scientific research.

A growing body of research supports stromal tumour-infiltrating lymphocyte (TIL) density in breast cancer to be a robust prognostic and predicive biomarker. The gold standard for stromal TIL density quantitation in breast cancer is pathologist visual assessment using haematoxylin and eosin-stained slides. Artificial intelligence/machine-learning algorithms are in development to automate the stromal TIL scoring process, and must be validated against a reference standard such as pathologist visual assessment. Visual TIL assessment may suffer from significant interobserver variability. To improve interobserver agreement, regulatory science experts at the US Food and Drug Administration partnered with academic pathologists internationally to create a freely available online continuing medical education (CME) course to train pathologists in assessing breast cancer stromal TILs using an interactive format with expert commentary. Here we describe and provide a user guide to this CME course, whose content was designed to improve pathologist accuracy in scoring breast cancer TILs. We also suggest subsequent steps to translate knowledge into clinical practice with proficiency testing.

Reproducible Reporting of the Collection and Evaluation of Annotations for Artificial Intelligence Models.

This work puts forth and demonstrates the utility of a reporting framework for collecting and evaluating annotations of medical images used for training and testing artificial intelligence (AI) models in assisting detection and diagnosis. AI has unique reporting requirements, as shown by the AI extensions to the Consolidated Standards of Reporting Trials (CONSORT) and Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) checklists and the proposed AI extensions to the Standards for Reporting Diagnostic Accuracy (STARD) and Transparent Reporting of a Multivariable Prediction model for Individual Prognosis or Diagnosis (TRIPOD) checklists. AI for detection and/or diagnostic image analysis requires complete, reproducible, and transparent reporting of the annotations and metadata used in training and testing data sets. In an earlier work by other researchers, an annotation workflow and quality checklist for computational pathology annotations were proposed. In this manuscript, we operationalize this workflow into an evaluable quality checklist that applies to any reader-interpreted medical images, and we demonstrate its use for an annotation effort in digital pathology. We refer to this quality framework as the Collection and Evaluation of Annotations for Reproducible Reporting of Artificial Intelligence (CLEARR-AI).

Patient-Reported Outcomes in Patients Undergoing Lumpectomy With and Without Defect Closure.

BACKGROUND: The effect of lumpectomy defect repair (a level 1 oncoplastic technique) on patient-reported breast satisfaction among patients undergoing lumpectomy has not yet been investigated. METHODS: Patients undergoing lumpectomy at our institution between 2018 and 2020 with or without repair of their lumpectomy defect during index operation, comprised our study population. The BREAST-Q quality-of-life questionnaire was administered preoperatively, and at 6 months, 1 year, and 2 years postoperatively. Satisfaction and quality-of-life domains were compared between those who did and did not have closure of their lumpectomy defect, and compared with surgeon-reported outcomes. RESULTS: A total of 487 patients met eligibility criteria, 206 (42%) had their partial mastectomy defect repaired by glandular displacement. Median breast volume, as calculated from the mammogram, was smaller in patients undergoing defect closure (826 cm3 vs. 895 cm3, p = 0.006). There were no statistically significant differences in satisfaction with breasts (SABTR), physical well-being of the chest (PWB-CHEST), or psychosocial well-being (PsychWB) scores between the two cohorts at any time point. While patients undergoing defect closure had significantly higher sexual well-being (SexWB) scores compared with no closure (66 vs. 59, p = 0.021), there were no predictors of improvement in SexWB scores over time on multivariable analysis. Patients' self-reported scores positively correlated with physician-reported outcomes. CONCLUSIONS: Despite a larger lumpectomy-to-breast volume ratio among patients undergoing defect repair, satisfaction was equivalent among those whose defects were or were not repaired at 2 years postsurgery. Defect repair was associated with clinically relevant improvement in patient-reported sexual well-being.

Recommendations for Performance Evaluation of Machine Learning in Pathology: A Concept Paper From the College of American Pathologists.

CONTEXT.­: Machine learning applications in the pathology clinical domain are emerging rapidly. As decision support systems continue to mature, laboratories will increasingly need guidance to evaluate their performance in clinical practice. Currently there are no formal guidelines to assist pathology laboratories in verification and/or validation of such systems. These recommendations are being proposed for the evaluation of machine learning systems in the clinical practice of pathology. OBJECTIVE.­: To propose recommendations for performance evaluation of in vitro diagnostic tests on patient samples that incorporate machine learning as part of the preanalytical, analytical, or postanalytical phases of the laboratory workflow. Topics described include considerations for machine learning model evaluation including risk assessment, predeployment requirements, data sourcing and curation, verification and validation, change control management, human-computer interaction, practitioner training, and competency evaluation. DATA SOURCES.­: An expert panel performed a review of the literature, Clinical and Laboratory Standards Institute guidance, and laboratory and government regulatory frameworks. CONCLUSIONS.­: Review of the literature and existing documents enabled the development of proposed recommendations. This white paper pertains to performance evaluation of machine learning systems intended to be implemented for clinical patient testing. Further studies with real-world clinical data are encouraged to support these proposed recommendations. Performance evaluation of machine learning models is critical to verification and/or validation of in vitro diagnostic tests using machine learning intended for clinical practice.

Initial interactions with the FDA on developing a validation dataset as a medical device development tool.

Quantifying tumor-infiltrating lymphocytes (TILs) in breast cancer tumors is a challenging task for pathologists. With the advent of whole slide imaging that digitizes glass slides, it is possible to apply computational models to quantify TILs for pathologists. Development of computational models requires significant time, expertise, consensus, and investment. To reduce this burden, we are preparing a dataset for developers to validate their models and a proposal to the Medical Device Development Tool (MDDT) program in the Center for Devices and Radiological Health of the U.S. Food and Drug Administration (FDA). If the FDA qualifies the dataset for its submitted context of use, model developers can use it in a regulatory submission within the qualified context of use without additional documentation. Our dataset aims at reducing the regulatory burden placed on developers of models that estimate the density of TILs and will allow head-to-head comparison of multiple computational models on the same data. In this paper, we discuss the MDDT preparation and submission process, including the feedback we received from our initial interactions with the FDA and propose how a qualified MDDT validation dataset could be a mechanism for open, fair, and consistent measures of computational model performance. Our experiences will help the community understand what the FDA considers relevant and appropriate (from the perspective of the submitter), at the early stages of the MDDT submission process, for validating stromal TIL density estimation models and other potential computational models. © 2023 The Authors. The Journal of Pathology published by John Wiley & Sons Ltd on behalf of The Pathological Society of Great Britain and Ireland. This article has been contributed to by U.S. Government employees and their work is in the public domain in the USA.

Digital pathology operations at a tertiary cancer center: Infrastructure requirements and operational cost.

Whole slide imaging is revolutionizing the field of pathology and is currently being used for clinical, educational, and research initiatives by an increasing number of institutions. Pathology departments have distinct needs for digital pathology systems, yet the cost of digital workflows is cited as a major barrier for widespread adoption by many organizations. Memorial Sloan Kettering Cancer Center (MSK) is an early adopter of whole slide imaging with incremental investments in resources that started more than 15 years ago. This experience and the large-scale scan operations led to the identification of required framework components of digital pathology operations. The cost of these components for the 2021 digital pathology operations at MSK were studied and calculated to enable an understanding of the operation and benchmark the accompanying costs. This paper describes the unique infrastructure cost and the costs associated with the digital pathology clinical operation use cases in a large, tertiary cancer center. These calculations can serve as a blueprint for other institutions to provide the necessary concepts and offer insights towards the financial requirements for digital pathology adoption by other institutions.

Future Practices of Breast Pathology Using Digital and Computational Pathology.

Pathology clinical practice has evolved by adopting technological advancements initially regarded as potentially disruptive, such as electron microscopy, immunohistochemistry, and genomic sequencing. Breast pathology has a critical role as a medical domain, where the patient's pathology diagnosis has significant implications for prognostication and treatment of diseases. The advent of digital and computational pathology has brought about significant advancements in the field, offering new possibilities for enhancing diagnostic accuracy and improving patient care. Digital slide scanning enables to conversion of glass slides into high-fidelity digital images, supporting the review of cases in a digital workflow. Digitization offers the capability to render specimen diagnoses, digital archival of patient specimens, collaboration, and telepathology. Integration of image analysis and machine learning-based systems layered atop the high-resolution digital images offers novel workflows to assist breast pathologists in their clinical, educational, and research endeavors. Decision support tools may improve the detection and classification of breast lesions and the quantification of immunohistochemical studies. Computational biomarkers may help to contribute to patient management or outcomes. Furthermore, using digital and computational pathology may increase standardization and quality assurance, especially in areas with high interobserver variability. This review explores the current landscape and possible future applications of digital and computational techniques in the field of breast pathology.

Ki67 proliferation index in medullary thyroid carcinoma: a comparative study of multiple counting methods and validation of image analysis and deep learning platforms.

AIMS: The International Medullary Thyroid Carcinoma Grading System, introduced in 2022, mandates evaluation of the Ki67 proliferation index to assign a histological grade for medullary thyroid carcinoma. However, manual counting remains a tedious and time-consuming task. METHODS AND RESULTS: We aimed to evaluate the performance of three other counting techniques for the Ki67 index, eyeballing by a trained experienced investigator, a machine learning-based deep learning algorithm (DeepLIIF) and an image analysis software with internal thresholding compared to the gold standard manual counting in a large cohort of 260 primarily resected medullary thyroid carcinoma. The Ki67 proliferation index generated by all three methods correlate near-perfectly with the manual Ki67 index, with kappa values ranging from 0.884 to 0.979 and interclass correlation coefficients ranging from 0.969 to 0.983. Discrepant Ki67 results were only observed in cases with borderline manual Ki67 readings, ranging from 3 to 7%. Medullary thyroid carcinomas with a high Ki67 index (≥ 5%) determined using any of the four methods were associated with significantly decreased disease-specific survival and distant metastasis-free survival. CONCLUSIONS: We herein validate a machine learning-based deep-learning platform and an image analysis software with internal thresholding to generate accurate automatic Ki67 proliferation indices in medullary thyroid carcinoma. Manual Ki67 count remains useful when facing a tumour with a borderline Ki67 proliferation index of 3-7%. In daily practice, validation of alternative evaluation methods for the Ki67 index in MTC is required prior to implementation.

Quality Management System in Clinical Digital Pathology Operations at a Tertiary Cancer Center.

Digital pathology workflows can improve pathology operations by allowing reliable and fast retrieval of digital images, digitally reviewing pathology slides, enabling remote work and telepathology, use of computer-aided tools, and sharing of digital images for research and educational purposes. The need for quality systems is a prerequisite for successful clinical-grade digital pathology adoption and patient safety. In this article, we describe the development of a structured digital pathology laboratory quality management system (QMS) for clinical digital pathology operations at Memorial Sloan Kettering Cancer Center (MSK). This digital pathology-specific QMS development stemmed from the gaps that were identified when MSK integrated digital pathology into its clinical practice. The digital scan team in conjunction with the Department of Pathology and Laboratory Medicine quality team developed a QMS tailored to the scanning operation to support departmental and institutional needs. As a first step, systemic mapping of the digital pathology operations identified the prescan, scan, and postscan processes; instrumentation; and staffing involved in the digital pathology operation. Next, gaps identified in quality control and quality assurance measures led to the development of standard operating procedures and training material for the different roles and workflows in the process. All digital pathology-related documents were subject to regulatory review and approval by departmental leadership. The quality essentials were developed into an extensive Digital Pathology Quality Essentials framework to specifically address the needs of the growing clinical use of digital pathology technologies. Using the unique digital experience gained at MSK, we present our recommendations for QMS for large-scale digital pathology operations in clinical settings.

Spatial analyses of immune cell infiltration in cancer: current methods and future directions: A report of the International Immuno-Oncology Biomarker Working Group on Breast Cancer.

Modern histologic imaging platforms coupled with machine learning methods have provided new opportunities to map the spatial distribution of immune cells in the tumor microenvironment. However, there exists no standardized method for describing or analyzing spatial immune cell data, and most reported spatial analyses are rudimentary. In this review, we provide an overview of two approaches for reporting and analyzing spatial data (raster versus vector-based). We then provide a compendium of spatial immune cell metrics that have been reported in the literature, summarizing prognostic associations in the context of a variety of cancers. We conclude by discussing two well-described clinical biomarkers, the breast cancer stromal tumor infiltrating lymphocytes score and the colon cancer Immunoscore, and describe investigative opportunities to improve clinical utility of these spatial biomarkers. © 2023 The Pathological Society of Great Britain and Ireland.

Pitfalls in machine learning-based assessment of tumor-infiltrating lymphocytes in breast cancer: A report of the International Immuno-Oncology Biomarker Working Group on Breast Cancer

The clinical significance of the tumor-immune interaction in breast cancer is now established, and tumor-infiltrating lymphocytes (TILs) have emerged as predictive and prognostic biomarkers for patients with triple-negative (estrogen receptor, progesterone receptor, and HER2-negative) breast cancer and HER2-positive breast cancer. How computational assessments of TILs might complement manual TIL assessment in trial and daily practices is currently debated. Recent efforts to use machine learning (ML) to automatically evaluate TILs have shown promising results. We review state-of-the-art approaches and identify pitfalls and challenges of automated TIL evaluation by studying the root cause of ML discordances in comparison to manual TIL quantification. We categorize our findings into four main topics: (1) technical slide issues, (2) ML and image analysis aspects, (3) data challenges, and (4) validation issues. The main reason for discordant assessments is the inclusion of false-positive areas or cells identified by performance on certain tissue patterns or design choices in the computational implementation. To aid the adoption of ML for TIL assessment, we provide an in-depth discussion of ML and image analysis, including validation issues that need to be considered before reliable computational reporting of TILs can be incorporated into the trial and routine clinical management of patients with triple-negative breast cancer. © 2023 The Authors. The Journal of Pathology published by John Wiley & Sons Ltd on behalf of The Pathological Society of Great Britain and Ireland.

Digital pathology systems enabling quality patient care.

Pathology laboratories are undergoing digital transformations, adopting innovative technologies to enhance patient care. Digital pathology systems impact clinical, education, and research use cases where pathologists use digital technologies to perform tasks in lieu of using glass slides and a microscope. Pathology professional societies have established clinical validation guidelines, and the US Food and Drug Administration have also authorized digital pathology systems for primary diagnosis, including image analysis and machine learning systems. Whole slide images, or digital slides, can be viewed and navigated similar to glass slides on a microscope. These modern tools not only enable pathologists to practice their routine clinical activities, but can potentially enable digital computational discovery. Assimilation of whole slide images in pathology clinical workflow can further empower machine learning systems to support computer assisted diagnostics. The potential enrichment these systems can provide is unprecedented in the field of pathology. With appropriate integration, these clinical decision support systems will allow pathologists to increase the delivery of quality patient care. This review describes the digital pathology transformation process, applicable clinical use cases, incorporation of image analysis and machine learning systems in the clinical workflow, as well as future technologies that may further disrupt pathology modalities to deliver quality patient care.

Changes in the Diagnoses of Breast Core Needle Biopsies on Second Review at a Tertiary Care Center: Implications for Surgical Management.

Core needle biopsy (CNB) of breast lesions is routine for diagnosis and treatment planning. Despite refinement of diagnostic criteria, the diagnosis of breast lesions on CNB can be challenging. At many centers, including ours, confirmation of diagnoses rendered in other laboratories is required before treatment planning. We identified CNBs first diagnosed elsewhere that were reviewed in our department over the course of 1 year because the patients sought care at our center and in which a change in diagnosis had been recorded. The outside and in-house CNB diagnoses were then classified based on Breast WHO Fifth Edition diagnostic categories. The impact of the change in diagnosis was estimated based on the subsequent surgical management. Findings in follow-up surgical excisions (EXCs) were used for validation. In 2018, 4950 outside cases with CNB were reviewed at our center. A total of 403 CNBs diagnoses were discrepant. Of these, 147 had a change in the WHO diagnostic category: 80 (54%) CNBs had a more severe diagnosis and 44 (30%) a less severe diagnosis. In 23 (16%) CNBs, the change of diagnostic category had no impact on management. Intraductal proliferations (n=54), microinvasive carcinoma (n=18), and papillary lesions (n=35) were the most disputed diagnoses. The in-house CNB diagnosis was confirmed in most cases with available excisions. Following CNB reclassification, 22/147 (15%) lesions were not excised. A change affecting the surgical management at our center occurred in 2.5% of all CNBs. Our results support routine review of outside breast CNB as a clinically significant practice before definitive treatment.

Pilot study to evaluate tools to collect pathologist annotations for validating machine learning algorithms.

Purpose: Validation of artificial intelligence (AI) algorithms in digital pathology with a reference standard is necessary before widespread clinical use, but few examples focus on creating a reference standard based on pathologist annotations. This work assesses the results of a pilot study that collects density estimates of stromal tumor-infiltrating lymphocytes (sTILs) in breast cancer biopsy specimens. This work will inform the creation of a validation dataset for the evaluation of AI algorithms fit for a regulatory purpose. Approach: Collaborators and crowdsourced pathologists contributed glass slides, digital images, and annotations. Here, "annotations" refer to any marks, segmentations, measurements, or labels a pathologist adds to a report, image, region of interest (ROI), or biological feature. Pathologists estimated sTILs density in 640 ROIs from hematoxylin and eosin stained slides of 64 patients via two modalities: an optical light microscope and two digital image viewing platforms. Results: The pilot study generated 7373 sTILs density estimates from 29 pathologists. Analysis of annotations found the variability of density estimates per ROI increases with the mean; the root mean square differences were 4.46, 14.25, and 26.25 as the mean density ranged from 0% to 10%, 11% to 40%, and 41% to 100%, respectively. The pilot study informs three areas of improvement for future work: technical workflows, annotation platforms, and agreement analysis methods. Upgrades to the workflows and platforms will improve operability and increase annotation speed and consistency. Conclusions: Exploratory data analysis demonstrates the need to develop new statistical approaches for agreement. The pilot study dataset and analysis methods are publicly available to allow community feedback. The development and results of the validation dataset will be publicly available to serve as an instructive tool that can be replicated by developers and researchers.

Expression of novel neuroendocrine markers in breast carcinomas: a study of INSM1, ASCL1, and POU2F3.

INSM1, ASCL1, and POU2F3 are novel transcription factors involved in neuroendocrine (NE) differentiation of neoplasms in several organs, but data on their expression in breast carcinomas (BCs) are limited. We retrospectively evaluated the expression of these markers in a series of 97 BCs (58 with NE morphology and 39 with otherwise uncommon morphology) tested prospectively using immunohistochemistry (IHC). Nuclear staining in >50% of the cells was used as the positive cut-off. Thirty-two of the 97 BCs (33%) were INSM1-positive. INSM1-positivity correlated significantly with histologic type and presence of stromal mucin. INSM1 also correlated with synaptophysin and chromogranin, established markers of NE differentiation (P < .0001 and P = .0023, respectively). In BC with NE morphology, the expression of INSM1 supported NE differentiation, and INSM1 was more specific than synaptophysin and more sensitive and specific than chromogranin. INSM1 was the most expressed NE marker in 17 BCs. INSM1-positive BCs included 56% of solid papillary BCs, 88% of BCs with solid papillary features, and 75% of high-grade NE carcinomas. Of 35 BCs tested for POU2F3 and ASCL1, only 1 and 4 cases were positive, respectively. Our results show that INSM1 is a sensitive marker of NE differentiation in BC and should be included with synaptophysin and chromogranin in the IHC panel used to evaluate NE differentiation in BC with NE morphology. ASCL1 and POU2F3 are uncommon in BC and their routine assessment is not warranted.

Development of Training Materials for Pathologists to Provide Machine Learning Validation Data of Tumor-Infiltrating Lymphocytes in Breast Cancer.

The High Throughput Truthing project aims to develop a dataset for validating artificial intelligence and machine learning models (AI/ML) fit for regulatory purposes. The context of this AI/ML validation dataset is the reporting of stromal tumor-infiltrating lymphocytes (sTILs) density evaluations in hematoxylin and eosin-stained invasive breast cancer biopsy specimens. After completing the pilot study, we found notable variability in the sTILs estimates as well as inconsistencies and gaps in the provided training to pathologists. Using the pilot study data and an expert panel, we created custom training materials to improve pathologist annotation quality for the pivotal study. We categorized regions of interest (ROIs) based on their mean sTILs density and selected ROIs with the highest and lowest sTILs variability. In a series of eight one-hour sessions, the expert panel reviewed each ROI and provided verbal density estimates and comments on features that confounded the sTILs evaluation. We aggregated and shaped the comments to identify pitfalls and instructions to improve our training materials. From these selected ROIs, we created a training set and proficiency test set to improve pathologist training with the goal to improve data collection for the pivotal study. We are not exploring AI/ML performance in this paper. Instead, we are creating materials that will train crowd-sourced pathologists to be the reference standard in a pivotal study to create an AI/ML model validation dataset. The issues discussed here are also important for clinicians to understand about the evaluation of sTILs in clinical practice and can provide insight to developers of AI/ML models.

Efficient Visualization of Whole Slide Images in Web-based Viewers for Digital Pathology.

CONTEXT.­: Wide adoption of digital pathology requires efficient visualization and navigation in Web-based digital slide viewers, which is poorly defined. OBJECTIVE.­: To define and quantify relevant performance metrics for efficient visualization of cases and slides in digital slide viewers. DESIGN.­: With a universal slide viewer used in clinical routine diagnostics, we evaluated the impact of slide caching, compression type, tile, and block size of whole slide images generated from Philips, Leica, and 3DHistech scanners on streaming performance on case, slide, and field of view levels. RESULTS.­: Two hundred thirty-nine pathologists routinely reviewed 60 080 whole slide images over 3 months. The median time to open a case's slides from the laboratory information system was less than 4 seconds, the time to change to a slide within the case was less than 1 second, and the time to render the adjacent field of view when navigating the slide was less than one-quarter of a second. A whole slide image's block size and a viewer tile size of 1024 pixels showed best performance to display a field of view and was preferrable over smaller tiles due to fewer mosaic effects. For Philips, fastest median slide streaming pace was 238 ms per field of view and for 3DHistech, 125 ms. For Leica, the fastest pace of 108 ms per field of view was established with block serving without decompression. CONCLUSIONS.­: This is the first study to systematically assess user-centric slide visualization performance metrics for digital viewers, including time to open a case, time to change a slide, and time to change a field of view. These metrics help to improve the viewer's configuration, leading to an efficient visualization baseline that is widely accepted among pathologists using routine digital pathology.