Accelerating pharmaceutical R&D with a user-friendly AI system for histopathology image analysis.

A system for analysis of histopathology data within a pharmaceutical R&D environment has been developed with the intention of enabling interdisciplinary collaboration. State-of-the-art AI tools have been deployed as easy-to-use self-service modules within an open-source whole slide image viewing platform, so that non-data scientist users (e.g., clinicians) can utilize and evaluate pre-trained algorithms and retrieve quantitative results. The outputs of analysis are automatically cataloged in the database to track data provenance and can be viewed interactively on the slide as annotations or heatmaps. Commonly used models for analysis of whole slide images including segmentation, extraction of hand-engineered features for segmented regions, and slide-level classification using multi-instance learning are included and new models can be added as needed. The source code that supports running inference with these models internally is backed up by a robust CI/CD pipeline to ensure model versioning, robust testing, and seamless deployment of the latest models. Examples of the use of this system in a pharmaceutical development workflow include glomeruli segmentation, enumeration of podocyte count from WT-1 immuno-histochemistry, measurement of beta-1 integrin target engagement from immunofluorescence, digital glomerular phenotyping from periodic acid-Schiff histology, PD-L1 score prediction using multi-instance learning, and the deployment of the open-source Segment Anything model to speed up annotation.

Generative Modeling of Histology Tissue Reduces Human Annotation Effort for Segmentation Model Development.

Segmentation of histology tissue whole side images is an important step for tissue analysis. Given enough annotated training data, modern neural networks are capable of accurate reproducible segmentation; however, the annotation of training datasets is time consuming. Techniques such as human-in-the-loop annotation attempt to reduce this annotation burden, but still require vast initial annotation. Semi-supervised learning-a technique which leverages both labeled and unlabeled data to learn features-has shown promise for easing the burden of annotation. Towards this goal, we employ a recently published semi-supervised method, datasetGAN, for the segmentation of glomeruli from renal biopsy images. We compare the performance of models trained using datasetGAN and traditional annotation and show that datasetGAN significantly reduces the amount of annotation required to develop a highly performing segmentation model. We also explore the usefulness of datasetGAN for transfer learning and find that this method greatly enhances the performance when a limited number of whole slide images are used for training.

A tool for federated training of segmentation models on whole slide images.

The largest bottleneck to the development of convolutional neural network (CNN) models in the computational pathology domain is the collection and curation of diverse training datasets. Training CNNs requires large cohorts of image data, and model generalizability is dependent on training data heterogeneity. Including data from multiple centers enhances the generalizability of CNN-based models, but this is hindered by the logistical challenges of sharing medical data. In this paper, we explore the feasibility of training our recently developed cloud-based segmentation tool (Histo-Cloud) using federated learning. Using a dataset of renal tissue biopsies we show that federated training to segment interstitial fibrosis and tubular atrophy (IFTA) using datasets from three institutions is not found to be different from a training by pooling the data on one server when tested on a fourth (holdout) institution's data. Further, training a model to segment glomeruli for a federated dataset (split by staining) demonstrates similar performance.

A user-friendly tool for cloud-based whole slide image segmentation with examples from renal histopathology.

Background: Image-based machine learning tools hold great promise for clinical applications in pathology research. However, the ideal end-users of these computational tools (e.g., pathologists and biological scientists) often lack the programming experience required for the setup and use of these tools which often rely on the use of command line interfaces. Methods: We have developed Histo-Cloud, a tool for segmentation of whole slide images (WSIs) that has an easy-to-use graphical user interface. This tool runs a state-of-the-art convolutional neural network (CNN) for segmentation of WSIs in the cloud and allows the extraction of features from segmented regions for further analysis. Results: By segmenting glomeruli, interstitial fibrosis and tubular atrophy, and vascular structures from renal and non-renal WSIs, we demonstrate the scalability, best practices for transfer learning, and effects of dataset variability. Finally, we demonstrate an application for animal model research, analyzing glomerular features in three murine models. Conclusions: Histo-Cloud is open source, accessible over the internet, and adaptable for segmentation of any histological structure regardless of stain.

Histo-Fetch - On-the-Fly Processing of Gigapixel Whole Slide Images Simplifies and Speeds Neural Network Training.

BACKGROUND: Training convolutional neural networks using pathology whole slide images (WSIs) is traditionally prefaced by the extraction of a training dataset of image patches. While effective, for large datasets of WSIs, this dataset preparation is inefficient. METHODS: We created a custom pipeline (histo-fetch) to efficiently extract random patches and labels from pathology WSIs for input to a neural network on-the-fly. We prefetch these patches as needed during network training, avoiding the need for WSI preparation such as chopping/tiling. RESULTS & CONCLUSIONS: We demonstrate the utility of this pipeline to perform artificial stain transfer and image generation using the popular networks CycleGAN and ProGAN, respectively. For a large WSI dataset, histo-fetch is 98.6% faster to start training and used 7535x less disk space.

A cloud-based tool for federated segmentation of whole slide images.

It is commonly known that diverse datasets of WSIs are beneficial when training convolutional neural networks, however sharing medical data between institutions is often hindered by regulatory concerns. We have developed a cloud-based tool for federated WSI segmentation, allowing collaboration between institutions without the need to directly share data. To show the feasibility of federated learning on pathology data in the real world, We demonstrate this tool by segmenting IFTA from three institutions and show that keeping the three datasets separate does not hinder segmentation performance. This pipeline is deployed in the cloud for easy access for data viewing and annotation by each site's respective constituents.

PodoSighter: A Cloud-Based Tool for Label-Free Podocyte Detection in Kidney Whole-Slide Images.

BACKGROUND: Podocyte depletion precedes progressive glomerular damage in several kidney diseases. However, the current standard of visual detection and quantification of podocyte nuclei from brightfield microscopy images is laborious and imprecise. METHODS: We have developed PodoSighter, an online cloud-based tool, to automatically identify and quantify podocyte nuclei from giga-pixel brightfield whole-slide images (WSIs) using deep learning. Ground-truth to train the tool used immunohistochemically or immunofluorescence-labeled images from a multi-institutional cohort of 122 histologic sections from mouse, rat, and human kidneys. To demonstrate the generalizability of our tool in investigating podocyte loss in clinically relevant samples, we tested it in rodent models of glomerular diseases, including diabetic kidney disease, crescentic GN, and dose-dependent direct podocyte toxicity and depletion, and in human biopsies from steroid-resistant nephrotic syndrome and from human autopsy tissues. RESULTS: The optimal model yielded high sensitivity/specificity of 0.80/0.80, 0.81/0.86, and 0.80/0.91, in mouse, rat, and human images, respectively, from periodic acid-Schiff-stained WSIs. Furthermore, the podocyte nuclear morphometrics extracted using PodoSighter were informative in identifying diseased glomeruli. We have made PodoSighter freely available to the general public as turnkey plugins in a cloud-based web application for end users. CONCLUSIONS: Our study demonstrates an automated computational approach to detect and quantify podocyte nuclei in standard histologically stained WSIs, facilitating podocyte research, and enabling possible future clinical applications.

A Distributed System Improves Inter-Observer and AI Concordance in Annotating Interstitial Fibrosis and Tubular Atrophy.

Histologic examination of interstitial fibrosis and tubular atrophy (IFTA) is critical to determine the extent of irreversible kidney injury in renal disease. The current clinical standard involves pathologist's visual assessment of IFTA, which is prone to inter-observer variability. To address this diagnostic variability, we designed two case studies (CSs), including seven pathologists, using HistomicsTK- a distributed system developed by Kitware Inc. (Clifton Park, NY). Twenty-five whole slide images (WSIs) were classified into a training set of 21 and a validation set of four. The training set was composed of seven unique subsets, each provided to an individual pathologist along with four common WSIs from the validation set. In CS 1, all pathologists individually annotated IFTA in their respective slides. These annotations were then used to train a deep learning algorithm to computationally segment IFTA. In CS 2, manual and computational annotations from CS 1 were first reviewed by the annotators to improve concordance of IFTA annotation. Both the manual and computational annotation processes were then repeated as in CS1. The inter-observer concordance in the validation set was measured by Krippendorff's alpha (KA). The KA for the seven pathologists in CS1 was 0.62 with CI [0.57, 0.67], and after reviewing each other's annotations in CS2, 0.66 with CI [0.60, 0.72]. The respective CS1 and CS2 KA were 0.58 with CI [0.52, 0.64] and 0.63 with CI [0.56, 0.69] when including the deep learner as an eighth annotator. These results suggest that our designed annotation framework refines agreement of spatial annotation of IFTA and demonstrates a human-AI approach to significantly improve the development of computational models.

In Silico Multi-Compartment Detection Based on Multiplex Immunohistochemical Staining in Renal Pathology.

With the rapid advancement in multiplex tissue staining, computer hardware, and machine learning, computationally-based tools are becoming indispensable for the evaluation of digital histopathology. Historically, standard histochemical staining methods such as hematoxylin and eosin, periodic acid-Schiff, and trichrome have been the gold standard for microscopic tissue evaluation by pathologists, and therefore brightfield microscopy images derived from such stains are primarily used for developing computational pathology tools. However, these histochemical stains are nonspecific in terms of highlighting structures and cell types. In contrast, immunohistochemical stains use antibodies to specifically detect and quantify proteins, which can be used to specifically highlight structures and cell types of interest. Traditionally, such immunofluorescence-based methods are only able to simultaneously stain a limited number of target proteins/antigens, typically up to three channels. Fluorescence-based multiplex immunohistochemistry (mIHC) is a new technology that enables simultaneous localization and quantification of numerous proteins/antigens, allowing for the possibility to detect a wide range of histologic structures and cell types within tissue. However, this method is limited by cost, specialized equipment, technical expertise, and time. In this study, we implemented a deep learning-based pipeline to synthetically generate in silico mIHC images from brightfield images of tissue slides-stained with routinely used histochemical stains, in particular PAS. Our tool was trained using fluorescence-based mIHC images as the ground-truth. The proposed pipeline offers high contrast detection of structures in brightfield imaged tissue sections stained with standard histochemical stains. We demonstrate the performance of our pipeline by computationally detecting multiple compartments in kidney biopsies, including cell nuclei, collagen/fibrosis, distal tubules, proximal tubules, endothelial cells, and leukocytes, from PAS-stained tissue sections. Our work can be extended for other histologic structures and tissue types and can be used as a basis for future automated annotation of histologic structures and cell types without the added cost of actually generating mIHC slides.

User friendly, cloud based, whole slide image segmentation.

Convolutional neural networks, the state of the art for image segmentation, have been successfully applied to histology images by many computational researchers. However, the translatability of this technology to clinicians and biological researchers is limited due to the complex and undeveloped user interface of the code, as well as the extensive computer setup required. We have developed a plugin for segmentation of whole slide images (WSIs) with an easy to use graphical user interface. This plugin runs a state-of-the-art convolutional neural network for segmentation of WSIs in the cloud. Our plugin is built on the open source tool HistomicsTK by Kitware Inc. (Clifton Park, NY), which provides remote data management and viewing abilities for WSI datasets. The ability to access this tool over the internet will facilitate widespread use by computational non-experts. Users can easily upload slides to a server where our plugin is installed and perform the segmentation analysis remotely. This plugin is open source and once trained, has the ability to be applied to the segmentation of any pathological structure. For a proof of concept, we have trained it to segment glomeruli from renal tissue images, demonstrating it on holdout tissue slides.

Generative modeling for renal microanatomy.

Generative adversarial networks (GANs) have received immense attention in the field of machine learning for their potential to learn high-dimensional and real data distribution. These methods do not rely on any assumptions about the data distribution of the input sample and can generate real-like samples from latent vector space based on unsupervised learning. In the medical field, particularly, in digital pathology expert annotation and availability of a large set of training data is costly and the study of manifestations of various diseases is based on visual examination of stained slides. In clinical practice, various staining information is required to improve the pathological diagnosis process. But when the sampled tissue to be examined is limited, the final diagnosis made by the pathologist is based on limited stain styles. These limitations can be overcome by studying the usability and reliability of generative models in the field of digital pathology. To understand the usability of the generative models, we synthesize in an unsupervised manner, high resolution renal microanatomical structures like renal glomerulus in thin tissue histology images using state-of-art architectures like Deep Convolutional Generative Adversarial Network (DCGAN) and Enhanced Super-Resolution Generative Adversarial Network (ESRGAN). Successful generation of such structures will lead to obtaining a large set of labeled data for further developing supervised algorithms for disease classification and understanding progression. Our study suggests while GAN is able to attain formalin fixed and paraffin embedded tissue image quality, GAN requires further prior knowledge as input to model intrinsic micro-anatomical details, such as capillary wall, urinary pole, nuclei placement, suggesting developing semi-supervised architectures by using these above details as prior information. Also, the generative models can be used to create an artificial effect of staining without physically tampering the histopathological slide. To demonstrate this, we use a CycleGAN network to transform Hematoxylin and eosin (H&E) stain to Periodic acid-Schiff (PAS) stain and Jones methenamine silver (JMS) stain to PAS stain. In this way GAN can be employed to translate different renal pathology stain styles when the relevant staining information is not available in the clinical settings.

Generative modeling for label-free glomerular modeling and classification.

Generative modeling using GANs has gained traction in machine learning literature, as training does not require labeled datasets. This is perfect for applications in biological datasets, where large labeled datasets are often difficult and expensive to acquire. However, generative models offer no easy way to encode real images into feature-sets, something that is desirable for network explainability and may yield potentially informative image features. For this reason, we test a VAE-GAN architecture for label-free modeling of glomerular structural features. We show that this network can generate realistic looking synthetic images, and be used to interpolate between images. To prove the biological relevance of the network encodings, we classify small-labeled sets of encoded glomeruli by biopsy Tervaert class and for the presence of sclerosis, obtaining a Cohen's kappa values of 0.87 and 0.78 respectfully.

Computational Segmentation and Classification of Diabetic Glomerulosclerosis.

BACKGROUND: Pathologists use visual classification of glomerular lesions to assess samples from patients with diabetic nephropathy (DN). The results may vary among pathologists. Digital algorithms may reduce this variability and provide more consistent image structure interpretation. METHODS: We developed a digital pipeline to classify renal biopsies from patients with DN. We combined traditional image analysis with modern machine learning to efficiently capture important structures, minimize manual effort and supervision, and enforce biologic prior information onto our model. To computationally quantify glomerular structure despite its complexity, we simplified it to three components consisting of nuclei, capillary lumina and Bowman spaces; and Periodic Acid-Schiff positive structures. We detected glomerular boundaries and nuclei from whole slide images using convolutional neural networks, and the remaining glomerular structures using an unsupervised technique developed expressly for this purpose. We defined a set of digital features which quantify the structural progression of DN, and a recurrent network architecture which processes these features into a classification. RESULTS: Our digital classification agreed with a senior pathologist whose classifications were used as ground truth with moderate Cohen's kappa κ = 0.55 and 95% confidence interval [0.50, 0.60]. Two other renal pathologists agreed with the digital classification with κ1 = 0.68, 95% interval [0.50, 0.86] and κ2 = 0.48, 95% interval [0.32, 0.64]. Our results suggest computational approaches are comparable to human visual classification methods, and can offer improved precision in clinical decision workflows. We detected glomerular boundaries from whole slide images with 0.93±0.04 balanced accuracy, glomerular nuclei with 0.94 sensitivity and 0.93 specificity, and glomerular structural components with 0.95 sensitivity and 0.99 specificity. CONCLUSIONS: Computationally derived, histologic image features hold significant diagnostic information that may augment clinical diagnostics.

An integrated iterative annotation technique for easing neural network training in medical image analysis.

Neural networks promise to bring robust, quantitative analysis to medical fields. However, their adoption is limited by the technicalities of training these networks and the required volume and quality of human-generated annotations. To address this gap in the field of pathology, we have created an intuitive interface for data annotation and the display of neural network predictions within a commonly used digital pathology whole-slide viewer. This strategy used a 'human-in-the-loop' to reduce the annotation burden. We demonstrate that segmentation of human and mouse renal micro compartments is repeatedly improved when humans interact with automatically generated annotations throughout the training process. Finally, to show the adaptability of this technique to other medical imaging fields, we demonstrate its ability to iteratively segment human prostate glands from radiology imaging data.

Automated erythrocyte detection and classification from whole slide images.

Blood smear is a crucial diagnostic aid. Quantification of both solitary and overlapping erythrocytes within these smears, directly from their whole slide images (WSIs), remains a challenge. Existing software designed to accomplish the computationally extensive task of hematological WSI analysis is too expensive and is widely unavailable. We have thereby developed a fully automated software targeted for erythrocyte detection and quantification from WSIs. We define an optimal region within the smear, which contains cells that are neither too scarce/damaged nor too crowded. We detect the optimal regions within the smear and subsequently extract all the cells from these regions, both solitary and overlapped, the latter of which undergoes a clump splitting before extraction. The performance was systematically tested on 28 WSIs of blood smears obtained from 13 different species from three classes of the subphylum vertebrata including birds, mammals, and reptiles. These data pose as an immensely variant erythrocyte database with diversity in size, shape, intensity, and textural features. Our method detected [Formula: see text] more cells than that detected from the traditional monolayer and resulted in a testing accuracy of 99.14% for the classification into their respective class (bird, mammal, or reptile) and a testing accuracy of 84.73% for the classification into their respective species. The results suggest the potential employment of this software for the diagnosis of hematological disorders, such as sickle cell anemia.

Computational detection and quantification of human and mouse neutrophil extracellular traps in flow cytometry and confocal microscopy.

Neutrophil extracellular traps (NETs) are extracellular defense mechanisms used by neutrophils, where chromatin is expelled together with histones and granular/cytoplasmic proteins. They have become an immunology hotspot, implicated in infections, but also in a diverse array of diseases such as systemic lupus erythematosus, diabetes, and cancer. However, the precise assessment of in vivo relevance in different disease settings has been hampered by limited tools to quantify occurrence of extracellular traps in experimental models and human samples. To expedite progress towards improved quantitative tools, we have developed computational pipelines to identify extracellular traps from an in vitro human samples visualized using the ImageStream® platform (Millipore Sigma, Darmstadt, Germany), and confocal images of an in vivo mouse disease model of aspergillus fumigatus pneumonia. Our two in vitro methods, tested on n = 363/n =145 images respectively, achieved holdout sensitivity/specificity 0.98/0.93 and 1/0.92. Our unsupervised method for thin lung tissue sections in murine fungal pneumonia achieved sensitivity/specificity 0.99/0.98 in n = 14 images. Our supervised method for thin lung tissue classified NETs with sensitivity/specificity 0.86/0.90. We expect that our approach will be of value for researchers, and have application in infectious and inflammatory diseases.