An initial game-theoretic assessment of enhanced tissue preparation and imaging protocols for improved deep learning inference of spatial transcriptomics from tissue morphology.

The application of deep learning to spatial transcriptomics (ST) can reveal relationships between gene expression and tissue architecture. Prior work has demonstrated that inferring gene expression from tissue histomorphology can discern these spatial molecular markers to enable population scale studies, reducing the fiscal barriers associated with large-scale spatial profiling. However, while most improvements in algorithmic performance have focused on improving model architectures, little is known about how the quality of tissue preparation and imaging can affect deep learning model training for spatial inference from morphology and its potential for widespread clinical adoption. Prior studies for ST inference from histology typically utilize manually stained frozen sections with imaging on non-clinical grade scanners. Training such models on ST cohorts is also costly. We hypothesize that adopting tissue processing and imaging practices that mirror standards for clinical implementation (permanent sections, automated tissue staining, and clinical grade scanning) can significantly improve model performance. An enhanced specimen processing and imaging protocol was developed for deep learning-based ST inference from morphology. This protocol featured the Visium CytAssist assay to permit automated hematoxylin and eosin staining (e.g. Leica Bond), 40×-resolution imaging, and joining of multiple patients' tissue sections per capture area prior to ST profiling. Using a cohort of 13 pathologic T Stage-III stage colorectal cancer patients, we compared the performance of models trained on slide prepared using enhanced versus traditional (i.e. manual staining and low-resolution imaging) protocols. Leveraging Inceptionv3 neural networks, we predicted gene expression across serial, histologically-matched tissue sections using whole slide images (WSI) from both protocols. The data Shapley was used to quantify and compare marginal performance gains on a patient-by-patient basis attributed to using the enhanced protocol versus the actual costs of spatial profiling. Findings indicate that training and validating on WSI acquired through the enhanced protocol as opposed to the traditional method resulted in improved performance at lower fiscal cost. In the realm of ST, the enhancement of deep learning architectures frequently captures the spotlight; however, the significance of specimen processing and imaging is often understated. This research, informed through a game-theoretic lens, underscores the substantial impact that specimen preparation/imaging can have on spatial transcriptomic inference from morphology. It is essential to integrate such optimized processing protocols to facilitate the identification of prognostic markers at a larger scale.

A Recognition System for Diagnosing Salivary Gland Neoplasms Based on Vision Transformer.

Salivary gland neoplasms (SGNs) represent a group of human neoplasms characterized by a remarkable cyto-morphological diversity, which frequently poses diagnostic challenges. Accurate histological categorization of salivary tumors is crucial to make precise diagnoses and guide decisions regarding patient management. Within the scope of this study, a computer-aided diagnosis model utilizing Vision Transformer, a cutting-edge deep-learning model in computer vision, has been developed to accurately classify the most prevalent subtypes of SGNs. These subtypes include pleomorphic adenoma, myoepithelioma, Warthin's tumor, basal cell adenoma, oncocytic adenoma, cystadenoma, mucoepidermoid carcinoma and salivary adenoid cystic carcinoma. The dataset comprised 3046 whole slide images (WSIs) of histologically confirmed salivary gland tumors, encompassing nine distinct tissue categories. SGN-ViT exhibited impressive performance in classifying the eight salivary gland tumors, achieving an accuracy of 0.9966, an AUC value of 0.9899, precision of 0.9848, recall of 0.9848, and an F1-score of 0.9848. When compared to benchmark models, SGN-ViT surpassed them in terms of diagnostic performance. In a subset of 100 WSIs, SGN-ViT demonstrated comparable diagnostic performance to that of the chief pathologist while significantly reducing the diagnosis time, indicating that SGN-ViT held the potential to serve as a valuable computer-aided diagnostic tool for salivary tumors, enhancing the diagnostic accuracy of junior pathologists.

Development of a multi-scanner facility for data acquisition for digital pathology artificial intelligence.

Whole slide imaging (WSI) of pathology glass slides using high-resolution scanners has enabled the large-scale application of artificial intelligence (AI) in pathology, to support the detection and diagnosis of disease, potentially increasing efficiency and accuracy in tissue diagnosis. Despite the promise of AI, it has limitations. 'Brittleness' or sensitivity to variation in inputs necessitates that large amounts of data are used for training. AI is often trained on data from different scanners but not usually by replicating the same slide across scanners. The utilisation of multiple WSI instruments to produce digital replicas of the same slides will make more comprehensive datasets and may improve the robustness and generalisability of AI algorithms as well as reduce the overall data requirements of AI training. To this end, the National Pathology Imaging Cooperative (NPIC) has built the AI FORGE (Facilitating Opportunities for Robust Generalisable data Emulation), a unique multi-scanner facility embedded in a clinical site in the NHS to (1) compare scanner performance, (2) replicate digital pathology image datasets across WSI systems, and (3) support the evaluation of clinical AI algorithms. The NPIC AI FORGE currently comprises 15 scanners from nine manufacturers. It can generate approximately 4,000 WSI images per day (approximately 7 TB of image data). This paper describes the process followed to plan and build such a facility. © 2024 The Author(s). The Journal of Pathology published by John Wiley & Sons Ltd on behalf of The Pathological Society of Great Britain and Ireland.

Slideflow: deep learning for digital histopathology with real-time whole-slide visualization.

Deep learning methods have emerged as powerful tools for analyzing histopathological images, but current methods are often specialized for specific domains and software environments, and few open-source options exist for deploying models in an interactive interface. Experimenting with different deep learning approaches typically requires switching software libraries and reprocessing data, reducing the feasibility and practicality of experimenting with new architectures. We developed a flexible deep learning library for histopathology called Slideflow, a package which supports a broad array of deep learning methods for digital pathology and includes a fast whole-slide interface for deploying trained models. Slideflow includes unique tools for whole-slide image data processing, efficient stain normalization and augmentation, weakly-supervised whole-slide classification, uncertainty quantification, feature generation, feature space analysis, and explainability. Whole-slide image processing is highly optimized, enabling whole-slide tile extraction at 40x magnification in 2.5 s per slide. The framework-agnostic data processing pipeline enables rapid experimentation with new methods built with either Tensorflow or PyTorch, and the graphical user interface supports real-time visualization of slides, predictions, heatmaps, and feature space characteristics on a variety of hardware devices, including ARM-based devices such as the Raspberry Pi.

Comparing Accuracy of Helicobacter pylori Identification Using Traditional Hematoxylin and Eosin-Stained Glass Slides With Digital Whole Slide Imaging.

With advancements in the field of digital pathology, there has been a growing need to compare the diagnostic abilities of pathologists using digitized whole slide images against those when using traditional hematoxylin and eosin (H&E)-stained glass slides for primary diagnosis. One of the most common specimens received in pathology practices is an endoscopic gastric biopsy with a request to rule out Helicobacter pylori (H. pylori) infection. The current standard of care is the identification of the organisms on H&E-stained slides. Immunohistochemical or histochemical stains are used selectively. However, due to their small size (2-4 µm in length by 0.5-1 µm in width), visualization of the organisms can present a diagnostic challenge. The goal of the study was to compare the ability of pathologists to identify H. pylori on H&E slides using a digital platform against the gold standard of H&E glass slides using routine light microscopy. Diagnostic accuracy rates using glass slides vs digital slides were 81% vs 72% (P = .0142) based on H&E slides alone. When H. pylori immunohistochemical slides were provided, the diagnostic accuracy was significantly improved to comparable rates (96% glass vs 99% digital, P = 0.2199). Furthermore, differences in practice settings (academic/subspecialized vs community/general) and the duration of sign-out experience did not significantly impact the accuracy of detecting H. pylori on digital slides. We concluded that digital whole slide images, although amenable in different practice settings and teaching environments, does present some shortcomings in accuracy and precision, especially in certain circumstances and thus is not yet fully capable of completely replacing glass slide review for identification of H. pylori. We specifically recommend reviewing glass slides and/or performing ancillary stains, especially when there is a discrepancy between the degree of inflammation and the presence of microorganisms on digital images.

Implementation of Digital Pathology and Artificial Intelligence in Routine Pathology Practice.

The advent of affordable technology has significantly influenced the practice of digital pathology, leading to its growing adoption within the pathology community. This review article aimed to outline the latest developments in digital pathology, the cutting-edge advancements in artificial intelligence (AI) applications within this field, and the pertinent United States regulatory frameworks. The content is based on a thorough analysis of original research articles and official United States Federal guidelines. Findings from our review indicate that several Food and Drug Administration-approved digital scanners and image management systems are establishing a solid foundation for the seamless integration of advanced technologies into everyday pathology workflows, which may reduce device and operational costs in the future. AI is particularly transforming the way morphologic diagnoses are automated, notably in cancers like prostate and colorectal, within screening initiatives, albeit challenges such as data privacy issues and algorithmic biases remain. The regulatory environment, shaped by standards from the Food and Drug Administration, Centers for Medicare & Medicaid Services/Clinical Laboratory Improvement Amendments, and College of American Pathologists, is evolving to accommodate these innovations while ensuring safety and reliability. Centers for Medicare & Medicaid Services/Clinical Laboratory Improvement Amendments have issued policies to allow pathologists to review and render diagnoses using digital pathology remotely. Moreover, the introduction of new digital pathology Current Procedural Terminology codes designed to complement existing pathology Current Procedural Terminology codes is facilitating reimbursement processes. Overall, these advancements are heralding a new era in pathology that promises enhanced diagnostic precision and efficiency through digital and AI technologies, potentially improving patient care as well as bolstering educational and research activities.

Current status and challenges in HER2 IHC assessment: scoring survey results in Japan.

PURPOSE: This study aimed to assess the concordance of human epidermal growth factor receptor 2 (HER2) expression scoring by immunohistochemistry (IHC) among practicing pathologists in Japan, given the challenging nature of scoring and the critical role of HER2 status in breast cancer management. METHODS: Whole slide images (WSI) from 20 invasive breast cancer cases (1 representative WSI per case) selected to represent a diverse IHC scores and staining patterns were used in an online survey involving seven reference pathologists who established consensus HER2 IHC scores (0 to 3 +) decided by majority interpretation. Participating pathologists nationwide scored the same 20 WSI cases online using the American Society of Clinical Oncology/College of American Pathologists (ASCO/CAP) 2018 guidelines. Deidentified case metadata were registered in the uPath system. RESULTS: A total of 144 participating pathologists responded. The scoring results of the participating pathologists most commonly agreed with the consensus IHC score, followed by a ± 1 point deviation and no survey responses with > 1 point deviation. The mean percentage of agreement with the consensus score for all 20 cases was 63.4%. In cases where the reference pathologists' scores were discordant, the participating pathologists also showed a lower concordance rate. CONCLUSION: This study highlighted the current status of HER2 expression scoring by IHC for breast cancer among pathologists in Japan. These findings underscore the challenges in HER2 IHC scoring cases and emphasize the need for improved standardization and training, especially in the evolving landscape of HER2-targeted therapies.

Automated whole slide morphometry of sural nerve biopsy using machine learning.

AIM: The morphometry of sural nerve biopsies, such as fibre diameter and myelin thickness, helps us understand the underlying mechanism of peripheral neuropathies. However, in current clinical practice, only a portion of the specimen is measured manually because of its labour-intensive nature. In this study, we aimed to develop a machine learning-based application that inputs a whole slide image (WSI) of the biopsied sural nerve and automatically performs morphometric analyses. METHODS: Our application consists of three supervised learning models: (1) nerve fascicle instance segmentation, (2) myelinated fibre detection and (3) myelin sheath segmentation. We fine-tuned these models using 86 toluidine blue-stained slides from various neuropathies and developed an open-source Python library. RESULTS: Performance evaluation showed (1) a mask average precision (AP) of 0.861 for fascicle segmentation, (2) box AP of 0.711 for fibre detection and (3) a mean intersection over union (mIoU) of 0.817 for myelin segmentation. Our software identified 323,298 nerve fibres and 782 fascicles in 70 WSIs. Small and large fibre populations were objectively determined based on clustering analysis. The demyelination group had large fibres with thinner myelin sheaths and higher g-ratios than the vasculitis group. The slope of the regression line from the scatter plots of the diameters and g-ratios was higher in the demyelination group than in the vasculitis group. CONCLUSION: We developed an application that performs whole slide morphometry of human biopsy samples. Our open-source software can be used by clinicians and pathologists without specific machine learning skills, which we expect will facilitate data-driven analysis of sural nerve biopsies for a more detailed understanding of these diseases.

An automated slide scanning system for membrane filter imaging in diagnosis of urogenital schistosomiasis.

Traditionally, automated slide scanning involves capturing a rectangular grid of field-of-view (FoV) images which can be stitched together to create whole slide images, while the autofocusing algorithm captures a focal stack of images to determine the best in-focus image. However, these methods can be time-consuming due to the need for X-, Y- and Z-axis movements of the digital microscope while capturing multiple FoV images. In this paper, we propose a solution to minimise these redundancies by presenting an optimal procedure for automated slide scanning of circular membrane filters on a glass slide. We achieve this by following an optimal path in the sample plane, ensuring that only FoVs overlapping the filter membrane are captured. To capture the best in-focus FoV image, we utilise a hill-climbing approach that tracks the peak of the mean of Gaussian gradient of the captured FoVs images along the Z-axis. We implemented this procedure to optimise the efficiency of the Schistoscope, an automated digital microscope developed to diagnose urogenital schistosomiasis by imaging Schistosoma haematobium eggs on 13 or 25 mm membrane filters. Our improved method reduces the automated slide scanning time by 63.18% and 72.52% for the respective filter sizes. This advancement greatly supports the practicality of the Schistoscope in large-scale schistosomiasis monitoring and evaluation programs in endemic regions. This will save time, resources and also accelerate generation of data that is critical in achieving the targets for schistosomiasis elimination.

Artificial Intelligence Recognition Model Using Liquid-Based Cytology Images to Discriminate Malignancy and Histological Types of Non-Small-Cell Lung Cancer.

INTRODUCTION: Artificial intelligence image recognition has applications in clinical practice. The purpose of this study was to develop an automated image classification model for lung cancer cytology using a deep learning convolutional neural network (DCNN). METHODS: Liquid-based cytology samples from 8 normal parenchymal (N), 22 adenocarcinoma (ADC), and 15 squamous cell carcinoma (SQCC) surgical specimens were prepared, and 45 Papanicolaou-stained slides were scanned using whole-slide imaging. The final dataset of 9,141 patches consisted of 2,737 N, 4,756 ADC, and 1,648 SQCC samples. Densenet-121 was used as the DCNN to classify N versus malignant (ADC+SQCC) and ADC versus SQCC images. AdamW optimizer and 5-fold cross-validation were used in the training. RESULTS: For malignancy prediction, the sensitivity, specificity, and accuracy were 0.97, 0.85, and 0.94, respectively, in the patch-level classification, and 0.92, 0.88, and 0.91, respectively, in the case-level classification. For SQCC prediction, the sensitivity, specificity, and accuracy were 0.86, 0.91, and 0.90, respectively, in the patch-level classification and 0.73, 0.82, and 0.78, respectively, in the case-level classification. CONCLUSION: The DCNN model performed excellently in predicting malignancy and histological types of lung cancer. This model may be useful for predicting cytopathological diagnosis in clinical situations by reinforcing training.

Digital whole-slide imaging of changes in amyloid after peripheral blood stem cell transplantation in patients with amyloid light-chain amyloidosis.

Peripheral blood stem cell transplantation (PBSCT) has made amyloid light-chain (AL) amyloidosis treatable. After PBSCT, hematological complete remission (HCR) can be achieved, leading to improved renal prognosis. The purpose of this study was to evaluate whether whole slide imaging of biopsy samples shows a post-treatment reduction in amyloid deposits in patients with AL amyloidosis. Patients were divided into three groups: Group A (n = 8), not eligible for PBSCT and treated with other therapies; Group B (n = 11), treated with PBSCT and achieved HCR; and Group C (n = 5), treated with PBSCT but did not achieve HCR. Clinical findings and amyloid deposition in glomeruli, interstitium, and blood vessels were compared before and after treatment using digital whole-slide imaging. Proteinuria and hypoalbuminemia improved more in Group B than in the other groups, and in Group B, amyloid deposition improved more in the glomeruli than in the interstitium and blood vessels. The long-term renal and survival prognosis was better in Group B than in the other groups. PBSCT can be expected to improve long-term clinical and renal histological prognosis in patients with AL amyloidosis who achieve HCR. Amyloid disappearance from renal tissue may take a long time even after clinical HCR.

Vision transformer introduces a new vitality to the classification of renal pathology.

Recent advancements in computer vision within the field of artificial intelligence (AI) have made significant inroads into the medical domain. However, the application of AI for classifying renal pathology remains challenging due to the subtle variations in multiple renal pathological classifications. Vision Transformers (ViT), an adaptation of the Transformer model for image recognition, have demonstrated superior capabilities in capturing global features and providing greater explainability. In our study, we developed a ViT model using a diverse set of stained renal histopathology images to evaluate its effectiveness in classifying renal pathology. A total of 1861 whole slide images (WSI) stained with HE, MASSON, PAS, and PASM were collected from 635 patients. Renal tissue images were then extracted, tiled, and categorized into 14 classes on the basis of renal pathology. We employed the classic ViT model from the Timm library, utilizing images sized 384 × 384 pixels with 16 × 16 pixel patches, to train the classification model. A comparative analysis was conducted to evaluate the performance of the ViT model against traditional convolutional neural network (CNN) models. The results indicated that the ViT model demonstrated superior recognition ability (accuracy: 0.96-0.99). Furthermore, we visualized the identification process of the ViT models to investigate potentially significant pathological ultrastructures. Our study demonstrated that ViT models outperformed CNN models in accurately classifying renal pathology. Additionally, ViT models are able to focus on specific, significant structures within renal histopathology, which could be crucial for identifying novel and meaningful pathological features in the diagnosis and treatment of renal disease.

A Microscope Setup and Methodology for Capturing Hyperspectral and RGB Histopathological Imaging Databases.

The digitization of pathology departments in hospitals around the world is now a reality. The current commercial solutions applied to digitize histopathological samples consist of a robotic microscope with an RGB-type camera attached to it. This technology is very limited in terms of information captured, as it only works with three spectral bands of the visible electromagnetic spectrum. Therefore, we present an automated system that combines RGB and hyperspectral technology. Throughout this work, the hardware of the system and its components are described along with the developed software and a working methodology to ensure the correct capture of histopathological samples. The software is integrated by the controller of the microscope, which features an autofocus functionality, whole slide scanning with a stitching algorithm, and hyperspectral scanning functionality. As a reference, the time to capture and process a complete sample with 20 regions of high biological interest using the proposed method is estimated at a maximum of 79 min, reducing the time required by a manual operator by at least three times. Both hardware and software can be easily adapted to other systems that might benefit from the advantages of hyperspectral technology.

An Artificial Intelligent System for Prostate Cancer Diagnosis in Whole Slide Images.

In recent years a significant demand to develop computer-assisted diagnostic tools to assess prostate cancer using whole slide images has been observed. In this study we develop and validate a machine learning system for cancer assessment, inclusive of detection of perineural invasion and measurement of cancer portion to meet clinical reporting needs. The system analyses the whole slide image in three consecutive stages: tissue detection, classification, and slide level analysis. The whole slide image is divided into smaller regions (patches). The tissue detection stage relies upon traditional machine learning to identify WSI patches containing tissue, which are then further assessed at the classification stage where deep learning algorithms are employed to detect and classify cancer tissue. At the slide level analysis stage, entire slide level information is generated by aggregating all the patch level information of the slide. A total of 2340 haematoxylin and eosin stained slides were used to train and validate the system. A medical team consisting of 11 board certified pathologists with prostatic pathology subspeciality competences working independently in 4 different medical centres performed the annotations. Pixel-level annotation based on an agreed set of 10 annotation terms, determined based on medical relevance and prevalence, was created by the team. The system achieved an accuracy of 99.53% in tissue detection, with sensitivity and specificity respectively of 99.78% and 99.12%. The system achieved an accuracy of 92.80% in classifying tissue terms, with sensitivity and specificity respectively 92.61% and 99.25%, when 5x magnification level was used. For 10x magnification, these values were respectively 91.04%, 90.49%, and 99.07%. For 20x magnification they were 84.71%, 83.95%, 90.13%.

Evaluation of A Computer-Aided Detection Software for Prostate Cancer Prediction: Excellent Diagnostic Accuracy Independent of Preanalytical Factors.

Prostate cancer (PCa) is the most common noncutaneous cancer in men in the Western world. In addition to accurate diagnosis, Gleason grading and tumor volume estimates are critical for patient management. Computer-aided detection (CADe) software can be used to facilitate the diagnosis and improve the diagnostic accuracy and reporting consistency. However, preanalytical factors such as fixation and staining of prostate biopsy specimens and whole slide images (WSI) on scanners can vary significantly between pathology laboratories and may, therefore, impact the quality of WSI and utility of CADe algorithms. We evaluated the performance of a CADe software in predicting PCa on WSIs of prostate biopsy specimens and focused on whether there were any significant differences in image quality between WSIs obtained on different scanners and specimens from different histopathology laboratories. Thirty prostate biopsy specimens from 2 histopathology laboratories in the United States were included in this study. The hematoxylin and eosin slides of the biopsy specimens were scanned on 3 scanners, generating 90 WSIs. These WSIs were then analyzed using a CADe software (INIFY Prostate, Algorithm), which identified and annotated all areas suspicious for PCa and calculated the tumor volume (percentage area of the biopsy core involved). Study pathologists then reviewed the Algorithm's annotations and tumor volume calculation to confirm the diagnosis and identify benign glands that were misclassified as cancer (false positive) and cancer glands that were misclassified as benign (false negative). The CADe software worked equally well on WSIs from all 3 scanners and from both laboratories, with similar sensitivity and specificity. The overall sensitivity was 99.4%, and specificity was 97%. The percentage of suspicious cancer areas calculated by the Algorithm was similar for all 3 scanners. For WSIs with small foci of cancer (<1 mm), the Algorithm identified all cancer glands (sensitivity, 100%). Preanalytical factors had no significant impact on whole slide imaging and subsequent application of a CADe software. Prediction accuracy could potentially be further improved by processing biopsy specimens in a centralized histology laboratory and training the Algorithm on WSIs from the same laboratory in order to minimize variations in preanalytical factors and optimize the diagnostic performance of the Algorithm.

Unstained Tissue Imaging and Virtual Hematoxylin and Eosin Staining of Histologic Whole Slide Images.

Tissue structures, phenotypes, and pathology are routinely investigated based on histology. This includes chemically staining the transparent tissue sections to make them visible to the human eye. Although chemical staining is fast and routine, it permanently alters the tissue and often consumes hazardous reagents. On the other hand, on using adjacent tissue sections for combined measurements, the cell-wise resolution is lost owing to sections representing different parts of the tissue. Hence, techniques providing visual information of the basic tissue structure enabling additional measurements from the exact same tissue section are required. Here we tested unstained tissue imaging for the development of computational hematoxylin and eosin (HE) staining. We used unsupervised deep learning (CycleGAN) and whole slide images of prostate tissue sections to compare the performance of imaging tissue in paraffin, as deparaffinized in air, and as deparaffinized in mounting medium with section thicknesses varying between 3 and 20 µm. We showed that although thicker sections increase the information content of tissue structures in the images, thinner sections generally perform better in providing information that can be reproduced in virtual staining. According to our results, tissue imaged in paraffin and as deparaffinized provides a good overall representation of the tissue for virtually HE-stained images. Further, using a pix2pix model, we showed that the reproduction of overall tissue histology can be clearly improved with image-to-image translation using supervised learning and pixel-wise ground truth. We also showed that virtual HE staining can be used for various tissues and used with both 20× and 40× imaging magnifications. Although the performance and methods of virtual staining need further development, our study provides evidence of the feasibility of whole slide unstained microscopy as a fast, cheap, and feasible approach to producing virtual staining of tissue histology while sparing the exact same tissue section ready for subsequent utilization with follow-up methods at single-cell resolution.

Whole-Slide Imaging, Mutual Information Registration for Multiplex Immunohistochemistry and Immunofluorescence.

Multiplex immunohistochemistry/immunofluorescence (mIHC/mIF) is a developing technology that facilitates the evaluation of multiple, simultaneous protein expressions at single-cell resolution while preserving tissue architecture. These approaches have shown great potential for biomarker discovery, yet many challenges remain. Importantly, streamlined cross-registration of multiplex immunofluorescence images with additional imaging modalities and immunohistochemistry (IHC) can help increase the plex and/or improve the quality of the data generated by potentiating downstream processes such as cell segmentation. To address this problem, a fully automated process was designed to perform a hierarchical, parallelizable, and deformable registration of multiplexed digital whole-slide images (WSIs). We generalized the calculation of mutual information as a registration criterion to an arbitrary number of dimensions, making it well suited for multiplexed imaging. We also used the self-information of a given IF channel as a criterion to select the optimal channels to use for registration. Additionally, as precise labeling of cellular membranes in situ is essential for robust cell segmentation, a pan-membrane immunohistochemical staining method was developed for incorporation into mIF panels or for use as an IHC followed by cross-registration. In this study, we demonstrate this process by registering whole-slide 6-plex/7-color mIF images with whole-slide brightfield mIHC images, including a CD3 and a pan-membrane stain. Our algorithm, WSI, mutual information registration (WSIMIR), performed highly accurate registration allowing the retrospective generation of an 8-plex/9-color, WSI, and outperformed 2 alternative automated methods for cross-registration by Jaccard index and Dice similarity coefficient (WSIMIR vs automated WARPY, P < .01 and P < .01, respectively, vs HALO + transformix, P = .083 and P = .049, respectively). Furthermore, the addition of a pan-membrane IHC stain cross-registered to an mIF panel facilitated improved automated cell segmentation across mIF WSIs, as measured by significantly increased correct detections, Jaccard index (0.78 vs 0.65), and Dice similarity coefficient (0.88 vs 0.79).

A Deep Learning Approach for Histology-Based Nucleus Segmentation and Tumor Microenvironment Characterization.

Microscopic examination of pathology slides is essential to disease diagnosis and biomedical research. However, traditional manual examination of tissue slides is laborious and subjective. Tumor whole-slide image (WSI) scanning is becoming part of routine clinical procedures and produces massive data that capture tumor histologic details at high resolution. Furthermore, the rapid development of deep learning algorithms has significantly increased the efficiency and accuracy of pathology image analysis. In light of this progress, digital pathology is fast becoming a powerful tool to assist pathologists. Studying tumor tissue and its surrounding microenvironment provides critical insight into tumor initiation, progression, metastasis, and potential therapeutic targets. Nucleus segmentation and classification are critical to pathology image analysis, especially in characterizing and quantifying the tumor microenvironment (TME). Computational algorithms have been developed for nucleus segmentation and TME quantification within image patches. However, existing algorithms are computationally intensive and time consuming for WSI analysis. This study presents Histology-based Detection using Yolo (HD-Yolo), a new method that significantly accelerates nucleus segmentation and TME quantification. We demonstrate that HD-Yolo outperforms existing WSI analysis methods in nucleus detection, classification accuracy, and computation time. We validated the advantages of the system on 3 different tissue types: lung cancer, liver cancer, and breast cancer. For breast cancer, nucleus features by HD-Yolo were more prognostically significant than both the estrogen receptor status by immunohistochemistry and the progesterone receptor status by immunohistochemistry. The WSI analysis pipeline and a real-time nucleus segmentation viewer are available at https://github.com/impromptuRong/hd_wsi.

Deep Learning for Predicting Effect of Neoadjuvant Therapies in Non-Small Cell Lung Carcinomas With Histologic Images.

Neoadjuvant therapies are used for locally advanced non-small cell lung carcinomas, whereby pathologists histologically evaluate the effect using resected specimens. Major pathological response (MPR) has recently been used for treatment evaluation and as an economical survival surrogate; however, interobserver variability and poor reproducibility are often noted. The aim of this study was to develop a deep learning (DL) model to predict MPR from hematoxylin and eosin-stained tissue images and to validate its utility for clinical use. We collected data on 125 primary non-small cell lung carcinoma cases that were resected after neoadjuvant therapy. The cases were randomly divided into 55 for training/validation and 70 for testing. A total of 261 hematoxylin and eosin-stained slides were obtained from the maximum tumor beds, and whole slide images were prepared. We used a multiscale patch model that can adaptively weight multiple convolutional neural networks trained with different field-of-view images. We performed 3-fold cross-validation to evaluate the model. During testing, we compared the percentages of viable tumor evaluated by annotator pathologists (reviewed data), those evaluated by nonannotator pathologists (primary data), and those predicted by the DL-based model using 2-class confusion matrices and receiver operating characteristic curves and performed a survival analysis between MPR-achieved and non-MPR cases. In cross-validation, accuracy and mean F1 score were 0.859 and 0.805, respectively. During testing, accuracy and mean F1 score with reviewed data and those with primary data were 0.986, 0.985, 0.943, and 0.943, respectively. The areas under the receiver operating characteristic curve with reviewed and primary data were 0.999 and 0.978, respectively. The disease-free survival of MPR-achieved cases with reviewed and primary data was significantly better than that of the non-MPR cases (P<.001 and P=.001), and that predicted by the DL-based model was almost identical (P=.005). The DL model may support pathologist evaluations and can offer accurate determinations of MPR in patients.

An Automated Pipeline for Differential Cell Counts on Whole-Slide Bone Marrow Aspirate Smears.

The pathologic diagnosis of bone marrow disorders relies in part on the microscopic analysis of bone marrow aspirate (BMA) smears and the manual counting of marrow nucleated cells to obtain a differential cell count (DCC). This manual process has significant limitations, including the analysis of only a small subset of optimal slide areas and nucleated cells, as well as interobserver variability due to differences in cell selection and classification. To address these shortcomings, we developed an automated machine learning-based pipeline for obtaining 11-component DCCs on whole-slide BMAs. This pipeline uses a sequential process of identifying optimal BMA regions with high proportions of marrow nucleated cells, detecting individual cells within these optimal areas, and classifying these cells into 1 of 11 DCC components. Convolutional neural network models were trained on 396,048 BMA region, 28,914 cell boundary, and 1,510,976 cell class images from manual annotations. The resulting automated pipeline produced 11-component DCCs that demonstrated a high statistical and diagnostic concordance with manual DCCs among a heterogeneous group of testing BMA slides with varying pathologies and cellularities. Additionally, we demonstrated that an automated analysis can reduce the intraslide variance in DCCs by analyzing the whole slide and marrow nucleated cells within all optimal regions. Finally, the pipeline outputs of region classification, cell detection, and cell classification can be visualized using whole-slide image analysis software. This study demonstrates the feasibility of a fully automated pipeline for generating DCCs on scanned whole-slide BMA images, with the potential for improving the current standard of practice for utilizing BMA smears in the laboratory analysis of hematologic disorders.