Accelerating pharmaceutical R&D with a user-friendly AI system for histopathology image analysis.

A system for analysis of histopathology data within a pharmaceutical R&D environment has been developed with the intention of enabling interdisciplinary collaboration. State-of-the-art AI tools have been deployed as easy-to-use self-service modules within an open-source whole slide image viewing platform, so that non-data scientist users (e.g., clinicians) can utilize and evaluate pre-trained algorithms and retrieve quantitative results. The outputs of analysis are automatically cataloged in the database to track data provenance and can be viewed interactively on the slide as annotations or heatmaps. Commonly used models for analysis of whole slide images including segmentation, extraction of hand-engineered features for segmented regions, and slide-level classification using multi-instance learning are included and new models can be added as needed. The source code that supports running inference with these models internally is backed up by a robust CI/CD pipeline to ensure model versioning, robust testing, and seamless deployment of the latest models. Examples of the use of this system in a pharmaceutical development workflow include glomeruli segmentation, enumeration of podocyte count from WT-1 immuno-histochemistry, measurement of beta-1 integrin target engagement from immunofluorescence, digital glomerular phenotyping from periodic acid-Schiff histology, PD-L1 score prediction using multi-instance learning, and the deployment of the open-source Segment Anything model to speed up annotation.

AI-based histologic scoring enables automated and reproducible assessment of enrollment criteria and endpoints in NASH clinical trials.

Clinical trials in nonalcoholic steatohepatitis (NASH) require histologic scoring for assessment of inclusion criteria and endpoints. However, guidelines for scoring key features have led to variability in interpretation, impacting clinical trial outcomes. We developed an artificial intelligence (AI)-based measurement (AIM) tool for scoring NASH histology (AIM-NASH). AIM-NASH predictions for NASH Clinical Research Network (CRN) grades of necroinflammation and stages of fibrosis aligned with expert consensus scores and were reproducible. Continuous scores produced by AIM-NASH for key histological features of NASH correlated with mean pathologist scores and with noninvasive biomarkers and strongly predicted patient outcomes. In a retrospective analysis of the ATLAS trial, previously unmet pathological endpoints were met when scored by the AIM-NASH algorithm alone. Overall, these results suggest that AIM-NASH may assist pathologists in histologic review of NASH clinical trials, reducing inter-rater variability on trial outcomes and offering a more sensitive and reproducible measure of patient therapeutic response.

Integration of deep learning-based histopathology and transcriptomics reveals key genes associated with fibrogenesis in patients with advanced NASH.

Nonalcoholic steatohepatitis (NASH) is the most common chronic liver disease globally and a leading cause for liver transplantation in the US. Its pathogenesis remains imprecisely defined. We combined two high-resolution modalities to tissue samples from NASH clinical trials, machine learning (ML)-based quantification of histological features and transcriptomics, to identify genes that are associated with disease progression and clinical events. A histopathology-driven 5-gene expression signature predicted disease progression and clinical events in patients with NASH with F3 (pre-cirrhotic) and F4 (cirrhotic) fibrosis. Notably, the Notch signaling pathway and genes implicated in liver-related diseases were enriched in this expression signature. In a validation cohort where pharmacologic intervention improved disease histology, multiple Notch signaling components were suppressed.

Distinct hepatic immunological patterns are associated with the progression or inhibition of hepatocellular carcinoma.

To discover distinct immune responses promoting or inhibiting hepatocellular carcinoma (HCC), we perform a three-dimensional analysis of the immune cells, correlating immune cell types, interactions, and changes over time in an animal model displaying gender disparity in nonalcoholic fatty liver disease (NAFLD)-associated HCC. In response to a Western diet (WD), animals mount acute and chronic patterns of inflammatory cytokines, respectively. Tumor progression in males and females is associated with a predominant CD8+ > CD4+, Th1 > Th17 > Th2, NKT > NK, M1 > M2 pattern in the liver. A complete rescue of females from HCC is associated with an equilibrium Th1 = Th17 = Th2, NKT = NK, M1 = M2 pattern, while a partial rescue of males from HCC is associated with an equilibrium CD8+ = CD4+, NKT = NK and a semi-equilibrium Th1 = Th17 > Th2 but a sustained M1 > M2 pattern in the liver. Our data suggest that immunological pattern-recognition can explain immunobiology of HCC and guide immune modulatory interventions for the treatment of HCC in a gender-specific manner.

A Machine Learning Approach to Liver Histological Evaluation Predicts Clinically Significant Portal Hypertension in NASH Cirrhosis.

BACKGROUND AND AIMS: The hepatic venous pressure gradient (HVPG) is the standard for estimating portal pressure but requires expertise for interpretation. We hypothesized that HVPG could be extrapolated from liver histology using a machine learning (ML) algorithm. APPROACH AND RESULTS: Patients with NASH with compensated cirrhosis from a phase 2b trial were included. HVPG and biopsies from baseline and weeks 48 and 96 were reviewed centrally, and biopsies evaluated with a convolutional neural network (PathAI, Boston, MA). Using trichrome-stained biopsies in the training set (n = 130), an ML model was developed to recognize fibrosis patterns associated with HVPG, and the resultant ML HVPG score was validated in a held-out test set (n = 88). Associations between the ML HVPG score with measured HVPG and liver-related events, and performance of the ML HVPG score for clinically significant portal hypertension (CSPH) (HVPG ≥ 10 mm Hg), were determined. The ML-HVPG score was more strongly correlated with HVPG than hepatic collagen by morphometry (ρ = 0.47 vs. ρ = 0.28; P < 0.001). The ML HVPG score differentiated patients with normal (0-5 mm Hg) and elevated (5.5-9.5 mm Hg) HVPG and CSPH (median: 1.51 vs. 1.93 vs. 2.60; all P < 0.05). The areas under receiver operating characteristic curve (AUROCs) (95% CI) of the ML-HVPG score for CSPH were 0.85 (0.80, 0.90) and 0.76 (0.68, 0.85) in the training and test sets, respectively. Discrimination of the ML-HVPG score for CSPH improved with the addition of a ML parameter for nodularity, Enhanced Liver Fibrosis, platelets, aspartate aminotransferase (AST), and bilirubin (AUROC in test set: 0.85; 95% CI: 0.78, 0.92). Although baseline ML-HVPG score was not prognostic, changes were predictive of clinical events (HR: 2.13; 95% CI: 1.26, 3.59) and associated with hemodynamic response and fibrosis improvement. CONCLUSIONS: An ML model based on trichrome-stained liver biopsy slides can predict CSPH in patients with NASH with cirrhosis.

A Machine Learning Approach Enables Quantitative Measurement of Liver Histology and Disease Monitoring in NASH.

BACKGROUND AND AIMS: Manual histological assessment is currently the accepted standard for diagnosing and monitoring disease progression in NASH, but is limited by variability in interpretation and insensitivity to change. Thus, there is a critical need for improved tools to assess liver pathology in order to risk stratify NASH patients and monitor treatment response. APPROACH AND RESULTS: Here, we describe a machine learning (ML)-based approach to liver histology assessment, which accurately characterizes disease severity and heterogeneity, and sensitively quantifies treatment response in NASH. We use samples from three randomized controlled trials to build and then validate deep convolutional neural networks to measure key histological features in NASH, including steatosis, inflammation, hepatocellular ballooning, and fibrosis. The ML-based predictions showed strong correlations with expert pathologists and were prognostic of progression to cirrhosis and liver-related clinical events. We developed a heterogeneity-sensitive metric of fibrosis response, the Deep Learning Treatment Assessment Liver Fibrosis score, which measured antifibrotic treatment effects that went undetected by manual pathological staging and was concordant with histological disease progression. CONCLUSIONS: Our ML method has shown reproducibility and sensitivity and was prognostic for disease progression, demonstrating the power of ML to advance our understanding of disease heterogeneity in NASH, risk stratify affected patients, and facilitate the development of therapies.

QUANTIFICATION OF RETINAL CAPILLARY NONPERFUSION IN DIABETICS USING WIDE-FIELD OPTICAL COHERENCE TOMOGRAPHY ANGIOGRAPHY.

PURPOSE: To combine advances in high-speed, wide-field optical coherence tomography angiography (OCTA) with image processing methods for semiautomatic quantitative analysis of capillary nonperfusion in patients with diabetic retinopathy (DR). METHODS: Sixty-eight diabetic patients (73 eyes), either without retinopathy or with different degrees of retinopathy, were prospectively recruited for volumetric swept-source OCTA imaging using 12 mm × 12 mm fields centered at the fovea. A custom, semiautomatic software algorithm was used to quantify areas of capillary nonperfusion. RESULTS: The mean percentage of nonperfused area was 0.1% (95% confidence interval: 0.0-0.4) in the eyes without DR; 2.1% (95% confidence interval: 1.2-3.7) in the nonproliferative DR eyes (mild, moderate, and severe), and 8.5% (95% confidence interval: 5.0-14.3) in the proliferative DR eyes. The percentage of nonperfused area increased in a statistically significant manner from eyes without DR, to eyes with nonproliferative DR, to eyes with proliferative DR. CONCLUSION: Capillary nonperfusion area in the posterior retina increases with increasing DR severity as measured by swept-source OCTA. Quantitative analysis of retinal nonperfusion on wide-field OCTA may be useful for early detection and monitoring of disease in patients with diabetes and DR.

Controlling for Artifacts in Widefield Optical Coherence Tomography Angiography Measurements of Non-Perfusion Area.

The recent clinical adoption of optical coherence tomography (OCT) angiography (OCTA) has enabled non-invasive, volumetric visualization of ocular vasculature at micron-scale resolutions. Initially limited to 3 mm × 3 mm and 6 mm × 6 mm fields-of-view (FOV), commercial OCTA systems now offer 12 mm × 12 mm, or larger, imaging fields. While larger FOVs promise a more complete visualization of retinal disease, they also introduce new challenges to the accurate and reliable interpretation of OCTA data. In particular, because of vignetting, wide-field imaging increases occurrence of low-OCT-signal artifacts, which leads to thresholding and/or segmentation artifacts, complicating OCTA analysis. This study presents theoretical and case-based descriptions of the causes and effects of low-OCT-signal artifacts. Through these descriptions, we demonstrate that OCTA data interpretation can be ambiguous if performed without consulting corresponding OCT data. Furthermore, using wide-field non-perfusion analysis in diabetic retinopathy as a model widefield OCTA usage-case, we show how qualitative and quantitative analysis can be confounded by low-OCT-signal artifacts. Based on these results, we suggest methods and best-practices for preventing and managing low-OCT-signal artifacts, thereby reducing errors in OCTA quantitative analysis of non-perfusion and improving reproducibility. These methods promise to be especially important for longitudinal studies detecting progression and response to therapy.

Comparing histologic evaluation of prostate tissue using nonlinear microscopy and paraffin H&E: a pilot study.

Rapid histological assessment of large areas of prostate tissue is required for many intraoperative consultation scenarios such as margin evaluation. Nonlinear microscopy (NLM) enables imaging of large (whole mount) specimens without freezing or cryotoming. This study demonstrates rapid histological imaging of unsectioned prostate cancer surgical specimens using nonlinear microscopy and compares features of prostate pathology to standard paraffin embedded H&E histology. Fresh or formalin fixed specimens were stained in 2.5 min with fluorescent nuclear and stromal dyes. Nonlinear microscopy images of unsectioned tissues were generated by nonlinear (two-photon) excitation of the fluorophores, where fluorescence is only emitted from tissue at the microscope focus, avoiding the need for physical sectioning. The images were displayed in real time using a color scale similar to H&E, then tissues were processed for standard paraffin embedded H&E histology. Seventy nonlinear microscopy and corresponding paraffin H&E images of fresh and fixed prostate specimens (15 cancer, 55 benign) from 24 patients were read by genitourinary pathologists to assess if nonlinear microscopy could achieve an equivalent evaluation to paraffin embedded H&E histology. Differences between nonlinear microscopy images and paraffin H&E slides, including cytoplasmic color and stromal density, were observed, however nonlinear microscopy images could be interpreted with minimal training. Nonlinear microscopy enabled visualization of benign, atrophic and hyperplastic glands and stroma, ejaculatory ducts, vasculature and inflammatory changes. Nonlinear microscopy enabled identification of typical and variants of adenocarcinoma, as well as Gleason patterns. Perineural invasion and extraprostatic extension could also be assessed. Nonlinear microscopy images closely resemble paraffin H&E slides and enable rapid assessment of normal prostate architecture, benign conditions, and carcinoma in freshly excised and fixed specimens. Nonlinear microscopy can image large regions of tissue, equivalent to multiple frozen section tissue blocks, within minutes because cryotoming/microtoming are not required, making it a promising technique for intraoperative consultation.

Constant linear velocity spiral scanning for near video rate 4D OCT ophthalmic and surgical imaging with isotropic transverse sampling.

Ultrahigh speed optical coherence tomography (OCT) systems with >100 kHz A-scan rates can generate volumes rapidly with minimal motion artifacts and are well suited for 4D imaging (volumes through time) applications such as intra-operative imaging. In such systems, high OCT data acquisition efficiency (defined as the fraction of usable A-scans generated during the total acquisition time) is desired to maximize the volumetric frame rate and sampling pitch. However, current methods for beam scanning using non-resonant and resonant mirror scanners can result in severe scan distortion and transverse oversampling as well as require acquisition dead times, which limit the acquisition efficiency and performance of ultrahigh speed 4D OCT. We introduce constant linear velocity spiral scanning (CLV-SC) as a novel beam scanning method to maximize the data acquisition efficiency of ultrahigh speed 4D OCT systems. We demonstrate that CLV-SC does not require acquisition dead times and achieves more uniform transverse sampling compared to raster scanning. To assess its clinical utility, we implement CLV-SC with a 400 kHz OCT system and image the anterior eye and retina of healthy adults at up to 10 volumes per second with isotropic transverse sampling, allowing B-scans with equal sampling pitch to be extracted from arbitrary locations within a single volume. The feasibility of CLV-SC for intra-operative imaging is also demonstrated using a 800 kHz OCT system to image simulated retinal surgery at 15 volumes per second with isotropic transverse sampling, resulting in high quality volume renders that enable clear visualization of surgical instruments and manipulation of tissue.

Real-time visualization and interaction with static and live optical coherence tomography volumes in immersive virtual reality.

Virtual reality (VR) head-mounted displays are an attractive technology for viewing intrasurgical optical coherence tomography (OCT) volumes because they liberate surgeons from microscope oculars. We demonstrate real-time, interactive viewing of OCT volumes in a commercial HTC Vive immersive VR system using previously reported ray casting techniques. Furthermore, we show interactive manipulation and sectioning of volumes using handheld controllers and guidance of mock surgical procedures in porcine eyes exclusively within VR. To the best of our knowledge, we report the first immersive VR-OCT viewer with stereo ray casting volumetric renders, arbitrary sectioning planes, and live acquisition support. We believe VR-OCT volume displays will advance ophthalmic surgery towards VR-integrated surgery.

Volumetric Measurement of Subretinal Blebs Using Microscope-Integrated Optical Coherence Tomography.

PURPOSE: We advance studies of subretinal treatments by developing a microscope-integrated optical coherence tomography (MIOCT) image-based method for measuring the volume of therapeutics delivered into the subretinal space. METHODS: A MIOCT image-based volume measurement method was developed and assessed for accuracy and reproducibility by imaging an object of known size in model eyes. This method then was applied to subretinal blebs created by injection of diluted triamcinolone. Bleb volumes obtained from MIOCT were compared to the intended injection volume and the surgeon's estimation of leakage. RESULTS: Validation of the image-based volume measurement method showed accuracy to ±1.0 µL (6.0% of measured volume) with no statistically significant variation under different imaging settings. When this method was applied to subretinal blebs, four of 11 blebs without surgeon-observed leakage yielded a mean volume of 32 ± 12.5 µL, in contrast to the intended 50 µL volume injected from the delivery device. This constituted a mean difference of -18 µL (mean percent error, 36 ± 25%). For all 11 blebs, the surgeon's estimations of leakage were significantly different from and showed no correlation with the volume loss based on image-based volume measurements (P < 0.001, paired t-test; intraclass correlation = 0). CONCLUSIONS: We validated an accurate and reproducible method for measuring subretinal volumes using MIOCT. Use of this method revealed that the intended volume might not be delivered into the subretinal space. MIOCT can allow for accurate assessment of subretinal dose delivered, which may have therapeutic implications in evaluating the efficacy and toxicity of subretinal therapies. TRANSLATIONAL RELEVANCE: Use of MIOCT can provide feedback on the accuracy of subretinal injection volumes delivered.

Real-Time Volumetric Imaging of Vitreoretinal Surgery with a Prototype Microscope-Integrated Swept-Source OCT Device.

PURPOSE: To evaluate the images produced in an initial surgical series of intraoperative near-real-time volumetric swept-source (SS) OCT imaging. DESIGN: Prospective translational study. PARTICIPANTS: Forty-one consecutive adult patients undergoing vitreoretinal surgery between July 22, 2014, and July 1, 2015, at the Duke University Eye Center who agreed to participate. METHODS: A novel microscope-integrated SS-OCT prototype captured volumetric renderings of imaging of macular surgery in near-real-time and showed them to the surgeon via a heads-up display through the microscope oculars. Then the images were analyzed formally after surgery. MAIN OUTCOME MEASURES: Image quality, successful capture of surgical instruments, maneuvers and associated retinal deformation volumetrically over time, and qualitative image analysis. RESULTS: Volumetric SS-OCT images were graded as acceptable in 92% of patients. Volumetric imaging of scraping and peeling procedures was achieved in 75% and 78% of patients in whom it was performed, respectively. Imaging provided the surgeon with near-real-time volumetric visualization of the position of the instrument relative to the retinal surface, flap initiation, flap removal, and retinal deformation during instrumentation via a heads-up display. CONCLUSIONS: This volumetric, microscope-integrated SS-OCT prototype seems to provide high-detail, near-real-time volumetric imaging of delicate maneuvers during macular surgery.

Characterization of Long Working Distance Optical Coherence Tomography for Imaging of Pediatric Retinal Pathology.

PURPOSE: We determined the feasibility of fovea and optic nerve head imaging with a long working distance (LWD) swept source optical coherence tomography (OCT) prototype in adults, teenagers, and young children. METHODS: A prototype swept source OCT system with a LWD (defined as distance from the last optical element of the imaging system to the eye) of 350 mm with custom fixation targets was developed to facilitate imaging of children. Imaging was performed in 49 participants from three age groups: 26 adults, 16 children 13 to 18 years old (teenagers), and seven children under 6 years old (young children) under an approved institutional review board protocol. The imaging goal was to acquire high quality scans of the fovea and optic nerve in each eye in the shortest time possible. OCT B-scans and volumes of the fovea and optic nerve head of each eligible eye were captured and graded based on four categories (lateral and axial centration, contrast, and resolution) and on ability to determine presence or absence of pathology. RESULTS: LWD-OCT imaging was successful in 88 of 94 eligible eyes, including seven of 10 eyes of young children. Of the successfully acquired OCT images, 83% of B-scan and volumetric images, including 86% from young children, were graded as high-quality scans. Pathology was observed in high-quality OCT images. CONCLUSIONS: The prototype LWD-OCT system achieved high quality retinal imaging of adults, teenagers, and some young children with and without pathology with reasonable alignment time. TRANSLATIONAL RELEVANCE: The LWD-OCT system can facilitate imaging in children.

Intraocular Pressure and Big Bubble Diameter in Deep Anterior Lamellar Keratoplasty: An Ex-Vivo Microscope-Integrated OCT With Heads-Up Display Study.

PURPOSE: To investigate the relationship between intraocular pressure (IOP) and big bubble (BB) formation in a model of deep anterior lamellar keratoplasty (DALK). DESIGN: Ex-vivo. METHODS: Corneoscleral buttons from human donors were loaded onto an artificial anterior chamber connected to a column of balanced salt solution. A surgeon-in-training learned to perform DALK via the BB technique using swept-source microscope-integrated optical coherence tomography (SS-MIOCT) with heads-up display (HUD). DALK procedures were performed at 6 different IOPs (5, 10, 15, 20, 30, or 40 mm Hg; n = 6 per group) in a randomized fashion, with the surgeon-in-training masked to the pressure and guided by SS-MIOCT with HUD. For a subset of corneas within each pressure group, DALK was performed on matching donor tissue at a control IOP. BB diameter was recorded, and a diameter exceeding the trephine diameter was considered optimal. RESULTS: Wilcoxon rank sum test showed a difference in BB diameter among the different pressure groups (mean ± SD of 7.75 ± 1.60, 8.33 ± 1.99, 10.9 ± 0.92, 9.08 ± 1.07, 6.67 ± 3.33, and 3.42 ± 3.77 mm in the 5, 10, 15, 20, 30, and 40 mm Hg groups, respectively; P = 0.0014). Per Tukey test, this difference was attributable to comparisons between the 40 mm Hg group and the 5, 10, 15, or 20 mm Hg groups (P = 0.04, 0.02, 0.0001, 0.004, respectively). CONCLUSIONS: In this ex-vivo model of DALK, the BB technique guided by SS-MIOCT with HUD yielded bubbles of optimal diameters only at physiologic pressures (10‒20 mm Hg). Extremely high IOP (40 mm Hg) resulted in BBs of significantly smaller diameter than BBs obtained at physiologic and low (5 mm Hg) IOPs.

Review of intraoperative optical coherence tomography: technology and applications [Invited].

During microsurgery, en face imaging of the surgical field through the operating microscope limits the surgeon's depth perception and visualization of instruments and sub-surface anatomy. Surgical procedures outside microsurgery, such as breast tumor resections, may also benefit from visualization of the sub-surface tissue structures. The widespread clinical adoption of optical coherence tomography (OCT) in ophthalmology and its growing prominence in other fields, such as cancer imaging, has motivated the development of intraoperative OCT for real-time tomographic visualization of surgical interventions. This article reviews key technological developments in intraoperative OCT and their applications in human surgery. We focus on handheld OCT probes, microscope-integrated OCT systems, and OCT-guided laser treatment platforms designed for intraoperative use. Moreover, we discuss intraoperative OCT adjuncts and processing techniques currently under development to optimize the surgical feedback derivable from OCT data. Lastly, we survey salient clinical studies of intraoperative OCT for human surgery.

Microscope-Integrated Optical Coherence Tomography Angiography in the Operating Room in Young Children With Retinal Vascular Disease.

Importance: Intraoperative optical coherence tomography (OCT) has gained traction as an important adjunct for clinical decision making during vitreoretinal surgery, and OCT angiography (OCTA) has provided novel insights in clinical evaluation of retinal diseases. To date, these two technologies have not been applied in combination to evaluate retinal vascular disease in the operating suite. Objective: To conduct microscope-integrated, swept-source OCTA (MIOCTA) in children with retinal vascular disease. Design, Setting, and Participants: In this case report analysis, OCT imaging in pediatric patients, MIOCTA images were obtained during examination under anesthesia from a young boy with a history of idiopathic vitreous hemorrhage and a female infant with familial exudative vitreoretinopathy. Main Outcomes and Measures: Side-by-side comparison of research MIOCT angiograms and clinically indicated fluorescein angiograms. Results: In 2 young children with retinal vascular disease, the MIOCTA images showed more detailed vascular patterns than were visible on the fluorescein angiograms although within a more posterior field of view. The MIOCTA system allowed visualization of small pathological retinal vessels in the retinal periphery that were obscured in the fluorescein angiograms by fluorescein staining from underlying, preexisting laser scars. Conclusions and Relevance: This is the first report to date of the use of MIOCTA in the operating room for young children with retinal vascular disease. Further optimization of this system may allow noninvasive detailed evaluation of retinal vasculature during surgical procedures and in patients who could not cooperate with in-office examinations.

Four-dimensional microscope- integrated optical coherence tomography to enhance visualization in glaucoma surgeries.

We report the first use of swept-source microscope-integrated optical coherence tomography (SS-MIOCT) capable of live four-dimensional (4D) (three-dimensional across time) imaging intraoperatively to directly visualize tube shunt placement and trabeculectomy surgeries in two patients with severe open-angle glaucoma and elevated intraocular pressure (IOP) that was not adequately managed by medical intervention or prior surgery. We performed tube shunt placement and trabeculectomy surgery and used SS-MIOCT to visualize and record surgical steps that benefitted from the enhanced visualization. In the case of tube shunt placement, SS-MIOCT successfully visualized the scleral tunneling, tube shunt positioning in the anterior chamber, and tube shunt suturing. For the trabeculectomy, SS-MIOCT successfully visualized the scleral flap creation, sclerotomy, and iridectomy. Postoperatively, both patients did well, with IOPs decreasing to the target goal. We found the benefit of SS-MIOCT was greatest in surgical steps requiring depth-based assessments. This technology has the potential to improve clinical outcomes.