Development and deployment of a histopathology-based deep learning algorithm for patient prescreening in a clinical trial.

Accurate identification of genetic alterations in tumors, such as Fibroblast Growth Factor Receptor, is crucial for treating with targeted therapies; however, molecular testing can delay patient care due to the time and tissue required. Successful development, validation, and deployment of an AI-based, biomarker-detection algorithm could reduce screening cost and accelerate patient recruitment. Here, we develop a deep-learning algorithm using >3000 H&E-stained whole slide images from patients with advanced urothelial cancers, optimized for high sensitivity to avoid ruling out trial-eligible patients. The algorithm is validated on a dataset of 350 patients, achieving an area under the curve of 0.75, specificity of 31.8% at 88.7% sensitivity, and projected 28.7% reduction in molecular testing. We successfully deploy the system in a non-interventional study comprising 89 global study clinical sites and demonstrate its potential to prioritize/deprioritize molecular testing resources and provide substantial cost savings in the drug development and clinical settings.

Distinct hepatic immunological patterns are associated with the progression or inhibition of hepatocellular carcinoma.

To discover distinct immune responses promoting or inhibiting hepatocellular carcinoma (HCC), we perform a three-dimensional analysis of the immune cells, correlating immune cell types, interactions, and changes over time in an animal model displaying gender disparity in nonalcoholic fatty liver disease (NAFLD)-associated HCC. In response to a Western diet (WD), animals mount acute and chronic patterns of inflammatory cytokines, respectively. Tumor progression in males and females is associated with a predominant CD8+ > CD4+, Th1 > Th17 > Th2, NKT > NK, M1 > M2 pattern in the liver. A complete rescue of females from HCC is associated with an equilibrium Th1 = Th17 = Th2, NKT = NK, M1 = M2 pattern, while a partial rescue of males from HCC is associated with an equilibrium CD8+ = CD4+, NKT = NK and a semi-equilibrium Th1 = Th17 > Th2 but a sustained M1 > M2 pattern in the liver. Our data suggest that immunological pattern-recognition can explain immunobiology of HCC and guide immune modulatory interventions for the treatment of HCC in a gender-specific manner.

A Machine Learning Approach to Liver Histological Evaluation Predicts Clinically Significant Portal Hypertension in NASH Cirrhosis.

BACKGROUND AND AIMS: The hepatic venous pressure gradient (HVPG) is the standard for estimating portal pressure but requires expertise for interpretation. We hypothesized that HVPG could be extrapolated from liver histology using a machine learning (ML) algorithm. APPROACH AND RESULTS: Patients with NASH with compensated cirrhosis from a phase 2b trial were included. HVPG and biopsies from baseline and weeks 48 and 96 were reviewed centrally, and biopsies evaluated with a convolutional neural network (PathAI, Boston, MA). Using trichrome-stained biopsies in the training set (n = 130), an ML model was developed to recognize fibrosis patterns associated with HVPG, and the resultant ML HVPG score was validated in a held-out test set (n = 88). Associations between the ML HVPG score with measured HVPG and liver-related events, and performance of the ML HVPG score for clinically significant portal hypertension (CSPH) (HVPG ≥ 10 mm Hg), were determined. The ML-HVPG score was more strongly correlated with HVPG than hepatic collagen by morphometry (ρ = 0.47 vs. ρ = 0.28; P < 0.001). The ML HVPG score differentiated patients with normal (0-5 mm Hg) and elevated (5.5-9.5 mm Hg) HVPG and CSPH (median: 1.51 vs. 1.93 vs. 2.60; all P < 0.05). The areas under receiver operating characteristic curve (AUROCs) (95% CI) of the ML-HVPG score for CSPH were 0.85 (0.80, 0.90) and 0.76 (0.68, 0.85) in the training and test sets, respectively. Discrimination of the ML-HVPG score for CSPH improved with the addition of a ML parameter for nodularity, Enhanced Liver Fibrosis, platelets, aspartate aminotransferase (AST), and bilirubin (AUROC in test set: 0.85; 95% CI: 0.78, 0.92). Although baseline ML-HVPG score was not prognostic, changes were predictive of clinical events (HR: 2.13; 95% CI: 1.26, 3.59) and associated with hemodynamic response and fibrosis improvement. CONCLUSIONS: An ML model based on trichrome-stained liver biopsy slides can predict CSPH in patients with NASH with cirrhosis.

A Machine Learning Approach Enables Quantitative Measurement of Liver Histology and Disease Monitoring in NASH.

BACKGROUND AND AIMS: Manual histological assessment is currently the accepted standard for diagnosing and monitoring disease progression in NASH, but is limited by variability in interpretation and insensitivity to change. Thus, there is a critical need for improved tools to assess liver pathology in order to risk stratify NASH patients and monitor treatment response. APPROACH AND RESULTS: Here, we describe a machine learning (ML)-based approach to liver histology assessment, which accurately characterizes disease severity and heterogeneity, and sensitively quantifies treatment response in NASH. We use samples from three randomized controlled trials to build and then validate deep convolutional neural networks to measure key histological features in NASH, including steatosis, inflammation, hepatocellular ballooning, and fibrosis. The ML-based predictions showed strong correlations with expert pathologists and were prognostic of progression to cirrhosis and liver-related clinical events. We developed a heterogeneity-sensitive metric of fibrosis response, the Deep Learning Treatment Assessment Liver Fibrosis score, which measured antifibrotic treatment effects that went undetected by manual pathological staging and was concordant with histological disease progression. CONCLUSIONS: Our ML method has shown reproducibility and sensitivity and was prognostic for disease progression, demonstrating the power of ML to advance our understanding of disease heterogeneity in NASH, risk stratify affected patients, and facilitate the development of therapies.

Comparing histologic evaluation of prostate tissue using nonlinear microscopy and paraffin H&E: a pilot study.

Rapid histological assessment of large areas of prostate tissue is required for many intraoperative consultation scenarios such as margin evaluation. Nonlinear microscopy (NLM) enables imaging of large (whole mount) specimens without freezing or cryotoming. This study demonstrates rapid histological imaging of unsectioned prostate cancer surgical specimens using nonlinear microscopy and compares features of prostate pathology to standard paraffin embedded H&E histology. Fresh or formalin fixed specimens were stained in 2.5 min with fluorescent nuclear and stromal dyes. Nonlinear microscopy images of unsectioned tissues were generated by nonlinear (two-photon) excitation of the fluorophores, where fluorescence is only emitted from tissue at the microscope focus, avoiding the need for physical sectioning. The images were displayed in real time using a color scale similar to H&E, then tissues were processed for standard paraffin embedded H&E histology. Seventy nonlinear microscopy and corresponding paraffin H&E images of fresh and fixed prostate specimens (15 cancer, 55 benign) from 24 patients were read by genitourinary pathologists to assess if nonlinear microscopy could achieve an equivalent evaluation to paraffin embedded H&E histology. Differences between nonlinear microscopy images and paraffin H&E slides, including cytoplasmic color and stromal density, were observed, however nonlinear microscopy images could be interpreted with minimal training. Nonlinear microscopy enabled visualization of benign, atrophic and hyperplastic glands and stroma, ejaculatory ducts, vasculature and inflammatory changes. Nonlinear microscopy enabled identification of typical and variants of adenocarcinoma, as well as Gleason patterns. Perineural invasion and extraprostatic extension could also be assessed. Nonlinear microscopy images closely resemble paraffin H&E slides and enable rapid assessment of normal prostate architecture, benign conditions, and carcinoma in freshly excised and fixed specimens. Nonlinear microscopy can image large regions of tissue, equivalent to multiple frozen section tissue blocks, within minutes because cryotoming/microtoming are not required, making it a promising technique for intraoperative consultation.

Constant linear velocity spiral scanning for near video rate 4D OCT ophthalmic and surgical imaging with isotropic transverse sampling.

Ultrahigh speed optical coherence tomography (OCT) systems with >100 kHz A-scan rates can generate volumes rapidly with minimal motion artifacts and are well suited for 4D imaging (volumes through time) applications such as intra-operative imaging. In such systems, high OCT data acquisition efficiency (defined as the fraction of usable A-scans generated during the total acquisition time) is desired to maximize the volumetric frame rate and sampling pitch. However, current methods for beam scanning using non-resonant and resonant mirror scanners can result in severe scan distortion and transverse oversampling as well as require acquisition dead times, which limit the acquisition efficiency and performance of ultrahigh speed 4D OCT. We introduce constant linear velocity spiral scanning (CLV-SC) as a novel beam scanning method to maximize the data acquisition efficiency of ultrahigh speed 4D OCT systems. We demonstrate that CLV-SC does not require acquisition dead times and achieves more uniform transverse sampling compared to raster scanning. To assess its clinical utility, we implement CLV-SC with a 400 kHz OCT system and image the anterior eye and retina of healthy adults at up to 10 volumes per second with isotropic transverse sampling, allowing B-scans with equal sampling pitch to be extracted from arbitrary locations within a single volume. The feasibility of CLV-SC for intra-operative imaging is also demonstrated using a 800 kHz OCT system to image simulated retinal surgery at 15 volumes per second with isotropic transverse sampling, resulting in high quality volume renders that enable clear visualization of surgical instruments and manipulation of tissue.

Real-time visualization and interaction with static and live optical coherence tomography volumes in immersive virtual reality.

Virtual reality (VR) head-mounted displays are an attractive technology for viewing intrasurgical optical coherence tomography (OCT) volumes because they liberate surgeons from microscope oculars. We demonstrate real-time, interactive viewing of OCT volumes in a commercial HTC Vive immersive VR system using previously reported ray casting techniques. Furthermore, we show interactive manipulation and sectioning of volumes using handheld controllers and guidance of mock surgical procedures in porcine eyes exclusively within VR. To the best of our knowledge, we report the first immersive VR-OCT viewer with stereo ray casting volumetric renders, arbitrary sectioning planes, and live acquisition support. We believe VR-OCT volume displays will advance ophthalmic surgery towards VR-integrated surgery.

Real-Time Volumetric Imaging of Vitreoretinal Surgery with a Prototype Microscope-Integrated Swept-Source OCT Device.

PURPOSE: To evaluate the images produced in an initial surgical series of intraoperative near-real-time volumetric swept-source (SS) OCT imaging. DESIGN: Prospective translational study. PARTICIPANTS: Forty-one consecutive adult patients undergoing vitreoretinal surgery between July 22, 2014, and July 1, 2015, at the Duke University Eye Center who agreed to participate. METHODS: A novel microscope-integrated SS-OCT prototype captured volumetric renderings of imaging of macular surgery in near-real-time and showed them to the surgeon via a heads-up display through the microscope oculars. Then the images were analyzed formally after surgery. MAIN OUTCOME MEASURES: Image quality, successful capture of surgical instruments, maneuvers and associated retinal deformation volumetrically over time, and qualitative image analysis. RESULTS: Volumetric SS-OCT images were graded as acceptable in 92% of patients. Volumetric imaging of scraping and peeling procedures was achieved in 75% and 78% of patients in whom it was performed, respectively. Imaging provided the surgeon with near-real-time volumetric visualization of the position of the instrument relative to the retinal surface, flap initiation, flap removal, and retinal deformation during instrumentation via a heads-up display. CONCLUSIONS: This volumetric, microscope-integrated SS-OCT prototype seems to provide high-detail, near-real-time volumetric imaging of delicate maneuvers during macular surgery.

Review of intraoperative optical coherence tomography: technology and applications [Invited].

During microsurgery, en face imaging of the surgical field through the operating microscope limits the surgeon's depth perception and visualization of instruments and sub-surface anatomy. Surgical procedures outside microsurgery, such as breast tumor resections, may also benefit from visualization of the sub-surface tissue structures. The widespread clinical adoption of optical coherence tomography (OCT) in ophthalmology and its growing prominence in other fields, such as cancer imaging, has motivated the development of intraoperative OCT for real-time tomographic visualization of surgical interventions. This article reviews key technological developments in intraoperative OCT and their applications in human surgery. We focus on handheld OCT probes, microscope-integrated OCT systems, and OCT-guided laser treatment platforms designed for intraoperative use. Moreover, we discuss intraoperative OCT adjuncts and processing techniques currently under development to optimize the surgical feedback derivable from OCT data. Lastly, we survey salient clinical studies of intraoperative OCT for human surgery.

Microscope-Integrated Optical Coherence Tomography Angiography in the Operating Room in Young Children With Retinal Vascular Disease.

Importance: Intraoperative optical coherence tomography (OCT) has gained traction as an important adjunct for clinical decision making during vitreoretinal surgery, and OCT angiography (OCTA) has provided novel insights in clinical evaluation of retinal diseases. To date, these two technologies have not been applied in combination to evaluate retinal vascular disease in the operating suite. Objective: To conduct microscope-integrated, swept-source OCTA (MIOCTA) in children with retinal vascular disease. Design, Setting, and Participants: In this case report analysis, OCT imaging in pediatric patients, MIOCTA images were obtained during examination under anesthesia from a young boy with a history of idiopathic vitreous hemorrhage and a female infant with familial exudative vitreoretinopathy. Main Outcomes and Measures: Side-by-side comparison of research MIOCT angiograms and clinically indicated fluorescein angiograms. Results: In 2 young children with retinal vascular disease, the MIOCTA images showed more detailed vascular patterns than were visible on the fluorescein angiograms although within a more posterior field of view. The MIOCTA system allowed visualization of small pathological retinal vessels in the retinal periphery that were obscured in the fluorescein angiograms by fluorescein staining from underlying, preexisting laser scars. Conclusions and Relevance: This is the first report to date of the use of MIOCTA in the operating room for young children with retinal vascular disease. Further optimization of this system may allow noninvasive detailed evaluation of retinal vasculature during surgical procedures and in patients who could not cooperate with in-office examinations.

Four-dimensional microscope- integrated optical coherence tomography to enhance visualization in glaucoma surgeries.

We report the first use of swept-source microscope-integrated optical coherence tomography (SS-MIOCT) capable of live four-dimensional (4D) (three-dimensional across time) imaging intraoperatively to directly visualize tube shunt placement and trabeculectomy surgeries in two patients with severe open-angle glaucoma and elevated intraocular pressure (IOP) that was not adequately managed by medical intervention or prior surgery. We performed tube shunt placement and trabeculectomy surgery and used SS-MIOCT to visualize and record surgical steps that benefitted from the enhanced visualization. In the case of tube shunt placement, SS-MIOCT successfully visualized the scleral tunneling, tube shunt positioning in the anterior chamber, and tube shunt suturing. For the trabeculectomy, SS-MIOCT successfully visualized the scleral flap creation, sclerotomy, and iridectomy. Postoperatively, both patients did well, with IOPs decreasing to the target goal. We found the benefit of SS-MIOCT was greatest in surgical steps requiring depth-based assessments. This technology has the potential to improve clinical outcomes.

Four-dimensional Microscope-Integrated Optical Coherence Tomography to Visualize Suture Depth in Strabismus Surgery.

The authors report the use of swept-source microscope-integrated optical coherence tomography (SS-MIOCT), capable of live four-dimensional (three-dimensional across time) intraoperative imaging, to directly visualize suture depth during lateral rectus resection. Key surgical steps visualized in this report included needle depth during partial and full-thickness muscle passes along with scleral passes. [J Pediatr Ophthalmol Strabismus. 2017;54:e1-e5.].

Impact of Microscope-Integrated OCT on Ophthalmology Resident Performance of Anterior Segment Surgical Maneuvers in Model Eyes.

PURPOSE: The integration of swept-source optical coherence tomography (SS-OCT) into the operating microscope enables real-time, tissue-level three-dimensional (3D) imaging to aid in ophthalmic microsurgery. In this prospective randomized controlled study, we evaluated the impact of SS microscope-integrated OCT (MI-OCT) on ophthalmology residents' performance of ophthalmic microsurgical maneuvers. METHODS: Fourteen ophthalmology residents from a single institution were stratified by year of training and randomized to perform four anterior segment surgical maneuvers on porcine eyes with (MI-OCT+) or without (MI-OCT-) direct intraoperative OCT guidance. Subsequently, both groups repeated the same maneuvers without MI-OCT feedback to test whether initial MI-OCT experience affected subsequent surgical performance. Finally, the MI-OCT- group was crossed over and allowed to repeat the same maneuvers with direct MI-OCT guidance. Each resident completed a survey at the completion of the study. RESULTS: With direct MI-OCT feedback, residents demonstrated enhanced performance in depth-based anterior segment maneuvers (corneal suture passes at 50% and 90% depth and corneal laceration repair) compared with the residents operating without MI-OCT. Microscope-integrated OCT+ residents continued to outperform the controls when both groups subsequently operated without MI-OCT. For clear corneal wound geometry, there was no statistically significant effect of MI-OCT as applied in this study. Overall, the resident surgeons rated their subjective experience of using MI-OCT very favorably. CONCLUSIONS: Microscope-integrated OCT feedback enhances performance of ophthalmology residents in select anterior segment surgical maneuvers. Microscope-integrated OCT represents a valuable tool in the surgical education of ophthalmology residents.

Optical Coherence Tomography for Retinal Surgery: Perioperative Analysis to Real-Time Four-Dimensional Image-Guided Surgery.

Magnification of the surgical field using the operating microscope facilitated profound innovations in retinal surgery in the 1970s, such as pars plana vitrectomy. Although surgical instrumentation and illumination techniques are continually developing, the operating microscope for vitreoretinal procedures has remained essentially unchanged and currently limits the surgeon's depth perception and assessment of subtle microanatomy. Optical coherence tomography (OCT) has revolutionized clinical management of retinal pathology, and its introduction into the operating suite may have a similar impact on surgical visualization and treatment. In this article, we review the evolution of OCT for retinal surgery, from perioperative analysis to live volumetric (four-dimensional, 4D) image-guided surgery. We begin by briefly addressing the benefits and limitations of the operating microscope, the progression of OCT technology, and OCT applications in clinical/perioperative retinal imaging. Next, we review intraoperative OCT (iOCT) applications using handheld probes during surgical pauses, two-dimensional (2D) microscope-integrated OCT (MIOCT) of live surgery, and volumetric MIOCT of live surgery. The iOCT discussion focuses on technological advancements, applications during human retinal surgery, translational difficulties and limitations, and future directions.

Novel microscope-integrated stereoscopic heads-up display for intrasurgical optical coherence tomography.

Intra-operative optical coherence tomography (OCT) requires a display technology which allows surgeons to visualize OCT data without disrupting surgery. Previous research and commercial intrasurgical OCT systems have integrated heads-up display (HUD) systems into surgical microscopes to provide monoscopic viewing of OCT data through one microscope ocular. To take full advantage of our previously reported real-time volumetric microscope-integrated OCT (4D MIOCT) system, we describe a stereoscopic HUD which projects a stereo pair of OCT volume renderings into both oculars simultaneously. The stereoscopic HUD uses a novel optical design employing spatial multiplexing to project dual OCT volume renderings utilizing a single micro-display. The optical performance of the surgical microscope with the HUD was quantitatively characterized and the addition of the HUD was found not to substantially effect the resolution, field of view, or pincushion distortion of the operating microscope. In a pilot depth perception subject study, five ophthalmic surgeons completed a pre-set dexterity task with 50.0% (SD = 37.3%) higher success rate and in 35.0% (SD = 24.8%) less time on average with stereoscopic OCT vision compared to monoscopic OCT vision. Preliminary experience using the HUD in 40 vitreo-retinal human surgeries by five ophthalmic surgeons is reported, in which all surgeons reported that the HUD did not alter their normal view of surgery and that live surgical maneuvers were readily visible in displayed stereoscopic OCT volumes.

Intrasurgical Human Retinal Imaging With Manual Instrument Tracking Using a Microscope-Integrated Spectral-Domain Optical Coherence Tomography Device.

PURPOSE: To characterize the first in-human intraoperative imaging using a custom prototype spectral-domain microscope-integrated optical coherence tomography (MIOCT) device during vitreoretinal surgery with instruments in the eye. METHODS: Under institutional review board approval for a prospective intraoperative study, MIOCT images were obtained at surgical pauses with instruments held static in the vitreous cavity and then concurrently with surgical maneuvers. Postoperatively, MIOCT images obtained at surgical pauses were compared with images obtained with a high-resolution handheld spectral-domain OCT (HHOCT) system with objective endpoints, including acquisition of images acceptable for analysis and identification of predefined macular morphologic or pathologic features. RESULTS: Human MIOCT images were successfully obtained before incision and during pauses in surgical maneuvers. MIOCT imaging confirmed preoperative diagnoses, such as epiretinal membrane, full-thickness macular hole, and vitreomacular traction and demonstrated successful achievement of surgical goals. MIOCT and HHOCT images obtained at surgical pauses in two cohorts of five patients were comparable with greater than or equal to 80% correlation in 80% of patients. Real-time video-imaging concurrent with surgical manipulations enabled, for the first time using this device, visualization of dynamic instrument-retina interaction with targeted OCT tracking. CONCLUSION: MIOCT is successful for imaging at surgical pauses and for real-time image guidance with implementation of targeted OCT tracking. Even faster acquisition speeds are currently being developed with incorporation of a swept-source MIOCT engine. Further refinements and investigations will be directed toward continued integration for real-time volumetric imaging of surgical maneuvers. TRANSLATIONAL RELEVANCE: Ongoing development of seamless MIOCT systems will likely transform surgical visualization, approaches, and decision-making.