Leveraging mid-infrared spectroscopic imaging and deep learning for tissue subtype classification in ovarian cancer.

Mid-infrared spectroscopic imaging (MIRSI) is an emerging class of label-free techniques being leveraged for digital histopathology. Modern histopathologic identification of ovarian cancer involves tissue staining followed by morphological pattern recognition. This process is time-consuming and subjective and requires extensive expertise. This paper presents the first label-free, quantitative, and automated histological recognition of ovarian tissue subtypes using a new MIRSI technique. This optical photothermal infrared (O-PTIR) imaging technique provides a 10× enhancement in spatial resolution relative to prior instruments. It enables sub-cellular spectroscopic investigation of tissue at biochemically important fingerprint wavelengths. We demonstrate that the enhanced resolution of sub-cellular features, combined with spectroscopic information, enables reliable classification of ovarian cell subtypes achieving a classification accuracy of 0.98. Moreover, we present a statistically robust analysis from 78 patient samples with over 60 million data points. We show that sub-cellular resolution from five wavenumbers is sufficient to outperform state-of-the-art diffraction-limited techniques with up to 235 wavenumbers. We also propose two quantitative biomarkers based on the relative quantities of epithelia and stroma that exhibit efficacy in early cancer diagnosis. This paper demonstrates that combining deep learning with intrinsic biochemical MIRSI measurements enables quantitative evaluation of cancerous tissue, improving the rigor and reproducibility of histopathology.

Multimodal high-resolution embryonic imaging with light sheet fluorescence microscopy and optical coherence tomography.

A high-resolution imaging system combining optical coherence tomography (OCT) and light sheet fluorescence microscopy (LSFM) was developed. LSFM confined the excitation to only the focal plane, removing the out of plane fluorescence. This enabled imaging a murine embryo with higher speed and specificity than traditional fluorescence microscopy. OCT gives information about the structure of the embryo from the same plane illuminated by LSFM. The co-planar OCT and LSFM instrument was capable of performing co-registered functional and structural imaging of mouse embryos simultaneously.

Multi-modal image sharpening in fourier transform infrared (FTIR) microscopy.

Mid-infrared Spectroscopic Imaging (MIRSI) provides spatially-resolved molecular specificity by measuring wavelength-dependent mid-infrared absorbance. Infrared microscopes use large numerical aperture objectives to obtain high-resolution images of heterogeneous samples. However, the optical resolution is fundamentally diffraction-limited, and therefore wavelength-dependent. This significantly limits resolution in infrared microscopy, which relies on long wavelengths (2.5 µm to 12.5 µm) for molecular specificity. The resolution is particularly restrictive in biomedical and materials applications, where molecular information is encoded in the fingerprint region (6 µm to 12 µm), limiting the maximum resolving power to between 3 µm and 6 µm. We present an unsupervised curvelet-based image fusion method that overcomes limitations in spatial resolution by augmenting infrared images with label-free visible microscopy. We demonstrate the effectiveness of this approach by fusing images of breast and ovarian tumor biopsies acquired using both infrared and dark-field microscopy. The proposed fusion algorithm generates a hyperspectral dataset that has both high spatial resolution and good molecular contrast. We validate this technique using multiple standard approaches and through comparisons to super-resolved experimentally measured photothermal spectroscopic images. We also propose a novel comparison method based on tissue classification accuracy.

Artificial intelligence and machine learning in nephropathology.

Artificial intelligence (AI) for the purpose of this review is an umbrella term for technologies emulating a nephropathologist's ability to extract information on diagnosis, prognosis, and therapy responsiveness from native or transplant kidney biopsies. Although AI can be used to analyze a wide variety of biopsy-related data, this review focuses on whole slide images traditionally used in nephropathology. AI applications in nephropathology have recently become available through several advancing technologies, including (i) widespread introduction of glass slide scanners, (ii) data servers in pathology departments worldwide, and (iii) through greatly improved computer hardware to enable AI training. In this review, we explain how AI can enhance the reproducibility of nephropathology results for certain parameters in the context of precision medicine using advanced architectures, such as convolutional neural networks, that are currently the state of the art in machine learning software for this task. Because AI applications in nephropathology are still in their infancy, we show the power and potential of AI applications mostly in the example of oncopathology. Moreover, we discuss the technological obstacles as well as the current stakeholder and regulatory concerns about developing AI applications in nephropathology from the perspective of nephropathologists and the wider nephrology community. We expect the gradual introduction of these technologies into routine diagnostics and research for selective tasks, suggesting that this technology will enhance the performance of nephropathologists rather than making them redundant.

Automated Osteosclerosis Grading of Clinical Biopsies Using Infrared Spectroscopic Imaging.

Osteosclerosis and myefibrosis are complications of myeloproliferative neoplasms. These disorders result in excess growth of trabecular bone and collagen fibers that replace hematopoietic cells, resulting in abnormal bone marrow function. Treatments using imatinib and JAK2 pathway inhibitors can be effective on osteosclerosis and fibrosis; therefore, accurate grading is critical for tracking treatment effectiveness. Current grading standards use a four-class system based on analysis of biopsies stained with three histological stains: hematoxylin and eosin (H&E), Masson's trichrome, and reticulin. However, conventional grading can be subjective and imprecise, impacting the effectiveness of treatment. In this Article, we demonstrate that mid-infrared spectroscopic imaging may serve as a quantitative diagnostic tool for quantitatively tracking disease progression and response to treatment. The proposed approach is label-free and provides automated quantitative analysis of osteosclerosis and collagen fibrosis.

TIMING 2.0: high-throughput single-cell profiling of dynamic cell-cell interactions by time-lapse imaging microscopy in nanowell grids.

MOTIVATION: Automated profiling of cell-cell interactions from high-throughput time-lapse imaging microscopy data of cells in nanowell grids (TIMING) has led to fundamental insights into cell-cell interactions in immunotherapy. This application note aims to enable widespread adoption of TIMING by (i) enabling the computations to occur on a desktop computer with a graphical processing unit instead of a server; (ii) enabling image acquisition and analysis to occur in the laboratory avoiding network data transfers to/from a server and (iii) providing a comprehensive graphical user interface. RESULTS: On a desktop computer, TIMING 2.0 takes 5 s/block/image frame, four times faster than our previous method on the same computer, and twice as fast as our previous method (TIMING) running on a Dell PowerEdge server. The cell segmentation accuracy (f-number = 0.993) is superior to our previous method (f-number = 0.821). A graphical user interface provides the ability to inspect the video analysis results, make corrective edits efficiently (one-click editing of an entire nanowell video sequence in 5-10 s) and display a summary of the cell killing efficacy measurements. AVAILABILITY AND IMPLEMENTATION: Open source Python software (GPL v3 license), instruction manual, sample data and sample results are included with the Supplement (https://github.com/RoysamLab/TIMING2). SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

Deep learning for FTIR histology: leveraging spatial and spectral features with convolutional neural networks.

Current methods for cancer detection rely on tissue biopsy, chemical labeling/staining, and examination of the tissue by a pathologist. Though these methods continue to remain the gold standard, they are non-quantitative and susceptible to human error. Fourier transform infrared (FTIR) spectroscopic imaging has shown potential as a quantitative alternative to traditional histology. However, identification of histological components requires reliable classification based on molecular spectra, which are susceptible to artifacts introduced by noise and scattering. Several tissue types, particularly in heterogeneous tissue regions, tend to confound traditional classification methods. Convolutional neural networks (CNNs) are the current state-of-the-art in image classification, providing the ability to learn spatial characteristics of images. In this paper, we demonstrate that CNNs with architectures designed to process both spectral and spatial information can significantly improve classifier performance over per-pixel spectral classification. We report classification results after applying CNNs to data from tissue microarrays (TMAs) to identify six major cellular and acellular constituents of tissue, namely adipocytes, blood, collagen, epithelium, necrosis, and myofibroblasts. Experimental results show that the use of spatial information in addition to the spectral information brings significant improvements in the classifier performance and allows classification of cellular subtypes, such as adipocytes, that exhibit minimal chemical information but have distinct spatial characteristics. This work demonstrates the application and efficiency of deep learning algorithms in improving the diagnostic techniques in clinical and research activities related to cancer.

Selecting optimal features from Fourier transform infrared spectroscopy for discrete-frequency imaging.

Tissue histology utilizing chemical and immunohistochemical labels plays an important role in biomedicine and disease diagnosis. Recent research suggests that mid-infrared (IR) spectroscopic imaging may augment histology by providing quantitative molecular information. One of the major barriers to this approach is long acquisition time using Fourier-transform infrared (FTIR) spectroscopy. Recent advances in discrete frequency sources, particularly quantum cascade lasers (QCLs), may mitigate this problem by allowing selective sampling of the absorption spectrum. However, DFIR imaging only provides a significant advantage when the number of spectral samples is minimized, requiring a priori knowledge of important spectral features. In this paper, we demonstrate the use of a GPU-based genetic algorithm (GA) using linear discriminant analysis (LDA) for DFIR feature selection. Our proposed method relies on pre-acquired broadband FTIR images for feature selection. Based on user-selected criteria for classification accuracy, our algorithm provides a minimal set of features that can be used with DFIR in a time-frame more practical for clinical diagnosis.

SIproc: an open-source biomedical data processing platform for large hyperspectral images.

There has recently been significant interest within the vibrational spectroscopy community to apply quantitative spectroscopic imaging techniques to histology and clinical diagnosis. However, many of the proposed methods require collecting spectroscopic images that have a similar region size and resolution to the corresponding histological images. Since spectroscopic images contain significantly more spectral samples than traditional histology, the resulting data sets can approach hundreds of gigabytes to terabytes in size. This makes them difficult to store and process, and the tools available to researchers for handling large spectroscopic data sets are limited. Fundamental mathematical tools, such as MATLAB, Octave, and SciPy, are extremely powerful but require that the data be stored in fast memory. This memory limitation becomes impractical for even modestly sized histological images, which can be hundreds of gigabytes in size. In this paper, we propose an open-source toolkit designed to perform out-of-core processing of hyperspectral images. By taking advantage of graphical processing unit (GPU) computing combined with adaptive data streaming, our software alleviates common workstation memory limitations while achieving better performance than existing applications.

Compositional prior information in computed infrared spectroscopic imaging.

Compositional prior information is used to bridge a gap in the theory between optical coherence tomography (OCT), which provides high-resolution structural images by neglecting spectral variation, and imaging spectroscopy, which provides only spectral information without significant regard to structure. A constraint is proposed in which it is assumed that a sample is composed of N distinct materials with known spectra, allowing the structural and spectral composition of the sample to be determined with a number of measurements on the order of N. We present a forward model for a sample with heterogeneities along the optical axis and show through simulation that the N-species constraint allows unambiguous inversion of Fourier transform interferometric data within the spatial frequency passband of the optical system. We then explore the stability and limitations of this model and extend it to a general 3D heterogeneous sample.

On the importance of image formation optics in the design of infrared spectroscopic imaging systems.

Infrared spectroscopic imaging provides micron-scale spatial resolution with molecular contrast. While recent work demonstrates that sample morphology affects the recorded spectrum, considerably less attention has been focused on the effects of the optics, including the condenser and objective. This analysis is extremely important, since it will be possible to understand effects on recorded data and provides insight for reducing optical effects through rigorous microscope design. Here, we present a theoretical description and experimental results that demonstrate the effects of commonly-employed cassegranian optics on recorded spectra. We first combine an explicit model of image formation and a method for quantifying and visualizing the deviations in recorded spectra as a function of microscope optics. We then verify these simulations with measurements obtained from spatially heterogeneous samples. The deviation of the computed spectrum from the ideal case is quantified via a map which we call a deviation map. The deviation map is obtained as a function of optical elements by systematic simulations. Examination of deviation maps demonstrates that the optimal optical configuration for minimal deviation is contrary to prevailing practice in which throughput is maximized for an instrument without a sample. This report should be helpful for understanding recorded spectra as a function of the optics, the analytical limits of recorded data determined by the optical design, and potential routes for optimization of imaging systems.

GPU-Accelerated RSF Level Set Evolution for Large-Scale Microvascular Segmentation.

Microvascular networks are challenging to model because these structures are currently near the diffraction limit for most advanced three-dimensional imaging modalities, including confocal and light sheet microscopy. This makes semantic segmentation difficult, because individual components of these networks fluctuate within the confines of individual pixels. Level set methods are ideally suited to solve this problem by providing surface and topological constraints on the resulting model, however these active contour techniques are extremely time intensive and impractical for terabyte-scale images. We propose a reformulation and implementation of the region-scalable fitting (RSF) level set model that makes it amenable to three-dimensional evaluation using both single-instruction multiple data (SIMD) and single-program multiple-data (SPMD) parallel processing. This enables evaluation of the level set equation on independent regions of the data set using graphics processing units (GPUs), making large-scale segmentation of high-resolution networks practical and inexpensive. We tested this 3D parallel RSF approach on multiple data sets acquired using state-of-the-art imaging techniques to acquire microvascular data, including micro-CT, light sheet fluorescence microscopy (LSFM) and milling microscopy. To assess the performance and accuracy of the RSF model, we conducted a Monte-Carlo-based validation technique to compare results to other segmentation methods. We also provide a rigorous profiling to show the gains in processing speed leveraging parallel hardware. This study showcases the practical application of the RSF model, emphasizing its utility in the challenging domain of segmenting large-scale high-topology network structures with a particular focus on building microvascular models.

Rapid hyperspectral photothermal mid-infrared spectroscopic imaging from sparse data for gynecologic cancer tissue subtyping.

Ovarian cancer detection has traditionally relied on a multi-step process that includes biopsy, tissue staining, and morphological analysis by experienced pathologists. While widely practiced, this conventional approach suffers from several drawbacks: it is qualitative, time-intensive, and heavily dependent on the quality of staining. Mid-infrared (MIR) hyperspectral photothermal imaging is a label-free, biochemically quantitative technology that, when combined with machine learning algorithms, can eliminate the need for staining and provide quantitative results comparable to traditional histology. However, this technology is slow. This work presents a novel approach to MIR photothermal imaging that enhances its speed by an order of magnitude. Our method significantly accelerates data collection by capturing a combination of highresolution and interleaved, lower-resolution infrared band images and applying computational techniques for data interpolation. We effectively minimize data collection requirements by leveraging sparse data acquisition and employing curvelet-based reconstruction algorithms. This approach enhances imaging speed without compromising image quality and ensures robust tissue segmentation. This method resolves the longstanding trade-off between imaging resolution and data collection speed, enabling the reconstruction of high-quality, high-resolution images from undersampled datasets and achieving a 10X improvement in data acquisition time. We assessed the performance of our sparse imaging methodology using a variety of quantitative metrics, including mean squared error (MSE), structural similarity index (SSIM), and tissue subtype classification accuracies, employing both random forest and convolutional neural network (CNN) models, accompanied by Receiver Operating Characteristic (ROC) curves. Our statistically robust analysis, based on data from 100 ovarian cancer patient samples and over 65 million data points, demonstrates the method's capability to produce superior image quality and accurately distinguish between different gynecological tissue types with segmentation accuracy exceeding 95%. Our work demonstrates the feasibility of integrating rapid MIR hyperspectral photothermal imaging with machine learning in enhancing ovarian cancer tissue characterization, paving the way for quantitative, label-free, automated histopathology. It represents a significant leap forward from traditional histopathological methods, offering profound implications for cancer diagnostics and treatment decision-making.

Real-time interactive data mining for chemical imaging information: application to automated histopathology.

BACKGROUND: Vibrational spectroscopic imaging is now used in several fields to acquire molecular information from microscopically heterogeneous systems. Recent advances have led to promising applications in tissue analysis for cancer research, where chemical information can be used to identify cell types and disease. However, recorded spectra are affected by the morphology of the tissue sample, making identification of chemical structures difficult. RESULTS: Extracting features that can be used to classify tissue is a cumbersome manual process which limits this technology from wide applicability. In this paper, we describe a method for interactive data mining of spectral features using GPU-based manipulation of the spectral distribution. CONCLUSIONS: This allows researchers to quickly identify chemical features corresponding to cell type. These features are then applied to tissue samples in order to visualize the chemical composition of the tissue without the use of chemical stains.

Rapid spectral-domain localization.

We present a method to dynamically image structures at nanometer spatial resolution with far-field instruments. We propose the use of engineered nanoprobes with distinguishable spectral responses and the measurement of coherent scattering, rather than fluorescence. Approaches such as PALM/STORM have relied on the rarity of emission events in time to distinguish signals from distinct probes. By distinguishing signals in the spectral domain, we enable the acquisition of data in a multiplex fashion and thus circumvent the fundamental problem of slow data acquisition of current techniques. The described method has the potential to image dynamic systems with a spatial resolution only limited to the size of the scattering probes.

NetMets: software for quantifying and visualizing errors in biological network segmentation.

One of the major goals in biomedical image processing is accurate segmentation of networks embedded in volumetric data sets. Biological networks are composed of a meshwork of thin filaments that span large volumes of tissue. Examples of these structures include neurons and microvasculature, which can take the form of both hierarchical trees and fully connected networks, depending on the imaging modality and resolution. Network function depends on both the geometric structure and connectivity. Therefore, there is considerable demand for algorithms that segment biological networks embedded in three-dimensional data. While a large number of tracking and segmentation algorithms have been published, most of these do not generalize well across data sets. One of the major reasons for the lack of general-purpose algorithms is the limited availability of metrics that can be used to quantitatively compare their effectiveness against a pre-constructed ground-truth. In this paper, we propose a robust metric for measuring and visualizing the differences between network models. Our algorithm takes into account both geometry and connectivity to measure network similarity. These metrics are then mapped back onto an explicit model for visualization.

Computational insights on coronary artery function.

Collateral arteries may act as natural bypasses that reduce hypoperfusion after a coronary blockage. 3D imaging of neonatal and adult mouse hearts, plus human fetal and diseased adult hearts, is now used to computationally predict flow within the heart, and understand the cardioprotective role of collateral arteries in vivo.

Polarization Sensitive Photothermal Mid-Infrared Spectroscopic Imaging of Human Bone Marrow Tissue.

Collagen quantity and integrity play an important role in understanding diseases such as myelofibrosis (MF). Label-free mid-infrared spectroscopic imaging (MIRSI) has the potential to quantify collagen while minimizing the subjective variance observed with conventional histopathology. Infrared (IR) spectroscopy with polarization sensitivity provides chemical information while also estimating tissue dichroism. This can potentially aid MF grading by revealing the structure and orientation of collagen fibers. Simultaneous measurement of collagen structure and biochemical properties can translate clinically into improved diagnosis and enhance our understanding of disease progression. In this paper, we present the first report of polarization-dependent spectroscopic variations in collagen from human bone marrow samples. We build on prior work with animal models and extend it to human clinical biopsies with a practical method for high-resolution chemical and structural imaging of bone marrow on clinical glass slides. This is done using a new polarization-sensitive photothermal mid-infrared spectroscopic imaging scheme that enables sample and source independent polarization control. This technology provides 0.5 µm spatial resolution, enabling the identification of thin (≈1 µm) collagen fibers that were not separable using Fourier Transform Infrared (FT-IR) imaging in the fingerprint region at diffraction-limited resolution ( ≈ 5 µm). Finally, we propose quantitative metrics to identify fiber orientation from discrete band images (amide I and amide II) measured under three polarizations. Previous studies have used a pair of orthogonal polarization measurements, which is insufficient for clinical samples since human bone biopsies contain collagen fibers with multiple orientations. Here, we address this challenge and demonstrate that three polarization measurements are necessary to resolve orientation ambiguity in clinical bone marrow samples. This is also the first study to demonstrate the ability to spectroscopically identify thin collagen fibers (≈1 µm diameter) and their orientations, which is critical for accurate grading of human bone marrow fibrosis.

Prostacyclin Promotes Degenerative Pathology in a Model of Alzheimer's Disease.

Alzheimer's disease (AD) is a progressive neurodegenerative disorder that is the most common form of dementia in aged populations. A substantial amount of data demonstrates that chronic neuroinflammation can accelerate neurodegenerative pathologies. In AD, chronic neuroinflammation results in the upregulation of cyclooxygenase and increased production of prostaglandin H2, a precursor for many vasoactive prostanoids. While it is well-established that many prostaglandins can modulate the progression of neurodegenerative disorders, the role of prostacyclin (PGI2) in the brain is poorly understood. We have conducted studies to assess the effect of elevated prostacyclin biosynthesis in a mouse model of AD. Upregulated prostacyclin expression significantly worsened multiple measures associated with amyloid-ß (Aß) disease pathologies. Mice overexpressing both Aß and PGI2 exhibited impaired learning and memory and increased anxiety-like behavior compared with non-transgenic and PGI2 control mice. PGI2 overexpression accelerated the development of Aß accumulation in the brain and selectively increased the production of soluble Aß42. PGI2 damaged the microvasculature through alterations in vascular length and branching; Aß expression exacerbated these effects. Our findings demonstrate that chronic prostacyclin expression plays a novel and unexpected role that hastens the development of the AD phenotype.

Multiplex protein-specific microscopy with ultraviolet surface excitation.

Immunohistochemical techniques, such as immunofluorescence (IF) staining, enable microscopic imaging of local protein expression within tissue samples. Molecular profiling enabled by IF is critical to understanding pathogenesis and is often involved in complex diagnoses. A recent innovation, known as microscopy with ultraviolet surface excitation (MUSE), uses deep ultraviolet (≈280 nm) illumination to excite labels at the tissue surface, providing equivalent images without fixation, embedding, and sectioning. However, MUSE has not yet been integrated into traditional IF pipelines. This limits its application in more complex diagnoses that rely on protein-specific markers. This paper aims to broaden the applicability of MUSE to multiplex immunohistochemistry using quantum dot nanoparticles. We demonstrate the advantages of quantum dot labels for protein-specific MUSE imaging on both paraffin-embedded and intact tissue, significantly expanding MUSE applicability to protein-specific applications. Furthermore, with recent innovations in three-dimensional ultraviolet fluorescence microscopy, this opens the door to three-dimensional IF imaging with quantum dots using ultraviolet excitation.