Hierarchical discriminative learning improves visual representations of biomedical microscopy.

Learning high-quality, self-supervised, visual representations is essential to advance the role of computer vision in biomedical microscopy and clinical medicine. Previous work has focused on self-supervised representation learning (SSL) methods developed for instance discrimination and applied them directly to image patches, or fields-of-view, sampled from gigapixel whole-slide images (WSIs) used for cancer diagnosis. However, this strategy is limited because it (1) assumes patches from the same patient are independent, (2) neglects the patient-slide-patch hierarchy of clinical biomedical microscopy, and (3) requires strong data augmentations that can degrade downstream performance. Importantly, sampled patches from WSIs of a patient's tumor are a diverse set of image examples that capture the same underlying cancer diagnosis. This motivated HiDisc, a data-driven method that leverages the inherent patient-slide-patch hierarchy of clinical biomedical microscopy to define a hierarchical discriminative learning task that implicitly learns features of the underlying diagnosis. HiDisc uses a self-supervised contrastive learning framework in which positive patch pairs are defined based on a common ancestry in the data hierarchy, and a unified patch, slide, and patient discriminative learning objective is used for visual SSL. We benchmark HiDisc visual representations on two vision tasks using two biomedical microscopy datasets, and demonstrate that (1) HiDisc pretraining outperforms current state-of-the-art self-supervised pretraining methods for cancer diagnosis and genetic mutation prediction, and (2) HiDisc learns high-quality visual representations using natural patch diversity without strong data augmentations.

Toward digital histopathological assessment in surgery for central nervous system tumors using stimulated Raman histology.

OBJECTIVE: Intraoperative neuropathological assessment with conventional frozen sections supports the neurosurgeon in optimizing the surgical strategy. However, preparation and review of frozen sections can take as long as 45 minutes. Stimulated Raman histology (SRH) was introduced as a novel technique to provide rapid high-resolution digital images of unprocessed tissue samples directly in the operating room that are comparable to conventional histopathological images. Additionally, SRH images are simultaneously and easily accessible for neuropathological judgment. Recently, the first study showed promising results regarding the accuracy and feasibility of SRH compared with conventional histopathology. Thus, the aim of this study was to compare SRH with conventional H&E images and frozen sections in a large cohort of patients with different suspected central nervous system (CNS) tumors. METHODS: The authors included patients who underwent resection or stereotactic biopsy of suspected CNS neoplasm, including brain and spinal tumors. Intraoperatively, tissue samples were safely collected and SRH analysis was performed directly in the operating room. To enable optimal comparison of SRH with H&E images and frozen sections, the authors created a digital databank that included images obtained with all 3 imaging modalities. Subsequently, 2 neuropathologists investigated the diagnostic accuracy, tumor cellularity, and presence of diagnostic histopathological characteristics (score 0 [not present] through 3 [excellent]) determined with SRH images and compared these data to those of H&E images and frozen sections, if available. RESULTS: In total, 94 patients with various suspected CNS tumors were included, and the application of SRH directly in the operating room was feasible in all cases. The diagnostic accuracy based on SRH images was 99% when compared with the final histopathological diagnosis based on H&E images. Additionally, the same histopathological diagnosis was established in all SRH images (100%) when compared with that of the corresponding frozen sections. Moreover, the authors found a statistically significant correlation in tumor cellularity between SRH images and corresponding H&E images (p < 0.0005 and R = 0.867, Pearson correlation coefficient). Finally, excellent (score 3) or good (2) accordance between diagnostic histopathological characteristics and H&E images was present in 95% of cases. CONCLUSIONS: The results of this retrospective analysis demonstrate the near-perfect diagnostic accuracy and capability of visualizing relevant histopathological characteristics with SRH compared with conventional H&E staining and frozen sections. Therefore, digital SRH histopathology seems especially useful for rapid intraoperative investigation to confirm the presence of diagnostic tumor tissue and the precise tumor entity, as well as to rapidly analyze multiple tissue biopsies from the suspected tumor margin. A real-time analysis comparing SRH images and conventional histological images at the time of surgery should be performed as the next step in future studies.

Novel rapid intraoperative qualitative tumor detection by a residual convolutional neural network using label-free stimulated Raman scattering microscopy.

Determining the presence of tumor in biopsies and the decision-making during resections is often dependent on intraoperative rapid frozen-section histopathology. Recently, stimulated Raman scattering microscopy has been introduced to rapidly generate digital hematoxylin-and-eosin-stained-like images (stimulated Raman histology) for intraoperative analysis. To enable intraoperative prediction of tumor presence, we aimed to develop a new deep residual convolutional neural network in an automated pipeline and tested its validity. In a monocentric prospective clinical study with 94 patients undergoing biopsy, brain or spinal tumor resection, Stimulated Raman histology images of intraoperative tissue samples were obtained using a fiber-laser-based stimulated Raman scattering microscope. A residual network was established and trained in ResNetV50 to predict three classes for each image: (1) tumor, (2) non-tumor, and (3) low-quality. The residual network was validated on images obtained in three small random areas within the tissue samples and were blindly independently reviewed by a neuropathologist as ground truth. 402 images derived from 132 tissue samples were analyzed representing the entire spectrum of neurooncological surgery. The automated workflow took in a mean of 240 s per case, and the residual network correctly classified tumor (305/326), non-tumorous tissue (49/67), and low-quality (6/9) images with an inter-rater agreement of 89.6% (κ = 0.671). An excellent internal consistency was found among the random areas with 90.2% (Cα = 0.942) accuracy. In conclusion, the novel stimulated Raman histology-based residual network can reliably detect the microscopic presence of tumor and differentiate from non-tumorous brain tissue in resection and biopsy samples within 4 min and may pave a promising way for an alternative rapid intraoperative histopathological decision-making tool.

Rapid Automated Analysis of Skull Base Tumor Specimens Using Intraoperative Optical Imaging and Artificial Intelligence.

BACKGROUND: Accurate specimen analysis of skull base tumors is essential for providing personalized surgical treatment strategies. Intraoperative specimen interpretation can be challenging because of the wide range of skull base pathologies and lack of intraoperative pathology resources. OBJECTIVE: To develop an independent and parallel intraoperative workflow that can provide rapid and accurate skull base tumor specimen analysis using label-free optical imaging and artificial intelligence. METHODS: We used a fiber laser-based, label-free, nonconsumptive, high-resolution microscopy method (<60 seconds per 1 × 1 mm2), called stimulated Raman histology (SRH), to image a consecutive, multicenter cohort of patients with skull base tumor. SRH images were then used to train a convolutional neural network model using 3 representation learning strategies: cross-entropy, self-supervised contrastive learning, and supervised contrastive learning. Our trained convolutional neural network models were tested on a held-out, multicenter SRH data set. RESULTS: SRH was able to image the diagnostic features of both benign and malignant skull base tumors. Of the 3 representation learning strategies, supervised contrastive learning most effectively learned the distinctive and diagnostic SRH image features for each of the skull base tumor types. In our multicenter testing set, cross-entropy achieved an overall diagnostic accuracy of 91.5%, self-supervised contrastive learning 83.9%, and supervised contrastive learning 96.6%. Our trained model was able to segment tumor-normal margins and detect regions of microscopic tumor infiltration in meningioma SRH images. CONCLUSION: SRH with trained artificial intelligence models can provide rapid and accurate intraoperative analysis of skull base tumor specimens to inform surgical decision-making.

OpenSRH: optimizing brain tumor surgery using intraoperative stimulated Raman histology.

Accurate intraoperative diagnosis is essential for providing safe and effective care during brain tumor surgery. Our standard-of-care diagnostic methods are time, resource, and labor intensive, which restricts access to optimal surgical treatments. To address these limitations, we propose an alternative workflow that combines stimulated Raman histology (SRH), a rapid optical imaging method, with deep learning-based automated interpretation of SRH images for intraoperative brain tumor diagnosis and real-time surgical decision support. Here, we present OpenSRH, the first public dataset of clinical SRH images from 300+ brain tumors patients and 1300+ unique whole slide optical images. OpenSRH contains data from the most common brain tumors diagnoses, full pathologic annotations, whole slide tumor segmentations, raw and processed optical imaging data for end-to-end model development and validation. We provide a framework for patch-based whole slide SRH classification and inference using weak (i.e. patient-level) diagnostic labels. Finally, we benchmark two computer vision tasks: multiclass histologic brain tumor classification and patch-based contrastive representation learning. We hope OpenSRH will facilitate the clinical translation of rapid optical imaging and real-time ML-based surgical decision support in order to improve the access, safety, and efficacy of cancer surgery in the era of precision medicine. Dataset access, code, and benchmarks are available at https://opensrh.mlins.org.

Rapid, label-free detection of diffuse glioma recurrence using intraoperative stimulated Raman histology and deep neural networks.

BACKGROUND: Detection of glioma recurrence remains a challenge in modern neuro-oncology. Noninvasive radiographic imaging is unable to definitively differentiate true recurrence versus pseudoprogression. Even in biopsied tissue, it can be challenging to differentiate recurrent tumor and treatment effect. We hypothesized that intraoperative stimulated Raman histology (SRH) and deep neural networks can be used to improve the intraoperative detection of glioma recurrence. METHODS: We used fiber laser-based SRH, a label-free, nonconsumptive, high-resolution microscopy method (<60 sec per 1 × 1 mm2) to image a cohort of patients (n = 35) with suspected recurrent gliomas who underwent biopsy or resection. The SRH images were then used to train a convolutional neural network (CNN) and develop an inference algorithm to detect viable recurrent glioma. Following network training, the performance of the CNN was tested for diagnostic accuracy in a retrospective cohort (n = 48). RESULTS: Using patch-level CNN predictions, the inference algorithm returns a single Bernoulli distribution for the probability of tumor recurrence for each surgical specimen or patient. The external SRH validation dataset consisted of 48 patients (recurrent, 30; pseudoprogression, 18), and we achieved a diagnostic accuracy of 95.8%. CONCLUSION: SRH with CNN-based diagnosis can be used to improve the intraoperative detection of glioma recurrence in near-real time. Our results provide insight into how optical imaging and computer vision can be combined to augment conventional diagnostic methods and improve the quality of specimen sampling at glioma recurrence.

Near real-time intraoperative brain tumor diagnosis using stimulated Raman histology and deep neural networks.

Intraoperative diagnosis is essential for providing safe and effective care during cancer surgery1. The existing workflow for intraoperative diagnosis based on hematoxylin and eosin staining of processed tissue is time, resource and labor intensive2,3. Moreover, interpretation of intraoperative histologic images is dependent on a contracting, unevenly distributed, pathology workforce4. In the present study, we report a parallel workflow that combines stimulated Raman histology (SRH)5-7, a label-free optical imaging method and deep convolutional neural networks (CNNs) to predict diagnosis at the bedside in near real-time in an automated fashion. Specifically, our CNNs, trained on over 2.5 million SRH images, predict brain tumor diagnosis in the operating room in under 150 s, an order of magnitude faster than conventional techniques (for example, 20-30 min)2. In a multicenter, prospective clinical trial (n = 278), we demonstrated that CNN-based diagnosis of SRH images was noninferior to pathologist-based interpretation of conventional histologic images (overall accuracy, 94.6% versus 93.9%). Our CNNs learned a hierarchy of recognizable histologic feature representations to classify the major histopathologic classes of brain tumors. In addition, we implemented a semantic segmentation method to identify tumor-infiltrated diagnostic regions within SRH images. These results demonstrate how intraoperative cancer diagnosis can be streamlined, creating a complementary pathway for tissue diagnosis that is independent of a traditional pathology laboratory.

Label-free imaging of amyloid plaques in Alzheimer's disease with stimulated Raman scattering microscopy.

One of the key pathological features of Alzheimer's disease (AD) is the existence of extracellular deposition of amyloid plaques formed with misfolded amyloid-ß (Aß). The conformational change of proteins leads to enriched contents of ß sheets, resulting in remarkable changes of vibrational spectra, especially the spectral shifts of the amide I mode. Here, we applied stimulated Raman scattering (SRS) microscopy to image amyloid plaques in the brain tissue of an AD mouse model. We have demonstrated the capability of SRS microscopy as a rapid, label-free imaging modality to differentiate misfolded from normal proteins based on the blue shift (~10 cm-1) of amide I SRS spectra. Furthermore, SRS imaging of Aß plaques was verified by antibody staining of frozen thin sections and fluorescence imaging of fresh tissues. Our method may provide a new approach for studies of AD pathology, as well as other neurodegenerative diseases associated with protein misfolding.

Rapid intraoperative histology of unprocessed surgical specimens via fibre-laser-based stimulated Raman scattering microscopy.

Conventional methods for intraoperative histopathologic diagnosis are labour- and time-intensive, and may delay decision-making during brain-tumour surgery. Stimulated Raman scattering (SRS) microscopy, a label-free optical process, has been shown to rapidly detect brain-tumour infiltration in fresh, unprocessed human tissues. Here, we demonstrate the first application of SRS microscopy in the operating room by using a portable fibre-laser-based microscope and unprocessed specimens from 101 neurosurgical patients. We also introduce an image-processing method - stimulated Raman histology (SRH) - which leverages SRS images to create virtual haematoxylin-and-eosin-stained slides, revealing essential diagnostic features. In a simulation of intraoperative pathologic consultation in 30 patients, we found a remarkable concordance of SRH and conventional histology for predicting diagnosis (Cohen's kappa, κ > 0.89), with accuracy exceeding 92%. We also built and validated a multilayer perceptron based on quantified SRH image attributes that predicts brain-tumour subtype with 90% accuracy. Our findings provide insight into how SRH can now be used to improve the surgical care of brain tumour patients.

Improving the accuracy of brain tumor surgery via Raman-based technology.

Despite advances in the surgical management of brain tumors, achieving optimal surgical results and identification of tumor remains a challenge. Raman spectroscopy, a laser-based technique that can be used to nondestructively differentiate molecules based on the inelastic scattering of light, is being applied toward improving the accuracy of brain tumor surgery. Here, the authors systematically review the application of Raman spectroscopy for guidance during brain tumor surgery. Raman spectroscopy can differentiate normal brain from necrotic and vital glioma tissue in human specimens based on chemical differences, and has recently been shown to differentiate tumor-infiltrated tissues from noninfiltrated tissues during surgery. Raman spectroscopy also forms the basis for coherent Raman scattering (CRS) microscopy, a technique that amplifies spontaneous Raman signals by 10,000-fold, enabling real-time histological imaging without the need for tissue processing, sectioning, or staining. The authors review the relevant basic and translational studies on CRS microscopy as a means of providing real-time intraoperative guidance. Recent studies have demonstrated how CRS can be used to differentiate tumor-infiltrated tissues from noninfiltrated tissues and that it has excellent agreement with traditional histology. Under simulated operative conditions, CRS has been shown to identify tumor margins that would be undetectable using standard bright-field microscopy. In addition, CRS microscopy has been shown to detect tumor in human surgical specimens with near-perfect agreement to standard H & E microscopy. The authors suggest that as the intraoperative application and instrumentation for Raman spectroscopy and imaging matures, it will become an essential component in the neurosurgical armamentarium for identifying residual tumor and improving the surgical management of brain tumors.

Detection of human brain tumor infiltration with quantitative stimulated Raman scattering microscopy.

Differentiating tumor from normal brain is a major barrier to achieving optimal outcome in brain tumor surgery. New imaging techniques for visualizing tumor margins during surgery are needed to improve surgical results. We recently demonstrated the ability of stimulated Raman scattering (SRS) microscopy, a nondestructive, label-free optical method, to reveal glioma infiltration in animal models. We show that SRS reveals human brain tumor infiltration in fresh, unprocessed surgical specimens from 22 neurosurgical patients. SRS detects tumor infiltration in near-perfect agreement with standard hematoxylin and eosin light microscopy (κ = 0.86). The unique chemical contrast specific to SRS microscopy enables tumor detection by revealing quantifiable alterations in tissue cellularity, axonal density, and protein/lipid ratio in tumor-infiltrated tissues. To ensure that SRS microscopic data can be easily used in brain tumor surgery, without the need for expert interpretation, we created a classifier based on cellularity, axonal density, and protein/lipid ratio in SRS images capable of detecting tumor infiltration with 97.5% sensitivity and 98.5% specificity. Quantitative SRS microscopy detects the spread of tumor cells, even in brain tissue surrounding a tumor that appears grossly normal. By accurately revealing tumor infiltration, quantitative SRS microscopy holds potential for improving the accuracy of brain tumor surgery.

Label-free DNA imaging in vivo with stimulated Raman scattering microscopy.

Label-free DNA imaging is highly desirable in biology and medicine to perform live imaging without affecting cell function and to obtain instant histological tissue examination during surgical procedures. Here we show a label-free DNA imaging method with stimulated Raman scattering (SRS) microscopy for visualization of the cell nuclei in live animals and intact fresh human tissues with subcellular resolution. Relying on the distinct Raman spectral features of the carbon-hydrogen bonds in DNA, the distribution of DNA is retrieved from the strong background of proteins and lipids by linear decomposition of SRS images at three optimally selected Raman shifts. Based on changes on DNA condensation in the nucleus, we were able to capture chromosome dynamics during cell division both in vitro and in vivo. We tracked mouse skin cell proliferation, induced by drug treatment, through in vivo counting of the mitotic rate. Furthermore, we demonstrated a label-free histology method for human skin cancer diagnosis that provides comparable results to other conventional tissue staining methods such as H&E. Our approach exhibits higher sensitivity than SRS imaging of DNA in the fingerprint spectral region. Compared with spontaneous Raman imaging of DNA, our approach is three orders of magnitude faster, allowing both chromatin dynamic studies and label-free optical histology in real time.

Stimulated Raman Scattering Microscopy with a Robust Fibre Laser Source.

Stimulated Raman Scattering microscopy allows label-free chemical imaging and has enabled exciting applications in biology, material science, and medicine. It provides a major advantage in imaging speed over spontaneous Raman scattering and has improved image contrast and spectral fidelity compared to coherent anti-Stokes Raman. Wider adoption of the technique has, however, been hindered by the need for a costly and environmentally sensitive tunable ultra-fast dual-wavelength source. We present the development of an optimized all-fibre laser system based on the optical synchronization of two picosecond power amplifiers. To circumvent the high-frequency laser noise intrinsic to amplified fibre lasers, we have further developed a high-speed noise cancellation system based on voltage-subtraction autobalanced detection. We demonstrate uncompromised imaging performance of our fibre-laser based stimulated Raman scattering microscope with shot-noise limited sensitivity and an imaging speed up to 1 frame/s.

Coherent Raman tissue imaging in the brain.

Imaging in neuroscience has been dramatically impacted by the advent of multiphoton microscopy. Multiphoton-excited fluorescence (MPF) in combination with endogenous fluorophores or labeling by fluorescent molecules has proven to be particularly powerful. However, endogenous fluorescence is limited to relatively few molecular species, and practical labeling schemes do not exist for many classes of molecules. Coherent Raman scattering (CRS) techniques, including coherent anti-Stokes Raman scattering and stimulated Raman scattering, allow imaging without the need for staining or fluorescent labeling. Such label-free imaging is desirable in biomedical research, because labeling often perturbs the function of small metabolite and drug molecules and may be too toxic to use in vivo. CRS techniques have similar imaging parameters to MPF, making use of pulsed near-infrared lasers to deliver high-sensitivity, high spatial resolution in three dimensions and rapid image acquisition. In this introduction, we will discuss the basic principles of CRS imaging, present the instrumentation requirements for high-speed CRS imaging, and show an example of imaging brain tumors and healthy tissue based on their intrinsic vibrational signatures. This discussion is intended to introduce the benefits and tradeoffs associated with different CRS techniques and show one example of the powerful capabilities of label-free chemical imaging.

Rapid, label-free detection of brain tumors with stimulated Raman scattering microscopy.

Surgery is an essential component in the treatment of brain tumors. However, delineating tumor from normal brain remains a major challenge. We describe the use of stimulated Raman scattering (SRS) microscopy for differentiating healthy human and mouse brain tissue from tumor-infiltrated brain based on histoarchitectural and biochemical differences. Unlike traditional histopathology, SRS is a label-free technique that can be rapidly performed in situ. SRS microscopy was able to differentiate tumor from nonneoplastic tissue in an infiltrative human glioblastoma xenograft mouse model based on their different Raman spectra. We further demonstrated a correlation between SRS and hematoxylin and eosin microscopy for detection of glioma infiltration (κ = 0.98). Finally, we applied SRS microscopy in vivo in mice during surgery to reveal tumor margins that were undetectable under standard operative conditions. By providing rapid intraoperative assessment of brain tissue, SRS microscopy may ultimately improve the safety and accuracy of surgeries where tumor boundaries are visually indistinct.

Multicolor stimulated Raman scattering microscopy with a rapidly tunable optical parametric oscillator.

Stimulated Raman scattering (SRS) microscopy allows label-free chemical imaging based on vibrational spectroscopy. Narrowband excitation with picosecond lasers creates the highest signal levels and enables imaging speeds up to video-rate, but it sacrifices chemical specificity in samples with overlapping bands compared to broadband (multiplex) excitation. We develop a rapidly tunable picosecond optical parametric oscillator with an electro-optical tunable Lyot filter, and demonstrate multicolor SRS microscopy with synchronized line-by-line wavelength tuning to avoid spectral artifacts due to sample movement. We show sensitive imaging of three different kinds of polymer beads and live HeLa cells with moving intracellular lipid droplets.

Multicolored stain-free histopathology with coherent Raman imaging.

Conventional histopathology with hematoxylin & eosin (H&E) has been the gold standard for histopathological diagnosis of a wide range of diseases. However, it is not performed in vivo and requires thin tissue sections obtained after tissue biopsy, which carries risk, particularly in the central nervous system. Here we describe the development of an alternative, multicolored way to visualize tissue in real-time through the use of coherent Raman imaging (CRI), without the use of dyes. CRI relies on intrinsic chemical contrast based on vibrational properties of molecules and intrinsic optical sectioning by nonlinear excitation. We demonstrate that multicolor images originating from CH(2) and CH(3) vibrations of lipids and protein, as well as two-photon absorption of hemoglobin, can be obtained with subcellular resolution from fresh tissue. These stain-free histopathological images show resolutions similar to those obtained by conventional techniques, but do not require tissue fixation, sectioning or staining of the tissue analyzed.

Label-free live-cell imaging of nucleic acids using stimulated Raman scattering microscopy.

Imaging of nucleic acids is important for studying cellular processes such as cell division and apoptosis. A noninvasive label-free technique is attractive. Raman spectroscopy provides rich chemical information based on specific vibrational peaks. However, the signal from spontaneous Raman scattering is weak and long integration times are required, which drastically limits the imaging speed when used for microscopy. Coherent Raman scattering techniques, comprising coherent anti-Stokes Raman scattering (CARS) and stimulated Raman scattering (SRS) microscopy, overcome this problem by enhancing the signal level by up to five orders of magnitude. CARS microscopy suffers from a nonresonant background signal, which distorts Raman spectra and limits sensitivity. This makes CARS imaging of weak transitions in spectrally congested regions challenging. This is especially the case in the fingerprint region, where nucleic acids show characteristic peaks. The recently developed SRS microscopy is free from these limitations; excitation spectra are identical to those of spontaneous Raman and sensitivity is close to shot-noise limited. Herein we demonstrate the use of SRS imaging in the fingerprint region to map the distribution of nucleic acids in addition to proteins and lipids in single salivary gland cells of Drosophila larvae, and in single mammalian cells. This allows the imaging of DNA condensation associated with cell division and opens up possibilities of imaging such processes in vivo.

Multicolor stimulated Raman scattering (SRS) microscopy.

Stimulated Raman scattering (SRS) microscopy has opened up a wide range of biochemical imaging applications by probing a particular Raman-active molecule vibrational mode in the specimen. However, the original implementation with picosecond pulse excitation can only realize rapid chemical mapping with a single Raman band. Here we present a novel SRS microscopic technique using a grating-based pulse shaper for excitation and a grating-based spectrograph for detection to achieve simultaneous multicolor SRS imaging with high sensitivity and high acquisition speeds. In particular, we used linear combination of the measured CH2 and CH3 stretching signals to map the distributions of protein and lipid contents simultaneously.

Coherent Raman scanning fiber endoscopy.

Coherent Raman scattering methods allow for label-free imaging of tissue with chemical contrast and high spatial and temporal resolution. However, their imaging depth in scattering tissue is limited to less than 1 mm, requiring the development of endoscopes to obtain images deep inside the body. Here, we describe a coherent Raman endoscope that provides stimulated Raman scattering images at seven frames per second using a miniaturized fiber scanner, a custom-designed objective lens, and an optimized scheme for collection of scattered light from the tissue. We characterize the system and demonstrate chemical selectivity in mouse tissue images.